QA/Execution/Web Testing/Docs/Automation/Testcases/ConventionsAmoTests

Conventions in AMO test cases

Element Locators

• Element locators are defined outside of the test case to centralize their definition and simplify test case code.
• Locators may contain %d or %s formatting specifiers to support elements in a list, or a common locator for sibling elements that differ by a short segment.
• Locators are defined in a combination of Xpath and CSS formats. CSS is the preferred format for POM.

Locators in Pre-POM test cases

• Defined in AMOlocators module. The AMOlocators class is divided into sub-classes that contain locators by page which is further sub-classed into sections of the page.
• Index or option subsitution is done with the % operator. For example, for the locator to select one of the sort options on the AMO home is:

In AMOlocators a locator is defined for links to sort options on a page (note the %s near the end of the string)

 sortLink_Keyed =
   “css=div[class~='featured-inner'] div[class='listing-header'] ul li a[href*='?browse=%s']”

To click on the ‘popular’ sort option:

 sort_locator = sortLink_Keyed % ‘popular’
 selenium.click(sort_locator)

Locators in POM test cases

• Defined in the page classes. Defined as strings or a Locator object.
• The POM Locator class is a child class of string which contains methods with_index( ) and with_option( ) for inserting values. For example:

In a page class, the locator for items in the themes category list:

 loc = locator.Locator
 theme_category_link =
   loc("css=div[class~='other-categories'] h3:contains('Themes') + ul + ul li:nth-child(%d) > a")

In a test case, to create a locator for the 2nd category in the list :

 cat_locator = theme_category_link.with_index(2)

If a list requires an offset to index an element (i.e. the 1st element is accessed by “:nth-child(2)” ) the page class may define an alternate locator class which defines with_index( ) to include the offset.

In the case of the Themes landing page the ThemesLoc class maps a single index into the (row, column) pair the page is implemented in.

Parameter driven tests

Test cases use values from TCparams to drive the test.

• Option lists such as user type and Mozilla applications and sort options are used in nested for-loops.
• Values such as maxItemsPerPage or maxCategories are used to limit how many items on a page or number of categories are included in the test run.

(The TCparams class contains three child classes which contains different values for the same option/variable name. In the child classes an option list may contain all valid values or a subset. A max* variable will contain a numeric value to partially drive the length of the test.  The suite driver passes one of the child classes to the test case, overriding options indicated in the tests.py file)

The iterating through option lists has been implemented in two ways:

A) Simple nesting example

 for user_type in tc_parms.userTypeList:
 
     # login or not, as per user_type
     ...
     for app_name in tc_params.mozillaApplicationOptions:
 
         # select the application from the navigation menu
         ...
                         # eventually the destination page is reached
 
                         #  get the count of add-ons on the page
                         addon_count = selenium.get_xpath_count(addon_item_locator)
                         # determine the maximum add-ons to check on this page
                         max_count = min(addon_count, tc_params.maxItemsPerPage)
                         for addon_index in range(1, max_count + 1):
                              # process an add-on

Because of the multiple nested loops the innermost loop can be deeply indented resulting in less space for statements when the 80 character maximum limit is followed.

B) Option tuples

To avoid deep indention from multiple options lists, another implementation creates tuples of all combinations of options.

 param_list = [tc_parms.userTypeList,
                    tc_params.mozillaApplicationOptions, ...]
 
 # create tuples from parameter lists
 param_combos = shared_lib.combos_from_lists(param_list)
 
 for param_tuple in param_combos:
 
     user_type = param_tuple[0]
     app_name = parm_tuple[1]
     ...
     # login or not, as per user_type
     ...
     # select the application from the navigation menu
     ...
     # eventually the destination page is reached
     ...
     # process add-ons like in above code

shared_lib is the shared library where combos_from_lists is defined: general_functions in pre-POM test cases, data_functions in POM.

Verifying a Variety of Add-ons

In an effort to check a variety of add-ons in page layout tests, two mechanisms are used to weakly mix the add-ons:

• multiple sort options
• multiple categories

While it could be argued that traversing multiple pages of one category and one sort option also provides a variety of add-ons, the current implementation does not require a decision on the “best” sort option or category for a test case.

Breadcrumbs, Verbose mode, Tracing back to a page

In test cases that iterate through multiple add-ons, the test cases breadcrumbs provide context to when something happened during a test run. The TC breadcrumbs contain the current sequence of parameters and items being processed.  They can appear in exceptions and verbosity messages.

Tracing back to a page: The text “User: anonymous, App: firefox, Sort: popular, Add-on #2 ‘Firefox sync’” appearing in an exception would indicate the error occurred on the 2nd add-on on the page that’s sorted with the Popular option, etc.

When testcases generate exceptions with extended information, the third part of the extended information is the breadcrumbs string.

Verbose mode:

• pre-POM test cases typically use the variable CONSOLE_TRACE to indicate when breadcrumbs are printed to the console (ex: 0 = no verbose messages, 1 = print options being processed, 2 = print options and add-ons being processed). Currently they are turned off in all test cases.
• The POM test cases don’t have a verbose mode convention. A print statement using the breadcrumbs variable could be added to a test case as needed.

Building Breadrumbs

Pre-POM test cases use string variables named tc_breadcrumbs* and build breadcrumbs them by concatenating a new value to “parent” breadcrumbs. (The tc_* variable name prefix distinguishes it from the breadcrumb element that often appears in the page header.)

The following example sets the breadcrumbs which contains the current options (user, application, etc) and the next add-on in the list that will be processed. Based on the verbosity level the breadcrumbs string is printed.

 for addon_index in range(1, max_addons + 1):
  
    f = '%s, add-on #%s'
    tc_breadcrumbs_addon = f % (tc_breadcrumbs_options, addon_index)

    if CONSOLE_TRACE >= 2: print tc_breadcrumbs_addon

The POM test cases use TestCastBreadcrumbs object that's defined in the testcase_breadcrumbs module which stores a list of (type, value) pairs. The breadcrumbs are updated with an add(type, value) method. A __str__ method creates a formated string with the labels and values.

The SavedExceptions data type contains a breadcrumbs attribute.  POM test cases typically use the breadcrumbs attribute that's inside a SavedExceptions object, typically named ex.

 for addon_index in range(1, max_addons + 1):
  
    ex.breadcrumbs.add('add-on', '#%d' % addon_index)
 
    if CONSOLE_TRACE >= 2: print ex.breadcrumbs

Although the example has a conditional print, most POM test cases haven't implemented a verbosity.

Saved Exceptions

(see Troubleshooting AMO test runs doc for introduction to saved execeptions.)

• Pre-POM test cases implemention:
  • List of exceptions saved in a list
  • Error summary counts saved in a dictionary
  • save_exception function is defined locally in each test case, taking the error summary, error detail and TC breadcrumbs as parameters.
  • saved exceptions detail and summary counts are printed by teardown( ) using a for-loop
• POM test cases implementation:
  • The SavedExceptions class is defined in the saved_exceptions module. The object contains the exception list, error summary counts, and breadcrumbs. It also has a has_exceptions attribute which is True when any exceptions have been saved.
  • The class method save_exception(error_summary, error_detail) updates the exception detail list and error summary count. The internal breadcrumb attribute of the object is added to the detailed exception message.
  • The class method print_exceptions_to_stderr( ) is called from teardown( ) to print the error summary count, exception details, and raise an exception.

Skipping Exceptions ( aka exception to the exception )

Sometimes a verification in a test fails for a narrow set of conditions and the issue is not expected to be fixed soon. An actual example is a verification that expects add-on descriptions on on the browse and detail pages to be the same (accounting for the possibility the browse page description is truncated). A particular add-on has an embedded URL in the description which is followed by a space on only one of the two pages.

Because the problem is minor it’s undesirable to disable the entire test in the suite, disable the verification in the test, or dilute the verification to avoid the condition (i.e. compare only the first 25 chars of the descriptions). The solution has been to add logic to the exception handling that also checks for the limited condition when the exception should be skipped. The result is the test had the potential of finishing green without losing visibility to the exception-to-the-exception.

Implementation in test cases often takes the form similar to

 if conditionX:
     m1 = “this did not match that”
     m2 = “value found ‘%s’, expected ‘%s’ “ % (valueA, valueB)

     #  Bugzilla bug 123456
     #  skip exception if condition Y exists
     if conditionY:
         print “. skipped exception”, m1, “bug 123456”, m2
    else:
         save_exception(m1, m2)      # or  raise Exception, m1+m2

Points to note

• The word Bugzilla is included in the comment to provide a keyword for source code searches.
• The bug# appears in the comment, and more importantly in the print statement.
• The information that would appear in the exception appears in the print statement
• conditionY may include add-on IDs or sub-strings to search for, to keep the condition very narrow. The POM amo_page class has variables that define a list of add-ons related to particular bugs.
• conditions X and Y appear in separate statements so that logic for the exception and exception-to-the-exception is clear.

Normally, a test case would not entirely skip add-ons because of a known condition in one element. Although sometimes an add-on is be skipped completely if a condition affects multiple elements or checks and would require an exception to multiple exceptions..