MozillaWiki

Auto-tools/Projects/Robocop/WritingTests: Difference between revisions

Page Discussion

Auto-tools/Projects/Robocop/WritingTests (view source)

Revision as of 14:20, 9 March 2016

6,365 bytes added ,  9 March 2016
→‎Logging
 
(26 intermediate revisions by 5 users not shown)
Line 1: Line 1:
= What's this? =
Robotium is a test framework created to make it easy to write powerful and robust automatic black-box test cases for Android applications. Robocop provides a wrapper around Robotium making it even easier to write and execute UI tests for native Fennec.
 
New tests should probably use the UITest base class - see the documentation at [[Mobile/Fennec/Android/UITest|UITest]]. Most of the information below will also be relevant to users of UITest (though the information should probably be curated).


Robotium is a test framework created to make it easy to write powerful and robust automatic black-box test cases for Android applications. Robocop provides a wrapper around Robotium making it even easier to write and execute UI tests for native Fennec.
== Creating a new test ==
These tests will extend BaseTest.


= Creating a new test =
The best way to create a new Robocop test is to copy and modify an existing one -- see mobile/android/base/tests.


The best way to create a new Robocop test is to copy and modify an existing one -- see mobile/android/base/tests.  
(We are currently upgrading our tests from Robotium 3.6 to 4.2 [{{bug|912590}}], so the APIs used in the tests may be out of date. It might be helpful to take a look at the [http://code.google.com/p/robotium/wiki/Changelog Robotium changelog] to see some of the new API methods.)


Create a new file in the test directory named test[YourFeature].java.in.
Create a new file in the test directory named test[YourFeature].java.in.
Line 20: Line 23:
</pre>
</pre>


For convenience, you can also extend one of the provided abstract classes, like BaseTest. This derives from ActivityInstrumentationTestCase2 but also provides a bunch of boilerplate and utility functions.
For convenience, you can also extend one of the provided abstract classes, like BaseTest, PixelTest, or AboutHomeTest. BaseTest derives from ActivityInstrumentationTestCase2 but also provides a bunch of boilerplate and utility functions.
<pre>
<pre>
public class testMyFeature extends BaseTest
public class testMyFeature extends BaseTest
Line 32: Line 35:
</pre>
</pre>


If you extend BaseTest, the setUp() and tearDown() are provided for you, but you can still override them and do additional setup/teardown if needed; just make sure you call super.setUp() and super.tearDown() in your overiddes as necessary.
If you extend one of the provided abstract classes, the setUp() and tearDown() are provided for you, but you can still override them and do additional setup/teardown if needed; just make sure you call super.setUp() and super.tearDown() in your overiddes as necessary.
 
Finally, add your new test file to the test manifest: mobile/android/base/tests/robocop.ini:
<pre>
[testAllPagesTab]
[testHistoryTab]
...
[testMasterPassword]
[testDeviceSearchEngine]
[testMyFeature] # NEW!
</pre>
 
=== Creating a new Content Provider test ===
 
We have infrastructure for testing content providers in an isolated environment. This means that we ensure that any updates to the databases behind content providers will not affect or be affected by Firefox.
 
To write a content provider test, follow the same file naming and Java packaging guidelines described for UI tests, but create a class that inherits from ''ContentProviderTest'' instead of ''BaseTest''. Each Content Provider test class must have the same three methods as UI tests (setUp(), test[YourFeature]() and tearDown) with one important difference: the ''setUp()'' method should call the following parent method:
 
<pre>
protected void setUp(String className, String authorityField);
</pre>
 
Where ''className'' is the class of your content provider and ''authorityField'' is the name of the property in ''BrowserContract'' containing your content provider AUTHORITY string. For example, for BrowserProvider, the ''setUp()'' method would look something like:
 
<pre>
public setUp() throws Exception {
    super.setUp("@ANDROID_PACKAGE_NAME@.db.BrowserProvider", "AUTHORITY");
    // Then other setup operations...
}
</pre>


Finally, add your new test file to the test manifest: mobile/android/base/tests/robocop.ini.
After this is called, ''mProvider'' will point to an isolated instance of your content provider and you can make insert/query/update/delete calls on it as expected. For more details and sample code on how to implement Content Provider tests, see testBrowserProvider.java.in.


= APIs  =
== APIs  ==
Robotium itself provides a rich API through the Solo class - javadocs for Solo are available at [1].
Robotium itself provides a rich API through the Solo class - [http://robotium.googlecode.com/svn/doc/index.html javadocs] [1] for Solo are available.


"Robocop" provides an additional API to make common tasks easier. The main interfaces are Actions, Elements, and Driver.  
"Robocop" provides an additional API to make common tasks easier. The main interfaces are Actions, Elements, and Driver.  
Line 52: Line 84:
   //Sends a drag action across the screen
   //Sends a drag action across the screen
   void drag(int startingX, int endingX, int startingY, int endingY)
   void drag(int startingX, int endingX, int startingY, int endingY)
  // Run a sql query on the specified database
  public Cursor querySql(String dbPath, String sql);
</pre>
</pre>


Line 82: Line 116:
   int getGeckoWidth();
   int getGeckoWidth();
</pre>
</pre>
See <objdir>/mobile/android/base/R.java to find ids that can be used with Driver.findElement()


Finally, an evolving set of test base classes - BaseTest, PixelTest, etc - can be leveraged for some types of tests.
Finally, an evolving set of test base classes - BaseTest, PixelTest, etc - can be leveraged for some types of tests.


= Usable IDs =
== Tips ==
The following is a list of ids that can be used with Driver.findElement(). Most of the following ids have not been tested, so might have unexpeced results, or require increased APIs for them. To know how a given object is used, in mobile/android/base, grep R.id.[id-name] *
=== Event timing ===
(from Objdir/mobile/android/base/R.java#id)
A recurring issue when writing UI tests is the timing of events. To enter a URL, you need to click on the awesome bar and then send the key events for the text. If you click() and then immediately sendKeys(), the text probably won't get to the awesome bar. If you sleep() briefly before and after clicking, the task will probably succeed...but how long are those sleep() calls? Will the test still work on other devices, reliably? Avoid the temptation to scatter sleep() throughout your test. Whenever possible, wait for events or other feedback to verify the UI is in the required state before proceeding. For example, many tests will want to wait for startup before starting the test: driver.waitForGeckoEvent("Gecko:Ready").
=== Use waitForCondition instead of sleep() ===
If you cannot find an event to wait for and you are still tempted to sleep(), consider using waitForCondition() instead. waitForConditon() gives you a simple way to poll for a condition for a specified maximum amount of time. This provides the advantage that the test can proceed as soon as the condition is met, reducing test time.
 
=== Logging ===
Robocop logs to both logcat and a file log via Assert.dumpLog (or FennecNativeDriver.log). Test code can use these, or log as a side-effect of Assert.is, Assert.ok, etc. Do not call android.util.Log.i/w/e/etc directly.
 
'''Robocat''' is a command line tool to transform the JSON robocop dumps into logcat into a human-readable format: https://github.com/pocmo/Robocat
 
=== Indirect Database Operations ===
Robocop UI operations (eg click on a certain button) may initiate database operations (eg create a new record in a database table). When performing these actions manually, the effect appears to occur instantly, but of course the database operation takes some time, and often occurs on a background thread, independent of the UI. Robocop tests that indirectly initiate database operations and then immediately check for the effect of that operation (eg search for text in a table) may fail, or fail intermittently. It is generally better to loop, periodically checking for the effect to occur. See testBookmark for an example.


<pre>
=== Screenshots ===
abouthome_content
Robocop takes a screenshot when a test fails. See http://gbrownmozilla.wordpress.com/2013/03/26/screenshots-for-robocop/ for details.
add_tab
 
addons
=== Do not rely on view indices ===
address_bar
Some Robotium APIs allow a test to specify a view by index; for instance, Solo.clickOnButton(int index). Selecting a button with a constant index may work for a broad set of circumstances only to fail under a different version of Android, or when a new button is added, or views are simply organized differently. In most cases, using something like Solo.clickOnButton(String text) has proven to be more reliable.
agent_mode
 
all_pages_list
=== Check and report return values; use BaseTest.waitForText ===
awesome_bar
Intermittent test failures can be very difficult to debug and often need to be debugged based entirely on the log from the failed test. Ignoring robotium return codes can make the task more complicated. For example, consider boolean Solo.waitForText(String). waitForText blocks until the specified text is displayed, or until a maximum timeout is reached. If the text is found, waitForText returns true; if it times out, false. Tests rarely assert based on waitForText, but use it to wait until a certain view is displayed:
awesome_screen
  clickOnSomething()
awesomebar_button
  Solo.waitForText(someTitle)
awesomebar_tabs
  clickOnSomethingElse()
awesomebar_text
  mAsserter.ok(...)
background
 
bookmark
If clickOnSomething fails to bring up the expected view, the test might only fail on the mAsserter.ok(), even though the problem is much earlier. So it's better to use something like:
bookmark_icon
  clickOnSomething()
bookmark_title
  mAsserter.ok(Solo.waitForText(someTitle), ...)
bookmark_url
  clickOnSomethingElse()
bookmarks_list
  mAsserter.ok(...)
browser_toolbar
or:
close
  clickOnSomething()
container
  if (!Solo.waitForText(someTitle))
doorhanger_choices
      mAsserter.dumpLog(...)
doorhanger_container
  clickOnSomethingElse()
doorhanger_title
  mAsserter.ok(...)
favicon
or use BaseTest.waitForText, which conveniently logs on failure:
forward
  clickOnSomething()
gecko_layout
  waitForText(someTitle)
grid
  clickOnSomethingElse()
history_list
  mAsserter.ok(...)
info
 
list
=== Be specific: Use clickOnButton or clickOnView in preference to clickOnText ===
main_layout
We have seen several intermittent failures when using Solo.clickOnText because the specified text appears in more than one place. It's really easy to write a test that says clickOnText("OK"), intending of course to click on the OK button and not the prompt that says "Click OK to proceed"! Worse yet, that test might work sometimes but not others, since Robotium will simply click on the first view with the specified text. Using view-specific functions, like clickOnButton(String text) addresses many of these issues; look over the screen very carefully to catch the others! (You can also enumerate views to identify the view intended, then use clickOnView(View).)
notification_image
=== Be careful when relying on scrolling in waitForText ===
notification_progressbar
Solo.waitForText (and some other Robotium functions) will conveniently scroll down in a list to find text that is not displayed. But it will not scroll up again! Consider a test like this:
notification_text
notification_title
outline
plugin_container
preferences
quit
recommended_addon_list
reload
save_as_pdf
screenshot
select_list
share
site_security
stop
tabs
tabs_count
title
url
</pre>


= Tips =
  <open a menu or display a list>
=== Event timing ===
  mAsserter.ok(waitForText("Item 1"), ...
A recurring issue when writing UI tests is the timing of events. To enter a URL, you need to click on the awesome bar and then send the key events for the text. If you click() and then immediately sendKeys(), the text probably won't get to the awesome bar. If you sleep() briefly before and after clicking, the task will probably succeed...but how long are those sleep() calls? Will the test still work on other devices, reliably? Avoid the temptation to scatter sleep() throughout your test. Whenever possible, wait for events or other feedback to verify the UI is in the required state before proceeding. For example, many tests will want to wait for startup before starting the test: driver.waitForGeckoEvent("Gecko:Ready").
  mAsserter.ok(waitForText("Item 2"), ...
=== Activities ===
Most tests will access UI views using FennecNativeDriver.findElement, which takes an activity as an argument. Some tests will only interact with one activity, but others may need to use multiple activities. For instance, when there are multiple tabs, there may be multiple awesome bar activities; use the wrong activity and you'll be accessing an unexpected element / view! See testNewTab for an example with multiple activities.


= More Information =
* If both Item 1 and Item 2 are visible, this test passes.
* If Item 1 occurs first in the list, with Item 2 later, and Item 2 is off the bottom of the screen, then the first waitForText succeeds immediately, the second waitForText scrolls down, then succeeds, and the test passes.
* If Item 2 occurs first in the list, with Item 1 later, and Item 1 is off the bottom of the screen, then the first waitForText scrolls down to find Item 1. If that operation scrolls Item 2 off the top of the screen, then the second waitForText will fail!


[1] http://code.google.com/p/robotium/
Note that the test is therefore dependent on screen size and orientation, as well as Robotium scrolling behavior.


[2] https://github.com/jayway/robotium/tree/master/robotium-solo/src/main/java/com/jayway/android/robotium/solo
== Useful links ==
#[[Mobile/Fennec/Android/UITest|Our UITest base class documentation]]
#Latest Robotium javadocs: http://robotium.googlecode.com/svn/doc/index.html
#Robotium project page: http://code.google.com/p/robotium/
#Robotium changelog for versions: http://code.google.com/p/robotium/wiki/Changelog
#Robotium GitHub repo: https://github.com/jayway/robotium/tree/master/robotium-solo/src/main/java/com/jayway/android/robotium/solo
Skaspari
118

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/402595...1120946"