Mobile/Fennec TestDev: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
** NOTE this is a work in progress **


== What is TestDev and Fennec ==
Fennec TestDev is a part of the Fennec project aimed at bringing automation to Fennec specific areas (such as panning, tabs, bookmarks).  Automation will be using existing automation frameworks that Firefox uses, but new test cases that are specific to Fennec.
== Have a Question? ==


== What is TestDev and Fennec ==
Not quite sure how to get involved?  Have some great ideas?  Maybe something isn't clear or you have a better suggestion?  Feel free to contact me and I can point you in the right direction!


Fennec TestDev is a part of the Fennec project aimed at bringing automation to Fennec specific areas (such as panning, tabs, bookmarks).  Automation will be using existing automation frameworks that Firefox uses, but new test cases that are only useful for Fennec.
Contact Info:
* email: jmaher at mozilla dot com
* irc: jmaher


== Areas to focus on ==
== Areas to focus on ==


* Tabs - {{bug|485855}}
** Loading and switching between tabs
** Test maximum tabs? (>4 ?)
** Slide over and display tab form
** + button, X button on tab preview
** Tab focus when adding, closing tab
** Tab preview (full screen, or any black/white?  test with a solid color page)
** Verify softkeyboard for entering URL on new tab


* Bookmarks - {{bug|486490}}
=== Tabs ===
** Add a bookmark and verifies URI exists
 
** Verify panning in the bookmark page
* Tracked in: {{bug|485855}} - zad (got the basics done!)
** Test adding, editing, deleting tags
* Recommended harness: (browser-chrome)
** Test adding, moving, editing names, deleting folders
 
** Test adding, editing url, editing title, deleting bookmarks
The tabs UI is new and unique to Fennec.  We have different limitations and display issues to work with.  This project is UI automation to cover the basic tab functionality, UI handling of the tabs and browser interaction.
** Test adding, moving, editing, deleting subfolders
 
** Verify editing bookmark via star icon (instead of bookmark icon)
=== Bookmarks ===
** Verify clicking on a bookmark loads the web page
 
** Verify softkeyboard launches for all text fields
* Tracked in: {{bug|486490}} - happyhans (waiting on review)
* Recommended harness: (browser-chrome)
 
Just like the tabs, the Bookmark UI in Fennec is new and unique to Fennec.  A lot of work has been done to make this mobile and small screen friendly.  We have most of the functionality of the Firefox bookmarks, but a different user flow.  This project entails writing UI automation to cover the bookmark usage scenarios, layout, functionality, and edge cases.
 
=== Preferences ===
 
* Tracked in: {{bug|489158}}
* Recommended harness: (browser-chrome)
 
In Fennec we have a limited amount of Preferences and options as compared to Firefox.  These require us to navigate the UI to display the preferences pane and turn on/off the preferences.  It is a simple UI compared to Firefox.  This project utilizes the UI to automate the viewing, editing, and testing of the preferences. 
 
=== Panning / Zoom ===
 
* Tracked in: {{bug|489160}}
* Recommended harness: (browser-chrome)
 
Fennec has a whole new UI to accommodate the stylus method of input rather than a mouse.  With this UI, we allow panning (dragging the screen around) as well as double click to zoom in/out.  There are also special pan states to view the tab pane and config pane.  This creates a whole set of interesting test cases since this is the core navigation method for the mobile browser. 
 
This project focuses heavily on UI automation and will most likely be done in browser-chrome, but could end up utilizing other tools or test methods.  This is a challenging project since there is potential for a lot of test cases and harder to solve problems like verifying the pan or zoom is accurate.
 
=== Download Manager ===


* Preferences
* Tracked in: {{bug|489161}}
** Verify panning of preference list
* Recommended harness: (browser-chrome)
** Verify on/off retains state
** Would be nice to verify core functionality


* Panning / Zoom
Download Manager just had a complete overhaul and is ready for some automation.  This has the same functionality as you would expect from a download manager on the desktop, except it is mobile friendly.  Developing the automation will require downloads of files and UI automation to get to the download manager and verify the display of the data. 
** Verify panning up, down, left, right on a large page
** Verify multiple pans on a long page top to bottom to top
** Verify double tap zooms in and allows panning
** Verify double tap zooms in, double tap zooms out, and panning still works


* Download Manager
=== Awesome bar / Search providers / History ===
** Verify panning in list of downloaded items
** Verify large amount of items in download list
** Verify searching (including softkeyboard)
** Verify clearlist button works as expected


* Awesome bar / Search providers / History
* Tracked in: {{bug|489163}}
** Verify Awesome Bar displays history/bookmarks
* Recommended harness: (browser-chrome)
** Verify we can use the search providers that we have by default
** Verify clear history removes history and retains bookmarks


* Tel protocol and integrated phone support
As all of the other Fennec projects, this involves testing the awesome bar which has similar functionality to Firefox.  There are existing unittests which cover the query and return values in the awesomebar.  For this project we need to test it from a UI perspective, specifically the unique Fennec features.
 
=== Tel protocol and integrated phone support ===
** Verify support of the tel: protocol
** Verify support of the tel: protocol
** Verify we send the correct phone number and handle error conditions
** Verify we send the correct phone number and handle error conditions


* Notification bar prompts
=== Notification bar prompts ===
** Test Password prompt (remember, yes, no)
** Test Password prompt (remember, yes, no)
** Test Geolocation prompt (remember, yes, no)
** Test Geolocation prompt (remember, yes, no)


* Hardkeys on N810
=== Hardkeys on N810 ===
** Verify Full screen button
** Verify Full screen button
** Verify dpad navigation (up, down, left, right, center)
** Verify dpad navigation (up, down, left, right, center)
** Verify ESC key
** Verify ESC key


* Hardkeys on HTC Touch Pro
=== Hardkeys on HTC Touch Pro ===
** Verify screen rotation
** Verify screen rotation
** Others?
=== NSPR Unit Tests ===
NSPR is the core piece of porting that we need to do for each new platform.  We need to ensure we have a solid fully comprehensive test suite which covers all the necessary api's that we rely on.
In porting to Symbian there is discussion around doing this: {{bug|432430}} {{bug|501172}}
These live in the source tree here:
http://mxr.mozilla.org/mozilla-central/source/nsprpub/pr/tests/
Actions todo:
* verify these run on existing platforms
* verify these are reporting accurately
* verify we have comprehensive coverage with these tests (??)
=== Performance ===
Taras has come up with a great start on a [[http://blog.mozilla.com/tglek/2009/04/23/benchmarks-and-instrumentation-for-fennecetc/ Fennec performance story]].  From a QA perspective we need to outline the tests that make sense on mobile (not just a general web browser on a desktop).  This project will also require taking what Taras creates for performance testing and working to get it integrated into talos! 
Here are my initial thoughts on performance scenarios we should be measuring:
* Application launch time
* Time to load a webpage - do this on various pages(difficult since we are dealing with live pages on the internet) - Tp
* Time to pan to the bottom of a webpage (including the number of hiccups and false starts to the pan) - do this on various pages - tPan
* Time to display the tab and config panes
* Time to browse back one page in history (including snapping the config pane, clicking the back button)
* Time to enter a URL with a large awesome bar history
* Time to close Fennec
* Time to open a new tab
* Time to open a new tab and load a page
* Time to Zoom in/out on a page (test with multiple pages of various content/length) - tZoom
A further portion of this project is to take the performance tests and expand those into stress tests.  I can see these types of tests (measure time, memory usage, ???):
* Open a 3 new tabs and load pages in them, then close. Repeat 5000 times. 
* Pan up and down a long page, zoom in every 3rd pan, out after the following pan 5000 times.
* browse back and forth through a history of 1000 pages using the navigation buttons
* exercise the URL bar for 1000 different pages


== Other TestDev related todo's ==
== Other TestDev related todo's ==
* Clearly define methods for:
** Excluding test cases from the Firefox tree
** Creating unique test cases for Fennec that only run on Fennec


* Follow up on existing Unittest bugs and get resolution (either in workarounds, or fixes)
* Follow up on existing Unittest bugs and get resolution (either in workarounds, or fixes)

Latest revision as of 15:04, 30 June 2009

What is TestDev and Fennec

Fennec TestDev is a part of the Fennec project aimed at bringing automation to Fennec specific areas (such as panning, tabs, bookmarks). Automation will be using existing automation frameworks that Firefox uses, but new test cases that are specific to Fennec.

Have a Question?

Not quite sure how to get involved? Have some great ideas? Maybe something isn't clear or you have a better suggestion? Feel free to contact me and I can point you in the right direction!

Contact Info:

  • email: jmaher at mozilla dot com
  • irc: jmaher

Areas to focus on

Tabs

  • Tracked in: bug 485855 - zad (got the basics done!)
  • Recommended harness: (browser-chrome)

The tabs UI is new and unique to Fennec. We have different limitations and display issues to work with. This project is UI automation to cover the basic tab functionality, UI handling of the tabs and browser interaction.

Bookmarks

  • Tracked in: bug 486490 - happyhans (waiting on review)
  • Recommended harness: (browser-chrome)

Just like the tabs, the Bookmark UI in Fennec is new and unique to Fennec. A lot of work has been done to make this mobile and small screen friendly. We have most of the functionality of the Firefox bookmarks, but a different user flow. This project entails writing UI automation to cover the bookmark usage scenarios, layout, functionality, and edge cases.

Preferences

  • Tracked in: bug 489158
  • Recommended harness: (browser-chrome)

In Fennec we have a limited amount of Preferences and options as compared to Firefox. These require us to navigate the UI to display the preferences pane and turn on/off the preferences. It is a simple UI compared to Firefox. This project utilizes the UI to automate the viewing, editing, and testing of the preferences.

Panning / Zoom

  • Tracked in: bug 489160
  • Recommended harness: (browser-chrome)

Fennec has a whole new UI to accommodate the stylus method of input rather than a mouse. With this UI, we allow panning (dragging the screen around) as well as double click to zoom in/out. There are also special pan states to view the tab pane and config pane. This creates a whole set of interesting test cases since this is the core navigation method for the mobile browser.

This project focuses heavily on UI automation and will most likely be done in browser-chrome, but could end up utilizing other tools or test methods. This is a challenging project since there is potential for a lot of test cases and harder to solve problems like verifying the pan or zoom is accurate.

Download Manager

  • Tracked in: bug 489161
  • Recommended harness: (browser-chrome)

Download Manager just had a complete overhaul and is ready for some automation. This has the same functionality as you would expect from a download manager on the desktop, except it is mobile friendly. Developing the automation will require downloads of files and UI automation to get to the download manager and verify the display of the data.

Awesome bar / Search providers / History

  • Tracked in: bug 489163
  • Recommended harness: (browser-chrome)

As all of the other Fennec projects, this involves testing the awesome bar which has similar functionality to Firefox. There are existing unittests which cover the query and return values in the awesomebar. For this project we need to test it from a UI perspective, specifically the unique Fennec features.

Tel protocol and integrated phone support

    • Verify support of the tel: protocol
    • Verify we send the correct phone number and handle error conditions

Notification bar prompts

    • Test Password prompt (remember, yes, no)
    • Test Geolocation prompt (remember, yes, no)

Hardkeys on N810

    • Verify Full screen button
    • Verify dpad navigation (up, down, left, right, center)
    • Verify ESC key

Hardkeys on HTC Touch Pro

    • Verify screen rotation
    • Others?

NSPR Unit Tests

NSPR is the core piece of porting that we need to do for each new platform. We need to ensure we have a solid fully comprehensive test suite which covers all the necessary api's that we rely on.

In porting to Symbian there is discussion around doing this: bug 432430 bug 501172

These live in the source tree here: http://mxr.mozilla.org/mozilla-central/source/nsprpub/pr/tests/

Actions todo:

  • verify these run on existing platforms
  • verify these are reporting accurately
  • verify we have comprehensive coverage with these tests (??)

Performance

Taras has come up with a great start on a [Fennec performance story]. From a QA perspective we need to outline the tests that make sense on mobile (not just a general web browser on a desktop). This project will also require taking what Taras creates for performance testing and working to get it integrated into talos!

Here are my initial thoughts on performance scenarios we should be measuring:

  • Application launch time
  • Time to load a webpage - do this on various pages(difficult since we are dealing with live pages on the internet) - Tp
  • Time to pan to the bottom of a webpage (including the number of hiccups and false starts to the pan) - do this on various pages - tPan
  • Time to display the tab and config panes
  • Time to browse back one page in history (including snapping the config pane, clicking the back button)
  • Time to enter a URL with a large awesome bar history
  • Time to close Fennec
  • Time to open a new tab
  • Time to open a new tab and load a page
  • Time to Zoom in/out on a page (test with multiple pages of various content/length) - tZoom

A further portion of this project is to take the performance tests and expand those into stress tests. I can see these types of tests (measure time, memory usage, ???):

  • Open a 3 new tabs and load pages in them, then close. Repeat 5000 times.
  • Pan up and down a long page, zoom in every 3rd pan, out after the following pan 5000 times.
  • browse back and forth through a history of 1000 pages using the navigation buttons
  • exercise the URL bar for 1000 different pages

Other TestDev related todo's

  • Follow up on existing Unittest bugs and get resolution (either in workarounds, or fixes)

References

  • Mochitest
  • Browser-chrome
  • Mochitest-chrome
  • Reftest
  • Fennec Automation
  • Building Fennec
Retrieved from "https://wiki.allizom.org/index.php?title=Mobile/Fennec_TestDev&oldid=152758"