MozillaWiki

Mobile/Testing: Difference between revisions

< Mobile
(formatting tweak)
 
(346 intermediate revisions by 13 users not shown)
Line 1: Line 1:
= Status =
= About =
[https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bmobile-testing%5D mobile testing bugs]
== Overview  ==


{| cellspacing="1" cellpadding="1" border="1" style="width: 578px; height: 194px;"
We use automated testing of our mobile browser to ensure that new releases get better, not worse. As with Firefox for Desktop, we are concerned both with testing the correctness of the browser (no crashes, pages should render correctly) and performance (new releases should be faster than old ones).
|-
! scope="col" | 
! scope="col" | standalone
! scope="col" | production
! scope="col" | native UI
! scope="col" | number running
|-
! scope="row" | reftests
| align="center" | yes
| align="center" | yes
| align="center" | no
| align="center" | 3557/6113
|-
! scope="row" | mochitests
| align="center" | yes
| align="center" | yes
| align="center" | yes
| align="center" | 17873/230492
|-
! scope="row" | browser chrome
| align="center" | no
| align="center" | yes
| align="center" | no
| align="center" | 422/422
|-
! scope="row" | xpcshell
| align="center" | yes
| align="center" | no
| align="center" | yes 
| align="center" | 848/1202
|-
! scope="row" | js reftests
| align="center" | no
| align="center" | yes
| align="center" | no
| align="center" | 54570/55357
|-
! scope="row" | crash tests
| align="center" | yes
| align="center" | yes
| align="center" | no
| align="center" | 1880/1888
|-
! scope="row" | talos
| align="center" | no  
| align="center" | yes
| align="center" | yes 
| align="center" | 9/9
|}


== Reftests ==
Note that the scope of this page is restricted to Firefox for Android (Fennec).
* over half the tests are not run on mobile
* most of this is due to not running large directories of tests
* somebody needs to go in and get more details as to what is not running.
* does running tests locally work?  I find a lot of failures when I don't have the proper resolution and use the --ignore-window-size flag.


== Mochitests ==
== Documentation ==
* currently running 11 directories
* last week releng had 16 more directories running in staging which are green (m5-8), those should be turned on this week
* layout/style has 96000+ tests and :mw22 is looking at cleaning those tests up
** some require scrollbars and we don't have those on mobile
** 1 has a e10s requirement which isn't an obvious fix
* content/* tests have patches to run with e10s and on mobile {{bug|668283}}
** seems to be blocked on {{bug|621363}}


== Browser Chrome ==
* [[Mobile/Testing/Architectural_Overview|Architectural Overview]] of our mobile automated testing systems.
[https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bmobile-testing%5D%20%5Bbrowser-chrome%5D browser-chrome test bugs]
* For information on running the same tests we run in automation on your desktop, see the [https://wiki.mozilla.org/Mobile/Fennec/Android/Testing testing] for Android wiki page.
(Broken out of Mochitests because of special requirements)
the requires to be run out of the package-tests directory.  When doing a 'make package-tests', we create a tests.jar file which we copy to the device (in the profile directory) and run the tests from there.


== XPCShell ==
= Status Meetings =
Patches landed: developers can run xpcshell on Android via ADB. About 2/3 of the tests pass; bugs opened for the remainder:
Interested parties meet every other week to discuss the current status of testing on Mobile and coordinate the required work between teams.
[https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bmobile-testing%5D%20%5Bxpcshell%5D xpcshell test bugs]


== JS Reftests ==
* Notes, Schedule, and Zoom Link: https://docs.google.com/document/d/1hxZ0xLFbtXA3AKZltYwL21lRA8qRvrwVjSlQO4dfv0U/edit?usp=sharing
* we already run the majority of these in production
* Email aerickson@mozilla.com to be added.
* we need to create a link to the jstests.list file so we can run these on a developer machine
* we should document the commented out tests with more details


== Crash tests ==
== Archive of Notes ==
* almost all are running, we should look into the tests we have turned off in the manifest files and document them better.
[[Template:MobileTesting|template]]


== Talos ==
Create a new weekly agenda from the [[Template:MobileTesting|template]]:
* we don't run tp5, but we have tp4m. 
* we need to turn off ts, txul and replace with ts_paint and tpaint
* tpan/tzoom/tp4m are failing frequently {{bug|662936}}


= Status Meetings =
<createbox>
There will be weekly meetings to discuss the current status of testing on Mobile and coordinate the required work between teams.
align=left
Details
type=create
* Wednesdays @ 10:30am PST/PDT
preload=Template:MobileTesting
* Meeting in Warp Core
default={{#time: m_d_y | wednesday}}
* Vidyo in Warp Core
prefix=Mobile/Testing/
* #mobile for back channel
== Notes ==
[[template]]


Q1
</createbox>
# [[Mobile/Testing/01_25_12 | 01/25/12]]
# NOW USING GOOGLE DOC, ABOVE
# [[Mobile/Testing/01_18_12 | 01/18/12]]
# [[Mobile/Testing/08_15_18 | 08/15/18]]
# [[Mobile/Testing/01_11_12 | 01/11/12]]
# [[Mobile/Testing/08_01_18 | 08/01/18]]
# [[Mobile/Testing/01_04_12 | 01/04/12]]
# [[Mobile/Testing/07_18_18 | 07/18/18]]
# [[Mobile/Testing/07_04_18 | 07/04/18]]  Cancelled!
# [[Mobile/Testing/06_20_18 | 06/20/18]]
# [[Mobile/Testing/06_06_18 | 06/06/18]]
# [[Mobile/Testing/05_23_18 | 05/23/18]]
# [[Mobile/Testing/05_09_18 | 05/09/18]]
# [[Mobile/Testing/04_25_18 | 04/25/18]]
# [[Mobile/Testing/04_11_18 | 04/11/18]]
# [[Mobile/Testing/03_28_18 | 03/28/18]]
# [[Mobile/Testing/03_14_18 | 03/14/18]]
# [[Mobile/Testing/02_28_18 | 02/28/18]]
# [[Mobile/Testing/02_14_18 | 02/14/18]]
# [[Mobile/Testing/01_31_18 | 01/31/18]]
# [[Mobile/Testing/01_17_18 | 01/17/18]]


2010
Older
Q4
# [[Mobile/Testing/Archive | Archive of notes from 2011 - 2017]]
# [[Mobile/Testing/12_28_11 | 12/28/11]]
# [[Mobile/Testing/12_21_11 | 12/21/11]]
# [[Mobile/Testing/12_14_11 | 12/14/11]]
# [[Mobile/Testing/12_07_11 | 12/07/11]]
# [[Mobile/Testing/11_30_11 | 11/30/11]]
# [[Mobile/Testing/11_23_11 | 11/23/11]]
# [[Mobile/Testing/11_16_11 | 11/16/11]]
# [[Mobile/Testing/11_09_11 | 11/09/11]]
# [[Mobile/Testing/11_02_11 | 11/02/11]]
# [[Mobile/Testing/10_26_11 | 10/26/11]]
# [[Mobile/Testing/10_19_11 | 10/19/11]]
# [[Mobile/Testing/10_12_11 | 10/12/11]]
# [[Mobile/Testing/10_05_11 | 10/05/11]]
Q3
# [[Mobile/Testing/09_28_11 | 09/28/11]]
# [[Mobile/Testing/09_21_11 | 09/21/11]]
# [[Mobile/Testing/08_24_11 | 08/24/11]]
# [[Mobile/Testing/08_17_11 | 08/17/11]]
# [[Mobile/Testing/08_08_11 | 08/08/11]]
# [[Mobile/Testing/08_01_11 | 08/01/11]]
# [[Mobile/Testing/07_06_11 | 07/06/11]]

Latest revision as of 20:18, 24 September 2020

About

We use automated testing of our mobile browser to ensure that new releases get better, not worse. As with Firefox for Desktop, we are concerned both with testing the correctness of the browser (no crashes, pages should render correctly) and performance (new releases should be faster than old ones).

Note that the scope of this page is restricted to Firefox for Android (Fennec).

Documentation

  • Architectural Overview of our mobile automated testing systems.
  • For information on running the same tests we run in automation on your desktop, see the testing for Android wiki page.

Status Meetings

Interested parties meet every other week to discuss the current status of testing on Mobile and coordinate the required work between teams.

Archive of Notes

template

Create a new weekly agenda from the template:

<createbox> align=left type=create preload=Template:MobileTesting default=09_10_25 prefix=Mobile/Testing/

</createbox>

  1. NOW USING GOOGLE DOC, ABOVE
  2. 08/15/18
  3. 08/01/18
  4. 07/18/18
  5. 07/04/18 Cancelled!
  6. 06/20/18
  7. 06/06/18
  8. 05/23/18
  9. 05/09/18
  10. 04/25/18
  11. 04/11/18
  12. 03/28/18
  13. 03/14/18
  14. 02/28/18
  15. 02/14/18
  16. 01/31/18
  17. 01/17/18

Older

  1. Archive of notes from 2011 - 2017
Retrieved from "https://wiki.allizom.org/index.php?title=Mobile/Testing&oldid=1231060"