QA/Automation/Projects/MTBF: Difference between revisions

From MozillaWiki
< QA‎ | Automation‎ | Projects
Jump to navigation Jump to search
No edit summary
Line 69: Line 69:
         "blocks":[987035],
         "blocks":[987035],
         "block_type":"contains"
         "block_type":"contains"
        "status":"NEW, REOPENED"
     }
     }
</bugzilla>
</bugzilla>


=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1034494 MTBF Marionette Meta Bug] ===
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1034494 MTBF Marionette Meta Bug] ===

Revision as of 09:11, 21 August 2014

Overview

The MTBF tests are a suite of Firefox OS tests built on top of the Gaiatest (Gaia UI Tests) Framework. The tests run on real Firefox OS devices, and use Marionette to drive the device's UI.

Mean time between failures (MTBF) is the predicted elapsed time between inherent failures of a system during operation. MTBF can be calculated as the arithmetic mean (average) time between failures of a system. The MTBF is typically part of a model that assumes the failed system is immediately repaired (mean time to repair, or MTTR), as a part of a renewal process. This is in contrast to the mean time to failure (MTTF), which measures average time to failures with the modeling assumption that the failed system is not repaired (infinite repair time).

Name: Mean Time Between Failures (MTBF)
Leads: Taiwan QA Team
Contributors: Paul Yang, Walter Chen
MDN: https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/MTBF_tests
Git Repository: https://github.com/Mozilla-TWQA/MTBF-Driver/


Backlog

  • Crash info detection
  • Report enhancement
  • Hang detection
  • Integrating mozbase b2g related modules
  • Datazilla integration

More Information

Recent Status

Date Version Details
- - -


Test Suites

Suite Version Tests
- - -

Current Bugs

MTBF Meta Bug

Bugzilla query error

Query options must be valid JSON.1


MTBF Marionette Meta Bug

Full Query
ID Summary Component Status Resolution Assigned to Whiteboard Blocking b2g
940911 [MTBF][Marionette] Marionette getcrashreport() Marionette RESOLVED WONTFIX [runner][affects=webqa loop] ---
940915 [Marionette] marionette's detecting of hung device Marionette RESOLVED FIXED ---
979184 [MTBF][Marionette] phone crashed due to Marionette exception Marionette RESOLVED WONTFIX [affects=b2gdevice] ---
984208 [Marionette Client] Modifying shuffle to put its seed outside Marionette RESOLVED FIXED Luv Agarwal(:lagarwal) [good first bug][lang=py][mentor=mdas] ---
987046 [Gaia-ui-tests] Discussion of Gaia-ui-tests/marionette restruct Gaia::UI Tests RESOLVED WONTFIX ---
990329 [MTBF] can't set keyboard.ftu to false and marionette client crashed (memory report attached) General RESOLVED WORKSFORME ---
1003043 [MTBF][Marionette] Some exception caught during test running Marionette RESOLVED INCOMPLETE [affects=mtbf][affects=b2gdevice] ---
1035039 [MTBF][Marionette] filedescriptor out of range in select() MTBF RESOLVED WORKSFORME ---
1035040 [MTBF][Marionette] Connection to Marionette server is lost. Check gecko.log (desktop firefox) or logcat (b2g) for errors. Marionette RESOLVED WORKSFORME [affects=b2gdevice] ---
1048833 [MTBF] Memory grows unexpectedly in system app Marionette RESOLVED FIXED Malini Das [:mdas] - Away, not checking bugmail 2.0+
1095817 Tens of thousands of copies of specialpowers Marionette RESOLVED INCOMPLETE [MemShrink:P1] ---
1141418 After 9 hours running, marionette server response incorrectly Marionette RESOLVED FIXED Ting-Yu Chou [:ting] (away) 2.2+
1168045 [MTBF][Marionette] Python Marionette network having issues after continuous browser tests Marionette RESOLVED INCOMPLETE ---
1185501 Support mozlog 3.0 in gaiatest Gaia::UI Tests VERIFIED FIXED Paul Yang [:pyang] (away) ---
1198950 [Marionette] All Jenkins jobs are failing almost all of the times - "Connection to Marionette server is lost" Gaia::UI Tests RESOLVED WONTFIX ---
1203060 Behavior change on bring back the browser from back end( from tab page to opened page), and this will cause the gaia_apps.js check app launch mechanism failed. Gaia::Browser RESOLVED FIXED Martijn Wargers (dead) ---
1206634 [MTBF] After 20 hours executing, test cases failed consequently Stability RESOLVED FIXED Ting-Yu Chou [:ting] (away) [MemShrink:P1] 2.5+
1218305 Can't allocate memory in MTBF Host machine Marionette RESOLVED WONTFIX ---
1220012 Leak nsPipe from GeckoDriver.mozBrowserClose Marionette RESOLVED FIXED ---

19 Total; 0 Open (0%); 18 Resolved (94.74%); 1 Verified (5.26%);


MTBF B2G Meta Bug

Full Query
ID Summary Component Status Resolution Assigned to Whiteboard Blocking b2g
936316 [Endurance Test] Turn FM Radio on and off Marionette RESOLVED WORKSFORME Robert Wood [:rwood] ---
979097 [MTBF] Phone crashes after few hours General RESOLVED DUPLICATE ---
979173 [MTBF] Phone get really slow when running MTBF on v1.3 branch Marionette RESOLVED FIXED Walter Chen[:ypwalter][:wachen] [perf-reviewed] ---
990837 [MTBF] Phone Hang/Crashed with no adb/fastboot Gaia::System RESOLVED WORKSFORME [systemsfe] ---
990975 [MTBF] Phone lost control and black background Stability RESOLVED DUPLICATE ---
997791 Investigate and fix tests fail to enable wifi during setup Wifi RESOLVED FIXED Chuck Lee [:chucklee, Triggered by NI, Review, and Feedback requests] [depend on sprd305892] 1.3T+
1001300 [Meta][MTBF] Phone would crash sometimes without any actions Stability RESOLVED WORKSFORME ---
1015811 [MTBF] APN issue for 7-mobile General RESOLVED FIXED Arthur Chen [:arthurcc][inactive after 6/30] ---
1031355 crash in eglsubAndroid.so@0x4046 Graphics RESOLVED WORKSFORME [b2g-crash] ---
1033922 [MTBF] master build 3G signal issue Stability RESOLVED WORKSFORME ---
1039445 [MTBF] homescreen not responding Gaia::System::Window Mgmt RESOLVED FIXED Etienne Segonzac (:etienne) [systemsfe] 2.0+
1041467 [MTBF] App only covers top half of the screen Gaia::System::Window Mgmt VERIFIED FIXED (Inactive after June) George Duan [:gduan] [:喬智] [p=2] 2.0+
1045976 [MTBF][B2G][Stability] 319 Flame failed and keep hunging Stability RESOLVED WORKSFORME 2.0+
1045980 [MTBF][B2G][Stability] 319 Flame failed and homescreen button doesn't work Stability RESOLVED FIXED 2.0+
1053027 [B2G] Manually submitting crash report MTBF RESOLVED FIXED ---
1056448 [MTBF][Cards View] It will stuck in cards view and can't select any app to launch within 30 minutes Gaia::System VERIFIED FIXED Sam Foster [:sfoster] (he/him) [ft:systems-fe][systemsfe] 2.0+
1064800 [MTBF] Marionette keeps script timeout exception after running 20 minutes DOM: Core & HTML VERIFIED FIXED Ting-Yu Chou [:ting] (away) [mtbf] 2.1+
1069691 [GonkIntegration] Unable to run b2g-info due to NUWA being detected as second B2G process GonkIntegration RESOLVED WONTFIX [2.1-flame-test-run-2] -
1072735 [MTBF][Wifi] Wifi showing strange warning Wifi RESOLVED WORKSFORME [mtbf] ---
1074063 [MTBF][B2G] Missing rendering for launched apps Gaia::System::Window Mgmt VERIFIED FIXED (inactive after 6/18) Alive Kuo [:alive] [mtbf] 2.1+
1076603 [MTBF][cards view] Stucked at card view and died Stability RESOLVED WORKSFORME [mtbf] ---
1076605 [MTBF][App Launch] Apps failed to launch, stuck at icon splash Gaia::System::Window Mgmt RESOLVED FIXED (inactive after 6/18) Alive Kuo [:alive] [mtbf] 2.1+
1077292 [MTBF][B2G] Phone shutdown due to high power consumption and chopped charging. Stability RESOLVED WORKSFORME [mtbf] -
1082290 Implement cgroup swappiness feature for low memory target (256 MB) Performance RESOLVED FIXED Dave Hylands [:dhylands] [caf priority: p2][CR 787261][ft:media] [2.2 target] ---
1083691 [MTBF][B2G][Crash Report] B2G 34.0 Crash Report [@ XPCConvert::JSData2Native(void*, JS::Handle<JS::Value>, nsXPTType const&, nsID const*, tag_nsresult*) ] Stability RESOLVED WORKSFORME ---
1084179 [MTBF][B2G][Stability][Camera] Camera App will be black with button visible Stability RESOLVED WORKSFORME ---
1092924 [MTBF][B2G] B2G crashed due to oom or other related issues. Vendcom RESOLVED INVALID [mtbf], [POVB] -
1094070 [MTBF][B2G][Camera][Media Server] camera screen become black with buttons or stating that the camera resource is not available Stability RESOLVED FIXED 2.1+
1102794 [MTBF][B2G][Cards View] Stucked in Cards View and Test ended MTBF RESOLVED WORKSFORME [systemsfe] -
1120541 Rocketbar appears instead of app icons near the app header Gaia::System VERIFIED FIXED (inactive after 6/18) Alive Kuo [:alive] [systemsfe] 2.2+
1122119 [System] Device randomly white-screens, becomes unresponsive, and spams computer (if connected) with 6 device folders. Hardware Abstraction Layer (HAL) VERIFIED FIXED Gabriele Svelto [:gsvelto] [3.0-Daily-Testing][fromAutomation] 2.2+
1130916 [MTBF][Stability] Phone hung with FM Radio app still shown Stability RESOLVED WORKSFORME -
1131892 [MTBF][B2G] phone crashed Wifi RESOLVED FIXED Kai-Zhen Li [:seinlin][:kli] [mtbf] 2.2+
1133454 [System] having a name attribute on search app Gaia RESOLVED FIXED (Inactive after June) George Duan [:gduan] [:喬智] ---
1141411 [MTBF][Power Consumption] shutdown when running MTBF Stability RESOLVED WORKSFORME -
1141948 [MTBF][Nexus5-L] Parsing error and no Low-memory killer parameters info in b2g-info Stability RESOLVED WORKSFORME -
1141962 [MTBF] Device random reboot with reason wdog_bark Stability RESOLVED WONTFIX Ting-Yu Chou [:ting] (away) ---
1142384 [MTBF] System crashed, wifi keep spinning Stability RESOLVED FIXED Patrick McManus [:mcmanus] [b2g-crash] 2.2+
1145015 [MTBF] crash in gfx/layers/opengl/OGLShaderProgram Graphics: Layers RESOLVED FIXED Jerry Shih[:jerry] (UTC+8) (inactive) 2.2+
1146861 [MTBF] Crash in media/audioTrack Vendcom RESOLVED WORKSFORME Alastor Wu [:alwu] [POVB] ---
1150323 [MTBF][Nexus5-L] Nexus 5 has been shut down after running MTBF for a while Stability RESOLVED WORKSFORME 2.2+
1154629 [MTBF][Nexus5-L] adb crashed after running MTBF for a while Stability RESOLVED WORKSFORME ---
1155065 [MTBF][HomeScreen] empty homescreen occurred Stability RESOLVED WORKSFORME ---
1155490 [MTBF][Nexus5-L][Camera] Crash at android::AudioSystem::get_audio_flinger Stability VERIFIED FIXED Hayden Huang [:haydenh] [flame-l-testing-needed] ---
1174585 [MTBF] W/SocketClient( 198): write error (Broken pipe) Stability RESOLVED WORKSFORME ---
1180604 [MTBF] flame device hang after 6 hours running Stability RESOLVED FIXED Ting-Yu Chou [:ting] (away) 2.5+
1184440 Create MTBF standalone testing package MTBF RESOLVED FIXED Paul Yang [:pyang] (away) ---
1194064 Kickoff MTBF testing on Aries MTBF RESOLVED FIXED Paul Yang [:pyang] (away) ---
1217736 [MTBF] crash in EMPTY: no crashing thread identified MTBF RESOLVED FIXED ---

49 Total; 0 Open (0%); 42 Resolved (85.71%); 7 Verified (14.29%);


MTBF-2014Q3

Full Query
ID Summary Component Status Resolution Assigned to Whiteboard Blocking b2g
989850 [MTBF] we need a regular report generated to those whom concern MTBF RESOLVED FIXED ---
1032092 [MTBF] log location should go with conf file MTBF RESOLVED FIXED ---
1034506 [MTBF] Better tuned the testvar mechanism for multiple start-ups of devices MTBF RESOLVED FIXED Walter Chen[:ypwalter][:wachen] ---
1034508 [MTBF] Detection of crash report should be implemented. MTBF RESOLVED FIXED ---
1034531 [MTBF][Jenkins] Multiple control of devices General RESOLVED FIXED ---
1040544 [Adding Component] Request of adding components Administration RESOLVED FIXED Dylan Hardison [:dylan] (he/him) ---
1046002 [MTBF][Jenkins] Jenkins caught exception when running MTBF MTBF RESOLVED WORKSFORME ---
1052310 [MTBF] Create a wiki page for all informations MTBF RESOLVED FIXED ---
1064800 [MTBF] Marionette keeps script timeout exception after running 20 minutes DOM: Core & HTML VERIFIED FIXED Ting-Yu Chou [:ting] (away) [mtbf] 2.1+

9 Total; 0 Open (0%); 8 Resolved (88.89%); 1 Verified (11.11%);


Documentation

MTBF Reports History

MTBF Trend

Moztrap MTBF Test Cases

References

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Automation/Projects/MTBF&oldid=1007844"