QA/Automation/Projects/MTBF: Difference between revisions

From MozillaWiki
< QA‎ | Automation‎ | Projects
Jump to navigation Jump to search
 
(34 intermediate revisions by 3 users not shown)
Line 5: Line 5:
Mean time between failures (MTBF) is the predicted elapsed time between inherent failures of a system during operation. MTBF can be calculated as the arithmetic mean (average) time between failures of a system. The MTBF is typically part of a model that assumes the failed system is immediately repaired (mean time to repair, or MTTR), as a part of a renewal process. This is in contrast to the mean time to failure (MTTF), which measures average time to failures with the modeling assumption that the failed system is not repaired (infinite repair time).
Mean time between failures (MTBF) is the predicted elapsed time between inherent failures of a system during operation. MTBF can be calculated as the arithmetic mean (average) time between failures of a system. The MTBF is typically part of a model that assumes the failed system is immediately repaired (mean time to repair, or MTTR), as a part of a renewal process. This is in contrast to the mean time to failure (MTTF), which measures average time to failures with the modeling assumption that the failed system is not repaired (infinite repair time).


[[File:Mtbf.png]]
Assuming T(i) is ith running time between two failures, MTBF will be '''''SUM ( T ( n ) ) / n'''''
Below is information for MTBF in Mozilla
{| style="width: 90%; margin: 0 0 1em 1em" |
{| style="width: 90%; margin: 0 0 1em 1em" |
  | style="width: 20%" |
  | style="width: 20%" |
Line 13: Line 20:
  |- valign="top"
  |- valign="top"
  | '''Leads:'''
  | '''Leads:'''
  | [mailto:twqa@mozilla.com Taiwan QA Team]
  | [mailto:twqa@mozilla.com Device QA Team]
  |- valign="top"
  |- valign="top"
  | '''Contributors:'''
  | '''Contributors:'''
Line 25: Line 32:
|}
|}


== Pass Criteria ==
Currently we set 100 hours as our pass criteria.  Crash, hang are regarded as failure.


== Backlog ==
* Crash info detection
* Report enhancement
* Hang detection
* Integrating mozbase b2g related modules
* Datazilla integration


=More Information=
== Test Scope ==
The scope of V2.1 is still under determination. We might still change it slightly.


== Recent Status ==
{| class="fullwidth-table sortable"
{| class="fullwidth-table sortable"
|-
|-
| style="background:#EFEFEF; width:20%" | '''Date'''  
| style="background:#EFEFEF; width:10%" | '''Version'''  
| style="background:#EFEFEF; width:14%" | '''Version'''  
| style="background:#EFEFEF; width:90%" | '''Test Cases'''  
| style="background:#EFEFEF; width:66%" | '''Details'''
|- valign="top"
| V2.0
| Dummy Tests, Keyboard * 4, Music * 3, Message * 1
|- valign="top"
|- valign="top"
| -
| V2.1
| -
| Dummy Tests, Keyboard * 4, Music * 3, Message * 1, FMRadio * 1, Dialer * 1, Cards View * 1, Camera * 1, Settings * 1, Browser * 6, Video * 1
| -
|}
|}


== Test Suite ==
* Keyboard - 1. number keyboard  2. add contact 3. email keyboard
* Music - play 10 sec music in song view
* Message - send a message to itself
* FMRadio - play and see radio is enable
* Dialer - make a call from call log
* Cards View - Open two apps and kill one
* Camera - take two pictures
* Settings - turn on and turn off wifi
* Browser - 1. browse a website 2. add a bookmark 3. browse from bookmark
* Video - play a very short video


== Test Suites ==
== Backlog ==
{| class="fullwidth-table sortable"
* More test suites (on-going)
|-
** airplane mode on/off
| style="background:#EFEFEF; width:20%" | '''Suite'''
* Crash info detection
| style="background:#EFEFEF; width:14%" | '''Version'''
* Report enhancement
| style="background:#EFEFEF; width:66%" | '''Tests'''
* Integrating mozbase b2g related modules
|- valign="top"
* Datazilla integration
| -
| -
| -
|}


== Current Bugs ==
== Current Bugs ==
Line 66: Line 77:
<bugzilla>
<bugzilla>
     {
     {
         "include_fields": "id, summary, component, status, resolution, assigned_to, whiteboard, cf_blocking_b2g",
         "include_fields": "id, summary, component, status, assigned_to, whiteboard, cf_blocking_b2g, last_change_time",
         "blocks":[987035],
         "blocks":[987035],
         "block_type":"contains",
         "block_type":"contains",
         "status":"NEW, REOPENED"
         "status":["NEW", "REOPENED"]
     }
     }
</bugzilla>
</bugzilla>
Line 76: Line 87:
<bugzilla>
<bugzilla>
     {
     {
         "include_fields": "id, summary, component, status, resolution, assigned_to, whiteboard, cf_blocking_b2g",
         "include_fields": "id, summary, component, status, assigned_to, whiteboard, cf_blocking_b2g, last_change_time",
         "blocks":[1034494],
         "blocks":[1034494],
         "block_type":"contains"
         "block_type":"contains",
        "status":["NEW", "REOPENED"]
     }
     }
</bugzilla>
</bugzilla>


=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1034485 MTBF B2G Meta Bug] ===
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1034485 MTBF B2G Meta Bug] ===
<bugzilla>
<bugzilla>
     {
     {
         "include_fields": "id, summary, component, status, resolution, assigned_to, whiteboard, cf_blocking_b2g",
         "include_fields": "id, summary, component, status, assigned_to, whiteboard, cf_blocking_b2g, last_change_time",
         "blocks":[1034485],
         "blocks":[1034485],
         "block_type":"contains"
         "block_type":"contains",
        "status":["NEW", "REOPENED"]
     }
     }
</bugzilla>
</bugzilla>


 
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1182954 MTBF-2015Q3] ===
=== [https://bugzilla.mozilla.org/show_bug.cgi?id=1032093 MTBF-2014Q3] ===
<bugzilla>
<bugzilla>
     {
     {
         "include_fields": "id, summary, component, status, resolution, assigned_to, whiteboard, cf_blocking_b2g",
         "include_fields": "id, summary, component, status, assigned_to, whiteboard, cf_blocking_b2g, last_change_time",
         "blocks":[1032093],
         "blocks":[1182954],
         "block_type":"contains"
         "block_type":"contains",
        "status":["NEW", "REOPENED"]
     }
     }
</bugzilla>
</bugzilla>

Latest revision as of 06:47, 21 July 2015

Overview

The MTBF tests are a suite of Firefox OS tests built on top of the Gaiatest (Gaia UI Tests) Framework. The tests run on real Firefox OS devices, and use Marionette to drive the device's UI.

Mean time between failures (MTBF) is the predicted elapsed time between inherent failures of a system during operation. MTBF can be calculated as the arithmetic mean (average) time between failures of a system. The MTBF is typically part of a model that assumes the failed system is immediately repaired (mean time to repair, or MTTR), as a part of a renewal process. This is in contrast to the mean time to failure (MTTF), which measures average time to failures with the modeling assumption that the failed system is not repaired (infinite repair time).


Mtbf.png

Assuming T(i) is ith running time between two failures, MTBF will be SUM ( T ( n ) ) / n


Below is information for MTBF in Mozilla

Name: Mean Time Between Failures (MTBF)
Leads: Device QA Team
Contributors: Paul Yang, Walter Chen
MDN: https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/MTBF_tests
Git Repository: https://github.com/Mozilla-TWQA/MTBF-Driver/

Pass Criteria

Currently we set 100 hours as our pass criteria. Crash, hang are regarded as failure.


Test Scope

The scope of V2.1 is still under determination. We might still change it slightly.

Version Test Cases
V2.0 Dummy Tests, Keyboard * 4, Music * 3, Message * 1
V2.1 Dummy Tests, Keyboard * 4, Music * 3, Message * 1, FMRadio * 1, Dialer * 1, Cards View * 1, Camera * 1, Settings * 1, Browser * 6, Video * 1

Test Suite

  • Keyboard - 1. number keyboard 2. add contact 3. email keyboard
  • Music - play 10 sec music in song view
  • Message - send a message to itself
  • FMRadio - play and see radio is enable
  • Dialer - make a call from call log
  • Cards View - Open two apps and kill one
  • Camera - take two pictures
  • Settings - turn on and turn off wifi
  • Browser - 1. browse a website 2. add a bookmark 3. browse from bookmark
  • Video - play a very short video

Backlog

  • More test suites (on-going)
    • airplane mode on/off
  • Crash info detection
  • Report enhancement
  • Integrating mozbase b2g related modules
  • Datazilla integration

Current Bugs

MTBF Meta Bug

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


MTBF Marionette Meta Bug

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


MTBF B2G Meta Bug

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


MTBF-2015Q3

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Documentation

MTBF Reports History

MTBF Trend

Moztrap MTBF Test Cases

References

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Automation/Projects/MTBF&oldid=1085774"