Firefox OS/Performance/Investigating Alerts: Difference between revisions
| Line 37: | Line 37: | ||
The last section of the alert is the details of the regression. For the given build, there could be many regressions found, we will list all of them in this section. | The last section of the alert is the details of the regression. For the given build, there could be many regressions found, we will list all of them in this section. | ||
[[File:Alert_details_device.png| | [[File:Alert_details_device.png|600px]] | ||
We outline which device model is used. The example here is Tarako, Other values are Hamachi as well. | We outline which device model is used. The example here is Tarako, Other values are Hamachi as well. | ||
| Line 43: | Line 43: | ||
After the device we mention what suite showed a regression. In this example we have 'vsize' as the suite: | After the device we mention what suite showed a regression. In this example we have 'vsize' as the suite: | ||
[[File:Alert_details_suite.png| | [[File:Alert_details_suite.png|600px]] | ||
The suite can be: | The suite can be: | ||
| Line 60: | Line 60: | ||
Next is the testname that was run inside of the suite. In this example we have 'email_memory': | Next is the testname that was run inside of the suite. In this example we have 'email_memory': | ||
[[File:Alert_details_testname.png| | [[File:Alert_details_testname.png|600px]] | ||
Other testnames can be: | Other testnames can be: | ||
Revision as of 18:18, 14 May 2014
When you get an email from fxos-perf-alerts@mozilla.com, there is a good chance it is an automated email.
We have two types of automated alerts:
- Ingestion Alerts (we were expecting data and never received it)
- Regression Alerts (the values of the reported data have changed)
Regression Alerts
A regression alert has a Title in this format
'[ALERT][B2G] <testname> regressed by <-x.x> in <suite>'
example
'[ALERT][B2G] email_memory regressed by -3.6 in vsize, vsize'
The first part of the Alert is the score:
The score can map directly to the confidence of the alert sustaining over time. The range is from 2-4 where 4 is .9999% sure. In the above example, we have a score of 3.32 which is .99932% confident this is a sustained regression.
Summary
The next section gives you direct links to the code that built B2G used for this test.
The gaia revision is a link to github that shows the specific changes to gaia.
The gecko revision is a link to github showing the current revision of gecko that we are using. bug 1010384 tracks getting these links to work.
Detailed Regressions
The last section of the alert is the details of the regression. For the given build, there could be many regressions found, we will list all of them in this section.
We outline which device model is used. The example here is Tarako, Other values are Hamachi as well.
After the device we mention what suite showed a regression. In this example we have 'vsize' as the suite:
The suite can be:
- vsize
- sms
- browser
- communication/contacts
- video
- fps
- music
- gallery
- and a few others
- bug 1010391 - document suite names and what each suite does
Next is the testname that was run inside of the suite. In this example we have 'email_memory':
Other testnames can be:
- email_memory
- communications/dialer_memory
- cold_load_time
- startup/startup_path_done
- etc.
- bug 1010396 - document test names in detail