FirefoxSummit/2006/ProposedSessions/CommunityTestingAndLitmus/Notes: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
mNo edit summary
Line 1: Line 1:
    <!-- start content -->
    <div class="boilerplate metadata" id="attention" style="background-color: #f3f9ff; margin: 0 2.5%; padding: 0 10px; border: 1px solid #aaa;">
<p><strong>All your base are belong to us!</strong><br />
Please edit [[Pelican shit|this page]] for great justice.
</p>
</div >
== Stats ==
== Stats ==
* Now nearly 3500 testcases
* Now nearly 3500 testcases

Revision as of 21:27, 15 January 2007

Stats

  • Now nearly 3500 testcases
  • About 850 active testers out of the 2000 who have accounts.
  • Not seeing much overlap between testdays, only a few people come back.
  • Of the top testers, several community members keeping pace with MoCo staff testers.
  • Dip in number of testers Aug/Sep -- one was Thunderbird which historically has less interest, and we had a lot of testdays around the same time, community getting burned out?

Test days

  • Seneca testday 2006-11-03 with 35 testers. Harness that kind of support in an ongoing basis? Seneca students assigned to come, most probably won't come back.
  • Trying to reward people with shirts and such, but we can't keep doing it to attract people
  • School/university support
  • Scheduling
    • Make testing calendar more widely known, publicize dates better
    • Don't schedule them at the list minute
  • Send automatic email to new Litmus testers after they've done some testing to introduce Mozilla QA + provide resources to get more involved.
  • More creative testdays--task oriented tests. The fun smoketest.
  • Let people create their own testcases and groups specific to content and sites they interact with regularly. Mix and match existing testcases and new tests.
  • Email out testday notifications to people who came to previous test days.

Manual test execution/Ways Litmus can help

  • Partner testing
    • Security flags for testcases
  • One-off tests (emails, hendryx data, etc...)
    • "Whiteboard" to track these tests, get them turned into real testcases
  • Pull data from Bugzilla. List of bugs to be verified, etc... Make Litmus show other QA activities.
  • Be better at vetting results + getting bugs found/filed in Bugzilla
  • Display with checkboxes for admins to check off when results have been appropriately dealt with
  • Programmatic regression analysis--determine when tests began failing and find the delta
  • Daily digest email from Litmus, giving failures and stats from the day's testing -- Bug 360996
  • Testcases in other languages
    • Let the localizers take care of the results
    • Do localizers really need tests in their own language?
    • Create new l10n suite of tests that are sensitive to locale changes
  • Easy way to tag tests/make a small list of tests to point people to around a specific area

Admin Interface Fixes

  • Test group editing is foobar'd
  • Adding testcases as a one-step operation again, not going through the hierarchy

Test Runs

  • Right now a very freeform testing interface. Need a better way to target testers to what we need help with right now
  • Adding an extra Test Run layer doesn't really add much functionality
  • Homepage to point people towards what we want to test
  • Limiting - test a particular build id on certain platforms
  • How is a run different then a group?
  • Run is a bucket of tests with a limited lifetime or permanent
  • Make it simpler, point people to the build and the tests--pimp it to them up front
  • Nomenclature - "run tests" and "test runs"
  • Admin reporting
    • Generate breakdown by subgroup, coverage data, etc...
  • "What is Litmus" box on the side of pages, tips