Mozilla Test Writing Workshop
Background
The Mozilla project has done a decent job of popularizing extension development, both in terms of teaching people how to write extensions and in terms of publicizing it as a method to enter the Mozilla Community. We aim to organize a test writing workshop to do the same thing for automated regression testing on the Mozilla Platform. We want to capture the most important take-aways from these sessions and create a video resource for online use later by students and community members.
Goals
- Elevate regression/automated test composition to a first order activity in the Mozilla community, equal in weight and prominence to extension development
- Bring together people from all across the Mozilla Community to discuss issues surrounding test development.
- Generate a high-quality, engaging product for future use by the community.
Proposed Topics
Some of the things that we're hoping to cover.
- What are the frameworks?
- How do the frameworks work?
- What problems are each testing framework created to solve?
- How do I write tests for each type of framework?
- What are the frameworks not good for? Where do we need to improve them? What new types of frameworks are needed?
- What is the mindset of the "perfect tester", how do you attain it?
- What is the philosophy/strategy of test development at Mozilla? How do you know when a feature or API is fully covered by a test? How do you plan your work? What's the most important type of test to start with?
Brainstorming The Frameworks/Test strategies we might cover
- Reducing a web page to a test case
- "normal" mochitest, browser-chrome mochitest, chrome mochitest
- XPCShell
- Javascript testing
- Memory Leak testing
- Performance Testing
- Reftests
Format
Our primary goal with this event is to create a reusable artifact that can be integrated into our existing MDC content and serve as a starting point for individuals that are interested in testing. It will also be a perfect starting point for developers that want to get involved with the Mozilla Project but are not ready to leap into feature/patch development yet. So, we really want to focus on bringing together the best minds of Mozilla test development for discussions and how-to workshops.
We do encourage and want an audience, but we are targeting these sessions and the material at people who are already thinking about Mozilla and want to get involved. We are not going to use our time talking about the project in general, instead we will focus on the targeted area of test development.
Also our primary motivation for this is to generate an engaging and informative (and as entertaining as possible) product from the interactions of the presenters and participants. So, it isn't an event for spectators, everyone will be participating at some level.
Logistics
- Dave Humphrey has offered us the use of the Seneca University facilities
- He's also volunteered the University film department to run the videography for the event
- Should it be a one or two day event? We are leaning toward a two day event - one of presentation and discussion, one of hands-on/how-to material.
- Should it be on a weekday/weekend day (i.e. friday/saturday) or on two weekdays?
- We don't want to generate large hours of material that would not be useful to people, so we want to have a very good idea of what and how to film the sessions. We don't want the "videotaped lecture" style of doing this.
- With the impending release of Firefox 3 and the constraints of the school year, it makes the most sense to schedule this event in mid-late September or early-mid October of this year.
Action Items
- Figure out the weekday/weekday vs weekday/weekend day question --> we are going to see if we can blend this into mfinkle's dev day perhaps condense the test material to one day.
- Figure out the number of people that would be interested in attending as participants (i.e. audience). Feel free to add yourself to the sign up sheet below.
- Finalize and invite the core presenters we want to come. Some early thoughts include members of the layout development team, folks responsible for maintaining and expanding the test harnesses, members of the Mozilla QA team, and other community members, and members from other projects. We really want to get a good cross-section of the people who are already involved in it. This event will depend on having the right people present and participate.
- Pitch to other Mozilla projects: Songbird, Miro, Thunderbird, A11y, Calendar, Seamonkey, Camino to see if they have members that want to attend.
Possible two day schedule (with some proposed topics)
We have six possible blocks of time to use on our breakdown of the proposed schedule. We have also discussed dividing the tests into front-end and back-end categories as each requires a slightly different mindset and a different suite of tools for testing.
| Time | Day 1 | Day 2 |
|---|---|---|
| 10 - 12 | Welcome, test strategy, philosophy, methodology | Front End Automation (mochitest chrome/browser) |
| 12 - 13 | LUNCH | LUNCH |
| 13 - 15 | Backend Testing (XPCShell) | Principles for Specialized Testing: Security, Web site reducing, layout etc |
| 1500 - 1530 | BREAK | BREAK |
| 1530 - 1730 | Front End Testing (mochitest, reftest) | Future of Test Dev - where we are going, where we need help and infrastructure improvements. |
Some thoughts on the rough schedule above
In order to squeeze this down to one day - I think we want to keep our 2 hour block sessions intact, so we will have to cut content. I think that we can do this by ensuring that each session would address each of the specific concerns within its "world view". For example, the outline of the Front End workshop could go something like this:
- Front End Workshop
- General idea of mochitests, some front-end test ideas
- Demonstrate putting together a simple mochitest
- Talk about a case studies of how problem X, Y, and Z were addressed in mochitest
- Talk about methodologies for other specialized front end tests like reducing a site to a reftest.
- Talk about what to test a front end for when looking for security bugs, crashers etc.
This type of format would give us quite a bit of flexibility when it comes to the filming of the sessions, and would enable us to film short pieces on specific topics which will be more reusable in the future than one long presentation on the topic would be. Also, short subsections like this will help to encourage more participation and interaction among the audience.
So, a possible one-day schedule could look something like this:
| Time | Day 1 |
|---|---|
| 10 - 12 | Welcome, Methodology, Philosophy |
| 12 - 13 | LUNCH |
| 13 - 15 | Back End Testing Topics |
| 1500 - 1530 | BREAK |
| 1530 - 1730 | Front End Testing Topics - with rap up and a quick Future-of discussion at the end |
- TODO: Issues with which to do first. On one level, starting with Front End testing and moving to back end testing seems like the natural progression, but I think that starting with back end testing and moving to front end testing is an easier progression to learn because the tests themselves and the technologies are build on each other better that way (INMO).
Sign Up Here If You Are Interested
| Name | Contact Info | Project | Role Interested In |
|---|---|---|---|
| Clint Talbert | ctalbert AT mozilla dotcom | Firefox & Calendar | Presenter & Participant & Co-Organizer |
| Carsten Book (Tomcat) | cbook AT mozilla.com | Firefox | Interested in Automated Memory Leak Testing - Automated Extension Performance/Leak Testing, |
| Dietrich Ayala | dietrich AT mozilla.com | Firefox | Interested in Automated UI Testing |
| your name | your contact | your project | your role |
Where will these videos live?
- Here's an idea - put them as part of the Mozilla University effort