MDN/QA/testplan

2016 QA Priorities

• Editing pages is a consistent and user friendly experience.
• Localized experiences persist when a user arrives, uses, and leaves the site.
• Feedback mechanisms on MDN exist and work.
• Topic and article pages behave as expected.
• The search experience is consistent.

Open Questions

• How do we prevent loss of polish over-time?
  • For example:
    • CSS tweaks that have unnoticed and unintended consequences.
    • User workflows that don't make sense.
• Data integrity
  • How is this maintained, persisted, prevented from regressing?
  • How is user content quality maintained & updated for relevancy over time?
• Are there bugzilla search flows that can be created for triaging
  • What needs manual verification?
  • Where test coverage is lacking?
• How is technical test debt tracked?
• Reliability discussion - https://etherpad.mozilla.org/MDN-reliability-plan
• l10n testing?
• Page load times

Outline of ownership of quality components

Shared responsibilities

**project management, developers, qa, UX, l10n**

• Deep knowledge of the application and the ability to identify areas of risk
• Provide feedback on feature definition to the project manager and team
• Identifying, creating, and maintaining the functional and end-to-end test automation
• Developing the appropriate mitigation strategies to lower risk of regressions landing in production
• Staying current on best practices and pushing the team to explore the applied relevancy
• Continually discussing and fine-tuning test coverage
• Opening avenues for community collaboration on the project

What are QA’s responsibilities

• Acting as a user advocate
• Leading discussions that identify and assign risk to user stories
• Driving exploratory testing efforts around feature areas that represent high risk

What are the dev team’s responsibilities

• Developing new features for the project and implementing fixes for issues
• Developing unit tests
• Maintaining code quality via code review and documentation of standards

ToDO Complete a risk assessment

• The team discusses and agrees on a set of risks that they are comfortable with assuming. Assumed risk isn’t a static conversation, but it is good to build a skeleton that the team can agree on and enhance over time.

• The product should be broken into user stories so a team can apply a hierarchy of risk. It is important to identify and engage as many stakeholders as possible, each individual will have a unique and important point of view.

• The list of stakeholders often include; project managers, developers, testers, UX, IT, DBAs, the security team, etc. Sometimes the product has high enough visibility in the organization that a proxy for the CEO is beneficial.

User stories and features are grouped into three categories

Cannot regress in production

• Features that users should never see broken in production (in order of throughput/traffic)
  • View a doc article (Zones & non-zones)
    • Live samples must continue to render and function
    • Edit a doc article
    • Translate a doc article
  • View the home page
  • Log in
  • View a user profile
  • Search docs
  • View a demo
  • Ensure we don't leak PII

Can regress in production for a finite amount of time

• Features that can go to production broken but need to be discovered and fixed within a specific time period

Can regress in production - fix when we notice it's broken

• Features that can regress on production and will be fixed once a user identifies the problem

Manual Testing

Feature testing

Not a dogmatic rule, in general new features or defect fixes should be manually tested and verified. Exploratory testing is the rule. Depending on the risk assigned to a feature, projects have the option of using waffle flags. The use of feature flags allows us to hide features from the general public while allowing a team or a group of beta testers to test on production with real data. An important note about feature verification

If a feature is deemed as low risk and it's functionality is easy to verify, anyone on the team is empowered to test and verify it. If it is a feature that requires deeper investigation and poses high risk, the QA team is the group who verifies it. If a feature set is big or user workflow is changed it is important to engage the test team to flush out defects and usability concerns.

• Features sent to QA for manual verification must include Bug numbers

Test Automation

Currently, writing and maintenance of the tests written using Intern are owned by the developer team. Once the tests stabilize discussion of phase two, joint ownership, will begin.

Currently work is being done to include the end-to-end tests in the build pipeline.

Blockers

• Bug 1186081 - [meta] Implement load-testing for MDN
• Bug 1184662 - Update intern tests to run reliably
• Bug 1184670 - Automatically run Intern test suite against stage & prod servers after deployments]

Overview

Feature work that falls into the category of “better never break” should see a heavy amount of exploratory testing before it is exposed to the public.

• Each pull request that includes a new feature that is exposed in the front-end UI should include theIntern.js test case(s).
• API end points should include end-to-end tests that exercise these routes.

A component of the automation strategy includes reviewing the amount of test coverage that follows a developers pull request. Often times the team is able to influence and enhance the depth of coverage by simply asking for more tests earlier in the workflow [Unit + Integration].

TODO Coverage Areas

End-to-end tests

• Integration tests
• API tests?

Stress and load testing

• locust.io
• https://etherpad.mozilla.org/MDN-load-testing
• Create and run stress test (based on the catalog) on MDN staging to learn max load limit of MDN code https://bugzilla.mozilla.org/showdependencytree.cgi?id=1186081
• Convert stress test into load test for use after stage deployments https://trello.com/b/p56Gwq46/mdndev-rocks
• https://bugzilla.mozilla.org/showdependencytree.cgi?id=1182182

Monitoring tools

• New Relic? DataDog?

Style Guides

• Link to style guides
  • code quality, linting tools, etc
  • What are the best practices for writing Intern tests?
  • Best practices for writing API endpoint tests?

One-time Events

Test Events

Test days are a great way to engage both longstanding contributors as well as onboarding new people on projects. As project leads and core community it is our role to create an engaging, fun, yet focused experience when people work on our projects. When drafting a test plan for it is important to know who your audience is as well as tightly define the parameters of the test day.

Each test day is an opportunity to tell the story behind why a project exists and why that application is important to the matrix of projects that Mozilla supports. Suggested components when crafting a test plan:

• Give a brief introduction of what the project is and why it’s important. For example; Mozillians.org is a community phonebook that is integral to helping people connect with one another as well as allows access to members only areas of Mozilla.
• Use social media to get the word out.
• Briefly discuss what precipitated the need for a test day - a new feature or set of features. Explain why there is a need for helping reduce project risk. This helps set the tone for the event and allows people to determine if they’re interested in helping on this project.
• Describe what expertise and environment are needed to participate. Provide information that points people to the information they need to get setup. A small suggestion, try to create a reusable body of documentation you can point people to.
• Teach people how and where to file bugs: what components make for a good bug, what to leave out of a bug. Include a Bugzilla URL with the correct GET parameters that point to the correct product, component, flags, etc.
• To track bugs filed during a test event, include a whiteboard flag that is unique and easily queryable.
• Include online and offline ways for people to get in touch with you. Emphasize pathways that lead people to your community: irc and mailing lists. Also include personal contact information.
• Lastly, warmly thank people for taking a moment to read your post. Test event’s don’t need to be real-time, but you should think of them as vehicles for creating lasting relationships that help coach people.

Using Social Media

Social media should be viewed as any medium that your team currently uses when creating one way conversations with the world. They are a great way to blast out a small amount of info, a breadcrumb trail, that leads people to an event.

• Use twitter to notify the world of an upcoming event, make it sound intriguing, and link to a blog post/test plan that contains all of the information.
• Use blog post to expand upon why the event is important, what people will learn, and how they can reach out for help.

Templates

Sample Testday

The latest feature set [include Bugzilla query] of the <name of project> has landed [include URL to where testing will take place] and is ready for testing. The <project name> a project that does X and is important because of Y.

What’s new in this release/what is being tested:

• This new feature(s) will allow users to do ..
• Combined with this old feature users can do ..
• .. etc ..

Test plan:

These are the areas to focus on .. This can be a list of bugs that need verification, a unified bugs that that represent a new feature set that you’d like people to test wholistically (exploratory testing, etc). Be very specific about what is being tested and what to ignore. Help set a cadence for what is relevant for this event.

Setup for manual or automated testing:

Create a list of things that will be needed to participate in this event. Include appropriate links/urls/screenshots/etc that help paint a colorful picture of what’s being worked on. Consider that some people learn through reading words, whereas others prefer a combination of images and text.

Filing bugs:

Give direction on how to file a good bug as well as

• Write good bugs that provide clear steps to reproduce the problem. Read this document for tips.
• Use this form to file new bugs. [Buzilla form that is pre-filled using GET params]
• Bugzilla etiquette - be polite and treat people with respect, we are a friendly community.
• IRC etiquette - same as Bugzilla; relax and have fun.

We really appreciate your enthusiasm and help with making the <project name> better. This is fully a community initiative and wouldn't exist without you.

Cheers,

<your name>

<url to your mozillians.org public profile>

oneanddone.mozilla.org

• For less time sensitive testing tasks we can use oneanddone.org as a task manager to aid in funneling community members into our onboarding flows.