Support/Q2HelpfulnessTests

This quarter we gave ourselves the goal of increasing the average helpfulness score of knowledge base articles by 2%. There are two basic ways to approach that – make articles more helpful (obviously) and get more people to an article that helps them on the first shot (thereby decreasing the likelihood of rating an article unhelpful).

Our plan is to run multivariate tests on the support home page (getting people to the right article) and on 2-3 of the top support articles (making things more helpful). For our tests on KB articles, we want to use some of our most trafficked pages. We also want to choose an article with a high helpfulness rating and one with a low rating.

Proposed articles to use for the test

• The Private browsing article is our 2nd most popular article (58763 views, wk of 4/12) and has a helpfulness rating of 85%.
• The Profiles article is our 5th most popular article (24640 views, wk of 4/12) and has a helpfulness rating of 37%
• The Pop-up blocker article is our 9th most popular article (17954 views, wk of 4/12) and has a helpfulness rating of 33%.

If we only pick two pages to test we might go with Profiles over Pop-up blocker because it gets almost 40% more traffic and has a similar helpfulness rating.

Article tests

Changing KB Article language

The current KB articles do a good job of speaking directly to the user and answering their questions. We'd like to test what impact changes to the language of the article have. For this test we'll work on rewriting each article, making these changes:

• Remove unnecessary information - There are almost always lots of tangential pieces of information related to a topic that could be presented. We should look to remove or not include anything that isn't necessary to solve the problem and get the user on their way.
• Use non-technical terms where possible - As SUMO contributors we're familiar with many technical terms and abbreviations. We shouldn't assume that people using the KB are familiar with any of these terms. We should explain things in as plain and simple language as possible.
• Use a conversational or friendly style - The articles should read more like your friend is explaining something to you and less like you are sitting in a lecture.

Here's the opening paragraph of the Profiles article:

• Firefox saves your personal information such as bookmarks, passwords, and user preferences in a set of files called your profile, which is stored in a separate location from the Firefox program files.

Here's an example of the kind of rewrite we propose to do:

• Did you know that all of the settings you make in Firefox, like your home page, what toolbars you use, saved passwords and bookmarks are all stored in a special profile folder? It's true. This article will tell you why this is a good thing, where you can find your profile folder, and give you links to related articles.

Embeding screencasts inline

Screencasts can be very helpful. We'd like to test if embeding them inline will encourage more people to watch them when available. None of the articles we're considering for the tests currently have screencasts so they'll need some created before the test can be run.

Changing KB Article layout

Currently, KB articles have an "Actions" box at the top giving people the option to edit or translate an article. While this is nice option that helps to reinforce that the KB is a community effort, it may be a distraction for someone trying to solve a problem. Instead we'd like to test whether swapping the related articles and search sections with the actions box will help people get to the correct article faster.

Start page tests

Improving performance

Improving page load times should make using and navigating the site easier and therefore help prevent people from becoming frustrated while searching for a solution. After looking at the size and loading times for various elements of the home page, it appears that removing the large background image (54k) at the top of the page will make the biggest impact. The screenshot examples below are very subtle because there is actually a tiled image (4k) behind the large background image.

Removing the background image changes the character of the SUMO home page quite a bit. Another option might be to use a more compressed version of the image. Here's an example of the image compressed to about 18k.

Simplifying the header and footer

Another test we'd like to run is simplifying the page header and footer. This would get rid of a lot of unnecessary information (the links to the rest of Mozilla aren't crucial since the site is about getting help) on each page, making it easier to find what's important.

Browsing Categories

Another idea that might help get people to the correct article faster would be to offer a better browsing option. Currently the home page is designed around searching with an additional list of popular articles for users who prefer to browse. One idea we'd like to test is to replace that list of popular articles with a few category sections. They could have headings like Crashes, Troubleshooting, or Tutorials with a short description of what other things are included. These would then link to a category page. We might also test whether categories work better above search (mockup on right).