Support/Q2HelpfulnessTests

This quarter we gave ourselves the goal of increasing the average helpfulness score of knowledge base articles by 2%. There are two basic ways to approach that – make articles more helpful (obviously) and get more people to an article that helps them on the first shot (thereby decreasing the likelihood of rating an article unhelpful).

Our plan is to run multivariate tests on the support home page (getting people to the right article) and on 2-3 of the top support articles (making things more helpful). For our tests on KB articles, we want to use some of our most trafficked pages. We also want to choose an article with a high helpfulness rating and one with a low rating.

Proposed articles to use for the test

• The Private browsing article is our 2nd most popular article (58763 views, wk of 4/12) and has a helpfulness rating of 85%.
• The Profiles article is our 5th most popular article (24640 views, wk of 4/12) and has a helpfulness rating of 37%
• The Pop-up blocker article is our 9th most popular article (17954 views, wk of 4/12) and has a helpfulness rating of 33%.

If we only pick two pages to test we might go with Profiles over Pop-up blocker because it gets almost 40% more traffic and has a similar helpfulness rating.

Article tests

Embeding screencasts inline

Screencasts can be very helpful. We'd like to test if embeding them inline will encourage more people to watch them when available. None of the articles we're considering for the tests currently have screencasts so they'll need some created before the test can be run.

Changing KB Article layout

Currently, KB articles have an "Actions" box at the top giving people the option to edit or translate an article. While this is nice option that helps to reinforce that the KB is a community effort, it may be a distraction for someone trying to solve a problem. Instead we'd like to test whether swapping the related articles and search sections with the actions box will help people get to the correct article faster.

Start page tests

Improving performance

Improving page load times should make using and navigating the site easier and therefore help prevent people from becoming frustrated while searching for a solution. After looking at the size and loading times for various elements of the home page, it appears that removing the large background image (54k) at the top of the page will make the biggest impact. The screenshot examples below are very subtle because there is actually a tiled image (4k) behind the large background image.

Removing the background image changes the character of the SUMO home page quite a bit. Another option might be to use a more compressed version of the image. Here's an example of the image compressed to about 18k.

Simplifying the header and footer

Another test we'd like to run is simplifying the page header and footer. This would get rid of a lot of unnecessary information (the links to the rest of Mozilla aren't crucial since the site is about getting help) on each page, making it easier to find what's important.

Browsing Categories

Another idea that might help get people to the correct article faster would be to offer a better browsing option. Currently the home page is designed around searching with an additional list of popular articles for users who prefer to browse. One idea we'd like to test is to replace that list of popular articles with a few category sections. They could have headings like Crashes, Troubleshooting, or Tutorials with a short description of what other things are included. These would then link to a category page. We might also test whether categories work better above search (mockup on right).