A series of usability tests were carried out by the wub in Sep/Oct 2012, as part of the help pages community fellowship. These were focused on the existing design and the proposed redesign of Help:Contents. These tests were carried out remotely through UserTesting.com. This page contains a summary of the results, along with links to view the test recordings.
For the previous in-person tests see Wikipedia:Help Project/Usability Testing 1.
Each user was given a scenario e.g. "You need help finding out how to cite Wikipedia in a class project."
They were then sent to Help:Contents (the current design) and given the following prompts:
Next they were sent to Help:Contents/B (the proposed redesign) and given the same prompts.
Finally a number of follow-up questions were asked.
Scenario: "You need help finding out how to cite Wikipedia in a class project." Test video
The user thought the page had "a lot of good information" and was also interested in the tip of the day. On the negative side he commented that the page "does throw a lot of information at you at the beginning", and has "too much vertical scroll". (Note that a Wiki Loves Monument banner was running, which increased amount of scrolling on both pages) Given the scenario he decided to click on Links and References to find the information. He then found Wikipedia:Citation tools under the References heading. He believed this met the goal of the scenario, although it actually covers adding citations to Wikipedia which is not what we were looking for. In fact how to cite Wikipedia is mentioned in Researching with Wikipedia which can be found via Help:Contents/Browsing Wikipedia.
Positive: "I like this page because it actually cites examples, these are things I might want to do." Negative: "There's still a lot of dead space at the top" By elimination he concluded that the scenario fell under "I have another question about Wikipedia", and went to the Frequently Asked Questions. "How do I cite Wikipedia" is the 5th question on that page, which he found successfully and was directed to Wikipedia:Citing Wikipedia which had the desired information.
Scenario: "An article you created on WIkipedia just got deleted because someone said it was not notable, and you want to get an answer from an actual person about how to fix this."
The user was initially a little confused by the page not looking like a normal Wikipedia article page, and pointed out that the blue colouring and formatting reminded him of a Google ad. His initial thought for the scenario was to click on "Asking questions" in the main body of the page, since it had the description "Contact another user". After looking round a bit more he spotted the "Ask questions" link in the header bar which actually leads to a different place. He noticed the different link targets and commented on how confusing it was, deciding to go with "Ask questions" as he believed it would allow him to contact "Wikipedia itself". He liked Wikipedia:Questions: "It has direct links to exactly where I expect to go, and it has explanations of where these links are going to take me.", and determined that the Help Desk would be the best place to ask.
"This is much less busy than the landing page I've seen previously" "This tells me that this is a help page" "The icons immediately are catching my attention. They're basically breaking up the information so I'm not confused by it like on the previous page" "Nothing that I really don't like about it. It's clear and it's to the point."
He goes via the "report a problem with an article" heading, to Wikipedia:Contact us/Article problem. Not the typical use for this heading, but it seems to serve well. He did point out that it would be nice if the heading itself was a link, but this isn't really possible as some sections require multiple links. Was happy with this page too, scanned the list not seeing his specific issue but quickly found the Help Desk link at the bottom.
Scenario: "You notice that an article about your favourite sports star has outdated information, but you’re not sure how to fix it yourself."
"Don't like the blue" "I use it [Wikipedia] probably on a weekly basis, but at the same time it's never a pleasure to look at is it?" "It's useful" "Aesthetically, visually, it's not nice."
At this point it became clear the user did not understand the test scenario, that he should be looking for help on the page he was sent to, and was expecting to see the article on the sports star itself. Therefore he skipped through the remainder of these tasks.
On going through to the new page, the user got somewhat back on track. "I like it because it's got big text. I like it because it seems to be in sections. I would like a border round it, but I do like the fact it's actually got headings etc." "Like the fact it's highlighted, it's bolder" (referring to bolding in headings).
Decides it would be best to click on Getting started under "I want to learn more about editing Wikipedia".
"Is this page what you expected it to be? Yeah, it's got a Tutorial, basically."
"The whole site is link-crazy, and I think that's its strength - it's links, everyone knows that. But at the same time there's no need for your tutorial pages and help pages to resemble the rest of Wikipedia." "Intuitive navigation doesn't work by just putting links everywhere"
Took a quick look at the tutorial and thought the content seemed quite good, although not the visual appearance.
Scenario: "You want to figure out how to upload a picture to a Wikipedia article and you can’t figure out how to add a caption to the picture"
Confused by the namespace checkboxes for the search box. Referring to main blue box: "I like that it's got all these categories" Decided to click "Files" link for the scenario. "It's not exactly what I expected to see but it's pretty good. I would have expected to see the help broken down into more categories. Maybe less on 'Other' and I would have expected to see a bigger 'Help' section." Thought about clicking "Guide to captions" next but then dismissed it as it was under the Policy and guidelines header. Instead went to "How to insert and use pictures in Wikipedia articles". Couldn't find captions anywhere in the table of contents, so clicked back.
Next went to "How to upload files". Still couldn't find it, so went back to Help:Contents and typed "how to add caption" in the search box, unticking all the checkboxes other than 'Help'. Somehow skipped over the link to Help:Visual file markup (terrible pagename!) and went to Help:Table#Captions! "Well that's the HTML... I don't know where I am. Very confusing."
Returned to Help:Contents once again and tried Editing Wikipedia link. Didn't see anything there. "I guess the best thing I found was with the search engine. I'm not having very much luck."
"I like this page better than the last help page. It tells me where to go"
"I think it would be under 'learn more about editing' because you're editing an article and you want to add a caption to an image."
Did try to click on the icon and the header at first, but then went to Getting started. Clicked Introduction to uploading images "Still not too clear on how to add a caption." Then went to the "Using an image" page. Missed it at first, but did then find the code to add a caption.
Scenario: "You want to figure out how to upload a picture to a Wikipedia article and you can’t figure out how to add a caption to the picture."
"The first thing that I probably don't like, is I just feel overwhelmed with text." "I like that it gives me information underneath the links, so that it gives me a better clue what I'm going to find here." "I like the colours, I like the blues and the greys" "When I just focused in on the main area, I liked it."
For the scenario, the user thinks the info could be under Files, Technical information, or Frequently asked questions, but most likely Files "because it specifically talks about dealing with images". She clicked on Files.
She was slightly confused by the "Upload an image to Commons" link, since she was expecting help rather than a direct link to carry out an action. Links she thought might be helpful were Images and other uploaded files, How to insert and use pictures in Wikipedia articles and Guide to captions (although she noted that the latter was under the Policy and Guidelines header), and she found it difficult to decide between them.
On clicking How to insert and use pictures in Wikipedia articles her instant reaction was "ugh, this looks overwhelming". She found that captions were mentioned in the lede, but could not find them in the table of contents or within the page.
The user then tried the link Guide to captions instead. However this page is part of the Manual of Style, and describes what to write in a good caption, rather than how to add it. At this point she gave up and moved on to the next task.
The user's instantaneous reaction to the new page was positive: "it just feels... nicer". She liked the icons and how "calming" the page was, but was less certain about which link to click. She eventually decided to click Getting started. On that page she quickly spotted and selected the Uploading images link (to one of the new tutorials).
She easily found the "Using an image" section, and was able to find out how to add a caption.