How to conduct User Testing

Introduction to usability tests

Do you need help with what questions to ask during user testing? Don't worry about a thing, and we got you covered. In this post, we will share our golden questions.

At UX Cambridge, we have conducted countless usability tests for our clients. Our blog post will cover how to test for mobile and desktop using qualitative and quantitative testing methods. 

Our mission is to assist you in conducting a usability test, and we will guide you through each step. Additionally, we will recommend essential recording equipment to ensure a professional interview.

Benefits of usability testing

Six benefits of user testing:

1. Identifying user needs via user testing enables your team to understand user preferences, goals, frustrations and pain points. Usability tests allow you to validate your assumptions and create user-centric products.
2. Finding usability issues will allow your team to solve and address problems head-on. UX researchers can then take these findings and optimise the design before development.
3. User feedback gives your team qualitative and quantitative insights, which helps you understand the overall user performance. It also allows you to present to stakeholders how users interact with software.
4. Improving user satisfaction is easy after conducting usability tests. It allows your team to create the next design iteration.
5. Reduce your development costs: Often, businesses don't invest in UX researchers because they need to understand the ROI benefits. By understanding your users the first time, you'll save truckloads of money; it's harder to change the user experience once it's developed. Test and test again is our advice to get users to perform specific tasks repeatedly until you have perfected your user experience.
6. Assess your information architecture by user testing. An excellent clickable prototype will enable your team to understand if the site structure makes sense by using scenarios that allow the target audience to navigate the platform.

Moderated task vs unmoderated user test

Usability tests are either moderated or unmoderated environments. Each has its advantages; in this article, we have primarily referred to moderated tests. Let's compare the two testing methods:

Moderated user testing

1. Moderated User testing: The test is conducted with a participant and moderator remotely or in person, which creates an intimate environment. As the moderator, you can ask the user to perform tasks on the software and observe user behaviour first-hand.
2. Real-time: As the test is real-time, we can probe and ask follow-up questions to identify problems.
3. Flexibility: This is one of our favourites. Sometimes, your users identify other problems you wouldn't have found otherwise. As the test is live, it becomes flexible, allowing you to pivot and adjust the questions in the live environment or after the session.
4. Participant support: The moderator guides the participants and supports them with tasks. Think of it like this: the moderator is there to relax the user, encouraging them to tell us valuable insights.
5. Qualitative data: Compared to unmoderated user tests, moderated user tests allow you to gather verbal cues and contextual information that otherwise would have been lost.

Unmoderated usability testing:

1. Participant independence: Unmoderated testing allows users to perform tasks remotely and alone. There is no direct interaction with a moderator. Therefore, they can complete assignments at their leisure.
2. Scalability: If you are a big business like Netflix, an unmoderated test allows more research participants to contribute to the study in a shorter period.
3. Cost and time efficiency: As no moderator exists, the test can be performed worldwide, opening the test to a broader geographical location.
4. Reduce bias: As it is unmoderated, the user performs the tasks without influence from the moderator. Therefore, they can lead to more realistic results.
5. A focus on metrics: An unmoderated test is more black and white, meaning it is quantitative and often focuses on success rates and completion time. However, the user cannot tell you their reasoning as to why they performed the task the way they did.

Your research goals will dictate which testing method you use. Time and scalability often play a contributing factor when testing. If you have time to conduct both, it's usually a great idea, especially when making big platform decisions.

Writing a script for a usability test

A customer script is required to ensure the test runs smoothly. The script will depend on the type of UX research you are conducting, for example, a 1-1 interview compared to a focus group. They can take sufficient time to plan.

Our team recommends spending time on templates; they save you hours. At the start of the interview, tell the user what to expect from the session and clarify that we are not testing you. We are testing the software and explain the testing method.

It's essential to record the session, where possible. Ensure your user has signed the consent form and agreed to be recorded, as this can bite you in the arse. In addition, if you need to screen share, please prompt the user. 

It's super important that the user drives during the test.

At UX Cambridge, we often start the usability test with a few basic questions: Could you tell us about your role? Please tell us about your experience with streaming services. It provides context and paints a picture of the user.

For 1-1 moderated tests, come up with a scenario; imagine that a friend has suggested a series, "Emily in Paris", and you want to learn more. Keep your questions as neutral as possible; this way, you're not putting any bias into the test.

Remember, user testing is all about how you ask the question. Keep it neutral and open, and don't ask closed questions like what you think about this card. Ask questions like What do you think of the information on this page? Users will often give you objective feedback.

If you need clarification that your questions are neutral, run a pilot test with someone and get them to provide feedback. Indeed, if you are new to user testing, it's excellent for you to learn; often, the biggest issue is timing and how long it will take for users to answer your questions.

Test with five users or more; this way, you understand whether your design is performing well. It will identify usability problems you can fix. Conduct different user tests within the study, such as card sorting, to learn customer behaviours.   

If you have a massive amount of customers and it's a significant design change. At UX Cambridge, our team recommends working out a percentage for user testing of 5-10%. Take the streaming project, for example. If you have 161 million users, conducting tests with 800k people would not be possible. That's why an unmoderated test method is more appropriate in this use case. 

Recruit test subjects from different geo-locations as the feedback will be different, and it will ensure reliable results.

When testing with real users, you often get subjective feedback, which leads to detailed qualitative insights.

Recruiting participants for usability tests

Recruiting for usability studies takes time and effort. One of the best methods is to set up an email campaign to seek out your target audience. 

Offer an incentive to bait the users into signing up for your user research. Nobody wants to give up their time for free. Usually, £1 per minute Is standard.

Outsourcing specific tasks like recruitment can significantly speed up the process if time is an issue. Some specialist recruitment companies specialise in finding people to test, and it saves you the hassle of finding actual users.

A screener is used to filter a test subject. It also helps spot bias tendencies before you commit to usability testing. For example, for the streaming project, we may ask questions like Have you used Netflix before, Disney Plus, or Amazon Prime? A screener often saves time and ensures the user fits your study criteria. Additionally, it's a brilliant way of gathering quantitive insights by asking preliminary questions first.

If your company plans to conduct hundreds of usability studies annually, invest in software like userinterviews.com. Invite, schedule, message, automate and sync your calendar with software that makes recruiting participants for a usability study easier.

Remote User testing

Remote user testing can be challenging, especially on mobile. I always get real users into the studio and test the user interface. If not, it usually happens over a Google Meet via a screen share. Therefore, it's sometimes easier to understand how the user feels in person.

If you're doing remote usability testing, it's all about preparation. Make sure that your equipment works and run a pilot test.

Mobile user testing

For mobile, you'll have to get the users to mirror their mobile devices onto the computer and then share their screens via Google Meet, Zoom or Teams.

If it's a new design and not an existing application, they must install the software, such as Figma, before the usability test can start. Figma works well for mobile testing; however, the user has to be tech-savvy to get this to work.

A top tip is ensuring they are on the same Wi-Fi network. Another is to send the user the details of the apps they need to download before conducting user testing. This way, the test will run smoother.

Desktop user testing

Getting your user to load Figma or Sketch is much more straightforward for a desktop test. Then, you can ask them to share their screen with you. Remember to record the session, and off you go!

Remote user testing can be complex, especially for mobile. I always try to get real users into the studio to test the user interface. If not, it usually happens over a Google Meet via a screen share. Therefore, it sometimes takes effort to understand the user's feelings online.

If you're doing a remote user test, it's all about preparation. Make sure that your equipment works and run a pilot test.

What device should I perform the usability test on

It would help if you always tried to test both the mobile and the desktop. However, sometimes, you can't do both. Check Google Analytics for the site to find out which version is more popular, then test the desktop or mobile prototype with your users.

Qualitative and quantitative research data

Qualitative user testing focuses on user experience. Most usability test results are qualitative data because they are open-ended. 

The information is usually a lengthy response. Therefore, it's more complex to interpret than a yes or no question. However, you often learn more about how the users feel, and it reveals golden nuggets of information.

Quantitative user testing lends itself to unmoderated tests. However, the initial part of the interview includes quantifiable answers.

For example, questions like Have you used Netflix before? The question is quantitative because the test results are numerical and predictable. Quantitative questions are straightforward when compared to qualitative insights.

Usability testing questions

Each usability test is unique and requires different questions depending on the testing situation. User testing uses scenarios. It's one of the best ways to create a natural and neutral environment. Our usability testing examples will help you gain confidence whilst conducting user testing. The team has completed hundreds of interviews, so here are some top questions you can ask during your subsequent usability study.

Three compelling user testing scenarios

1. Scenario 1: Imagine it is movie night, and you would like to pick a comedy movie. Please show us the process you would go through.
2. Scenario 2: We assume you would like to change your payment method. Please show us how you would do that.‍
3. Scenario 3: Think back to the last time you watched a movie. Please tell us about the experience.

Five questions whilst you conduct user testing

1. What did you expect to see?
2. What do you think about the information on this page?
3. Did you manage to complete your goal?
4. Is there anything missing from this page?
5. What will you do next?

Three types of questions to avoid during user research

Avoid compounding questions, which means two in one. For example, what are your thoughts on this page and please tell me why. Users find two-part questions more complex and often need to catch up on essential details. That's why it's always better to ask a follow-up question instead.

Avoid telling your users too much information. For example, I want you to go to your account settings page and change payment deals. Don't ask leading questions like this one, as it will add bias to your study.

Avoid direct yes and no questions. For example, should your name be underneath your avatar? Direct questions are a big no-no; you are after qualitative rather than quantitative data. Save direct questions for the survey.

Note-taking

If you have ever tried to take notes during a session, ask questions and listen to users' responses, it's super hard! If you ask someone to take notes, it makes the process easier.

During remote testing, the person who takes notes should mark down the success rates and the perceived user experience. Can the users perform the tasks?

Do the users understand the language on the page? Furthermore, any problems with the web design should be documented as part of the design process.

Follow the script during user testing

You have written a solid test plan for a reason, so follow it. I have seen many UX researchers go off script and ask leading questions. There is nothing worse than a loaded question.

Trust me, I have been there, done that, and got the loaded gun T-shirt. If you have years of experience like me, occasionally going off script is okay; however, I avoid it if I can.

Think about it this way: the more questions you ask ad hoc, the more chance you have of either bringing bias into the study or not finishing on time. 

If you need to find the answer to a juicy qualitative question, keep it short, sweet, and neutral.

After the first interview, I'll often add a couple of clarifying questions to the study. Then, in the following discussion, I'll ask that burning question. It's an excellent method to ensure you have time to think about the question correctly to get an unbiased answer.

Focus groups

Focus groups are entirely different and often much more complex to manage. Therefore, they are much easier to run in person. User testing is better 1-1 if you're in a focus group. It's better to bring physical printouts and ask for feedback.

Be careful of bias creeping into your study, as one user often becomes the leader. Don't let the sheep mentality, baa, into your research. 

Less is more, as your session will likely involve discussions and debates.

Focus groups have some positives; running a group opens people up and gets them to talk about their user experiences.

Other methods

UX designers often need to think outside of the box regarding user testing. Another excellent research method is to create an online community. They usually consist of unmoderated tasks, which users complete online.

An online community also benefits from a forum where users can learn and discuss problems. As a business, online communities are essential to user testing and allow you to collect insights on a larger scale.One negative is that they often cost more to implement than a standard round of user testing.

Software for user testing

Moderated usability testing software

There are many options for usability testing software; do your homework before committing. If you are going to conduct hundreds of interviews, we advise getting the software to manage participants and automate the process as much as possible. For moderated interviews, we recommend the following software:

1. Scheduling (optional): User interviews
2. Scheduling (Required): Google Sheets or Office 365
3. Video calling (Required): Google Meet or Teams
4. Testing software (Required): Figma or Sketch
5. Tracking progress (optional): Monday or Jira

We will cover unmoderated usability testing software another time. Our team uses Google for everything, so running sessions over a Google Meet makes sense. However, if your business is more Microsoft-based, you can run tests using Teams and Office 365.

Equipment for user testing

A good laptop or PC with an integrated webcam is a must! We recommend at least one external monitor for remote testing because it lets you watch the participant and open your notes simultaneously. The microphone is the essential equipment, so we have provided a list below:

Tried and tested mics

1. USB Mic: Blue Yeti Logitech Blue Yeti Snowball
2. USB Mic: Blue Yeti Logitech Blue Yeti
3. On-Camera Shotgun: RØDE VideoMicro II Ultra-Compact

Usability testing report

Creating a usability testing report is the final stage of the process. We often use Google Slides to present the results to stakeholders or team members. 

Our team often uses quotes for contextual findings. One of our top tips is to trim your interview videos to 3 to 5 minutes so observers can watch.

Another alternative is to hunt for patterns within the qualitative data and determine how many users echoed the same statement. Use an opportunity solution tree to track user findings, as it's one of the best methods. Here's a (Miro) Opportunity Solution Tree Template.

Conclusion

1. Define your research goals early to choose between a moderated or unmoderated study.
2. Write a test script and practice it before the user interviews.
3. Book a conference room or a quiet place to conduct the interviews
4. Take notes
5. Present your findings to the relevant stakeholders
6. Document your conclusions strategically. 
   Have fun! If you need us, contact info@uxcambridge.co.uk