User testing at mature startups

Five months ago I was tasked with conducting user testing of our new beta product. If you’ve ever set out to do user testing for the first time, you know how huge of an undertaking this can be. At a startup, this can be even more difficult when you don’t have the same foundation and resources as an established organization.

I took to researching how to tackle this and I felt like I was prepared to give it a go. What I learned firsthand is that establishing a clear strategy upfront helps ensure that the session goes smoothly. It’s more than writing up questions — being prepared to combat the unknowns will set you up to get quality feedback in your sessions.

–––

Establish an Empathetic Environment

Waste time upfront to save time in the end.

The more comfortable the user is with the interview the more insights you’ll receive. Before diving into the product establish a casual tone to the interview. This can be as simple as asking how their day is going, or to explain more about their role at their company. By doing this, you’ll create an environment where the interviewee feels comfortable speaking freely.

Answer Questions with Questions

Why do you think that? What do you think that means? How would you expect that to behave?

Throughout a testing session, the participant will definitely ask you questions on how something might work or what something does. It’s really easy to just answer the participant’s questions but, the key is to turn their question back at them. Yes, it feels really uncomfortable at first, but eventually, it helps get an understanding of what’s causing confusion and more accurately diagnoses user problems. Also, don’t forget to follow it up with a “why”.

Don’t Dwell on the Bugs

If something unexpected happens — follow up with the participant later.

If you’re doing user testing then you’ve probably acknowledged that your product is far from perfect. Things sometimes don’t work as expected — that is okay. It is good to preface whether your product is in beta or not prior to beginning the testing session. Here are two helpful tips for combating mid-session bugs.

First, anticipate the buggy areas. You’re likely spending a lot of time with your product and are aware of the imperfections. Create a plan B for when those known issues happen.

Second, prepare a short statement for when those unexpected bugs appear such as, “That seems like a flaw in the system. I’ll work with the product and engineering team to fix that. I’m happy to let you know when we’ve updated this issue.”

Allow Exploration

It’s only natural that a user will move in an unexpected direction. Don’t steer them away until they become totally stuck.

This is a hard one. If part of your testing session is validating a specific workflow there’s a chance that the participant will not complete the flow as you would have hoped. Let them continue down the path they’ve chosen until they become totally confused. Once this happens, question the participant to learn more about the reason why they moved in the direction they did.

Live in the Present

Don’t compare participants to previous sessions.

It’s a natural tendency to compare others. Try to avoid this. Each session should be private and unique on its own. Not sharing information from other sessions builds trust with participants that their session will truly be confidential.

Be Informed on Previous Design Decisions

Know the “why” behind how everything was designed.

On the flip side of Answer Questions with Questions, once the main questioning has been completed, the participant might ask you questions on specific features. Be prepared to explain the business goals and design decisions behind these features. It’s good to keep a short sentence in your back pocket if you aren’t informed on a decision. For example, “I wasn’t part of that initiative but I’m happy to follow up with you after our session.”