How to fight your own confirmation bias and your panelists unwillingness to be ‘bad’.
The No.1 challenge you have to navigate during user interviews is people's inability (or unwillingness) to disappoint you. They will want to first assume THEY are the problem, that THEY are the ones clicking on the wrong button, or misunderstanding something, before the realisation dawns on them that what you've brought to them is umm... full of friction.
Most panelists (i.e users who will be testing your product) will want to inundate you with reasons why they like what you're showing them and how it's 'such a good idea'. Testers tend to be a bunch of 'mums' who think every idea is a good idea, but you've got to take them through 'The Mum Test'. It is YOUR job to fight the unmerited glee to settle for that.
Before I get to the list of questions below, there are a couple of things that can help you tease out what your panelists true opinions are:
If it's prototype testing, let them know that YOU did not design the thing, you hired a 'guy' or someone else in a different team did it and you want to find out if they're doing a good job or not and that you hope your panelist can help you assess that. You want to remove the fear of offending you from your panelist.
If it's an in-person test, you reward anyone who find a bug or a friction point with sweets (surprisingly adults like these) or a dollar or a badge or something. You want to normalise the experience of delivering bad news to you that your product kind of sucks. That's the only way you'll know what to fix.
You make it explicit at the beginning of the session that the goal is to find out what 'issues' exist so you can solve it for them and the more direct and honest they are, the more helpful their feedback will be. Everyone likes to be helpful.
The questions you ask will depend on what kind of user interview you're doing. Usually, there's 4 testing exercises you do before launch, you can read more about them here.
Questions you might ask during a discovery exercise:
What are your current pain points or challenges in relation to the product or service category we are offering?
What do you like/dislike about similar products or services currently on the market?
What features or functionality are most important to you in a product or service like ours?
What would you be willing to pay for our product or service?
How often do you envision using our product or service?
What are your top priorities when considering a purchase in this category?
What would make you choose our product or service over competitors?
Can you think of any potential drawbacks or drawbacks of our product or service?
Are there any additional features or functionality you would like to see in our product or service?
How likely are you to recommend our product or service to friends or colleagues?
My favourite questions to ask during prototype testing:
Bad news only, what don't you like about product/feature/functionality/process/look?
What has been the most confusing part of this experience for you?
What would you add or remove from this design or process?
What do you think we might be forgetting?
What do you understand by X on this page?
What would prevent you from enjoying this product?
If you could add one feature to the product, what would it be?
How likely are you to recommend this product to a friend or colleague?
Questions during performance testing and beta launch:
Are you able to create an account? Did the OTP arrive on time?
Are all images loading correctly? Are buttons working?
If you put your phone on landscape mode, are you still able to scroll up and down?
How's the loading speed on your device?
Are you able to navigate through key tasks with relative ease?
How seamless is it for you to navigate through the pages as you need to?
Can you easily find (a button, a feature, a page) that you're looking for?
When facilitating user interviews as part of focus groups, user testing or just a good old 'talk-to-your-users' session, you might need to learn how to ask questions like a therapist would. You want to capture the emotions and the mental journey your user is going through when looking to answer your questions or test your product. These are great follow-up questions when a user responds a specific way or chooses a specific button over another.
Examples of things a Therapist would Say:
1. Something happens that seems to surprise them. For instance, they click on a link and say “Oh” or “Hmmm” when the new page appears. You could ask: “Is that what you expected to happen?”
2. The participant makes a comment, and you’re not sure what triggered it. You could ask: “Was there something in particular that made you think that?”
3. The participant suggests concern that he’s not giving you what you need.“No, this is very helpful.” “This is exactly what we need.”
You can read more about it Things a Therapist would Say here.
User testing as a beginner:
As a PM, you'll encounter your very first user testing exercise. Or, an opportunity to stop limping through the process and do it well.
I've created a user-testing playbook that helped me facilitate cross-country testing exercises, leading to my promotion just 8 months into my first product role. This playbook is now available on Gumroad and is filled with Notion templates that will help you plan the logistics, communicate with users effectively, and record their feedback.
With this playbook which includes other resources that are helpful for a new PM, you can take the guesswork out of user testing and ensure that your products are designed to meet the needs of your users. Don't miss out on this opportunity to take your product strategy to the next level.
You can get the Testing Textbook here: