Learning with User Interviews

* This content is primarily targeted at early-stage EdTech Entrepreneurs, but should be useful for other early-stage entrepreneurs as well. 

Talk to the right people

To get the most beneficial information, you’ll need to define and select the users you want — and explicitly screen out the ones that won’t provide you with helpful feedback. You will need to define strict inclusion and exclusion criteria which will help your team find the right fit.

Every scenario calls for a different recruiting approach, for example:

1. Test a getting-started experience with people from the target audience who haven’t used your product.

2. Test engagement or activation with users who signed up but don’t actively use your product.

3. Test task completion or usability with people who use (or would use) the feature you are designing.

Selecting the right users is a critical step in the process — one that many teams overlook! Take the time to seriously consider which users will help you answer your questions before you start recruiting.

*Getting feedback from friends and family seems better than nothing, but reactions from “real” users will be much more eye-opening and provide you with the information you require.

Inclusion and exclusion criteria

• Strictly defining the inclusion and exclusion criteria will help your team find the right fit people to interview

‍

Work with your team to define the inclusion & exclusion criteria. 

1. What is your inclusion criteria?
2. What is your exclusion criteria?
3. The name which 5 -10 users fit this profile.
4. Reach out to them and secure 5 confirmed users for testing the prototype.

‍

The process of putting your prototype in front of potential users and getting valuable feedback that your team can use going forward to improve your solution/offering.

Testing. Why do it?

To make data-informed decisions

• Testing is a process that is meant to:

• Derive deep understanding of your product and its users
• Inform the understanding of users and how they think, behave and feel in relation to your product
• Weed out issues before releasing the product

• An effective way to carry out testing is by conducting 1-on-1 interviews with a few potential users (five is typically enough)
• The benefits of having 1-on-1 interviews are:

• You have undivided attention and can focus on each user fully
• You are able to ask direct questions to each of the users. This also allows for flexibility for the interviewer to take the conversation in a certain direction.
• More information is shared by the user and their responses are not affected by other candidates

‍

The Five-Act Interview

To get the most useful information out of your interviews you will need to carry them out effectively. We are going to take the learnings from Google Ventures on what they call The Five-Act Interview to guide you on how to conduct your 1-on-1’s effectively.

The Five-Act interview is a user interview structure for testing a prototype, that has 5 main steps from the start to the end of the interview:

Act 1 / Welcome (3 min): 

Greet the user and describe how the interview will work, talking about why you are carrying out the interview. Be clear that you are not testing the user, but the design through the user’s eyes.

Act 2 / Context questions (10 min):

These are meant to help you learn a bit about the user before you get into the interview, e.g. What do you do for work?, What hobbies do you have?

Act 3 / Introduce the prototype (3 min):

Show the user the prototype. Make sure the user feels comfortable to give you any feedback they may have. Let the user know that there is no wrong answer.

Act 4 / Task (20 - 30 min):

Give the user the device you are using to test with and provide them with tasks to complete. Mostly watch what they do and occasionally ask questions to get them to think aloud. Paraphrase what the user has said to make sure you understand what they mean. (Use questions in the interview guide)

THINK ALOUD - You will need to continue encouraging your user to say everything that they are thinking as they are performing the tasks and explore the prototype. This will likely feel unnatural/awkward for them, but it will help you and your team to learn about a lot of things that you would have probably overlooked.

Act 5 / Quick Debrief (5 min) - 

At the end of the interview, ask the user to summarise all the information in three main categories: What did you like? What did you not like? What are you not sure about? Thank your user!

‍

General Interview Questions

Below are some of the questions included in the interview template that we have created for you. These questions are meant to help you gain valuable insights from the 5 users that you will interview. Note that the questions are open so as to encourage the user to be honest and provide you with the most valuable feedback possible.

1. Before {{product}}, what tools were you using to {{do the thing your product does}}?

2. When did you realize you needed a tool like ours? What was going on in your world that caused you to start looking for one?

3. When you tried {{product}}, what were your general thoughts about the solution?

Follow this link for the full interview template

The next section is going to explain in a little more detail what ‘open-ended questions’ are, and why you and your team will need to use these in your interviews.

Open-Ended Questions

• These are free-form survey questions that allow a respondent to answer a question with more than just a yes/no. 
• These questions allow the respondent to answer based on their complete knowledge, feeling, and understanding, and the response is not limited to a set of options.
• When testing your prototype it is important to ensure that the bulk of your questions in the interview are open-ended. This will ensure that you get more helpful information from the interview that you can use to make your product better.
• When testing your prototype it is also important to avoid leading questions as they result in the user’s answers being influenced by the interviewer. These can be described as questions that prompt or encourage the user to give a specific, wanted answer.  

See below some examples of open-ended questions:

1. What do you think of this product?

This question is a great way to kick off the interview and allows you up to capture a user’s thoughts word-for-word.

2. How can we improve this product?

When asked this way, users are encouraged to focus on the parts of the product that are currently insufficient or those they don’t like. 

3. If you were in charge of this product what would you change?

You’ll find that users often share bigger picture concepts with you when responding to this question.

Top Tip for Interviews

Another effective method that is sometimes utilised for testing prototypes: 

• You pretend that you are testing the prototype on behalf of someone else
• This will likely ensure that you get more honest feedback from the user 
• The user generally feels more comfortable to provide critical feedback, and will not hold back in fear of hurting your feelings (Unlike if they knew that this solution was your hard work or "baby")

Team Preparation for Interviews

• Set a tight deadline for when the interviews should be done (narrow it to one or two days to finish all 5). Setting a clear deadline will probably be very motivating for your team.
• It is advisable to record the interviews so that only the interviewer is present in the room. This will hopefully help to get the user comfortable enough, to be honest, and provide both negative and positive feedback. Find a way to record the interviews well, which will enable your team to see the non-verbal cues as well as the user’s interaction with the solution. After the interviews, the rest of the team can watch them with the interviewer providing more context as needed.
• It takes a few days to recruit, so don’t wait until the last minute to do this. Try to avoid Mondays and days right before or after holidays; participants will be less likely to show up.
• As the last aspect of your team preparation, go through some of the interview etiquette pointers coming up next. After doing all the technical preparations, you want to be prepared on the social and professional front as well to ensure the success of your interviews.

Interview Etiquette

As you prepare for your interviews, you need to keep in mind a few things that will help the user you are interviewing feel at ease and potentially open up more. Below are the set of ‘rules’ that we are going to refer to as the Interview Etiquette, that will help your interview go well.

1. Friendly welcome & be a good host. Welcome the user and put him/her at ease. Throughout the interview, keep the user’s comfort in mind. Use body language to make yourself friendlier. Smile! Explain that you’re looking for honest and blunt feedback.

2. Curiosity mindset. Be authentically fascinated by your user’s reactions and thoughts. Ask the user to think aloud.

3. Context questions. Start with easy general questions, then transition to questions about the topic you’re trying to learn about. 

4. Introduce the prototype. Remind the user that some things might not work, and that’s exactly what you’re trying to figure out. Remind the user that you’re not testing him/her and there are no wrong answers or ways of interacting with the prototype. 

5. Tasks and nudges. Watch the user figure out the prototype on his/her own. Start with a simple nudge. Ask follow-up questions to help the user think aloud, as this can feel very unnatural to some.
6. Debrief. Ask questions that prompt the user to summarise. Thank the user for their time and show them out. Some people feel the need to compensate the user, so if you wish, offer to pay for transport or buy them a coffee or some other small item to show your appreciation.

Summary Checklist for User Interviews

• Select five users that fit your target profile
• Download the included interview guide with the essential questions and things to look out for, e.g. body language and facial expressions. Be sure to adjust the questions for your specific solution where necessary.
• Review prototype and interview guide with your team.
• Set up a test device to be used for interviews.
• Interview the five users 1-on-1 using the content in this deck to guide you.
• Debrief with your team after the interviews and summarise all findings.
• Plan next steps in relation to your team’s long-term goal.\

Post-Interview Team Analysis

• After you have conducted all the interviews, you will need to set aside some time as a team to review and consolidate your learnings.
• Allow all the team members who were watching the interviews to mention everything that stood out for them in the interviews, as well as other things they hadn’t noticed about the prototype before the interview.
• Thereafter, group similar thoughts/ideas together and come up with themes to describe the main issues or focus points.
• Lastly, work as a team to rate the ‘themes’ according to importance/capacity/resources/any other dominating factors that can affect the implementation of changes. 
• After identifying all the main themes from the interviews, use the assessment guide in the next section, which will help you decide on a reasonable way forward.

Assessing Testing Feedback

There are 5 broad decisions that your team will need to make following the testing of the prototype:

1. Discard: The team can decide to stop working on the prototype following a unanimous decision that the prototype is not worth continued investment.

2. Evolve: The team can decide to adapt the prototype in one or a combination of the following ways:

• Test the current prototype in a new setting or context.
• Adapt the current prototype (and evaluation design) based on new learnings.
• Upgrade the design brief and generate new prototypes.

3. Graduate to Pilot: If the test results are sufficiently positive and unambiguous, the team would fully test the prototype through a formal pilot.

4. Go to Scale: If the test results are so positive and unambiguous, and the risks are extremely low or manageable, then the team may scale the product further without any more testing. (Note - this is a very unlikely outcome and likely only the case if the prototype was a slight variation of an existing product that has already been in the market)

5. Keep Testing: If the results of the evaluation are not strong enough to make a decision, the team can upgrade the evaluation design and test again.

Key Take-Away

Building a prototype and testing it with 5 users is the quickest way to get valuable feedback that will help your team make the right decisions in line with your long-term goal.

‍

Post-Interview Analysis

The last aspect of your Prototyping & Testing section will be the analysis that you  and your team will carry out. During this step, you are going to look at all the notes you and your team wrote down when watching the interviews. This process will help you decide what is a reasonable way forward to take for your solution. Download the template for this activity below.

1. Put all the team notes together ( sticky notes on a board works well for this)
2. Look for patterns (e.g. positive, negative & neutrally reviewed aspects of your prototype)
3. Where should your team focus their energy going forward? (Particularly in relation to the 5 major decisions on the previous slide)
4. Review your long-term goal. Make a plan on what to do next.

*If at a later stage you want to test something else, find 5 different users and do not use the same people.

Assignment download

Post Interview Analysis Exercise (Word Doc)