"You're on Mute" - Lessons Learned After a Year of Conducting Remote User Research

July 21, 2021

Back in March 2020, the circumstances forced us to quickly adapt to a new mode of working — remotely. That change couldn’t slow our user research work down. Unfortunately, the research setup we were used to, with our own fully-equipped research lab, could no longer be an option so we had to adapt and reframe to a new remote reality.

Throughout the year we conducted diary studies, individual and collective interviews, and usability tested prototypes for both national and international projects. We ended up learning a lot with each one, perfecting and refining our remote methods, tools and tasks.

Now, after more than one year of engaging with users remotely, we want to reflect on the pitfalls of remote user research, share some of the lessons we learned and reflect on what’s going to be “the next normal” after Covid’s impact.

1 — No control over participant’s environment

We were used to conducting research in our in-house research lab. We knew we had good internet connection and all equipment ready and working for the session. The research room was a quiet place. We knew we weren’t going to be interrupted by a door bell ringing or a dog barking. We had almost full control over the environment. When moving to remote research we simply lost control over the conditions in which research takes place. Participants can join in from anywhere, from their home, to their workplace to a tree shade in the middle of the countryside (yup, true story 😅).

Photo by Nathan Dumlao on Unsplash

Lesson Learned #1

It’s pretty much impossible to control where our participants will join us from, the noises they have in the background (often hammering noises from construction work nearby or kids wanting attention), their wi-fi connection, etc. In such cases, the facilitator has to assess the situation and decide whether or not the interview has the necessary conditions to continue.

Even if you try to validate the participant’s conditions beforehand, there is no guarantee that on the research session day they are not elsewhere or with different conditions. The best thing to do is validate right before the beginning of the session.

One of our participants joined the interview in the middle of the countryside, under a tree, with his computer on top of the car hood. Of course, his internet connection was not ideal and we had to call off the test.

Sometimes asking the participant to move closer to the router or change to mobile data might solve the situation. Other times their internet connection is so poor it’s impossible to move on with the session.

No matter the issue, the facilitator’s decision should be based on how the research will be impacted, on how long the issue seems like is going to last, and if the session is going to provide the necessary insights.

2 — Participants might be tech-savvy or not so much

Research participants have different backgrounds and ages so they might or not be at ease with video conferencing tools. Fortunately, the increase of remote work over the past year and the need to find new ways to keep in touch with friends and family has made people more used to tools like Zoom, Microsoft Teams or Google Meet. They’ll likely know how to install the necessary app, join the call and share their screen.

Lesson Learned #2

In the first remote research sessions we conducted, we used to start with a set of introductory questions to make the participant feel more comfortable. Only then we moved to the prototype testing, when we asked them to share their screen so we could see what they were doing in the website or app we were testing. However, we’ve rapidly come to the conclusion that before making any questions it would be best to take a quick tech check, where we ask participants to share their screen. This allows us to make sure that everything’s working and that they’ll know how to do it when the prototype testing begins.

In the recruitment phase, the technical requirements for the session must be clearly explained. We share the Zoom link to the interview beforehand, inform participants whether they should join with their computer or smartphone or if they’re supposed to share their screen or not.

3 — Time management becomes more challenging

Participants running late, technical problems, distractions, background noises are just some of the things that might cause a remote research session to take a bit longer than a normal in-lab session.

Also, not everyone is a talker. There are those participants who only answer exactly what you ask them and often need a bit more prompting to share the information you’re looking for. Then there are those chatty ones who provide feedback freely with little prompting, and even go off-topic sometimes. In this last case we risk the session being longer than anticipated. The facilitator has the somehow difficult task to find ways to make the participant focus on what’s being asked or might even have the need to cut off a few questions to stay within schedule.

Photo by Fakurian Design on Unsplash

Lesson learned #3

When you have a full-day of remote sessions, it’s important that you make room for at least a 30 to 45-minute break between them. This will ensure that any delay with a session will not affect the next one and that the facilitator and the note taker can have some resting breaks in-between sessions.

In order to save some time during the session we also ask participants to fill the consent form beforehand.

4 — Take your script for a test drive

Imagine you’re 10 minutes into the remote session, you’ve done a few introductory questions, and you’re now moving on to the prototype testing. As the participant tries to accomplish the tasks you realise some feature in the prototype is not working properly or that you’re making irrelevant or repetitive questions.

That could all be avoid with a pilot test. It will help to identify whether you’re asking the right questions, if your prototype is working properly, and if your tasks fit the time you have for the session. It can easily be done with a colleague over a video call.

Lesson Learned #4

Sometimes we’re the ones writing the test script, other times the client sends us the test script for us to translate and apply. In either situation we always conduct a pilot test. It has helped us to remove repetitive questions and, most importantly, check if the prototype is working properly or not. Running it a couple of days before the sessions gives us enough time to get back to the client with any changes or clarifications needed.

5 — Always take notes

Our preferred tool — Zoom

There’s a wide variety of remote video conferencing and collaboration tools but we ended up realising Zoom was a good fit for us. Instead of a Zoom Meeting we found out it would be better to create a Zoom Webinar for our usability tests and user interviews. The facilitator starts the webinar and enters as a host. Then the participant enters as a panelist, which allows them to share their video, audio and screen. All team members (like the note taker) and the client team, enter the webinar as attendees. They don’t have their video or audio on but can see and hear everything, without disrupting the session.

One of the things we found out was that the client team would sometimes end up talking with each other using the Zoom chat. The problem here is that the participant will actually be able to read those messages, which can make them feel more uncomfortable or deviate their attention during the session. The solution then is to change the settings to “Allow attendees to chat with no one”.

How UX research is moving to the “next normal” after Covid’s impact

As things gradually open up globally, returning to face-to-face research might be too soon but eventually things will get back to normal. The pandemic will undoubtedly influence how people decide about the activities they do and the places they go for the next months, or even years.

The UX Matters report “Going Back to Face-to-Face Research After COVID-19” found that, as of June 2020, 71.8% of UK participants would be comfortable doing in-person research. Older people are less willing to participate whereas key workers are the most likely group to want to participante in face-to-face research. They concluded that even if people are currently uncomfortable with in-person research, they’re likely to be ready to participate soon.

Covid-19 risks vary considerably from location to location, and sometimes even from one week to the other, so the return face-to-face research might be a closer reality for some countries than for others.

For a safe return to our user research labs we’ll have to provide a safe environment based on the recommendations of local health officials. That will probably include a detailed hygiene protocol (cleaning and disinfection of facilities and hardware), personal protective equipment (face masks), social distancing and screening questions specific to Covid-19 health and safety.

User research is possibly more relevant now than ever. More people are now grocery shopping online, using their bank’s app, ordering their dinner through food delivery apps. Globally, 49% of consumers shop online more now than they did pre-COVID-19.