User research observation in a remote-first world

Today I conducted a research session without an observer (at the participant’s request) and I felt so much more connected with the person taking part. It got me thinking once again about research observation, now that we’re doing more remote work.

Traditionally, (in my experience) user research observation would happen by streaming a secure link into a separate room where the team would sit and watch, discuss and take notes together.

Moving to a remote-first world during the pandemic, it now seems to have been replaced by having an observer (or multiple observers!) sit on a Zoom call. This is a big bug bear for me and I’ve raised it a lot in the teams I work with. I’d love to know — how are you handling this? Is anyone setting up secure streaming links for remote observation, and how?

What I’ve been doing

In absence of a live stream link, I’ll invite the participant to the session, collect their consent and ask if they’d be happy with an observer. If so, I share a separate invite with the whole team, ask them to click attend if interested in observing then choose one (max two) to sit in on the session, observe and take notes. Then I’d add these chosen observers to the main invite.

• I try to keep at least 1 consistent observer throughout the research
• I limit observation to 1–2 people per session
• I always share observation guidance with whoever is observing
• I do a debrief after each session with all observers on a Slack huddle

Observers must confirm they’ve read and are happy with the guidance, which includes things like not sharing the participant’s name outside the session, taking verbatim notes, storing notes in a secure area (not locally), staying on mute etc. You might think this is a bit strict, but I once had a stakeholder take themselves off mute and start asking questions — hence the guidance!

There’s obviously problems with this approach too, such as bias in who is available or selected to observe and giving the whole team a chance to be involved. I’ve found these things can help, but there’s still a bunch of pros and cons of research observation done remotely. Things like…

Pros of research observation

• Research doesn’t happen in isolation
• Team are involved
• More likely for findings to land and have impact
• Can reduce bias
• Safety for the researcher (if a sensitive topic)
• Someone to debrief with
• Someone to take notes
• Researcher can focus on the session instead of capturing all the notes

BUT many of these pros I feel are counteracted by the cons of having people sat on the research call itself.

Cons of research observation (on a call)

• Can be intimidating for the participant
• Participant could be less likely to be honest (if they think the people on the call observing are part of the team designing this service)
• Researcher can be distracted or on the spot (I personally feel a small amount of pressure that I need to be doing ‘good’, because it’s usually a client or colleague watching, which may affect quality of the session)
• Less rapport between participant and researcher than you get 1:1
• Observer can un-mute themselves and derail the interview (unlikely but can happen)
• Anonymity of the participant
• Not all the team can be involved in the research (if limited to 1–2 observers)
• Bias —observer can get hung up on the findings from one session they have watched, if they haven’t had chance so sit in on them all

Don’t get me wrong, I certainly don’t think research should happen in isolation without observers — but I don’t know if observers sitting in on the call is a good thing either.

What do you think?

You might be thinking well can’t the team just watch the recordings? Yes, they can, but there are problems with that too. One is that most people don’t make the time to sit through 6 hours of recordings. Two is that it can help to watch sessions in real-time and with other people. This gives the team chance to discuss what came up and the researcher chance to debrief and process the session with the team, before the next one. This debrief is particularly important if a sensitive topic is being researched.

I guess if we’re talking about safety and anonymity, I have also noticed how softwares like Zoom show up the person’s name on the recording. The Home Office had software to edit faces and names out of the recordings — this is good, but takes a while. What else are people using to conduct the research? Perhaps there’s a tool out there where we can conduct sessions and stream them anonymously with the team?

In an ideal world, I’d love to set up a secure streaming link and have the team observe in a separate meeting, led by a facilitator (researcher).

Of course, we would always get informed consent to stream the session in this way, but I feel like this would work so much better than having people join the call itself.

• What do you think?
• How are you managing observation in remote user research?
• Have you used secure streaming links before? What software are you using and how’s it going?

Thanks for reading ❤️