Remote research: real users, real time, real research

Remote studies allow you to recruit subjects quickly, cheaply, and

immediately, and give you the opportunity to observe users as they

behave naturally in their own environment. In Remote Research,

Nate Bolt and Tony Tulathimutte teach you how to design and conduct

remote research studies, top to bottom, with little more than a

phone and a laptop.

“Nate and Tony use the attitude and power of Web 2.0 technologies to create a new way of thinking

about user experience research.”

MIKE KUNIAVSKY

Author, Observing the User Experience

“This tackles everything you need for remote research from soup to nuts.”

Jared Spool

CEO and Founding Principal, User Interface Engineering

“Bolt and Tulathimutte provide a delightfully clear and practical introduction to how to do

remote research.”

James D. Hollan

Professor of Cognitive Science and Computer Science, University of California, San Diego

“No longer must you only test your interfaces with people who come to your office or some expensive

lab. So stop doing boring research! Stop having a boring life! Read this book!”

Mark Trammell

User Researcher, Twitter

www.rosenfeldmedia.com

MORE ON REMOTE RESEARCH

www.rosenfeldmedia.com/books/remote-research/

R E M O T E R E S E A R C H

Real Users, Real Time, Real Research

by N ATE BOLT and TONY TUL ATHIMUT TE

foreword by Peter Merholz

REMOTE RESEARCH by N ATE BOLT and TONY TUL ATHIMUT TE

Rosenfeld Media

Brooklyn, New York

Remote Research

i

Enter code RRDE for 15% off any Rosenfeld Media product directly

purchased from our site: rosenfeldmedia.com

Nate Bolt

Tony Tulathimutte

Do

ii

Table of Contents

Table of Contents

How to Use This Book viii

Frequently Asked Questions xii

Foreword xviii

Chapter 1 1

Why Remote Research? 1

The Appeal of Lab Research 3

Is Lab Research Dead? 7

What’s Remote Research Good For? 15

When to Go Remote 23

Moderated vs. Automated 35

When to Use Which Remote Method 38

Chapter Summary 42

Chapter 2 44

Moderated Research: Setup 44

Gearing Up: Physical Equipment 45

Doing a Pilot Test Right Now 48

Preparing for a Real Study 53

Drafting the Research Documents 62

Chapter Summary 80

Chapter 3 82

Recruiting for Remote Studies 82

What’s “Live Recruiting”? 84

Live Recruiting Using Forms and Pop-ups 90

Designing the Screener 101

Paying Remote Recruits 110

Choosing Good Users and Spotting the Fakers 113

iii

Table of Contents

Table of Contents

Recruiting Slow? Don’t Sit Around—

Fix the Screener 117

Wait! Read On… 124

Chapter Summary 125

Chapter 4 127

Privacy and Consent 127

Certain Assumptions 128

Set Up Your Web Site’s Privacy Policy 130

Basic Participation Consent 132

Consent to Record 142

International Consent 146

Consent for Minors 156

Chapter Summary 163

Chapter 5 165

Moderating 165

Introducing the Study 167

Time-Aware Research and Going Off-Script 177

Smart Note Taking 182

Working with Observers 193

Quiet, Chatty, Bored, Drunk, and Mean 200

Ain’t Nothing Wrong with Using the Phone 208

Wrapping Up 213

Chapter Summary 214

Chapter 6 216

Automated Research 216

The Structure of an Automated Study 218

Recruiting for Automated Studies 222

iv

Table of Contents

Table of Contents

Different Kinds of Automated Research 227

Task Elicitation 229

Remote Card Sorting 250

Surveys 260

Other Methods 262

Where Do I Learn About Tools? 270

Chapter Summary 271

Chapter 7 273

Analysis and Reporting 273

Moderated Analysis 274

Automated Analysis 284

Reporting 295

Not All Research Is Research 310

Chapter Summary 314

Chapter 8 316

Remote Research Tools 316

Screen Sharing 318

Recording 343

Automated Tools and Services 354

Chapter Summary 376

Chapter 9 377

New Approaches to User Research 377

Top Secret (Reverse Screen Sharing and

Remote Access) 378

Mobile Device Research (Smartphones) 384

One-to-Many Remote Moderating (Video Games) 388

v

Table of Contents

Table of Contents

Portable Testing (Cars) 403

Staying Current 414

Chapter Summary 415

Chapter 10 416

The Challenges of Remote Testing 416

Legitimacy 417

Not Seeing the Users’ Faces 418

Technology Failures 420

Not as Inexpensive as You’d Think 423

Organizational Challenges of Web Recruiting 424

Getting the Right Recruits 425

Natural User Behavior 427

Multitasking 429

Security and Confidentiality 430

Persistent Negativity 431

Chapter Summary 433

Conclusion 435

Don’t Waste Your Life Doing

Pointless Research 435

Index 441

Acknowledgments 460

About the Authors 463

Remote Research

By Nate Bolt and Tony Tulathimutte

Rosenfeld Media, LLC

457 Third Street, #4R

Brooklyn, New York

11215 USA

On the Web: www.rosenfeldmedia.com

Please send errors to: [email protected]

Publisher: Louis Rosenfeld

Editor: Marta Justak

Copy Editor: Chuck Hutchinson

Interior Layout Tech: Danielle Foster

Cover Design: The Heads of State

Indexer: Nancy Guenther

Proofreader: Dan Foster

© 2010 Rosenfeld Media, LLC

All Rights Reserved

ISBN: 1-933820-44-6

ISBN-13: 978-1-933820-44-6

LCCN: 2009936925

Printed and bound in the United States of America

vii

Dedications

Dedications

Nate: To my family,

friends, colleagues,

teachers, and loved

ones

Thanks for listening

to me ramble on

about this field for

so long.

Tony: To Mike

Hardnett and Renee

Zalles

viii

How to Use This Book

How to Use This Book

Who Should Read This Book?

This book is about remote user research, which

is a method of using Internet tools and services

to conduct user research with participants who

are in another location. (User research, in turn,

is the field of studying how people interact

with technology.)

Are you a user experience/human-computer

interaction practitioner? If so, you’re totally

gonna love this book, especially if you’ve ever

been frustrated with current in-person or lab

methods of user research for any of the several

reasons we describe in Chapter 1. If you’re a

software or Web developer looking for insights

into your own (or your competitors’) designs,

or an interaction designer or consultant, you’ll

probably dig this book too.

Is there anyone this book isn’t for? You don’t

need to be a veteran user-experience researcher

to understand what we talk about in this book,

although we do focus mostly on the “remote”

ix

How to Use This Book

How to Use This Book

aspects of remote research. You won’t find much

advice on how to conduct user research in

general—for that, a great place to start is Mike

Kuniavsky’s Observing the User Experience.

What’s in This Book?

Remote Research is a how-to book about

remote research methods: using a phone

and the Internet to conduct user experience

research from a distance.

In the Introduction and Chapter 1, you’ll get

an overview of what remote research is all about,

when you should and shouldn’t use remote

methods, and the two main kinds of remote

research studies: moderated and automated.

In Chapters 2 through 5, you’ll learn how

to set up, recruit, and conduct a basic remote

moderated study. We describe a method called

“live recruiting,” which involves intercepting

visitors to your own Web site to participate in

your studies immediately. We also discuss the

x

How to Use This Book

How to Use This Book

privacy and consent issues around recruiting,

session recording, and remote participation.

In Chapter 6 we describe various automated

research methods, illustrating them with case

studies.

Chapter 7 provides advice about how to

analyze and report on the findings of remote

research studies.

Chapter 8 is a short guide to tools and services

you can use to fill the many technological

needs of remote research, including screen

sharing applications, recording software, and

several online automated research Web apps.

Chapter 9 illustrates how many of the basic

principles described earlier in the book can

be adapted and applied to special testing

circumstances, where normal remote testing

methods aren’t possible or desirable.

Chapter 10 is a review of the major challenges

you’ll face when planning, conducting, and

presenting a remote research study.

xi

How to Use This Book

How to Use This Book

What Comes with This Book?

This book comes with two companion Web

sites. The first is the Remote Research page

( www.rosenfeldmedia.com/books/remote￾research) on the Web site of our publisher,

Rosenfeld Media. The other site is our

companion Web site for the book, Remote

Usability (http://remoteusability.com). On both

sites you’ll be able to find detailed resources,

research document templates, supplementary

articles, and up-to-date lists of remote research

tools and services.

We’ve also made the book’s diagrams,

screenshots, and other illustrations available

under a Creative Common license for you to

download and include in your own presentations.

You’ll find the original illustrations and diagrams

from this book at http://www.flickr.com/photos/

rosenfeldmedia/sets/, or you can just double￾click the pushpin next to the image to see them in

high resolution.

xii

Frequently Asked Questions

Frequently

Asked Questions

What is remote user research, anyway? Is

it anything like focus groups or surveys?

Remote user research is simply a type of user

experience (UX) research that’s conducted

over the phone and Internet, instead of

in person. In general, UX research seeks

to understand how people interact with

technology. Unlike focus groups and surveys,

market research techniques that are used

to learn people’s opinions and preferences,

UX research focuses on studying people’s

behavior. In that sense, remote user research

isn’t really like market research; however, both

remote user research and market research can

be applied toward improving the design of

existing technologies and inspiring new ones.

See Chapter 1, page 3–4.

What kinds of remote research are there?

That’s a huge question, and we spend a good

chunk of this book introducing and describing

the many varieties and specialties of remote

research out there. In general, there are two

xiii

Frequently Asked Questions

branches of remote research: moderated

and automated. In a moderated study, a

researcher talks directly to the participants

as they use the interface in question, and it’s

good for obtaining rich, qualitative feedback.

In an automated study, you use online tools

and services to gather behavioral or written

feedback and information automatically,

without the researcher’s direct involvement.

For more about moderated testing, see

Chapter 2, page 53, and all of Chapter 5. For

more about automated testing, see Chapter 6.

I’m skeptical about remote research. If

it’s so great, why haven’t I heard of it?

A lot of the misgivings that people have about

remote research come from its novelty. The

method is still cutting edge, and the technique

requires a certain degree of know-how. Until

now, there hasn’t been a book you could learn

the method from—which is, of course, the

reason we wrote it.

Frequently

Asked Questions

xiv

Frequently Asked Questions

Still, lots of companies have done it, with great

success. We’ve done remote studies with Sony,

Autodesk, Greenpeace, AAA, HP, Genentech,

Wikipedia, UCSF Medical Center, the

Washington Post, Esurance, Princess

Cruises, Hallmark, Oracle, and Blue Shield of

California, among many others.

I’m still skeptical. Can you really get

valid behavioral feedback without

seeing your participants in person?

Since remote research is conducted over the

phone and Internet, many people worry about

missing “rich details” like facial expressions

and body language. First, we believe that for

most user research studies, the way that users

interact with the interface and their think￾aloud comments are the only really necessary

things to focus on. And on top of that, much

of the tone does come through the user’s voice

and language. We weigh the pros and cons

of in-person research and remote research in

Frequently

Asked Questions