Lee Carter on the Value of Direct Observation

A Q&A with Lee Carter where we explore the value of direct observation of customer environments to complement customer interviews. Lee shares details of a project he worked on where direct observation provided a valuable complement to a basic product specification.

Lee Carter on the Value of Direct Observation

Lee Carter also does customer interviews on behalf of products and this Q&A is an edited transcript of a conversation we had in May 2019. Carter is a strong proponent of going onsite for direct observation of the customer environment as a complement to customer interviews.

Direct Observation Provides Insights You Won’t Get From Interviews

Q: You are a strong advocate for the going onsite to be able to see the customer’s environment and how they operate in it as a complement to customer interviews. Can you tell me a little more about your experience with direct observation?

A: Well, the short answer is that observation will give you insights that you won’t get from just interviewing someone.

The best example I have of this is from a project I was part of back in 2014-2016.

Our team had been hired to rewrite a training software for teachers. The client was a non-profit agency that provided language and literacy training to early childhood education teachers. Participation and completion rates on the previous system had been abysmal, and the agency had received a state grant to rewrite their learning management system and courses.

The software we were designing would be used by people in a number of different roles: The teachers who would be receiving the training; the instructional designers who would be creating the courses on the LMS; and the program administrators who needed to report enrollment and completion data back to the state.

We were fortunate to have a very talented Project Manager on our team who was able to persuade the non-profit to arrange for us to visit a couple of the schools in the area that had received training on the old system.

If No One on the Team Has Seen the Customer’s Environment, That’s a Significant Risk

Q: I agree, if you’re going to introduce a product into their environment, and it’s the first time you’ve been in their environment, then that’s a problem. It’s fortunate you were able to not only gain access but also convince the team to visit.

A: Yes. Absolutely.

Q: Tell me about your visits.

A: In the morning we observed two different classrooms in a Head Start Center in a less affluent neighborhood. Then in the afternoon we visited one class in a school in a more affluent suburb.

Q: So did you just sit in the back of the room and observe?

A: At first, but after a while it started to feel unnatural just standing there, especially since there were six of us.

Q: Six? That’s a lot of observers.

A: Yeah, we brought the whole team. That probably seems odd, but we wanted the whole team there because we felt that it was important to have everyone’s observations and input.

Anyway, it started to feel unnatural just standing there, so we started participating in the classroom activities and talking with the kids, too.

Key Learnings

Q: What did you learn from observing the teachers in the classroom that you wouldn’t have learned from just interviewing them.

A: So, when I list out the bullet points in my head:

Their schedules were chaotic and unpredictable, with only short windows of time in which they could study during work hours
The computers provided by their schools were ancient and underpowered, and so it took forever for courses to load. Because of this the teachers preferred to study on their own mobile or tablet devices; but it was something they weren’t able to do with the existing courseware which was Flash-based
Some of the teachers confided in us that they found the vocabulary of the existing courseware difficult to comprehend
Teachers understood the value of the courseware, both for their students and their own professional development, and were enthusiastic about being able to complete the training.

Two Key Benefits: Rapport With Customer and Deeper Commitment by Team

Q: You believe that a good interviewer would not have uncovered these issues? Were there other benefits?

A: I think that some of these things could “in theory” have been uncovered from just interviewing the teachers, but being there in the customer’s environment does two really important things:

First, it helps build rapport with the person you’re interviewing. Not the small talk kind of rapport that happens when you’re sitting across a table or a Skype connection from someone.

By taking the time to travel to and physically be there with someone, you’re showing them that you consider them – and their work – to be important, to be valuable; and so I believe they’re more inclined to open up to you. Like they did about the vocabulary issue.

And hearing that their work environment is chaotic is not the same as being there and feeling what it’s like to be in the middle of the chaos. Frankly, that’s one of the big reasons we ended up advocating for and designing a system that allowed the instructional designers to break up the old monolithic, hour-plus courses into short, easy-to-consume modules.

The second important thing is that we became emotionally invested in the project. We found that we came to really like and admire the teachers. And we really liked the kids, too. Because of that experience, we wanted, on an emotional level, to make this project a success for them.

What Happened Next?

Q: After you developed the LMS, did you go back and observe teachers using it?

A: No. We really should have, and honestly I’m not sure why we didn’t. The people working at the non-profit reported back to us their satisfaction with the system, and also mentioned anecdotally that the teachers taking the courses were happy. But, yes, I wish we had gone back.

Q: How long did it take you to deliver the improved LMS?

A: It was a two-phase project. It took us about 3 months to add a responsive UI to the old LMS, communicating via an API. Then another 6-9 months where we did a complete rewrite of the LMS.

Q: After the second phase, what happened?

A: They just kept re-engaging our team for application enhancements – a teachers forum, enhanced reporting, and so on. In fact, when I left the company they had been a client for two years, and the last I heard they were still a client.

Q: What were the improved outcomes?

A: I wish I had percentage improvements and things like that, but this was five years ago and I just don’t remember.

What I can say is that when we started the project only 3,000 or so teachers had signed up on the old LMS over the previous several years; and today there are over 60,000 users on the system. Also, they’ve gone from offering a single learning track of 7 courses to four tracks and a total of 16 courses.

This ability to create lots of courses is something I’m really proud of – before our rewrite the instructional designers had to outsource the implementation of the previous Flash-based courses to specialized developers – it was expensive and took forever for new courses to be delivered. But we created a system that let them create courses on their own–quickly and with zero additional cost.

Net Net: Direct Observation Complements Customer Interviews

Q: Any other thoughts?

A: I’d say that observation and interviews go hand-in-hand. Observing people without talking to them will only get you so far (and maybe not very). Talking to people without observing them in their work environment will also only get you so far. So if you can do both, then do so, because you’ll create a better product in the long run.

Q: Thanks very much for your time and your insights.

SKMurphy Take

I blogged about the value of ethnographic methods (embedding yourself in the customer’s environment as an observer) in “Chip Conley: Reconceive Bewilderment As Curiosity”

“First, I quickly learned that I needed to strategically forget part of my historical work identity. […] More than anything, I listened and watched intently, with as little judgment or ego as possible. I imagined myself as a cultural anthropologist, intrigued and fascinated by this new habitat.”
Chip Conley in I Joined Airbnb at 52, and Here’s What I Learned About Age, Wisdom, and the Tech Industry

To the extent that you can observe customers in their natural habitat you can learn as much by keeping your eyes open as you do from keeping your ears open in a customer interview. Appreciative inquiry offers a sound approach to understanding what’s working in a prospect’s firm before suggesting changes.

I think direct observation is very important when no one on the team has prior work experience in the environment your product is designed to operate in. There are too many intangibles that you can overlook when you are not oriented on the “ground truth” of the customer’s situation.

One trap technical founders can fall into when selling is to believe that they are “bringing fire to the savages.” They see the benefits of their offering very clearly, but assume that their prospects are lost or clueless unless they adopt the new approach. Most of the time there is a working status quo that a new offering has to significantly outperform to foster adoption. Very few new technologies are fully form fit and function compatible; this further increases the cost of adoption and makes it less likely

One mindset you can adopt to guard against technical arrogance is appreciative inquiry, which assumes that there are a lot of things are that are good about the current situation and you should understand and appreciate them before suggesting any changes. Focus on clearly understanding the customer’s perception of the problem and the constraints that any improved solution must respect, some of these will be technical, many will be cultural.

M	T	W	T	F	S	S
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30

Lee Carter on the Value of Direct Observation