Why being data-driven and using your intuition aren’t as different as you think
Data-driven, data-informed, KPIs, success metrics, A/B testing, prototypes, insights, intuition, market research, surveys, usability testing, gut, user empathy… How on earth is a team supposed to decide what to do next? I am going to tell you the answer (yes that’s right, I’m actually giving an answer)— use your gut but make sure it’s really healthy first.
Ummm… healthy?
There is a wealth of information about how the gut is as complex a system as your brain, and possibly even more important. It is an entire ecosystem that needs to have the right inputs to keep your whole body healthy. Your ability to use your intuition to guide your product is much the same way. If you don’t feed your gut it’s going to make crappy decisions. If you feed it the right information (data) regularly, the answers will be much clearer.
Think about what it would be like to build a product for yourself. For you and only you. Exactly what you want. Do you think it would be hard to decide what features to put in? Probably not. It is easy because you have consumed all the data there is to consume about being you. Without a focus group. Without Google Analytics. Without having to do anything but live.
Your goal is to try (you will never quite get there) to get to that same state with your users. Know them like you know yourself. But since you haven’t lived their lives, you need to work really hard to do this.
Let’s back up a sec
If you’ve been a PM for while you’ve probably experienced one of these situations…
You spend a bunch of time with the metrics, find some insights, do feature brainstorming, comb through your roadmap and your backlog, do some customer feature validation, send out a survey, then put together a prioritization framework using value/cost model or Kano or asking your customers to spend 100 dollars or weighted goals (I’ll be writing that one too — this is my favorite). And then:
- Your stack ranked feature list comes out and everyone looks at it and says “Well duh. Why did we do all of that work, this is so obvious!”
- Key stakeholder(s) sees the list, insists it’s wrong and re-orders most of it, making your work essentially a waste of time.
What happened? In scenario 1, the team was brought along during all of the work above. Their guts all got primed with the same inputs. And so the resulting list matches their (very carefully, very intentionally fed) intuition. A common mistake as a PM is feeling discouraged that maybe the work was indeed a waste of time. It wasn’t (there’s me being black and white again!). Scenario 1 is unmitigated, wild success. Go tell some fellow PMs about it over celebratory drinks and maybe commiserate a little about how hard your job is and how difficult it is for anyone to understand but other PMs. If you tell me a story about this happening to you, you’re hired.
In scenario 2, the stakeholders weren’t part of the process and they probably think their intuition is why they got where they are today. This is (unfortunately) rampant at Apple. See a later article about managing up. This is really tough and you need to work hard to avoid it. But it’s not the topic for today.
Getting to gut health
Are you the user?
The less “like” the user the people on the team are, the more work you need to do to create an optimal microbiome. If you are a bunch of millennials trying to create an app for millennials then you don’t have to work as hard as if you are a very diverse team looking to make an app for people in retirement homes. So that’s the first question you should ask yourselves. But be careful. Are you really the “same” as your target users in ways that matter to the problem space?
As an example, when I worked on Microsoft Office, we looked at the metrics of our users for email. The average number of emails an employee received was an order of magnitude lower than what we experienced on a daily basis. A natural assumption is that we are knowledge workers and therefore our experience is very much like that of our knowledge worker customers. But that was not the case on the critical dimension of email. If we brainstormed without that knowledge, we would come up with a bunch of features for dealing with the onslaught of email and miss the problems our users were really experiencing.
Metrics
If you’ve got them, metrics are incredibly valuable food for the gut. If you don’t have them… get them. You are flying pretty blind without them. But a very common mistake people make is needing metrics to be “actionable”. The way to use metrics (beyond A/B testing which is another topic) is to immerse in them. Try a weekly meeting where you alternate the person running it and they bring a metrics topic to the table. The goal of the meeting is simply to discuss the topic. Why does the metric make sense? Why doesn’t it? What ideas does looking at it spark? What other questions do you have now? The actions from this meeting can certainly be tracked. You may put some work items on the list for the data team, you may add some instrumentation feature requests, you may put some features on the backlog. But make sure everyone knows the true purpose of the meeting is gut health.
Qualitative
When done well, I’m a huge fan of user research as a way to feed the gut. It’s hard to do well and can be costly. And it’s much harder for the left-brained among us to understand the value which makes it riskier to push for as a PM. You will find that your users don’t know the product nearly as well as you hoped they do which makes it harder to see how this helps you move the product forward. Customer interviews and moderated focus groups are two of the most common qualitative methods used.
We did some focus groups at Apple Maps and one of my key insights was that we teach each other how to use the app. Internally, this turned, in our collective guts, into “the” way the app was used. We are all, of course, heavy users of our product and often show each other things that we experienced that were not optimal. We watch the product designs evolve and so knew how to use the new features before they were implemented making it impossible to evaluate discoverability or sometimes even usability. I forgot things I knew when I joined the team. That many people don’t even know which maps app they are using. Most people have no idea about half the features the app is capable of. Even more interesting, different countries were like our team in that they evolved collectively a way to use the app (presumably through the same methods) that was distinctively different from other countries. Was that insight “actionable”? Not really. Did it help me make better decisions every day. Absolutely.
I feel like I have to address the faster horse thing here. You know what I’m talking about? The so often quoted reason for not doing qualitative research. Henry Ford apparently said (although there are some doubts about this) that if he had asked people what they wanted when he was building the first car, they would’ve said “a faster horse”. The implication being it’s a waste of time to ask people what they want. Well, I respectfully disagree. Here is what that conversation would look like with a good PM at the helm.
Customer: “What I really want is a faster horse”
PM: “Interesting. Tell me more about why that would help you.”
Customer: “Oh wow, thanks for asking. My horse sometimes goes slow and sometimes fast and it’s hard to control it. I guess it gets tired or hungry? What I really want is to get somewhere when I need to be there and not have it be so dependent on my horse’s current state.”
Now we have empathy. Hypothetically, if the people on the team all had a stable of horses so they could pick the freshest one every time, they may not have been able to empathize with the plight of the common man that only had one horse and had to use it every time. Maybe I’m stretching the analogy… a bit :). But hopefully you get the point. A suggested feature is just a starting point for understanding customer pain. And good user research isn’t asking for feature requests anyway. And if I hear that quote one more time I might actually explode. Just sayin’.
Surveys
If focus groups are tricky, surveys are rocket science. Seriously. Just asking a question has often primed the answer in some way that you may not have intended. But there is value in them if you (again) don’t try too hard to make them “actionable” but instead consider them food for the gut.
A common use is to ask the user to stack rank the feature list you are considering. Or give them $100 to spend on the feature set so they can weight their responses — giving $50 to the one they really want, for example. I haven’t found that particular use to add much to my gut. I suppose it might be a way to satisfy those whose guts haven’t been fed the whole way. But someone who really doesn’t agree with the ordering will probably bring up the faster horse, causing me (or you) to finally explode. Or simply overrule.
A better way is to try and validate a hypothesis by looking at metrics. Or to get scaled data about something that came up in a focus group that you find yourselves wondering if it is more broadly true. My favorite example here is the cup holder. In watching our users, we found that many of them put their phones in the cup holder while they were using Apple Maps. Most of us had fancy phone holders that displayed them safely in our view while driving (and a lot of us put them in the cup holder and didn’t tell anyone). We did an extensive survey and indeed, a significant percentage of our users put their phones in the cup holder. It became a topic in our design discussions; our Apple maps team guts were successfully primed with this information. One day about a year later we were working with another team at Apple and discussing the importance of something together. The details are confidential and also not that relevant. What is relevant is that if the user had his/her phone in the cup holder, we would all agree on one thing and if they didn’t, we would agree on another. The other team professed at one point that “almost no one has their phone in the cup holder”. And we fortunately had the data to say… uh, actually they do.
I hope you are now convinced that it’s not about using your intuition for some decisions and being data-driven for others. It’s about gathering the necessary “data” to build empathy for your users so that you can innovate on their behalf. The more you feed your gut and are able to really be the user, the more your ideas will truly serve them in the end and the better you will be able to recognize the ideas that will actually make their lives better. And that’s why we do this, right?
And just like the body’s gut health, this is not a one shot deal — eat well for a week and you’re all set. Your gut needs to be constantly fed or it will start building products for you again, instead of your users. Think about it like taking your daily dose of probiotics in the morning.