Some people in this world are just too lost in their own thoughts. That’s how my friends from childhood would describe me. I grew old, things changed a great deal, but still part of me stayed that way.

Every time an event is full of impressions, full of interesting people, full of good content and good food, I forget. I forget to ask for names of attendees. I forget to make pictures.

Fortunately I do remember some tips that helps me being more efficient in my daily life. I also remember from time to time follow up on that and apply some of the techniques I’ve learned.

Last week I attended a workshop prior to the Mind the Product conference in London. You got it. I have no pictures. No business cards. Only new hints about where to start when creating an evidence-driven company and what’s the role of product managers in driving that culture.

Here are the eight lessons I’ve learned:

1. It’s all about spotting user behaviour patterns. Stop talking about statistical significance. It’s ok to have bias. Don’t let these ghosts block you from using a good mix of quantitative and qualitative data you can get. Go out there and explore!

“You know you’re done with user research if you can predict what someone else will say based on their user demographics”. — Running Lean.

2. Tooling is a distraction from the actual problem. The following quote resonates a lot to me, as in our Product O’Clock meetings we’ve been talking about how product managers could easily get hold of statistics and share with others.

“Stop leading with process. Lead with the outcome. Then figure out the how to achieve the what”. — Barry O’Reilly.

3. Goals & metrics alignment. Invite all stakeholders from different areas to identify the key metrics for your product. Use the business model to frame this conversation. Derive from there the One Metric That Matters (OMTM). Follow-up on that by defining other parameters that will tell you how you can fine tune your product to reach your goals.
4. Yes, we’ve set One Metric that Matters, but it doesn’t work. Make sure that this metric is: Comparative (to other times, groups of users, competitors), a ratio or a rate (total users/monthly active users) and changeable (flexible enough to be changed at any time). This metric gives you hints to change the way you behave. If one of these elements is missing or failing, most probably you need to rethink your OMTM.

Short quiz: How would musicians of an orchestra measure their success with One Metric that Matters? In a theather where tickets are always sold out, where Marketing is the one generating leads, how can musicians ever understand which metric they have a direct influence on? You can find the answer in the end of this article.

5. Make your best to invalidate your assumptions. It’s the job of the product manager to ensure that everyone understands we are working with hypothesis only. And that these need to be tested. A quote from Aron Ezra, Offercraft, gives good context here:

“I still frequently run into skeptical executives who prefer trusting their gut to trusting ‘science’. Those are the folks who will be left behind”.

6. Radiate metrics in an open and regular manner. Make figures transparent to stakeholders. Show management snapshots of videos from user interviews. It’s a fine balance between showing relevant metrics to different audiences and being constantly transparent.
7. There’s a time to stop measuring and make a decision. Testing hypothesis that explain observations is important. You, fellow product manager, collect your own data. You ask another team to get data for you. You make a plan and set a maximum length to your tests. Latest in this stage you should understand when tests start to get too expensive and estimate their ROI (here’s a Google tool for that). When all pieces fall into place—trust, transparency and cost efficiency—you will know when to stop measuring. And which path to take.
8. It is a lot about trust. Once people—regardless of which place in the hierarchy they occupy—notice that you learned how to spur patterns and match data from different sources, they will feel more comfortable to trust your advice.

All these points boil down to something I’ve been craving to understand: How do we create an evidence-driven culture? It’s all about the people. Empowering evidence-driven teams means trust. In this context, a product manager is responsible to unstick the communication gap between business and tech teams. To involve business teams in product discovery. To collect their feedback and identify the difference between one-off feedback and trends.

Perhaps you got curious about this article. Perhaps I gave you some fancy ideas. You might need more detailed information on how to do this or that. I might have something to help you here. Maybe all you need is pragmatic tips, not this high level stuff. Examples of frameworks and models. Don’t be a stranger, just reach out.

—

QUESTIONS YOU MIGHT (OR NOT) ASK YOURSELF

What should I read after this article to understand more about metrics and experiments?

This book: “Statistics Done Wrong: The Woefully Complete Guide” by Alex Reinhart.

Tips from Laura Agudelo follow: This article Google’s heart framework for measuring UX  and this whitepaper: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications

How can I write an Analytics story that I can include in my user story? A working method that I experienced during the workshop: “To know whether (Feature X) is working, we want to track (Event Y). We’ll want to be able to compare by (user type, date, …) so that we understand (Explanation).”

What is the definition of statistical significance in the product management world? A result big enough to detect against the “background noise” of random variation in your groups.

What’s the answer to your short quiz? Musicians came up with a creative answer for which One Metric that Matters to track: Standing ovation. That turned out to be a very effective and efficient way to measure how big they could impact even more the business and potentially change their behaviour based on this feedback.

*Special thanks to our TrustYou designers Radu Luchian & Laura Agudelo for reviewing this article. Last but not least, big thanks to Andjelina Schmidt for helping with the introduction by finding the right words!