Can Data be Dangerous for your Startup

In our continuing quest to uncover the role of data and analytics in building a company, we turned to Patrick Vlaskovits, co-author of The Entrepreneur’s Guide to Customer Development: A Cheat Sheet to The Four Steps to the Epiphany and a regular blogger about Customer Development at Vlaskovits.com.

So, just to get started, what do you think in general terms about the role data plays in customer development?

Patrick: I think data is paramount - but I think we should understand about what we mean when we use the word “data”. Many people, especially those handicapped with a graduate level education, like myself, think that data is only interesting and can only be acted upon if it is “statistically significant”. In the context of early stage Customer Development, I believe this is well-intentioned, but ultimately, misguided.

We should be informed by data, but we shouldn’t let it walk us around on a leash. Evidence comes in a diversity of forms. It can be anecdotal, it can be in aggregate or it can be a trend line. If you take an open-minded approach to the types of evidence you’ll accept, and adjust for their biases/problems/problems accordingly, you’ll likely fare better in the chaos of startup-land than just simply jettisoning what you feel is low-quality data.

I should add, this is what my friend, psychologist and blogger, Seth Roberts calls “evidence snobbery”.

Seth defines evidence snobs as:

“An evidence snob is not someone who “want[s] evidence that something works.” An evidence snob is someone who disregards evidence — evidence that doesn’t reach a sufficient level of quality.” People often ask me, how many customer interviews must they do? They stress a need to fully remove all bias from the qualitative approach and have their results somehow become statistically significant. But if you go about the process of discovering your customer in a mechanical way, IMO you lose quite a lot.

In fact, there’s a good deal of art in the CustDev process and empathy for your customer plays a tremendous role. What I have been seeing is that the best entrepreneurs aren’t the ones who try to over-formalize Customer Development or make it a step-by-step formula. Instead, the best entrepreneurs are often very empathetic and have an intuitive and emotional understanding of the customer’s pain or his joy at a solution. They can read body-language in an in-person interview or sense verbal cues and nuance over the phone.

In short, when you are speaking to people, in one sense, you are flipping over stones in a stream and looking to find gold. Because there isn’t yet enough quantity, you rely on quality. And in my opinion there are too many evidence/data snobs who are excited about the possibilities of Customer Development, but don’t understand the qualitative aspect of this approach.

Do you add any sort of structure to these interviews?

Patrick: I generally don’t like doing a mechanical question, response, question, response type pattern. I have two or three themes that I play with and let the person do as much talking as possible. But there’s a natural rigor in the customer discovery process itself. It’s the rigor of a reporter hunting down a story or a hunting dog sniffing for a pheasant. When people get really excited and lean over the table, that’s interesting. I can usually extract a pain point out of that. And then I verify that pain point to an excruciating degree with subsequent interviews.

How many times must you verify a new pain point in interviews before you call it reliable?

I’m not comfortable giving you a number for the sweet spot of interviews because people take that literally. The truth is, it may be 1 time or 100 times before you find something. Or it could be that you are terrible at customer development and you never uncover a serious pain or someone lobs you some gold and you simply fail to recognize what it is. Or it could be that you’re excellent at uncovering actionable data really quickly. I wish there was a magic number. There’s just too much chaos to say.

So where do numbers come in customer discovery?

I personally like keeping discovery interviews qualitative. Eric Ries talks about getting away from the future hypothetical. So if people say I would pay for this and this and this, test them on it. Have them sign a purchase order or a letter of intent. It helps validate data and provides a more robust signal.

In one of your blog posts, you mention the philosophical overlap between Nassim Nicholas Taleb’s **The Black Swan and Steve Blank’s The Fours Steps to the Epiphany. Can you explain that a bit more?**

Both of those books amplify each other. In fact, I’d venture to say they are the same book, just written from different angles. I am of the opinion that when you are building a tech startup, you are trying to expose yourself to a positive Black Swan event. As a founder, you have a hunch about where a certain market is going or what impact a certain technology will have. Taleb argues that one can’t manufacture Black Swans, but I feel that savvy entrepreneurs can employ Customer Development to increase the probability of stumbling across one.

So this might be a bit philosophical, but are you saying that product market fit exists and is waiting to be discovered rather than manufactured? Or that even certain technologies are somehow fated to be developed by virtue of some complex form of evolution?

While this is obviously very debatable, yes, I believe it’s more of a process of uncovering rather than creating. And there’s no true formula to a process that is quite idiosyncratic for each company. In fact, there’s a battle being fought now in CustDev circles via email and in the blogosphere about this. There are those who are trying to over-formalize some of the processes of Customer Development and read it too literally – and moreover, it is not what Steve Blank had intentioned when he wrote The Four Steps to the Epiphany.

In my opinion, trying to make CustDev formulaic, well–that’s a road to perdition. Some of the techniques that may work now, might not work as well in two, five, ten years, so there is no point writing a hyper-specific rule book because no book can know exactly the context you are operating in and when you are operating there.

That’s probably why Steve Blank’s most popular talk is titled Why Accountants Don’t Run Startups.

Steve has always said that entrepreneurs are artists, not accountants and that getting out of the building isn’t to sum up the responses you get but to acquire a deep understanding of the customer.

The only time pure numerical methods work are in the constrained corner cases of existing markets. In new and re-segmented markets doing CustDev by the numbers would have gotten Henry Ford a faster horse or Steve Jobs a faster iMac (not the iPod or iPhone.)

So, IMO we should draw direction and inspiration from the meta-principles from Steve Blank and Eric Ries and apply them to the domain/context of our specific startup. IMO simply “getting out of the building” gives us the best way to get the cues to build something that people want.

My interpretation of the Customer Development meta-rules is as follows:

0) There are no rules. 1) Question and test your assumptions. 2) Get out of the Building. 3) Iterate.

However you do that is entirely up to you.

Editor’s note:

Figuring out the role of numbers in solving a complex human problem isn’t easy. Data is crucial for every company, but it’s hard to generalize about the type of metrics you should collect. You can focus in on important metrics by vertical. So a social gaming company will track a certain set of numbers, which will be quite different from the numbers an E-Commerce venture cares about. Data is crucial, but takes different forms for different companies.

Data, however, doesn’t necessarily dovetail neatly into the customer development methodology, which relies on extracting qualitative intelligence about your customers and market. You can certainly try to quantify the qualitative, but that misses the point. Customer Development represents a set of meta-principles that guide you through the turmoil that precedes product-market fit (and has a lot to say about what happens after). Or, to borrow a phrase from Juanita Marquez from Snow Crash, Customer Development helps you “condense fact from the vapor of nuance.” Data, like your shadow, should follow you everywhere. But try to let customer development be customer development. Or, more simply, trust in the qualitative.

Note: Special thanks to Steve Blank for reviewing this article!