Off the Charts: A Smart Bear Knows What to Measure

We called up Jason Cohen of A Smart Bear to tap his analytics experience. In addition to running WPEngine, Jason also likes to consult with around a dozen startups at a time through Capital Thought and serves as a mentor at the Capital Factory, a great incubator in Texas. We’ve heard he cooks up some of the smartest dashboards in town, so we’re definitely all ears…

PS. Chartio is still in private beta. If you’d like to register, Signup!

Be Ruthless about Your Data

To kick off the interview, I asked Jason about where the sweet spot is for a startup tracking data. How much is too much?

Jason: You have to be ruthless about cutting away unimportant data. The dashboard is something you’ll look at daily, so only critical information should be there, otherwise you’re wasting time. For each chart, ask this: If I woke up tomorrow and this data didn’t go in the direction I expected, would I drop everything and find out why, because it’s that important to my business? If no, it doesn’t go on the dashboard.

Therefore, it’s actually worth your time to look at it every day.

But dashboard-style reporting doesn’t tell you that much–it’s “what” but not the “why.” So a good reporting system would let you dig around in an ad hoc, spur of the moment way. A good, flexible reports system involves having a lot of choices about what data to pull from, what data to compare side by side, where to zoom in and where to zoom out. As a business evolves, who knows what kind of questions you’ll want to ask? So I think your summary report should be ruthless about important data. But on top of that you should have the ability to ask very specific questions when you want to. In sum, the more data available to you the better (as long as you don’t look at them all at once).

Different Key Strokes for Different Folks

I asked Jason about the ultimate purpose of collecting data in a startup context:

Jason: It depends on what your goal is as a startup. If the goal is to raise a $1M round, then you want to be able to prove your model works, for example, in user acquisition. So you might want to show proof that once a user takes some certain action with your product, they’re very likely to take additional and valuable steps. You then raise capital so you can inject enough users into the system to start the engine.

Alternatively, there are companies that aim to just be profitable, like my own, WPEngine. In a company like ours, we don’t care about those abstract metrics as much. I just want to get signups and make sure they stick around and don’t leave. Those are more financially oriented questions.

Willful Ignorance

I asked Jason about whether companies place enough value on data: Jason: A lot of people don’t want to know the truth about their data. I find that at both the startup level and at bigger companies, a lot of people don’t want to know the truth about what’s going on. They don’t want to discover they are wrong, or bad at certain things or are failing. Some of the time, it’s just simple job preservation.

In the case of a big company, for example, you might run a marketing experiment to test a new channel. But when that channel fails, the failure is on you; you’ve wasted money. At a smaller startup, however, and especially if that startup has a Lean mentality, the fact that you spent a small amount of money and time on an experiment is a great thing. You learned. So some people revert to tracking vanity metrics that don’t actually speak to the core of how successful their product is at attracting and keeping users (e.g. hits).

So the culture of data accountability changes as you grow?

Jason: Yes and no. I see a lot of small companies that don’t care either. They refuse to do customer development.

Sometimes, I ask them, “Who are your alpha customers?”

They say, “I don’t have any of those.”

Ok, I say, “Who have you talked to about this idea?”

They answer, “My friends.”

And then when I suggest they speak to their customers, they never actually do. A lot of people don’t want to be proven wrong.

The movie “The Social Network” is a great example of this. People think a startup is as simple as the movie depicts in which a stubborn visionary builds what he wants to, despite his users. So it doesn’t matter that people don’t want to buy your product or that your data is showing low adoption. In the end, the market can’t distinguish between being a visionary and being wrong until it’s too late (for the visionary at least).

So in answer to your question about company size and a data culture, I feel like people from every sized company don’t want to know the truth. To be fair, there are a lot of big companies that do measure a lot of things. And bloggers tend to cover those companies, creating a kind of survivor bias and leading people to believe there’s more interest in data than actually exists.

So Much Data, So Little Time

Finally, I asked him how do inputs to product development change or grow with your company:

Jason: As companies get older, you get more inputs to your product development. As you get more data, you can learn more. The same thing applies to the data itself. If your site gets millions of views, you can get really subtle with testing. If you’re getting 100 hits, there’s not much you can do with that.

Well, that’s where qualitative analysis and customer calls come in, right?

Jason: Yes. You never want to stop doing qualitative analysis with customers. But the proportion of time you spend on that could diminish over time. You should try to hunt down those folks behind the metrics until there are so many that it’s overwhelming to do. And as long as all these conversations are converging on a set of facts, then doing more interviews won’t add much value. Until those conversations are converging on something, keep plugging away.

Editor’s note: We wanted to ask Jason about bears, but sadly ran out of time.