Consumer web companies (especially startups) should measure as much as they can; otherwise, they’re just prodding around in the dark.

The counter argument is that measuring everything discourages innovation and limits the usefulness of the data.

I think in one sense or another both arguments are correct.  On the one hand: “In God we trust, all others bring data.” And on the other: “paralysis by analysis.”

However, the optimal strategy is not simply a happy medium between accumulating and analyzing too much and too little data, but rather a balance between measuring and interpreting two different kinds of information: anecdotal vs. analytical.

Anecdotal

A few days ago I read How I almost ignored our single best source for customer feedback.  The author concludes that answering her startup’s support line helped her “really understand them [her business and it's customers], at an emotional level. It’s still early, but it looks like answering calls may not be a drag on the bottom line, but a boost.”

Aggregating large amounts of data about thousands of users and drawing broad conclusions about the “average user” is valuable, but nothing can replace an empathetic and thorough understanding of individual users.

Every user has a lifecycle and a story.  Interpreting and understanding this story, especially in the early days of the product/company, is invaluable.  You should know the name of your most active and passionate user…and they should know yours.  How did they find out about your site? What problem were they looking to solve and what itch did you scratch? Did they have any issues signing up? Have they called or emailed support? How did those conversations go? Were they ever frustrated? What did they love or want changed about the product? How many other people have they referred to the site? Do they plan on using it in the future? For what? How old are they? Where do they live?  Every point of contact between you (or your service) and your users is another chapter in their story.

Wufoo, an online form-builder, earned an early reputation for great customer service because they sent hand-written notes to their customers.  The hand-written notes did more for Wufoo than just showing their users how much they cared; it opened an intimate line of communication between the company and its users.

The guys from Airbnb, an online marketplace to rent out extra space (like a couch, bedroom, or house), visited the homes of almost every single one of their early users. Not only did they use their own product, they fostered genuine relationships with their most active users.

I’ve heard the guys from Airbnb talk about their company at a few YCombinator events, and they always read the same letter out loud. The letter is from a young woman in New York City . She was about to lose her apartment because she couldn’t afford the rent. But then she rented out her place on airbnb, and it literally changed her life. That’s a powerful anecdote, and it provides a deeper insight into airbnb than all the numbers and charts in the world.

Without anecdotal information:

• You can’t see the forest through the trees
• You may fail to pick up on leading indicators
• You may become detached from your actual users, which results in a whole bunch of problematic symptoms
• You become detached from your actual product or service
• You can miss big opportunities

Analytical

A pretty good definition of Analytics: Analytics involves studying data to research potential trends, to analyze the effects of certain decisions or events, or to evaluate the performance of a given tool or scenario. The goal of analytics is to improve the business by gaining knowledge which can be used to make improvements or changes.

Ultimately, websites use analytics to figure out two things:

1. How and why people come to your site. This helps you figure out, amongst other things, your sales and marketing ROI.

2. How users interact with your site (We use mixpanel).  This helps you figure out how to improve/optimize your product (i.e. how can you optimize your signup funnel, how do you increase engagement and retention, etc.)

You should start thinking about analytics sooner rather than later.  My company just started taking analytics seriously, and we’re definitely playing catch up.  We have been steadily increasing traffic and signups since launch, but we’re not exactly sure why. We’re figuring that out now, but it would have been good to know from the beginning. We are also starting to test the effects of different layouts and features in terms of signups, usage, and growth. Without analytics and data, interpreting those experiments would be a guessing game, and we’d have no idea whether we’re heading in the right direction.

However, focusing on analytics too much, too early can result in an obsession over data that’s not really (statistically) significant. If you get 8 hits a day to your homepage, for example, it probably doesn’t make sense to split test. Secondly, it make take some time to figure out what to measure. Do you care about number of signups? Transacting users? What are your key performance indicators? I think those are tough questions to answer before you have a real (and anecdotal) understanding of your product and your customers.

Without analytics, you will:

• leave a lot of potential on the table
• not be able to optimize your application
• not know where to focus your efforts/resources
• fail to gain a comprehensive picture of your application.

In the end, I think you want to balance anecdotes with analytics. I’m definitely not an authority on the subject (I’m still figuring it out for myself), but I wanted to share my current thoughts on the matter.  Let me know what you think!