When discussing Lean Startup methodologies with other entrepreneurs I’m frequently drawn back to an awesome presentation from Eric Ries’ Startup Lessons Learned Conference of 2010 by Kent Beck (Test Driven Development and Agile pioneer – http://www.justin.tv/startuplessonslearned/b/262656520 ) on how the Build-Measure-Learn loop should really be thought of as the Learn-Measure-Build loop. He draws inspiration from Test Driven Development (TDD) and it also just makes sense. You should always be starting out with what you want to learn and then think of how possibly you could measure it.

Step 1: What do you want to learn?  (Learn)

What’s the most pressing thing that you need to learn about your startup idea? In common Lean-lingo, of all the assumptions that make up your current guess at your business model, which is the riskiest? Which assumption would break your business if it were not true? This usually has to do with what customers want and will pay for.

Step 2: Is that reasonably testable? (Measure)

How can you test this efficiently? Ideally cheaply and quickly. Early on in my customer discovery process I prefer phone “Problem Interviews” with potential users. If I want to sell a kitchen knife that comes with a gnome that sharpens the knife when it needs it, my test can be “I believe that 75% of people I talk to will believe that dull knives are a problem they would pay to have solved”. You can find out the answer in a series of conversations. Hopefully you structure your interview freely enough to learn other things about their cooking habits and frustrations and imagine alternate problems that they would pay to solve if your guess doesn’t work out.

To be a test it must be able to clearly fail.

Fun activity: Look up the term “null hypothesis” and see how it applies to Lean Startup.

Other examples of tests:

• When I put $24.99 on my landing page I predict a get a 50% sign up rate.
• Users will forward our link to other users an average of 7 times.
• If I show a mockup of my proposed software, 50% of users will say they would pay $19.99 a month.

Sometimes you can’t quite measure what you want so you tweak what you learn to the to what can actually be tested.

Step 3: Build (but hopefully build nothing!)

Finally engage in the minimal activity that you can get the answers to your tests. Do the easiest one!

Best to Worst options of activity

1) Have a phone conversation
2) Show a PDF mockup and get answers
3) Fake the product or service just enough to get your answer (sign-up page with price, ad tests)
4) Cobble together the product from existing stuff (or services)
5) Create a “prototype” of the product
6) Create a decently functioning version of the product

To get the answer to some questions you may have to actually build it. But that is a last resort. If not a failure – its a very expensive option in terms of your startups most valuable capital – time. So match with the risk.

I had the privilege of mentoring at another fantastic Lean Startup Machine weekend in NYC this past weekend. There were examples of awesome validated learning and failing fast (Food Without Borders) and examples of where serendipity and  validation meet (Hangalong). Many thanks to Ryan MacCarrigan, Eugenia Koo and the whole LSM team of mentors and judges who made this event rock.

Why HangAlong won

They did a great job all around but I feel there were two areas in which they excelled. First, they competently executed on validated learning iterations. In other words, they nailed the basics. They had clear hypotheses and success criteria and had ample ample sample size (20 interviews) in each of three rounds of customer discovery interviews and experiments. This is harder than it sounds – many teams are not able to learn from their interview efforts because of bad question design, team-members not asking the same questions, solution bias etc.

Second, they also made a creative leap of faith guess about how people might want to use their product, they cleanly tested it and it turned out they were right and got excellent activation rate (80%) and ethusiasm from prospective customers. Everybody loves a happy validated epiphany. Can’t wait to see these guys move forward.

Why ThoughtBox won second place

ThoughtBox had a hunch that restaurants were missing out on customer feedback that could be given immediately after a meal. So got out of the building and spoke to a good number of people in the neighborhood about why they do or do not leave restaurant reviews. They found out that many people would love to help the restaurant with feedback but there wasn’t an easy way to do it anonymously and easily while they were still at the table.

But what they did next is why I feel they deserved a winning slot. They came up with a clever functional MVP – restaurant rating cards, whipped up at FedEx/Kinkos with a number that diners could text feedback to. They swarmed the neighborhood and found an owner of a restaurant who was on board to give this a try. And then my favorite: They also left cards without asking permission at another restaurant. I love this because anything that removes a perceived obstacle in conducting an experiment is simply a path to quicker learning. I especially like guerrilla  testing with another company’s user base (as long as its not causing too much harm).

The results showed few (or none?) filled out the cards – but it they had a reasonably large result set and they now have hunches on what their next experiments will be.

Why Childcare R’ Us got best Customer Development

They did an excellent job of getting out of the building and engaging customers (parents) by going where they hang out and conducting clear interviews with ample sample size. They also kept the ball going ‘in the building’ when they cool-headedly invalidated four successive pivots and now find themselves eager to explore a new area they’ve been lead to by customer interviews with parents.

Right on, looking forward to the next Lean Startup Machine. In my next post I’ll talk about recurring patterns that I’ve noticed with with teams that find themselves in a trouble spot during the weekend.

The Lean Startup movement says that to avoid the pain and waste associated with the historically high failure rate of startups, learning is the most valuable activity a startup can spend its precious time on. And the most valuable learning to be had is learning about your riskiest assumptions. Eric Ries makes the strong argument that in most cases (other than scientific breakthroughs or other problems that require major mojo secret sauce)  we don’t need to learn about whether something can be built (Technical Risk) – it can. But rather we need to learn whether we should build it (Market Risk). And for that we need to “get out of the building” (thank you Steve Blank) and talk to customers and learn about their problems and conduct brutally honest tests about specific assumptions.

Oh, but it hurts! Ego Risk

But there’s one problem. Humans and brutally honest tests – not so compatible. Why do so many startups that intellectually “know” that they should be testing their assumptions but still avoid setting up test that can fail? Or actually talking to customers. Because its painful to hear information that you don’t want to hear about something that is important to you. So besides Technical Risk and Market Risk I’d add a third type of risk to be aware of in startups – Ego Risk (thanks to a great conversation with Patrick Smith and Teague Hopkins at LSC DC last month!).

Ego Risk could be described as the anticipated pain of having to make a change to the carefully adorned vision of your startup that you’re spent many hours reveling in how it will change the world and transform your own life. This risk is great – and not least because its likely. So when we get feedback from customers or tests that contradicts the grand vision that we’re incubating we unconsciously avoid accepting that information   We tell ourselves we’re talking to the wrong market segment of customers (see my post “Why is it so hard for startups to focus on one market”), the feedback can’t be trusted because we’re missing just one feature or need a better logo (false negative), or its just an edge case. The effect of this is that we slow down the learning cycle and prevent our startup from making progress. Things we could have learned in two weeks take us six months. Things we could have learned in three months take us a year.

This pain avoidance is totally the norm. Its human. The Lean Startup methodology attempts to dig its screwdriver into one of the fundamental flawed gearboxes of the human entrepreneur and fix the fact that we’re most often driven by big dreams (we fall in love visions of the future) to start our own companies (reality-based endeavors).  How can we be dispassionate about running experiments that may drop the axe in a flash on our big dreams? Its anti-human.

Part of the problem is first time entrepreneurs tend to fuse “vision” (the greater problem domain to be solved) with “solution” (how we’re going to solve it). After all, its much more fun to day dream about a solution than a problem. So acknowledging that a solution may not be right feels like an attack on the vision. (I’ll be posting on this next week – Your Startups Solution is a Monster. Run!! )

How to beat the Ego Risk?

1. Be committed to a vision, not a solution

The best way to lower your Ego Risk is to set expectations up front with yourself ( and your team if you have one) that your commitment is to a vision rather than a specific solution. That vision may be as broad as “I want to start a company that is successful and transforms my life” or committed to a specific problem area “I want to make the babysitter finding process easier for young parents”. That way when the details of a specific solution may need to change, its less of a punch in the gut because your vision is still intct.

One of the coolest expressions of how a team can set their expectations is told in minute 31 of a 33 minute video of the founders of Aardvark doing a Q&A after presenting at  the Startup Lessons Learned conference in 2010 (http://www.justin.tv/startuplessonslearned/b/262666882). Max Ventilla says everybody on the team should know that  “…this is a sinking ship from day one and … we’re going to be totally uncompromising about when we’re going to abandon one ship and get on another”.

2. Accept that it will often feel like crap sometimes, that’s progress

Since humans can’t help hoping from time to time, when that hope gets crushed like a beetle under the reality microscope it will feel like crap for a bit. Think of it as progress. Usually feeling like crap is not fatal.

3. Regularly check-in, what feedback are we avoiding accepting? 

Check in regularly with yourself and your team to sense what areas of testing or customer feedback feel most risky from an emotional level. What answer would suck to hear the most. Then see if you’re putting off customer interviews, dragging your heels in that area to avoid that risk. Its probably the areas that are slowing down your learning.

Most startups should NOT be building a prototype before talking to customers. Likewise, I find in most cases extended mock-ups are just an easier way to waste time before you talk to customers. Yes Market Risk – No (Probably) Technical Risk. But the way we tend to avoid hearing what we don’t want (Ego Risk) and how it distorts our actions is also an important thing to keep your eye on during your constant steering adjustments to making sure your team is focusing on the most valuable activities.

Let me know any other thoughts folks have on hacking the human OS in this regard.

I attended the a Lean Startup Circle DC at the end of last year where Paul Singh of 500 Startups was the guest speaker. The topic was getting traction for your startup. While answering questions Paul gave a reasoning why a startup should focus on one type of user even if its technology solves a problem for a variety of users. He said something along the lines of (and I’m paraphrasing):

And of course you can then use the insights and language to inform how you describe how your solution solves a problem in their world.

This is not a new insight in product development. Geoffrey Moore talked about picking a beachhead market for technology products in Crossing the Chasm in1995. Offering specialized service for clients has been around since the beginning of management science.

But from my 15 years of experience in my own start-ups as well as working with other entrepreneurs it is stunning how often founders resist focusing on one type of customer (niche market, vertical, segment etc). I have personally experienced the stalled growth (and learning) due to a stubborn reluctance to specializing an offering for a single market.

Why is it so hard for many entrepreneurs and startups to pick a single type of user to begin with? Some thoughts:

Work has begun on a product before talking to any customers

I feel this is the number one reason most technology entrepreneurs struggle with focusing on a single customer group. I have done this at least twice with companies that I’ve founded. The founders have built a prototype of a ‘solution’ before they have dug deeply into how any one specific group of customers experiences the problem. This makes it harder to actually ‘focus’ on a specific customer group because you don’t really want to hear anything different than what you’ve guessed because you’re in the middle of executing on that guess.  Your biggest risk at this point is not Technical risk, yes Market risk, but most immediately: Ego risk (more about Ego Risk in my next post).

But, despite belief in the core startup’s vision, there is a nagging sense that the product probably isn’t ‘exactly right’ for a given group. So the founders lose trust in any single customer group as being a good test for their product and instead go after every customer group.

The knee-jerk urge to get busy with building a solution as soon as you have a vision is one of the fundamental human behavioral bugs that Lean Startup movement is trying to address. Its totally natural. After reveling in the awesomness of a new idea what do you want more than to actually see it, touch it feel it? When you’re foaming at the mouth to bring yourself into a new startup life what could feel more direct than actually building the thing?

But what it leads to is the awkward situation of having designed an awesome lure for fish in general but having to go around casting it again and again to discover which specific species of fish happens to like the lure you’ve already designed (probably none). In my experience building a new modified lure based on what you learn after a single specific species of fish will be much faster.

Specific is boring: Withdrawal pains from the epic vision

Intrinsic to the founding moment of a startup is the geyser of excitement about how a  product will solve a problem across a wide range of customers and life experiences. Each new use case is a spoke in the grand wheel of the core ‘vision’ of a new startup. “..and if we can make people levitate, the possibilities are endless!!”

Even if you’ve been fortunate enough to not have developed a product yet, it is still psychologically crushing to put aside the grand vision and work on an incarnation of the product for a specific user group  (a levitating stretcher that can move injured athletes from the field, or a levitating palette that can move a load to a construction site across a rocky terrain – so boring!!). It seems like we’re abandoning the vision if we begin work on only one part of it. Its sitting down when we’re pacing around excitedly. Its like someone telling you to walk in the hall when all you want to do is run. Its hard – and its normal.

But I don’t want to pick the wrong one – and ‘miss out’ 

We know our product can solve problems that are common across types of users – what if we pick the wrong one? We’d better try them all and ‘see which one bites’.

The problem with this is that its very hard to ‘try them all’. The very fact of presenting a ‘solution’ that is suitable to one of several markets actively undercuts its appeal to any single market. This is because the language these customers experience the problem in – the terminology, the ‘angle of approach’ of to problem – is different for each customer group.

I feel this mentality is supported by a founder actually knowing very little about any customer group. And therefore hedging their bets. Once you have a dozen conversations with people in a single customer group (market) you will begin to have confidence about whether or not the customer has a problem that your solution would address.

An invitation to awesomeness

So your killer vision is an invitation to awesomeness, not the main course. Go out and talk to people who live around the problem you’re trying to solve. Do enough listening and it’ll combine with the vision and you’ll find the sweet spot of a real business.