IN A NUTSHELL
We fall in love with our ideas and lose objectivity. The lean startup process relies on empirical evidence to more quickly optimise the product to deliver both value and growth. You need to build a minimum viable product, then run constant incremental tests with actual customers to gain valid insights from their real user experience. Comparing the research findings against pre-set benchmarks allows you to more accurately make decisions to either proceed or pivot.
Blinded – The harsh reality is most startups fail. Startups don’t starve – they drown (in having too many ideas). People fall in love with their ideas and can’t see their flaws. Furthermore, they do not know their audiences really well and critically how they relate to their product. In many cases the fundamental product concept was flawed – people were not wowed enough to want it. Rarely is the original eureka moment of product innovation the final product. It takes a lot of optimising to take-off in a competitive market. Startups need to be disciplined to succeed. Thus at the heart of The Lean Startup (TLS) is data and metrics.
The author was the chief technical officer at IMVU (http://www.imvu.com). They broke all the rules: they built a minimum viable product that was full of bugs; they charged money for it, and they kept changing it. They talked to their customers but did not do what they said. This approach of continuous innovation is now known as The Lean Startup (TLS) . By 2011, IMVU had created more than 60m avatars with annual revenues of $50m.
Key principles of The Lean Startup – Lean manufacturing is a process originated at Toyota to aid manufacturing. There are a number of key elements:
1) Validated learning – The heart of TLS is the principle of running frequent experiments to test each element of your concept until it has been honed to market success. All testing is aimed to rapidly learn what works/not works so to maximise the effectiveness of an organisation’s scarce resources.
2) Build-Measure-Learn – TLS encourages a rapid movement to a prototype (via a minimum viable product). Benchmarks are developed, then the product is put into market. The result either makes you persevere (continuing with incremental improvements) or pivot (i.e. make seismic changes).
3) UX – User experience is key as the way the customer uses your product could be very different from how you think they will. No amount of cross transference of learnings from other markets or brands will be valid – only by going out and doing real research on your market with your real consumers experiencing your product will you learn what is necessary for success in your specific market.
At IMVU, feedback from users said they wanted their Avatars to move around. This was a heavy technological ask so as an interim they gave them a limited functionality to just click where they wanted their avatar to go and it would ‘teleport’ itself there. Surprisingly, users preferred this ‘magic’.
4) Innovation accounting – TLS focuses on metrics – to remove as much subjectivity as possible to lead to empirically based decision-making.
5) Small Batch sizes – as small batch sizes allows more rapid learning and adaptation.
When IMVU entered the instant messaging market they had a clear hypothesis about what led to success in the market. And so they developed a dummy product along the lines of their hypothesis. Even though it was very rough, they resisted the temptation to hold back and develop it further (for fear of tarnishing their image) and instead pushed it out into the market (as better to find out you are wrong early on than after masses of product development). Their hypothesis was flawed, which led them to completely rethink their positioning. Indeed it took them many iterations before they finally stumbled upon a positioning that they could exploit. It’s only by really seeing what people do rather than relying on our own flawed beliefs can we let go of our prejudices and blinkered view.
Doing the right thing -There are many things to think about when setting up a company (e.g. sales, budget setting, recruitment etc etc) and doing all these things will make you feel you have been effective in the day – but you could be spinning wheels – you could be doing the wrong things efficiently because you still do not know who your customers are, what role your product plays, and if they will part with their money. TLS instead focuses on ensuring you are building the right product – it starts with testing your hypothesis about your product/concept with actual customers – as quickly as possible.
The most important question any company should answer is ‘What activities add value and which are a form of waste?’ Thus you need to test every element on an almost continuous basis to know which ones make a difference (and are worth pursuing) and which ones to stop wasting precious resources on (as most companies never had enough resources). So instead of building upon assumption, TLS relies on fact based evidence for each step via the ‘Build-Measure-Learn’ feedback process. By constant experimentation, you can move ahead with greater certainty. And if you do need to make bigger shifts (known as pivots) you do it much earlier on in the process, minimising wastage.
Intuit flagship product, Turbotax undergoes 500 different ‘micro’ changes in a two and a half month season. They will make a change on say a Thursday, run it over the weekend, read the results on Monday, come to a conclusion on Tuesday, then rebuild and retest on Thursday. When you only do limited tests, you can only choose one or two variations, so there is a fight over whose ideas wins. when you can do 500 tests, then everyone’s ideas can get tested. This way, it’s not middle or upper management making the decisions – it’s the actual customers – leading to more accurate (and timely) decision-making. This speed has meant that they have developed $50m revenue products in just 12 months instead of 5.5 years previously. By relying on data it allows them to quickly kill things that do not add value and instead focus on refining the bits that do.
Validated learning – Customers often do not know what they want. They find it difficult to envisage something they have not had past experience of. So asking them in advance of experience is often flawed. However, when they gets their hands onto it, then they can provide much greater clarity of response. One of the guiding principles of TLS is validated learning – it’s not what customers say that matters it’s what they do that is the empirical test of a product. Thus it’s key to give them a workable model of your product as soon as is feasibly possible and then track what they do (or don’t do) with it.
Waste vs. value – A critical question TLS forces is to quickly distinguish between waste and value – i.e. what adds value and what doesn’t (and hence is wasteful of resources). If a feature on a piece of software is not being used, then stop developing it. Instead use your scarce resources to work on the parts they do add value. The sooner you know the value items, the less you will waste time doing the wrong things. Spend 12 months developing a more finished version and if it bombs, then you have wasted a year. Develop a version in 3 months and if it bombs you have just wasted 3 months (and now have another 3 rounds to get it right). Thus the key principle is to learn fast (/fail fast).
When IMVU were developing their instant messaging product they run hundreds of experiments week after week to discover what would inspire their customers to become advocates (‘net promoter score’).
Removing emotion and subjectivity – You need to be ruthlessly objective and not fall in love with your ideas. Set benchmarks and stick to them – you must face the hard truth of the data – if the tests show it falls below benchmarks, then you need the strength to let go and change.
Buying success (too early) – Stakeholders may see all these endless negative test results as failure (which will make them lose faith). There is then the temptation to ‘buy’ artificial success (e.g. via marketing) but this will not lead to sustainable growth if the product is flawed. Furthermore it squanders limited resources which could have been better used further developing and testing the product. So it’s best to resist the temptation to accelerate growth before you have absolute certainty that you have a really powerful idea that can maximise the marketing investment.
The two driving factors for success: Value & Growth – Two critical questions to answer are: 1) The Value hypothesis – i.e. does your product or service really deliver value to your customers? and 2) The Growth hypothesis – i.e. how far and how fast can the product/ service grow? People find it very difficult to judge these two hypotheses in conceptual form so getting real market data is the only way to get validated data.
In 2004, three college sophomore arrived in silicon valley with their fledging social media network. With only 150,000 registered users it made little revenue, yet that summer they raised $500,000 in venture capital, and a year later another $12.7m. What made the Venture capitalists buy into Facebook was not the revenue stream at that time but the fact the two hypotheses of value and growth had been clearly demonstrated. Value was proven by the amount of time people were spending on the site (each day, every day); and the growth hypothesis was validated by the fact that over 75% of all Harvard undergraduates used Facebook within just 1 month of launch.
Build-Measure-Learn – At the startup of a business, information is more important than money because it is the critical element to success. Thus it’s imperative to learn quickly to hone the product quicker. The intent is to minimise the time it takes to circle round this loop. The quicker you can do this, the faster you learn and the more you can learn (as you can loop round more times, testing more elements).
The B-M-L model has three phases: First a hypothesis is formed. Then the product is developed that allows that hypothesis to be tested. Metrics/benchmarks are set to help determine success criteria. Then the product is tested with real consumers. Their usage data is then analysed against the benchmarks and these learnings drive decision-making. There are two routes to pursue – either to Proceed (if the results exceed the benchmarks) or Pivot (if the results dramatically under perform expectations). By testing quicker it allows some of these bigger ‘pivot’ decisions to be made faster and when it is easier to do (than when you have a whole infrastructure set up to operate in a certain direction). Speed is a competitive advantage. Learn to learn faster than your competition.
The 5 Whys – To help isolate the real cause of a problem, use the 5 W’s technique (e.g. Why did the machine stop? -> Because X failed -> Why did x fail? -> Because Y happened ->Why did Y happen? etc. In most cases it often tracks back to a human problem. By doing this you save time in the long run (and is usually easier/quicker/cheaper to solve before the problem escalates). The reality is that often a breakdown happens at a number of different levels so the 5 W’s creates a more robust solution as it picks up the errors at all the levels. Sadly in many organisations there seems to be more of a ‘5 blames’ culture. It needs to instead be seen as a ‘collective failure’ rather than an individual (as everyone else should also have spotted the problem). It’s recommended that teams tolerate mistakes first time round (as most times it’s caused by flawed systems rather than bad people) but to learn from them and not let the same mistake happen twice.
Genchi Gembutsu – ‘Go see for yourself’ (From Toyota) – you cannot be really sure you understand the problem unless you go see it for yourself first hand – it is unreliable to rely on other people’s reports. As Steve Blank once said, “Get out of the building and start learning”.
When they launched the 2004 version of the Sienna minivan, Toyota’s chief engineer on the project took a 53,000 mile road trip across America, Canada and Mexico, talking to minivan drivers and their families. His big insight was that the kids rule the minivan and they are the one’s who need to be wowed. This led to an improved Sienna which had 60% higher sales than the previous model.
When Intuit’s founder wanted to test his hypothesis that people would want to use computers to keep track of their expenses and pay bills, he picked up two phone books, and randomly called people to ask how they currently managed their finances. This cheap, quick and easy research help validate his hypothesis enough to proceed onto the next stage. It was not in-depth, checking out lots of potential options and pricing structures, but a quick piece of research focused on one key research question (all too often research becomes over complicated).
Minimum Viable Product (MVP) – To move rapidly through each cycle needs a product and this is what usually steals the time. Instead TLS relies on MVP’s – Minimum viable products – i.e. the rawest, quickest (cheapest) version that delivers the core functionality that is being tested (all the other bells and whistles, and glossy packaging are not developed at this stage as it takes time and detracts from what is being tested). It’s not meant to be 100% accurate – it’s meant to be the start of learning not the end of it. Whilst such small scale testing has its weaknesses, this is compensated by real in-market insights gained with people actually using your product. The secret of an MVP is to keep it as simple as possible.
Nick Swinmurn, the founder of Zappos, the online shoe retailer (worth in excess of $1bn) wanted to test his hypothesis that people would be prepared to buy shoes online. Rather than building up a warehouse full of shoes, he went to a local shoe store, took photos of their shoes, and posted their pictures on-line in a simple website. When customers clicked to buy, he would simply go back to the shoe store, buy the shoes and ship them out – thus holding no stock and no risk. This ‘cheap’ piece of research validated his hypothesis and helped him learn his way to success. What was critical was he had real behaviour data (not hypothetical ‘If we set up a website selling shoes would you buy from us?’ type spurious data). It also quickly taught him things he never really thought too much about such as discounting and returns policy.
VLS wanted to see if there was a market for laundrettes in India. Rather than commissioning large scale research, they mounted an industrial size washing machine on the back of a pick up truck and parked it on a street corner in Bangalore – the experiment cost less than $8,000 and proved that people would pay to have their clothes cleaned. They parked it on different street corners and experimented with different elements to answer different questions – such as speed and extra services like ironing. VLS now have 14 locations in India.
Groupon is one of the fastest growing companies of all times. The very first deal was 20 people buying a two-for-one deal on a pizza. Their first minimum viable product was a WordPress blog site where they sold T shirts. They had no on-line ordering mechanism so people had to email their order in. Now they are operating in more than 375 cities worldwide.
Sometimes it’s not possible to make the product – in which case trying to bring it alive as much as possible is the next best thing. When Dropbox was first being developed, it used a video to explain how to use Dropbox. This helped drive up the number of people prepared to take part in the beta tests from 5,000 to 75,000.
Food On The Table creates weekly meal plans for families based on their choices and the best deals on local ingredients. To first test their assumptions, they signed-up just one family and personally developed the food plans for them by visiting all their local stores. Clearly this was very costly in time but gave them invaluable learnings on the good and bad points of their offering.
Aardvark (a search engine to answer the questions Google can’t) initially tested their concept using people not technology to mimic the search engine functionality (because they knew it was not until they had validated their value and growth hypotheses should they start investing in the technology).
Quality levels at different life stages – The value of quality is target audience specific i.e. early adopters appear to be very forgiving of beta MVP’s (they like being at the forefront). So you can get away with an MVP at the start but once validated, then the product needs to be optimised for quality to appeal to new (less forgiving) audiences. For many mainstream marketers, the idea of putting out a substandard product is counter-intuitive as they see it damaging their brand reputation.
Innovation Accounting – Many brands stumble along in the zone of the ‘living dead’ – never quite taking off. The trouble with most people is they are naturally optimistic and assume 1) their product is brilliant and 2) things will get better. Entrepreneurs need to park their natural enthusiasm and face the harsh facts that proper financial accounting will testify to.
Innovation Accounting has three steps: 1) Use an MVP to establish a data baseline of where your company is right now 2) You then fine tune the ‘engine’ to move up from the baseline 3) If you do, then ‘Persevere’. If not then ‘Pivot’.
A key question to always ask is ‘How do you know if you are making your product better?’ The answer must be empiric. Metrics need to be actionable, accessible and auditable (i.e. credible).
At IMVU, they had monthly meetings where they reported back on a range of metrics such as registration rates, download rates, trial rates, repeat usage rates, conversion to purchase rates, customer counts and revenue. Even though they had been busy refining and changing the product, the bottom line metrics showed no shift – i.e. all their effort was not generating any return. The harsh reality of flat numbers made them re-look at everything they were doing. This led them to pivoting the product. If the company had not set clear benchmarks they could easily have carried on down this track for another year – and eventually into oblivion.
Beware ‘vanity metrics’ – It’s also key to make sure you look at the right kind of metrics (some give you a nice warm feeling and others put money in the bank). Cohort analysis gives a better reflection than cumulative total (which fool you into a space of positivity as the numbers grow each day), but with cohort analysis you just look at the behaviour trends of the latest 100 customers – and if they are not converting at a higher rate, then your product is not getting better. So avoid ‘vanity metrics’ and instead focus on ‘actionable’ metrics.
Grockit’s founder Farbood Nivi knew student to student peer based learning was effective. He saw the opportunity to use ‘social learning’ to develop it globally. So he designed a minimum viable product (i.e. an online video). Having validated his value and growth hypotheses, he was able to raise venture capital and embarked on developing his product. The trouble was initially they were relying on vanity metrics (such as total number of customers). Soon he switched to cohort analysis and using A/B testing was able to make rapid, effective progress.
‘Persevere’ or ‘Pivot’ – It takes courage and discipline to let go on your original beliefs about your product and change (that’s why you need to rely on data to overcome our inbuilt resistance). But the longer you leave it, the greater the costs.
When the numbers exceed your benchmark, then your hypothesis has been supported and you can ‘persevere’, continuing to refine along the same lines. When the results are well below benchmark, you need to shift the product or strategy to continue to meet the set vision. The faster you can pivot and learn, the greater the chance of taking off (before you run out of money).
KaChing first started off as an online game for amateur investors where they played the market but in a fantasy league. Their real aim was to use it to identify future traders and for those less successful traders, to offer them trading by those who did perform well. This meant they built up a sophisticated scoring system for each player to evaluate their investment prowess. However their strategy did not work, so they pivoted, abandoning the gaming option (and throwing away months of work) and instead allowing people to invest with professional managers who had been assessed through their tool. Today Wealthfront has over $180m invested with more than 40 professional managers.
The pivot is about optimising the product to market fit. When this happens, you get business traction. There are 10 kinds of pivot:
1) Zoom-in pivot: Where focus-in on one specific feature of the product
2) Zoom-out pivot: Where the original focus on a single feature gets subsumed into a wider offering
3) Platform pivot: A shift from just one product to a range of offerings
4) Customer need pivot: Where you solve a different problem for the same customer
5) Customer segment pivot: A shift in who the customer focus is
6) Business Architecture pivot: A major shift on the strategy for financial growth (e.g. a shift from low volume/high margin to high volume/low margin)
7) Value capture pivot: How it moneterises itself
8) Engine of growth pivot: How it changes its engine of growth (e.g. from viral or sticky to paid growth)
9) Channel pivot: Moving to a new distribution channel
10) Technology pivot: Finding a new way of delivering the same benefit
Batch size – If you stuff a series of envelopes, seal them, and put a stamp on them, it makes intuitive sense to do each operation in turn – i.e. stuff them all, then seal them all, then stamp them all. But research has shown that it’s faster to instead complete each envelope one at a time. In business, it saves time by identifying problems early on. For example, if the envelope’s seal is faulty, then it’s better to find that out early on (otherwise you have to unpack all the envelopes). So it is in manufacturing. Toyota adopted small batch sizes to offer a competitive advantage (of variety.) This was made possible by reducing the changeover time from hours to minutes (SMED – Single minute exchange of die). Furthermore they adopted the Andon cord principle where anyone can stop the production line to resolve a problem. It might slow things down initially but stops recalls/refits later on.
Increasingly software updates are delivered one update at a time – almost on a continual basis (IMVU can make up to 50 mini changes a day). They then use matrix teams to work on one feature at a time. They then get single issue feedback that provides immediate and focused feedback.
For many industries they may assume these principles cannot apply to their business. But increasingly we are seeing software providing added functionality to hardware (from cars to fridges to sportswear), and increased customisation of products (e.g. BMW, Nike etc).
SGW Designworks were asked to build a complex field x-ray system to detect explosives. They delivered their first 3D model within 3 days. Their clients made a few amends and the final design with a new mock-up was agreed 5 days later. 40 production units were made just three and a half weeks after initiation of the project. SGW used the same process to deliver 8 other products in a 12 month period.
In the ‘school of one’ students are given a daily personalised ‘playlist’ of their learning tasks. This may be a mix of small group work, e-learning, one-on-one tutoring time or personal study time. Each activity has built-in feedback mechanisms to help set future activities.
The myth of large batch sizes – Large batch sizes feels intuitively right, and so you quickly get into large batch death spiral – fix one more bug…wait another week and we have that new feature…. The trouble is the longer you leave something the more perfect you feel it needs to be.
When Apple launched its long awaited iPhone 4, it had 1500 changes in one batch.
Adapt – With this speed of learning, an organisation needs to adapt fast enough to the changes required. It’s best to operate in smaller matrix teams, who are empowered to make fast and courageous decisions (through faster customer feedback).
Complex systems need simple rules (not complex ones) to navigate through – thus principles rather than detailed processes are better – hence Learn (/Fail) fast, MVP, Small batch sizes etc are all sound principles of a more efficient way of working.
This book is quite an influential book doing the rounds at present. It has some strong core concepts that can be applied in many areas of business.
That said, I question whether The Lean Startup principles can be applied to all businesses or launches (nothing in my experience is that perfect a system). I think he overemphasises the difference between TLS and normal business. In my experience every brand I have ever worked on does a lot of concept/product testing to hone its offering and to understand about its customer.
One of the problems of this model is the trouble in setting accurate benchmarks and this can then lead to flawed decision-making. Building from this the TLS model is very ‘metrics’ driven. We can also be misled by numbers as we can be misled by emotions. Numbers tend to take a singular focus when we live in a place of mass complexity. I therefore believe we need to take the numbers as a guide but then make a more rounded decision taking into account other factors that numbers can’t help map.