Running Great TestsUsing the Assumption Matrix

Brian Jessup, San Diego Program Lead

Why are we here?

Why are we here,talking about

running good tests and learning about our assumptions?

Best chance to succeed

KeepMySeat

Assumption:People on the go have an unmet need for a seat in shop.

Test:Built vaporware app. Sought greater than X% download and open rate.

Result:Not so much. Founded ripl.com instead. 500k business users.

Example courtesy of Chris d’Eon and Randy Gerson

Rebrandly

Assumption:Our target customer is a social media marketer.

Test:Identify and interview social media marketers.

Result:Retargeted their marketing to focus on ‘data-driven’ marketers. 9% growth wk/wk.

Example courtesy of Derric Haynie

Achieved9% growth wk/wk

Facebook

Assumption:Presented with hurdles to using our mobile app, our users will work up to X hours to overcome those hurdles.

Test:Purposely crash app for various periods of time.

Result:Users went to mobile web. No appreciable drop off in users over any period of delay.

How Can You Identify and Test Your Assumptions?

Framework

● Identify Assumptions ● Refine Assumptions

○ Get feedback if you can!● Decide On Most Risky/Impactful Assumption● Make Into Hypothesis With Measureable Test● Run Test● Repeat

Exercise

● Using Case Study○ Form groups of 3 people ○ Come up with 3 assumptions to test in each category○ Decide which is absolutely the most critical

assumption in group than as a whole○ Build Hypothesis

Categories of Assumptions● Customer

○ Who is the customer?○ Where are they?

● Problem○ Is this a top of mind problem?○ What is the value gap?○ How big is that market?

● Solution○ Is this the best way to solve the problem we’ve identified?

● Implementation○ Can this become self-sustainable?○ Can we build a community/audience quickly or cheaply enough?

Dr. John Gorrie & the Air Conditioner

It is 1842 and you are Dr. John Gorrie.

Dr. Gorrie, a physician from Florida, created what we think of as the modern air conditioner. Though mechanically powered (think steam-driven), it successfully creates ice that he uses to cool the rooms of his malaria patients.

He now has visions of central air conditioning, bypassing the ice, and is making plans to take over the world!

Exercise - Part 1

● Each person creates 3 assumptions to test in each category○ Customer○ Problem○ Solution○ Implementation

● Share with the group and revise

Exercise - Part 2

● Group decides on most critical assumption in each category○ Customer○ Problem○ Solution○ Implementation

Exercise - Part 3

● Each group shares most critical assumption in each category

● Everyone picks most critical assumption in each category○ Customer○ Problem○ Solution○ Implementation

Hypothesis Statement

● Should contain:○ What you believe○ How you will know you were right

Customer/Problem Hypothesis Statement

We believe customer type has a (need for / problem doing) action.

We will know this is true when we see (quantitative measurable outcome / qualitative observable outcome).

Solution Hypothesis Statement

We believe customer type will solve their problem by solution behavior.

We will know this is true when we see (quantitative measurable outcome / qualitative observable outcome).

Implementation Hypothesis Statement

We believe our company can provide solution by implementation method.

We will know this is true when we see (quantitative measurable outcome / qualitative observable outcome).

Test top assumption - be accountable - and repeat.

Case Study - What Really Happened?

Dr. John Gorrie

Died impoverished in 1855 blaming big ice (Frederic Tudor) for blocking his air conditioner from being successful.

Willis CarrierIn 1902, Willis Carrier strikes it rich by applying air conditioning to printing presses.

With the ability to control humidity (big ice can’t solve this!), the printers are more efficient in their operations, easily saving them the cost of the air conditioner.

Applications in consumer residential and auto don’t take off until 1950, but they have a massive effect - the sun belt, including Florida, see a huge influx of people who are now able to tolerate the heat.

Running Great TestsUsing the Assumption Matrix