H O W , W H E N , E X A M P L E S & P I T T F A L L S
L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
A/B-testing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Should I go for City?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Should I go for United?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
A/B-testing != CRO
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
CRO != user knowledge
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
A/B-testing
==
user knowledge
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
HOW & WHEN
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
EXAMPLES
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
PITTFALLS
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Ton Wessel ing | @ TonW
More testing
Quality Assurance
Lower pricing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Online Experiments – A/B-test ing
Winner!
Variation B
Variation A
Default
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Hyperl ink junky
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Digging through log f i les
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Trying to improve
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Over and over again
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
What most start -ups do
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
That is:
0 / 1 Testing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Why can this lead to fai lures?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
0/1 Testing: weekly change?
26 out of 44 weeks: over 5% change!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Testing should be l ike:
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Scienti f ic approach
A / B Testing
Live Online Research
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon – 2004!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth l ike AmazonG
erm
an
Ecom
me
rce
mark
et,
thank
you
André
Mory
s
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Experiments per year
1000+(but it did not start like that)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth curve
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Phase 1: test solut ions
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Fake adwords test ing
Do they
clickon your idea?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Fake facebook test ing
Do they
likeyour idea?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Fake landing page test ing
Do they
subscribeto your idea?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Make sure i t ’s not
0 / 1 Testing
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Once you bridged the gap
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
The growth phase!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Duplicate?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t ’s the web, i t ’s scalable!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
You need to optimize!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Over and over again
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t takes many experiments to win
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Maximize your test capacity
Time
Conversion%
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Speed up – more people
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth l ike AmazonG
erm
an
Ecom
me
rce
mark
et,
thank
you
André
Mory
s
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Real growth
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t takes heavy experimenting
250+ in a year
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Not every experiment is successful
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t takes a ful l stadium a week for 250+
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Or roughly
1000+conversion points
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Too slow? Energy wil l run out!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
And you wil l never get this
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
You need the real fast paced culture
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
How does Bi l l win?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Emotion vs Ratio
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Why is your user here?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
How can you
help your user ’s brain
to fullfi l it ’s needs?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t ’s not about A/B-test ing
Growth is about learning
how to serve brains!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Testing costs t ime
Use it to LEARN
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
FA C T & A C T – 7 s t e p s t o l e a r n
&
TellConcludeAnalyze
TestCreateAnalyzeFind
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
N o w : f o c u s o n C TA
Analyze
Test
Create
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Create = Data & Psychology
Digital Data Persuasion Psychology
Analyze
&
Experiment
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Wheel-of-Persuasion.com
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Persuasion on forms
+25% sales
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Same technique
Significant more
submits, but..
Less registrations
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A small sentence
+18%
sales
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
But also
+10% applications
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Motivat ion
Winner
(sales)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Win your own form A/B-test
http://testing.agency/manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
And social proof?
Many went before you
In november 1.675
people applied for
a loan at MoneYou.
Winner
(sales)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Chunk your USP’s
+14%
sales
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
With learning: Phase 3 becomes easier
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Testing becomes user knowledge
ConversionOp miza on MarketResearch
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Pitt fal ls
How NOT to test
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing can be
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Don’t t rust your A/B-test analyt ics
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t keeps you away from this fai lure
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Enough traff ic & conversions?
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Low signif icance levels
Big chance the winner is not a real winner
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Low power levels
Big chance a real winner is not recognized
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Tested percentage is a spread!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Example test-setup
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Average power: 40%
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
So:
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Another error warning:
You just go and dig unitil you find something
(you will always find something that seems true BUT is
not!)
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
OK, results are in
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Not signif icant
It’s not a winner, stop
But, it’s does not mean it’s a loser!
Average: 30% of your tests will have winners
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
More?
http://ondi.me/contentverve
http://ondi.me/conversionxl
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Conclude
Do nothing?or
Re-test?or
Implement?or
Scale?and
Continue or change test roadmap
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Knowledge l ibrary: ABtestguide.com
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
I t ’s not about A/B-test ing
Growth is about learning
how to serve brains!
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
So you wil l get this curve
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Growth l ike AmazonG
erm
an
Ecom
me
rce
mark
et,
thank
you
André
Mory
s
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
HOW & WHEN
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
EXAMPLES
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
A/B-test ing
PITTFALLS
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Slides & more
http://testing.agency/manchester
T o n W e s s e l i n g | @ T o n W # l e a n c o n f M a n c h e s t e r 2 0 1 4
Be l ike Bil l , and win!
H O W , W H E N , E X A M P L E S & P I T T F A L L S
L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
ht tp : / /testing.agency/manchester
Top Related