Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments
-
Upload
ton-wesseling -
Category
Data & Analytics
-
view
94 -
download
1
Transcript of Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments
![Page 1: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/1.jpg)
D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# E l i t e C a m p - @ To n W
When to run more experiments?
![Page 2: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/2.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
http://test ing.agency
![Page 3: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/3.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Today
the
ROAR model
![Page 4: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/4.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Methods used to improve conversion
![Page 5: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/5.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Methods companies plan to use
![Page 6: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/6.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
A/B-test ing is
HOT
![Page 7: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/7.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Everyone is pushing to test more
![Page 8: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/8.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Never sat isfied with conversion rates
![Page 9: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/9.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Start test ing or test more
![Page 10: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/10.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
NO Growth problems wi l l d issapear…
![Page 11: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/11.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
A/B-test ing is not a solut ion
It’s a methodology
![Page 12: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/12.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
I t ’s part of a:
Continuous Optimization
Program
![Page 13: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/13.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
And you need to know:
When
How What
![Page 14: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/14.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Growth curve – how i t was teached
![Page 15: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/15.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
the
ROAR model
![Page 16: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/16.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
![Page 17: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/17.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
![Page 18: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/18.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
![Page 19: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/19.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk!
![Page 20: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/20.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization!
![Page 21: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/21.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation!
![Page 22: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/22.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
![Page 23: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/23.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
![Page 24: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/24.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions per month!
1.000 conversions!per month!
![Page 25: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/25.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Stat ist ics
![Page 26: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/26.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Optimizat ion phases - ROAR
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions per month!
1.000 conversions!per month!
15% impact needed!
5% impact!needed!
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!
![Page 27: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/27.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Low number test ing?
Why can it hurt you?
![Page 28: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/28.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Warning: low significance levels
ü Big chance the winner is not a real winner
![Page 29: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/29.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Warning: low power levels
ü Big chance a real winner is not recognized
![Page 30: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/30.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner…
ü …with an average uplift of 10%
![Page 31: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/31.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Average: 75% significance & 40% power
![Page 32: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/32.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 10%+
![Page 33: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/33.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Yes, I ’ve got those 1000+ conversions
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 34: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/34.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
With 1000+ conversions a month
Your tes t capac i ty i s max .
20+ tests a year
wi th a s ta t i s t i ca l power o f 80% on pred ic ted up l i f t s o f 15%
![Page 35: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/35.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests with measureable impact
On average
1 out of 3
Simply truth: you’re just not always able to create a winner with enough impact
![Page 36: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/36.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
So in th is case, you’ve got impact
Every 7 or 8 weeks
![Page 37: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/37.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Slower? Energy wi l l run out! And your continous optimization program will die
![Page 38: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/38.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
What to test? Business cases!
Delivery –test with in stock. 4 weeks
or 2 days?!
![Page 39: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/39.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
What to test? Business cases!
Service charge, do it or not? Include in
price or not?!
![Page 40: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/40.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
What to test? Big design changes
Yes this can be done with an A/B-test tool, you only need A/B-test coders!
![Page 41: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/41.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
ROAR – Opt imizat ion – moving up
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 42: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/42.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Final ly you can test detai ls…
Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!
![Page 43: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/43.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
You should grow to
Customer Buying Reasons
![Page 44: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/44.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Reasons behind the customer journey
![Page 45: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/45.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
A cont inuous opt imizat ion team
Analyst Psychologyst
Designer Developer
Team Lead
![Page 46: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/46.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
They keep on asking:
Why are these users here?
![Page 47: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/47.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
And they want to know
How they can
help these users’ brains
to fullfil their needs?
![Page 48: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/48.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Data & Psychology
Digital Data Persuasion Psychology
Analyze &
Experiment
![Page 49: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/49.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
A lack of resources
![Page 50: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/50.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Test ing takes resources
Use it to LEARN
![Page 51: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/51.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find
![Page 52: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/52.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Know the company
Mission, Vision, Stategy & Goals
![Page 53: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/53.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Know your user ( journeys)
![Page 54: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/54.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Based on: asking – why are you here?
![Page 55: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/55.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find Create
And yes, your team needs an economic / consumer Psychologyst and a UX designer!
![Page 56: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/56.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: A var iat ion
![Page 57: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/57.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: B var iat ion
Need help with coding A/B-test variations like this one? – http://testing.agency!
![Page 58: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/58.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: B var iat ion
![Page 59: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/59.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: A var iat ion
![Page 60: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/60.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: B var iat ion
Need help with coding A/B-test variations like this one? – http://testing.agency!
![Page 61: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/61.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: A & B var iat ion
![Page 62: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/62.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: A var iat ion
![Page 63: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/63.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: B var iat ion
![Page 64: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/64.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: A var iat ion
![Page 65: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/65.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Tests that can be done: B var iat ion
![Page 66: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/66.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
FA C T & A C T
&
Tell Conclude Analyze
Test Create Analyze Find
Analyze
Your analyst will analyze the results!
![Page 67: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/67.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
We measure in Analyt ics software
![Page 68: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/68.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Calcu late resu l ts : ABTestGuide.com/calc
![Page 69: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/69.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Not s ignificant
ü It’s not a winner, stop
ü But, it does not mean it’s a losing one!
ü Average: 33% of your tests will have winners If you are optimizing dialogues!
![Page 70: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/70.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Choose what to analyze up front
ü You just go and dig until you find something
(you will always find something that seems true BUT is not!)
![Page 71: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/71.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Don’t stop the test once i t ’s s ignificant!
![Page 72: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/72.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
You just let i t run unt i l i t ’s s ignificant
![Page 73: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/73.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
You just let i t run unt i l i t ’s s ignificant
Your test has a fixed lenght!
It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!
![Page 74: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/74.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Be aware: Your test takes too long
ü Cookie deletion (you will lose 10% in two weeks)
They will re-enter the and pollute your samples!
![Page 75: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/75.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Test length: ABTestGuide.com/calc
![Page 76: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/76.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
T ip: Don’t test broken stuff
ü Always check browser compatibility!
ü Don’t break dynamic stuff!
![Page 77: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/77.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
T ip: Slow down the control down
ü Add the same code to the control
![Page 78: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/78.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
T ip: leave out s o m e v is i tors
ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)
![Page 79: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/79.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
T ip: Only test on fresh people
ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)
![Page 80: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/80.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
T ip: give them t ime to buy!
ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)
![Page 81: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/81.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
+ Optimization! + Automation!
Back to the ROAR model
Time span!
Con
vers
ions
per
mon
th!
Risk! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 82: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/82.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
ROAR – Opt imizat ion – moving up
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 83: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/83.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Test capacity
maximum number of
experiments per year that your site or app
can handle
![Page 84: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/84.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Regret
amount of money you are losing per year by
not using your full test capacity
![Page 85: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/85.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Calculate your test capic i ty
http://ondi.me/size!
![Page 86: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/86.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
ROAR – Automation
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 87: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/87.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Automation
![Page 88: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/88.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
You tested which locat ions are cr i t ical
![Page 89: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/89.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
You know which dia logues are cr i t ical
![Page 90: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/90.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Automation
Contextual bandits
If you are ready – give me a call!!
![Page 91: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/91.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?!
![Page 92: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/92.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
+ Optimization! + Automation!
ROAR – Automation
Time span!
Con
vers
ions
per
mon
th!
Risk! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 93: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/93.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
ROAR – You don’t stop opt imiz ing
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 94: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/94.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
ROAR – Maybe you’ l l never automate
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 95: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/95.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
ROAR – Re-think
Time span!
Con
vers
ions
per
mon
th!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!per month!
10.000 conversions per month!
![Page 96: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/96.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
The ul t imate Re-think resource
![Page 97: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/97.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
CRO br ings user knowledge
![Page 98: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/98.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
You wi l l make big impact on Discover
![Page 99: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/99.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Customer knowledge center
Making money f rom convers ion improvements
Creat ing more and more cus tomer ins igh ts Which can be monet ized
on even a la rger sca le
![Page 100: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/100.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Now, that’s
HOT
![Page 101: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/101.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
Done test ing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!
![Page 102: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/102.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
A/B-test ing
Don’t start to soon
![Page 103: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/103.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
A/B-test ing
Test the right way
![Page 104: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/104.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
A/B-test ing
Minimize regret
Run more experiments, maximize your capacity!
![Page 105: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/105.jpg)
Ton Wesseling | @ TonW Run more experiments #TonW
http://test ing.agency
![Page 106: Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments](https://reader034.fdocuments.in/reader034/viewer/2022042701/55ca2f20bb61ebe84d8b4689/html5/thumbnails/106.jpg)
D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# E l i t e C a m p - @ To n W
When to run more experiments?