An Inconvenient TruthAbout Optimization

FrosmoX16

Antti Sivula 25.10.2016

Optimization is easy! Or is it?

2

• AB testing is simple

• You have an idea for modifying your site so you go ahead

and create an AB test using a testing tool

• After a couple of days you already have a positive result

• This is easy, right?

Optimization is complicated

3

• Getting valid results requires correct testing

• Sample sizes and biases

• Seasonality and business cycles

• Invalid data

• False positives and false negatives

• A cornucopia of alternatives (each with their own pros and cons)

• AB testing vs. MVT

• Multi-armed bandits

• Machine learning

Getting real results is difficult

4

• Are you testing and measuring the right things?

• You have to understand and influence the entirety of your business

• Optimization can quickly become an overwhelming task

• Not enough authority

• Not enough data

• Not enough time

There is no silver bullet

Optimization requires time and effort

6

• Expectations regarding speed, simplicity and immediate impact of

optimization are often unrealistic

• The systems that we want to optimize are usually complex

• Understanding, selecting and implementing appropriate methods is difficult

• Data collection and analysis is paramount

Optimization works

What now?

8

• You should do optimization

• If you’re not optimizing, you are paying an opportunity cost

• Manage expectations and cultivate cooperation

• Lay the foundation for optimization

• Data collection and analysis

• Understand your customers

• Develop your organization

• Start with a single achievable step and work your way up

Thank You

www.frosmo.com

Antti SivulaR&D Specialist

antti.sivula@frosmo.com