State Farm presentation at the Chief Analytics Officer Forum East Coast USA

Contains CONFIDENTIAL information which may not be disclosed without

express written authorization

Do You Know or Do You Think You Know?

Incorporating Business Experiments Into

Strategy Development

Andy Pulkstenis

Director, Advanced Analytics

[email protected]


express written authorization 1

• Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance


What are we doing today?



What is a “designed experiment?”

Designed testing is a way of treating the business environment like a controlled experimental setting, randomly assigning strategies to customers as in a clinical drug study or lab

Ben

efit

s • Clear, accurate insights about causation • Ultra-clean data for model development = more accurate prediction • Enables strategy optimization at the customer level



Innovators, on Testing

“By thorough experimentation and trial-and-error in the market, [successful companies] iterate towards a strategy and a business model that really is viable.”

- Harvard Business School Professor Clayton Christensen, a recognized authority on “disruptive innovation”



- Rich Fairbank, Chairman & CEO Capital One

“What we’ve done is to create an innovation machine. This is not just a credit card strategy, testing is a revolution that can be applied to many businesses.”




- Jeff Bezos, Amazon Founder

“If you double the number of experiments you do per year, you’re going to double your inventiveness.”


Fun fact: Amazon once fired a team for implementing strategic change without proper statistical testing first!



How do businesses use testing?

Strategy Optimization

Targeted Messaging

Pricing

Real-time Website Design/Optimization

Customer-level Decisioning

Retention & Save Strategies

Product & Strategy Development

Operations Improvement



Specific business examples

Sources: “How to Design Smart Business Experiments” by Tom Davenport, HBR 2/2009 “The Discipline of Business Experimentation” by Stefan Thomke and Jim Manzi, HBR 12/2014

Do visible lobster tanks increase lobster sales?

Are higher bids driven by CC payment option?

Will opening stores an hour later lead to a drop in sales?

Nearly all aspects of business.

http://www.google.com/imgres?imgurl=http://s.filife.com/be/company/images/CapitalOneLogo_2.png&imgrefurl=http://www.filife.com/reviews/capital-one-health-insurance&h=220&w=220&sz=9&tbnid=X9m_HOdMeCd-rM:&tbnh=107&tbnw=107&prev=/images?q=capital+one+logo+-bank&hl=en&usg=__n9FMka1p-W6RMC9V7K6zMOtNXLo=&ei=dc-XS4jxCI2wtgeDi_nkAQ&sa=X&oi=image_result&resnum=3&ct=image&ved=0CAoQ9QEwAg



Typical Testing Evolution

None A/B “one factor at a time”

(OFAT) Multivariate

(MVT) MVT w/

covariate analysis

Pick “one winner for all” Increased business insight while progressing left to right

Enables customer or segment-level strategy/message

optimization

Companies typically flow through an evolution when embarking on testing agendas

Continuous testing

A pervading culture of evolving, continuous, and cyclical testing provides

strategic foundation

Many stop at A/B or OFAT through lack of knowledge or lack of skillset, missing out on benefits that increase with each successive milestone.



Which banner led to the most visits to the

“Agent Search” page?

Version B:

Version A: (Click leads to “Drive

Safe & Save” page)

(Click leads to “Agent

Search” page)



Answer: Version B led more people to the

Agent Search page



Which statefarm.com banner led to the most

visits to the “Drive Safe & Save” page?

Version B:

Version A: (Click leads to “Drive

Safe & Save” page)

(Click leads to “Agent

Search” page)



“Find an Agent” drove more people to the “Drive Safe &

Save” page than the “Drive Safe & Save” experience did!



Version A: Version B:

Which mailer generated the highest lift in

Life insurance sales?



Claims Contact Center

Claims Contact Center – “Out of Scope” Calls

Can proactive sharing of contact info reduce misdirected or “out of scope” Claims

calls?



Multivariate Discount Double Check Test

• Explored 7 different page features as tested ‘factors’ (text, graphics, layout,

images, colors, etc.)

• Results in 384 distinct web page variations (with a subset of 200+ valid

combinations)

• Only needed to physically test 58 combinations

• Improved quote starts by 50% and quote completes by 10%



My roadmap for launching testing

Be Persistent It will take legwork and determination to find the partner in step two

Be Ambitious Aim for moderate or greater business impact

Be Careful Carefully execute the approach with business partners

Be Ready Build the “success story” deck before you need it

Be A Teacher There is a significant initial education component

Be Opportunistic Follow the path of least resistance



Best Practices: Set Up for Success

Assemble the

right structure Identify key roles at the outset (business, stats/DOE, execution, structural or creative design)

Assemble the

right players

Don’t just fill seats with “good people.” If you need an SME in a certain area, then staff that role with an SME

Assemble the

right questions

A surprising amount of time in early DOE engagements is spent helping the business partner define the problem as a quantifiable hypothesis

Assemble the

right scope

Too ambitious and you fail spectacularly Not ambitious enough and you really don’t move the needle

Assemble the

right controls

Make sure to monitor execution and data collection from Day 1 and throughout the process



Common Arguments

Regulation

Initial resistance to statistical testing typically falls into

three categories:

Complication Organizational

Tradition



Regulation

Our industry is too heavily regulated

for this

Didn’t stop Finance, Pharma, and others

from successful transition to DOE

Even if allowed, the regulators would

never understand it

It’s not “fair” to randomly assign

strategies to customers in a test

Successful testing industries proactively

educate regulators on key concepts

• Current strategy isn’t “fair” either

• Jonas Salk & polio

• “Beautiful experiment over which the

epidemiologist could become quite

ecstatic but would make the

humanitarian shudder”

Perceived Barrier Response



Complication

Too complicated Nonsense – no more complicated than

predictive modeling, just new to your firm

Ignoring complex relationships doesn’t

mean they aren’t there

Start small, manageable

• Design according to cost constraints

• Much cheaper than hoping we guessed

correctly (think of it as “buying data”)

This can be handled by applying

“exclusions” to the design

Complexity of analysis

Too difficult to execute

Some combinations don’t make

business sense

Too expensive




Organizational Tradition

“We tried it once and we saw no

improvement” (i.e. learned nothing)

• You don’t hit a home run every time

• You learned that those strategies were

equivalent

• Often gain fresh, real insights that defy

historical logic or perceived wisdom

• “The greatest obstacle to discovery is not

ignorance - it is the illusion of knowledge.”

- Daniel Boorstin

“We intuitively know our business”


“Our external consulting firms tell us

all we need to know about course

correcting”

Designed testing consistently outperforms

consultant-provided solutions

“I have not failed 10,000 times. I have succeeded in proving that those 10,000 ways will not work. When I have eliminated the ways that will not work, I will find the way that will work.” - Thomas Edison

WD-1 WD-39 didn’t work!

Today WD-40 can be found in 4/5 US homes



Questions/Comments

[email protected]



Engagement Model

Business Partner

Statistical Design

Implementation/Execution

Structural and Creative Design Test

ing

Team

Ensure alignment with broader strategic goals

Strategic Approval

“Testing Council” of SME’s ensures alignment with

statistical best practices and production constraints

Structural Approval

• Statistical DOE • Production Environment Document key hypotheses,

findings, participants, and actions taken

Test Results Repository

Execute/Monitor/Analyze/Take Action

Ensure alignment with broader strategic goals

Strategic Approval

Execute/Monitor/Analyze/Take Action “Testing Council” of SME’s

ensures alignment with statistical best practices and

production constraints

Structural Approval



What do I mean by “test-and-learn culture?”

• Designed experiments inform (and determine) strategy development

• Understanding of when to settle for simple “A/B” testing and when to use more sophisticated multifactor tests

• Internal standards/best practices for test design • Formal internal governance

A true testing culture demonstrates the following elements:

State Farm presentation at the Chief Analytics Officer Forum East Coast USA

Data & Analytics

Transcript of State Farm presentation at the Chief Analytics Officer Forum East Coast USA