Testing & Optimization:A Deeper Look

January 16, 2013Brendan Regan

Marketing Optimization SpecialistAnalytics Pros

BestPractices – Seattle, WA – 12.6.12

Late last year, I presented “Building a Successful Optimization Program.”

TO GET THE SLIDES:http://www.slideshare.net/analyticspros/best-practices-optimizationpresentationbrendanregan

TODAY, we’re focusing in for a deeper look…

People | Process | ToolsPeople | Process | Tools

Why is Optimization “process” worth a

webinar?I’m glad you asked.

“Process is boring.”

Process is the stuff of superheroes!

Process lets you…

1. Achieve valid, effective tests2. Take full advantage of business

opportunities

Without Process With Process

Unclear, misleading results Statistically valid results

Results that don’t add $ Results that add to the bottom line

High effort, low return Reasonable effort, huge return

Opinion-based testing Data-driven testing

“Inconclusive” results (a costly tie) Clear winners and losers

Take advantage of opportunities

1. Time to market– How fast can you get

improvements in place?2. Test the right stuff, in the right

order– Low Investment, high Return

Maximize Return Over Time

Today, You Will…

1. Learn the Scientific Method…for Marketers

2. Learn a proven Optimization process

3. Discover 4 hidden BestPractices nuggets

The Scientific Method…for Marketers

1. Define the problem2. Research/observe the problem3. Form a hypothesis4. Conduct experiment5. Analyze experiment results6. Form a conclusion7. Socialize results

Look familiar?

Optimization Process

Optimization Process

Science Marketing

1) Business Goals

Image credit: Square2Marketing.com

1) Business Goals• In science-speak, this is “define

the problem”• Resist temptation to:– skip it– assume

• Multiple goals OK– prioritize

1) Business Goals• Example:–Conversion Rate = primary business goal– Forward to a friend engagement =

secondary– Test for Conversion Rate–Analyze/monitor F2F engagement

Hidden BestPractice Nugget #1:

Test as “close to” revenue as you can. Testing further upstream can be misleading.

2) Understand Target Audience

Image credit: Closed Loop Marketing

• In science-speak, this is “research the problem”

• Generally qualitative data (“market research”)– personas– feedback forms– focus groups– social media listening– chat transcripts– call center monitoring– demographics/psychographics

2) Understand Target Audience

3) Web Data Analysis

• In science-speak, this is “research the problem”•Web Analytics– High traffic points– Decision/conversion points– Persuasion points– High abandonment points

• Survey Data

3) Data Analysis (Quantitative)

4) Heuristic Analysis

Image credit: Printed Circuit University

• In science-speak, this is “research the problem”• Are “best practices” (rules of thumb)

applied?• Are you at parity with competitors

and industry leaders?

4) Heuristic Analysis

5) Capture Challenges (roadblocks)

Image credit: YoJoe.com

No, not this kind of Roadblock.

5) Capture Challenges (roadblocks)

Image credit: ArtsJournal.com

• Challenges presented by your site/marketing to your prospects• Ex: “Landing page lacks ‘social

proof’ content.”

5) Capture Challenges (roadblocks)

6) Challenges » Hypotheses

• Every challenge/roadblock has potential solution(s)• Best guess, best solution =

hypothesis• Ex: “Adding ‘social proof’

content will increase conversion rate.”

6) Challenges » Hypotheses

Hidden BestPractice Nugget #2:• Hypothesis: “Adding ‘social proof’

content will increase conversion rate.”• Research Question: “Which version of

social proof content will perform best?”

7) Design Experiment

• Formal documentation• Capture:– Goal(s)– Problem/challenge– Quantitative/Qualitative proof– Hypothesis– Specifications

7) Design Experiment

Hidden BestPractice Nugget #3:“Confidence level” - the statistical probability that your results are not due to chance.

Own your confidence level. Most tools are 95%. Get buy-in on tool’s default or other confidence level.

8) Design Variations

• Sketch/wireframe• Copy deck• Designs/mockups• Approval process–Per experiment design–Marketing/Legal blessing

8) Design Variations

9) Build Variations

• You may have to write code • You may “throw away” code• Page load speed needs to be

consistent!

9) Build Variations

10) Implement Experiment

• Integrate w/ testing tool• Test before you test– X-browser– Acceptance testing?

• Launch & Monitor– 7 days minimum to mitigate “day-of-week”– Patience, please

10) Implement Experiment

11) Analyze Results

No, not this kind of analysis.

11) Analyze Results

• Google Analytics integrations– Optimizely, Visual Website Optimizer, etc.

11) Analyze Results

•What does the result mean about the target audience?• Are the learnings applicable

elsewhere?– Other sites? – Other campaigns? – Other channels?

11) Analyze Results

12) Report Findings

• If +, tie it to revenue– “estimated”

12) Report Findings

• If +, tie it to revenue– “estimated”

• If -, what did you learn?

12) Report Findings

• Install winning version– You don’t make $ until you install

the winner!– Production-ready code, SEO, etc.

• Follow-up test ideas• Add to Marketing knowledge

base

13) Next Steps

Hidden BestPractice Nugget #4:

Optimize your Optimization. Stay lean, learn as you go, smooth out the process using Six Sigma or other methodologies.

Image credit: htt

p://rahuleink.wordpress.com

/2010/04/23/what-is-six-sigm

a/

Key Takeaways

1. Optimization Process is your friend2. Optimization Process maps to science…– …but bows to Marketing

3. Optimize your own Optimization efforts– “…the best is the enemy of the good.”

Voltaire (1694–1778)

Questions?