Post on 17-Aug-2015
Prioritize Your Roadmap
Strategy Consultant: Ryan Lillis
August 12th, 2015
An Optimizely Online Workshop
1. Prioritization Components 2. Components in Action 3. Tools for Organization & Prioritization
AgendaKey Focus Areas
Activity: Write down what takes effort to set up a test.
Effort
Teams (or People)•Power user(s) •Designer(s) •Developer(s) •Graphic designer(s) •Photography team •SEO and/or SEM team •IT department •QA department •Executive sponsor
•HTML •CSS •Javascript •Backend changes (reordering pages in a checkout flow)
•Advanced audience targeting conditions (cookies, ad campaigns, custom tags etc.)
Technical
Impact
•Higher Conversion Rate • Purchases • Sign ups • Engagement
•More of Something • Revenue • Pageviews
•Less of Something • Internal cost reductions
Hard Impact - QuantifiablePoll: What is the primary way a test’s impact is measured?
Impact
Soft Impact - Meaningful•Internal Buy-in
• Executive sponsorship • Excitement (test ideas from more people)
• Resource allocation (dedicated developer)
• Access (places testing is allowed to happen)
•Implementation (winners actually implemented)
Hard Impact - Quantifiable•Higher Conversion Rate
• Purchases • Sign ups • Engagement
•More of Something • Revenue • Pageviews
•Less of Something • Internal cost reductions
Activity: Write down the primary ways your tests impacts will be measured?
Effort vs. Impact
Test Name Effort Impact
Test 1 Low High
Test 2 Low Medium
Test 3 Low Low
Test 4 Medium High
Test 5 Medium Medium
Test 6 Medium Low
Test 7 High High
Test 8 High Medium
Test 9 High Low
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
Priority
First Second Third
Test Name Effort Impact
Test 1 Low High
Test 2 Low Medium
Test 3 Low Low
Test 4 Medium High
Test 5 Medium Medium
Test 6 Medium Low
Test 7 High High
Test 8 High Medium
Test 9 High Low
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
Prioritized by EffortPrioritized by Impact
Effort vs. ImpactPriority
First Second Third
Use Numbers Instead of Words & Sum the Scores
Quantify Effort & Impact
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
This impact framework…
Use Numbers Instead of Words & Sum the Scores
Quantify Effort & Impact
…becomes this mixed framework.Priority Test Name Effort Impact Score
1 Test 1 3 3 6
2 Test 2 2 3 5
3 Test 4 3 2 5
4 Test 3 1 3 4
5 Test 5 2 2 4
6 Test 7 3 1 4
7 Test 6 1 2 3
8 Test 8 2 1 3
9 Test 9 1 1 2
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
This impact framework…
Create a Classic, Detailed Rubric
Rubric Method #1
Effort Component 3 2 1
HTML No HTML requiredHTML required but can be done within Optimizely’s visual editor
Detailed, custom HTML needs to be created
Javascript / jQuery No Javascript requiredSimple manipulation of jQuery in Optimizely’s “Edit Code” section
Detailed, custom Javascript or jQuery needs to be created
Graphic Design No graphic design required
Quick graphic design element needed
Detailed graphic design work needed
Executive Sponsor Executive sponsor not needed [intentionally blank] Executive sponsor
approval needed
Example Effort Rubric
=
Create a Classic, Detailed Rubric
Rubric Method #1
Impact Component 3 2 1
Conversion RateTest has a high likelihood of directly impacting our conversion rate.
Test may or may not directly impact our conversion rate but should impact a leading indicator metric.
This test will likely only impact a leading indicator metric.
ExcitementThis test will generate a lot of internal excitement for testing and build momentum.
This test will interest and satisfy some team members.
This test will only matter to a few people and get little visibility regardless of the outcome.
ImplementationIf this test has a winning variation it will definitely be implemented immediately.
If this test has a winning variation it will probably be implemented or it will take some time before it is.
Even if this test produces a winning variation, there is only a small chance it will be implemented due to buy-in or technical constraints.
Example Impact Rubric
=
Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?
#1 Impossible! It simply cannot
be done!
Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?
#1 Impossible! It simply cannot
be done!
#2 We’ll create an
exploratory committee to discover this
method. We should have an answer when Tesla has a flying car.
Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?
#1 Impossible! It simply cannot
be done!
#2 We’ll create an
exploratory committee to discover this
method. We should have an answer when Tesla has a flying car.
#3 Create a test idea submission form that utilizes questions asking what
resources, skill-sets etc. are required for a given idea. The
responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a
score is generated and the list can be sorted by scores.
Poll: How can test ideas be collected in a way that incorporates rubric scoring…automatically?
#1 Impossible! It simply cannot
be done!
#2 We’ll create an
exploratory committee to discover this
method. We should have an answer when Tesla has a flying car.
#3 Create a test idea submission form that utilizes questions asking what
resources, skill-sets etc. are required for a given idea. The
responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a
score is generated and the list can be sorted by scores.
So the answer is #3. Now what?
1. Create a test idea submission form
2. Include questions about required resources & skill-sets etc. required.
3. Modify the spreadsheet with built in formulas that add or subtract points based on the responses.
4. Sort the spreadsheet based on the final prioritization score.
+
Auto-Prioritization Recipe
= +
Scoring: Common Spreadsheet Formulas
Forms: Google Forms Responses: Google Sheets
Scoring: Common Spreadsheet Formulas
Quick Chat: Name one* tool you can use to help with prioritization?
Forms: Google Forms Responses: Google Sheets
* Google Forms uses Google Sheets automatically.
Idea Prioritization Tools
Google Sheets
Automatically integrates with Google Forms, always
shareable in the cloud
Microsoft Excel
Tried and true with advanced functionality for
graphing etc., can be cloud-based
Smartsheet
Set up for seamless project management and timelines,
cloud-based
Atlassian’s JIRA
Ticketing system incorporating elements of
form submission tool and a sortable spreadsheet
Poll: Why would a “workflow management” tool help with prioritization?
•Choice #1 - A test can’t be the first priority if it needs the expertise of someone who’s on vacation.
•Choice #2 - A test can’t be the last priority if it’s part of a time-sensitive marketing campaign.
•Choice #3 - Any test idea submitted by my boss is always the top priority.
•Choice #4 - Both #1 and #2 are correct but definitely not #3.
Workflow & Project Management Tools
Smartsheet
Set up for seamless project management and timelines,
cloud-based
Atlassian’s JIRA
Ticketing system incorporating elements of
form submission tool and a sortable spreadsheet, also
can utilize “cards” that track the status of each
step
Trello JIRA
One of the most popular testing workflow
management tools, incorporates “cards”, to-do lists and is one of the more
visual systems out there
1. Prioritization Components 2. Components in Action 3. Tools for Organization & Prioritization
Original Agenda
Additional ResourcesOptiverse contains content on a variety of topics in a
variety of formats including our online user community, knowledge base, and learning academy!
Community Knowledge Base Learning Academy