MARK2038 Data Base Marketing Strategies II
Week 11
Instructor: Santo Ligotti Email: [email protected]
Testing, Metrics, and Post Analysis
This week
Testing, metrics, and post analysisIn-class assignment #5Structure/content of final test (July 18th, 2006)
Learning Objectives:
You just learned: why testing of DBM programs is important;4 steps you can take to test DBM programs; how to analyze the effectiveness of direct response campaigns including response rate, ROI and cost per response.
Campaign Management Process
1. Planning List Budget Offer/call to action Fulfillment Creative format Messages and copy Response device Testing process Response tracking Financial success
measures
CampaignPlanning
List Compilation
Implementation
Measurement
Campaign Management Process
2. List compilation Purchase response
lists/compiled lists Ensure any last-minute
field edits are complete Select list members Forward records to
agency/suppliers Flag records for inclusion
in CRM system
CampaignPlanning
List Compilation
Implementation
Measurement
Campaign Management Process
CampaignPlanning
List Compilation
Implementation
Measurement
3. Implementation Campaign is activated Customer inquiries and orders
are acted upon Information is received from
selected media channels
4. Measurement Monitor the results of the
campaign for effectiveness Input recommendations to
direct marketing planning
Time to MarketMarketing campaigns require an average of 2.5 months to implement.
Reducing Time to Market
The longer the campaign lead time,
The less likely the message will be relevant to its audience…
… and the less likely it will be “highly effective.”
A partnership between Marketing and Analytics will maximize campaign results
Involve the data analytics team at the beginning of the campaign to establish key business objectives, pre-analysis, targeting and key metrics/tracking
Continually integrate the data analytics team’s tracking and key insights into future campaigns to maximize ROI of all marketing initiatives
Getting the right mix, requires internal partnerships
With increasing pressure from shareholders/analysts to continually improve financial results, marketers need to able to illustrate that their campaigns are delivering strong results
In order to ensure marketing dollars are maximized, data analytics needs to become a key partner in the ongoing measurement & tracking of campaigns
A number of marketer’s are still struggling to demonstrate that their campaigns deliver quantifiable results
The Business Challenge
So how do we as marketers achieve this?
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Data Analytics is key to CRM Process
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Knowing Your Customer starts with Data Analytics
Analyze customer behaviour to determine key drivers of valueKnow recent key events and interaction with your company
Utilizing Data Analytics allows you to Identify Potential Customer Actions
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Continuously target and tailor offerings based on testing and learning
Marketing and Data Analytics allows you to Create Appropriate Message
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Explicitly manage the flow and sequence of marketing communications to each customer
Marketing Delivers the Message to the Customer
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Create a dynamic and consistent messaging and response capability at all customer touch (communication) points
Data Analytics allows you to Listen to the customers response
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Capture and remember relevant customer conversations
Data Analytics allows you to Track the Customer Responses and gain Insights
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Customer responds to the message Key Learning’s are integrated into future programs by marketing
Establishing a Test & Learn Partnership between marketing & data analytics will maximize results
ACTION
KNOW THECUSTOMER
LISTEN
DELIVERMESSAGE
CREATEAPPROPRIATE MESSAGE
IDENTIFYPOTENTIALCUSTOMERACTIONS
TEST ANDLEARN
Conduct sophisticated tests, share learning widely, and implement fast read and re-launch capability
The Concept of Testing
Why Test?Good economics: Use a sample to learn what works and what
doesn’t work before rolling to entire databaseContinuous Improvement Learn how to improve marketing programs to
ensure they’re the most effective
Testing Multiple Variables
Test all or some variablesWhy? Learning Loop: Generates constant feedback
on how to improve effectiveness of communications
Commonly tested variables: Lists Offers Creative execution Channel Content
Testing an Idea
Four Steps1 Plan Test
Define objectives Set up test and control groups
2 Execute Test3 Track Results4 Analyze Results
Response rate ROI Cost per response LTV
Example - Department Store
AssumptionsStore has a house credit card tied to customer database containing 400,000 men and women Store credit card allows capture of information about purchasesStore has new line of designer clothes for women, being promoted through print adsWould like to increase sales of new clothing lineDecide to test a direct mail program with a small group of women customers, before roll out to entire databaseOffer: If buy new suit by May 30, will receive a free piece of costume jewelry worth $20 by presenting this offer
Step 1: Plan Test
i) Define Marketing ObjectivesWhat are you trying to accomplish?Objectives should be measurable and time-bound.
Department Store Example: To increase sales to existing customers by 4% within 1
year. To achieve sales of new clothing line of $4.2 million. To increase LTV per customer from $80 to $125 over
next 12 months.
ii) Set up test and control groups
Test Group Control Group
Total Customers
Gets Offer Does NOT get offer
Step 1: Plan Test
Why use a Control Group?
• Allows you to measure the effect of the promotion versus not running it
• No offer or promotional piece sent to the control group
• Can be larger/smaller than test group
Set up test and control groupsQuery the database to determine how many women have credit cards in their name
Example - Department store200,000 women with department store credit card in their nameMust select 2 groups from this 200,000: Women who get the direct mail offer (Test) Women who do not get the DM offer (Control)
Step 1: Plan Test
Test & Control Groups: How large?
Cost considerations: make as small as possibleStatistical validity: make as large as possible
Rule of Thumb:Each group must be big enough so that you receive at least 500 responses from the promoted group
ExampleIf anticipate response rate of 2%Test group needs to be (500/2%) = 25,000
Example: Department StoreAnticipate response rate of 2.5%200K women in databaseTest group size = 500/.025 = 20,000
Test & Control Groups: How large?
Set up test and control groupsConstruct Test Group using ‘Nth’ method (per RFM)YOUR CONTROL WOULD BE THE SAME FOR THE ENTIRE MAILING UNIVERSE, REGARDLESS OF HOW MANY CELLS
Nth = Total customers in databaseTest Group Quantity
Example: Department store• Test group of 20K• Add another 20K for control group … total = 40K• Nth = 200,000/40,000 = 5• Select every 5th customer from master database
• That is, select customer record #5, #10, #15 ...
Step 1: Plan Test
Why use ‘Nth’ select?
Test and Control groups must be exact statistical replicas of the master databaseMust mirror the master database - will have the same percentage of people with similar characteristics: Same postal codeSame incomeSame # of childrenSame lifestyleSame purchase behaviour etc.
Step 2: Execute Test Execute Program among test group,
interacting normally with control group
Test Group Control Group
200k Women customers
No Mailed Offer
Step 3: Track Results
Assign a source codeA “source code” is assigned to each test variable to facilitate measurement and analysisA source code is a series of letters or numbers used to identify a particular offer Rule: different source code for each new variable
Example: Women who got offer: OFFERMAY03 Women who did not get offer:
NOOFFERMAY03
Step 4: Analyze Results
What is the key learning?
15%
10%
0%
2%
4%
6%
8%
10%
12%
14%
16%
Response Rate
Test Group Control Group
3,000responses
2,000responses
What is a response?
A response can be ...
Phoning a 1-800 number
Providing information (e.g. survey answers)
Entering a contest
Purchasing a product
Signing up for a service
Our example
Step 4: Analyze ResultsEvaluate success using a number of factors:
How did the program perform relative to objectives?Did the promotion come in on budget?
Metrics used to analyze performance:Response Rates Analysis (RR%)Cost per Response (CPR)Return on Investment (ROI)LTV
Response Rate Analysis
Calculate response rate for Test Group
Calculate response rate for control group
Step 2
Step 1
Calculate incremental lift between test and control groups
Step 3
First, calculate response rate for Test group
Department Store ExampleDirect mail offer: Get free piece of costume jewelry if buy suit by May 3020,000 mailed, 3,000 responded
Test RR% = Responder Quantity x 100=15% Test Quantity
Response Rate Analysis
Then calculate response rate for the Control group
Department Store Example20,000 in Control Group do not receive direct mail offer Still, 2,000 people respond to print advertising and buy a suit by May 30
Control RR% = Responder Quantity x 100 Control Quantity
Response Rate Analysis
Third, calculate % Lift between groups
% Lift = Test RR% – Control RR% x 100 Control RR%
EvaluationThe higher the lift, the betterPositive % Lift = Test performed better than ControlNegative % Lift = Control performed better than Test
Based on the Department Store example, what is the incremental lift percentage?
Response Rate Analysis
Cost per Response Analysis
Campaign Costs / Budget Include:Planning & Campaign Development Agency Costs (e.g. Fees, Creative
Development) List Development (e.g. data work)
Campaign Execution Printing, Laser/Lettershop, Postage
Response Costs The marketing cost associated with response
to a database marketing campaign BRC postage, data entry, offer fulfillment, call
centre
Cost per Response
Cost per response = Total cost of program
# responses
Department Store ExampleTotal program costs = $210,000 (includes campaign development, execution, response costs)Cost/response = $210,000/3,000 = $70
Evaluation: the lower the cost, the better
Return on Investment (ROI) Analysis
ROI = what you earn on a campaign relative to what you spent on a campaign
Evaluation: the higher, the better
Objective: To determine if you made money from your database marketing investment
Return on Investment Analysis
ROI = Revenue – Program Costs x 100 Program Costs
Department Store ExampleTotal program costs = $210,000 Sales revenue = $450/suit=(450*3000)3,000 responses to program
What is the ROI ?
Next step:Determine promotion effect on lifetime value Increased lifetime value, rather than immediate short-term payout, should be the real goal of database marketing Test effectiveness of alternative ways of
increasing LTV
Lifetime Value
Testing an Idea
Four Steps1 Plan Test
Define objectives Set up test and control groups
2 Execute Test3 Track Results4 Analyze Results
Response rate ROI Cost per response LTV
Metrics Example:CIBC: Direct Mail Creative
Execution Test
Example: CIBC Creative Test
• 3 different Direct Mail pieces created for launch of CIBC Adventura Gold Visa card
• Packages all the same except the outer envelope:» Cell A: High-end envelope & CIBC logo» Cell B: High-end envelope & Adventura logo» Cell C: High-end envelope & CIBC logo &
Aventura logo
Calculate the % lift, cost per response and ROI for each cellWhich envelope creative would you roll out to the entire database of customers?
Example: CIBC Creative Test
Example: CIBC Creative Test
Cell A Cell B Cell CCell Quantities Mail 50,000 50,000 50,000
Control 10,000 10,000 10,000
Responders Mail 5,000 2,500 7,500Control 500 200 1,000
Program Costs $100,000 $100,000 $100,000
Net Profit $500,000 $300,000 $500,000
Response Rate MailControlLift
Cost/Response
ROI
10,000
3,500
Example: CIBC Creative Test
CELL A CELL B CELL CCell Quantities Mail 50,000 50,000 50,000
Control 10,000 10,000 10,000
Responders 5,000 2,500 7,500 Control 500
Program Costs 100,000.00$ 100,000.00$ 100,000.00$
Response Rate Mail 10% 5% 15%ControlLift 100% 0% 200%
Cost/Response 20.00$ 40.00$ 13.33$
5%7%
40%
10,000
3,500
28.57
Example: CIBC Creative Test
ROI% = Revenue – Program Costs x 100Program Costs
Cell A Cell B Cell C% Lift vs. Control 100% 150% 50%
Net Profit $500,000 $300,000 $500,000
Program Costs $100,000 $100,000 $100,000
ROI
100% 40% 200%
Revenue
Example: CIBC Creative Test
ROI = Revenue – Program Costs x 100Program Costs
Cell A Cell B Cell C% Lift vs. Control 100% 150% 50%
Net Profit $500,000 $300,000 $500,000
Program Costs $100,000 $100,000 $100,000
ROI 400% 200% 400%
100% 40% 200%
Revenue
Based on the results, which envelope creative would you roll out to all customers?
Example: CIBC Creative Test
» Cell A: High-end Envelope + CIBC logo» Cell B: High-end envelope & Adventura logo» Cell C: High-end envelope & CIBC logo &
Adventura logo
In-class Exercise (Worth 10%)-Part 1
Read Luring ‘em back to school, Strategy Magazine, November 2003
Write 2 measurable, timebound objectives for the integrated marketing programs executed by CMC. What was the CMC strategy?What direct marketing tactics were used?How would you measure campaign success?
In-class Exercise-Part II:
Luring ‘em back to school Complete the following table comparing the differences between the direct mail and e-mail catalogue mailings. Which program appears to be more successful? Why?
DM EMcatalogues sent 60,000 78,000response rate 2.5% 1.5%total responses
mktg cost/catalogue $3.00 $0.25total catalogue mktg costsother mktg costs (agency fees etc.) $300,000 $300,000total mktg costs
avg profit /course $700 $700total profit
ROIcost/response
Statistical Significance
Statistical certainty is impossibleWe normally talk of level of confidence in statistical predictions In DM this is often 95% (19 out of 20 times) or 90% (18 out of 20 times) confidence - results will be repeated within an acceptable margin of errorThe confidence level set normally depends on financial risk
Where Are the Other 95% - the Direct Marketer’s Non-respondents
Research evidence suggest that it is all due to poor timing!!Not ready or unable to transact because: lack of funds don’t know how the product or service will perform domestic upheaval (e.g. moving house)
Is this the reason why repeat mailings and follow-ups are often successful?Also, is this the reason behind the possible discrepancy between test results and roll-up?
Selecting Response Channels
How do you want them to respond?The 3 main channels are:MailPhone InternetAdditional Channels include:
Mobile Devices
Response Channel Specific Metrics
Direct MailResponse Rate versus no mail groupCreative Tests-different letter versionsOffer Tests-different offer typesResponse Mechanisms (call/in-person)
TelemarketingResponse Rate versus no calll groupPercentage Right Party ConnectWrap code analysisCross and Up sellsCreative Testing-Scripts
InternetResponse Rate versus no contactView RateAbandon RateAccept RateClick Through RateRe-visit RateCreative Tests-different content pagesPush versus Pull tactics
Channel CombinationsResponse Rate versus single channel
Measurement
It’s not enough to count responses. Response does not indicate the level
of customer commitment. Measuring response doesn’t tell us
WHY consumers behave the way they do.
Response builds only limited knowledge of customer behaviour.
Beyond Response
What kind of people are responding?What other market segments are there?What offers trigger different groups to respond?How many ways can we present a message?Where are the overlaps in media used?What messages are appropriate for various media?
Performance Measurement
Historical data can be useful in evaluating the performance of similar marketing campaigns.
Performance MeasurementMEASURE OPERATIONALIZATION
Response rate Percentage of prospects contacted who replied
Number of inquiries Number of fulfillments
Number of qualified leads Number of leads who expressed interest that were converted into
sales or opportunities
New customers acquired Number of purchasers who had not purchased before
Customer lifetime value Net present value of customer over a specified period of time
Customer acquisition cost Total marketing costs divided by number of new customers
TEST
TEST
TEST
1. Products/Services
2. Media e.g.. Lists, print, Internet
3. The Offer
4. Formats/Layouts
5. Timing Schedules
Testing Variables
Common Experimental Designs
Split-run experimentCompare responses of campaign A to
campaign B using the same list (split in two)
Before-and-after experimentCompare the outcomes of campaign A
recipients to a control group that did not receive it.
Good Pre-test Design
A good experiment will measure the effect of ONE variable on another (response rate).
Compare, on a limited audience:1. (Offer A) vs. (Offer B) vs. (Offer C)2. (Creative A) vs. (Creative B)3. (Segment A) vs. (Segment B)
Bad Pre-test Design
Marketer attempts to:alter more than 1 variable per test
cell in the same experiment compare results in one medium to
anothertest different response channelssplit the list into test cells that are too
small (n<30 responses)
Next Week: Test Structure (25%)
Class Test: July 18th, 20062 hours
Final ExamResponsible for everything covered in class, including handoutsCovers Materials from Week 1-Week 10
StructureMultiple choice Short AnswerMetrics ProblemCase Study
Top Related