Cim & brilliant media introduction to econometrics
-
Upload
neil-charles -
Category
Business
-
view
2.557 -
download
0
description
Transcript of Cim & brilliant media introduction to econometrics
Maximising returns on yourcommunications investment
Marketing measurement and econometrics
1st November 2011
We know that we need to measure marketing
• To increase its effectiveness
• To reduce risk
• To justify the marketing budget
But marketing measurement is hard
• Marketing doesn’t always work quickly
• The effects are often not felt immediately
• So we end up not being sure if it’s working at all
Week-to-week sales movements are affected by many factors other than marketing
• Short-term sales movements due to advertising are difficult to pull apart from other factors
Distribution change (+30%)
Promotions and discounts
Seasonality
Weather
Above the line advertising
Competitor activity
Price change (+10%)
Random ‘noise’
5-10%
30-50%
30%
10-20%
5-6%
3- 5%
1-2%
~ 1%
Scale of weekly sales movements typically measured by an FMCG model
The full impact of marketing is also not felt immediately
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0
20
40
60
80
100
120
140
07-J
an-0
828
-Jan
-08
18-F
eb-0
810
-Mar
-08
31-M
ar-0
821
-Apr
-08
12-M
ay-0
802
-Jun
-08
23-J
un-0
814
-Jul
-08
04-A
ug-0
825
-Aug
-08
15-S
ep-0
806
-Oct
-08
27-O
ct-0
817
-Nov
-08
08-D
ec-0
829
-Dec
-08
19-J
an-0
909
-Feb
-09
02-M
ar-0
923
-Mar
-09
13-A
pr-0
904
-May
-09
25-M
ay-0
915
-Jun
-09
06-J
ul-0
927
-Jul
-09
17-A
ug-0
907
-Sep
-09
28-S
ep-0
919
-Oct
-09
09-N
ov-0
930
-Nov
-09
21-D
ec-0
911
-Jan
-10
01-F
eb-1
022
-Feb
-10
15-M
ar-1
005
-Apr
-10
26-A
pr-1
017
-May
-10
07-J
un-1
028
-Jun
-10
19-J
ul-1
009
-Aug
-10
30-A
ug-1
020
-Sep
-10
11-O
ct-1
001
-Nov
-10
22-N
ov-1
013
-Dec
-10
03-J
an-1
124
-Jan
-11
14-F
eb-1
107
-Mar
-11
28-M
ar-1
118
-Apr
-11
09-M
ay-1
130
-May
-11
Indexed Impact on Sales
TVRs
Yorkshire TVRs Shape of Impact on Sales (Index)
0
20
40
60
80
100
120
140
160
180W
eek
1W
eek
2W
eek
3W
eek
4W
eek
5W
eek
6W
eek
7W
eek
8W
eek
9W
eek
10W
eek
11W
eek
12W
eek
13W
eek
14W
eek
15W
eek
16W
eek
17W
eek
18W
eek
19W
eek
20W
eek
21W
eek
22W
eek
23W
eek
24W
eek
25W
eek
26W
eek
27W
eek
28W
eek
29W
eek
30W
eek
31W
eek
32W
eek
33W
eek
34W
eek
35W
eek
36W
eek
37W
eek
38W
eek
39W
eek
40W
eek
41W
eek
42W
eek
43W
eek
44W
eek
45W
eek
46W
eek
47W
eek
48W
eek
49W
eek
50W
eek
51W
eek
52
TVRs
/ S
ales
Impa
ct
TVRs Effect on Sales
Illustrated impact of TV airtime on sales
Continuous TV airtime and shape of impact on sales (Brilliant Media client example)
0.00
50.00
100.00
150.00
200.00
250.00
01-O
ct-1
005
-Oct
-10
09-O
ct-1
013
-Oct
-10
17-O
ct-1
021
-Oct
-10
25-O
ct-1
029
-Oct
-10
02-N
ov-1
006
-Nov
-10
10-N
ov-1
014
-Nov
-10
18-N
ov-1
022
-Nov
-10
26-N
ov-1
030
-Nov
-10
04-D
ec-1
008
-Dec
-10
12-D
ec-1
016
-Dec
-10
20-D
ec-1
024
-Dec
-10
28-D
ec-1
001
-Jan
-11
05-J
an-1
109
-Jan
-11
13-J
an-1
117
-Jan
-11
21-J
an-1
125
-Jan
-11
29-J
an-1
102
-Feb
-11
06-F
eb-1
110
-Feb
-11
14-F
eb-1
118
-Feb
-11
22-F
eb-1
126
-Feb
-11
02-M
ar-1
106
-Mar
-11
10-M
ar-1
114
-Mar
-11
18-M
ar-1
122
-Mar
-11
26-M
ar-1
130
-Mar
-11
03-A
pr-1
107
-Apr
-11
11-A
pr-1
115
-Apr
-11
19-A
pr-1
123
-Apr
-11
27-A
pr-1
1
New
Cus
tom
ers
Immediate TV Contribution TV Carryover contribution
New Customers
The effect of a TV burst can last well beyond the timing of the spots
Example of a highly impactful TV burst
Trying to identify marketing impact by looking for increases in sales, works occasionally
Year on year sales. Client was looking for the reason that sales increased from June 2011
-15%
-10%
-5%
0%
5%
10%
15%
20%
25%
30%
35%
03-J
an-1
1
10-J
an-1
1
17-J
an-1
1
24-J
an-1
1
31-J
an-1
1
07-F
eb-1
1
14-F
eb-1
1
21-F
eb-1
1
28-F
eb-1
1
07-M
ar-1
1
14-M
ar-1
1
21-M
ar-1
1
28-M
ar-1
1
04-A
pr-1
1
11-A
pr-1
1
18-A
pr-1
1
25-A
pr-1
1
02-M
ay-1
1
09-M
ay-1
1
16-M
ay-1
1
23-M
ay-1
1
30-M
ay-1
1
06-J
un-1
1
13-J
un-1
1
20-J
un-1
1
27-J
un-1
1
04-J
ul-1
1
11-J
ul-1
1
18-J
ul-1
1
25-J
ul-1
1
01-A
ug-1
1
08-A
ug-1
1
15-A
ug-1
1
22-A
ug-1
1
29-A
ug-1
1
Year
on
Year
Cha
nge
Unfortunately, more often it leads to confusion.Sales increased here, after TV was returned to normal.
-40%
-20%
0%
20%
40%
60%
80%
100%
03-J
an-1
1
10-J
an-1
1
17-J
an-1
1
24-J
an-1
1
31-J
an-1
1
07-F
eb-1
1
14-F
eb-1
1
21-F
eb-1
1
28-F
eb-1
1
07-M
ar-1
1
14-M
ar-1
1
21-M
ar-1
1
28-M
ar-1
1
04-A
pr-1
1
11-A
pr-1
1
18-A
pr-1
1
25-A
pr-1
1
02-M
ay-1
1
09-M
ay-1
1
16-M
ay-1
1
23-M
ay-1
1
30-M
ay-1
1
06-J
un-1
1
13-J
un-1
1
20-J
un-1
1
27-J
un-1
1
04-J
ul-1
1
11-J
ul-1
1
18-J
ul-1
1
25-J
ul-1
1
01-A
ug-1
1
08-A
ug-1
1
15-A
ug-1
1
22-A
ug-1
1
29-A
ug-1
1
Year
on
Year
Cha
nge
Sales Sales Year on YearTV Ratings Year on Year
Year on year sales. Client was looking for the reason that sales increased from June 2011
Weekly sales tracking that tries to pin down marketing, is rarely effective unless marketing uplifts are very large
• Tracking can lead to a singular focus on trying to explain the previous week’s sales
– Trying to explain away the movements that aren’t marketing is very difficult
– Often we end up blaming everything on the weather
A weekly retail sales tracking dashboard with a singular focus on weather (Brilliant Media client example)
Total sales TY 2,534.42 3,347.83 2,321.96 2,384.24 2,320.83 2,623.11 3,429.65Total sales LY 2,167.29 3,138.10 3,061.70 3,110.99 3,446.56 3,739.47 4,942.17Total budget sales 2,194.87 3,357.76 3,249.18 3,275.26 3,657.56 3,955.02 5,119.55
Sales vs budget 15.5% -0.3% -28.5% -27.2% -36.5% -33.7% -33.0%Sales vs LY 16.9% 6.7% -24.2% -23.4% -32.7% -29.9% -30.6%
2011 Weather
Temp 7.1°C 6.7°C 6.1°C 5.6°C 5.9°C 5.9°C 5.7°C
2010 Weather
We can achieve a lot, without complex statistics
• Track what we can measure
– Take care not to only spend on what we can track…
• Acknowledge the issues with marketing measurement
Direct response tracking completes a part of the picture
EconometricsReturn on investment
Budget allocationBudget setting
Sales forecastingTest and learn
Direct response trackingOptimisation within a marketing
channel, including:Colour vs. B&W
Ad sizeNewspaper titles
Web display placement
Consumer ResearchBrand tracking
Brand perceptionCreative tuningTarget audienceSegmentation
Competitor benchmarks
Direct response has problems, but it’s a good step for advertisers with a product that’s suited to being tracked
• Several mechanics allow us to track response– Bespoke numbers– Direct mail– Competitions– ‘Where did you hear about us?’
• Compares within channels only
– What about brand TV, or if your brand has a memorable telephone number that you don’t want to change?
– Many brands – such as FMCG - have no response mechanic
Direct response data lets us optimise within press, search or other channels with response metrics
• Database technology makes this type of reporting quick and relatively easy– Tool for both agencies and clients
‘Brand’ might be driving a lot of your direct activity
• True cost per acquisition is a combination of brand and direct
– Most advertisers don’t analyse to this depth (yet)
Large numbers of search clicks can be driven by TV (Brilliant Media client example)
0 10,000 20,000 30,000 40,000 50,000 60,000
CPC
Organic
CPC
Organic
CPC
Organic
CPC
Organic
Prod
uct 1
Prod
uct 2
Chea
pBr
and
Number of clicks
Base Driven by TV
But if you’re not careful, it can all get a little complicated
There’s a simple rule of thumb that avoids a lot of brand measurement issues
• If direct channels (including search) bring in sales at a cost lower than TV, then they’re making your marketing more efficient
– TV generates the interest in your product, whether you run search ads to convert it, or not
• But we’ll need econometrics to find out the cost per acquisition from TV
Econometrics solves some of our direct response measurement issues
WARC case studies incorporating econometrics
1
4
5
14
16
18
20
20
27
29
33
33
39
43
54
67
146
Tobacco
Utilities and services
Business and industrial
Wearing apparel
Govt. and non-profit
Motor and auto
Media and publishing
Travel, transport and tourism
Leisure and entertainment
Telecomms
Financial services
Toiletries and cosmetics
Household and domestic
Pharmaceutical and healthcare
Drink and beverage
Retail
Food
Number of WARC case studies referencing econometrics (to October 2011)
Econometrics adds a new set of information, that we can’t get from direct response tracking alone
EconometricsReturn on investment
Budget allocationBudget setting
Sales forecastingTest and learn
Direct response trackingOptimisation within a marketing
channel, including:Colour vs. B&W
Ad sizeNewspaper titles
Web display placement
Consumer ResearchBrand tracking
Brand perceptionCreative tuningTarget audienceSegmentation
Competitor benchmarks
Econometrics measures and then improves the effectiveness of advertising
Econometrics uses statistical models of sales to…
– Measure the effectiveness (return on investment) of past advertising campaigns
– Split marketing campaigns into their individual parts(TV, radio, outdoor etc.) and measure the effectiveness of each part of the marketingmix
– Forecast the effectiveness offuture advertising campaigns
– Use forecasts to produce amore effective marketing mix
-50
0
50
100
150
200
250
Wee
k 1
Wee
k 4
Wee
k 7
Wee
k 10
Wee
k 13
Wee
k 16
Wee
k 19
Wee
k 22
Wee
k 25
Wee
k 28
Wee
k 31
Wee
k 34
Wee
k 37
Wee
k 40
Wee
k 43
Wee
k 46
Wee
k 49
Wee
k 52
Wee
k 55
Wee
k 58
Wee
k 61
Wee
k 64
Wee
k 67
Wee
k 70
Wee
k 73
Wee
k 76
Wee
k 79
Wee
k 82
Wee
k 85
Wee
k 88
Wee
k 91
Wee
k 94
Wee
k 97
Wee
k 10
0W
eek
103
Wee
k 10
6W
eek
109
Wee
k 11
2W
eek
115
Wee
k 11
8W
eek
121
Wee
k 12
4W
eek
127
Wee
k 13
0W
eek
133
Wee
k 13
6W
eek
139
Wee
k 14
2W
eek
145
Wee
k 14
8W
eek
151
Wee
k 15
4
Sale
s (£
'000
s)
Press RadioTV Store OpeningsSeasonality ActualModel
In marketing, econometrics usually means…
• Proving the effectiveness of advertising in driving sales
• Measuring return on investment (ROI)
• Building a mathematical model of two to three years of historical sales data
• Concentrating hardest on major above the line spends
• Aiming to produce a more efficient media budget allocation
A wider definition is much more useful
Econometrics is a toolbox that helps you to test theories about
your marketing
“”
What’s the output?
1. Measurement of past advertising campaigns, split into the different media channels that were used.
Proof that past advertising added to sales and (hopefully!) was profitable
Return on investment calculations showing the individual profitability of each marketing channel
2. Forecasting and improvement of future campaigns
The really useful bit and why it’s worth investing in econometrics
We can use the model to forecast the effectiveness of potential media schedules and then choose the one with the highest returns.
How a (standard) model actually works
• The maths that goes into a model is complicated…
• But you really don’t need to understand it, to get a feeling for how econometrics works
tiitii
ti
i
ti TTP
P
S
S )(21
We start with a sales history; two to three years of weekly sales data
-50
0
50
100
150
200
250
Wee
k 1
Wee
k 4
Wee
k 7
Wee
k 10
Wee
k 13
Wee
k 16
Wee
k 19
Wee
k 22
Wee
k 25
Wee
k 28
Wee
k 31
Wee
k 34
Wee
k 37
Wee
k 40
Wee
k 43
Wee
k 46
Wee
k 49
Wee
k 52
Wee
k 55
Wee
k 58
Wee
k 61
Wee
k 64
Wee
k 67
Wee
k 70
Wee
k 73
Wee
k 76
Wee
k 79
Wee
k 82
Wee
k 85
Wee
k 88
Wee
k 91
Wee
k 94
Wee
k 97
Wee
k 10
0W
eek
103
Wee
k 10
6W
eek
109
Wee
k 11
2W
eek
115
Wee
k 11
8W
eek
121
Wee
k 12
4W
eek
127
Wee
k 13
0W
eek
133
Wee
k 13
6W
eek
139
Wee
k 14
2W
eek
145
Wee
k 14
8W
eek
151
Wee
k 15
4
Sale
s (£
'000
s)
Sales
and a model, that at this stage doesn’t know anything at all about sales movements
-50
0
50
100
150
200
250
Wee
k 1
Wee
k 4
Wee
k 7
Wee
k 10
Wee
k 13
Wee
k 16
Wee
k 19
Wee
k 22
Wee
k 25
Wee
k 28
Wee
k 31
Wee
k 34
Wee
k 37
Wee
k 40
Wee
k 43
Wee
k 46
Wee
k 49
Wee
k 52
Wee
k 55
Wee
k 58
Wee
k 61
Wee
k 64
Wee
k 67
Wee
k 70
Wee
k 73
Wee
k 76
Wee
k 79
Wee
k 82
Wee
k 85
Wee
k 88
Wee
k 91
Wee
k 94
Wee
k 97
Wee
k 10
0W
eek
103
Wee
k 10
6W
eek
109
Wee
k 11
2W
eek
115
Wee
k 11
8W
eek
121
Wee
k 12
4W
eek
127
Wee
k 13
0W
eek
133
Wee
k 13
6W
eek
139
Wee
k 14
2W
eek
145
Wee
k 14
8W
eek
151
Wee
k 15
4
Sale
s (£
'000
s)
SalesModel
The aim of model building is to produce a model (red line) that tracks actual sales as closely as possible – explaining why sales have moved in the past
Step 1:Add large, easy to measure factors
to the model
-50
0
50
100
150
200
250
Wee
k 1
Wee
k 4
Wee
k 7
Wee
k 10
Wee
k 13
Wee
k 16
Wee
k 19
Wee
k 22
Wee
k 25
Wee
k 28
Wee
k 31
Wee
k 34
Wee
k 37
Wee
k 40
Wee
k 43
Wee
k 46
Wee
k 49
Wee
k 52
Wee
k 55
Wee
k 58
Wee
k 61
Wee
k 64
Wee
k 67
Wee
k 70
Wee
k 73
Wee
k 76
Wee
k 79
Wee
k 82
Wee
k 85
Wee
k 88
Wee
k 91
Wee
k 94
Wee
k 97
Wee
k 10
0W
eek
103
Wee
k 10
6W
eek
109
Wee
k 11
2W
eek
115
Wee
k 11
8W
eek
121
Wee
k 12
4W
eek
127
Wee
k 13
0W
eek
133
Wee
k 13
6W
eek
139
Wee
k 14
2W
eek
145
Wee
k 14
8W
eek
151
Wee
k 15
4
Sale
s (£
'000
s)
Seasonality
Actual
Model
The bars show how many sales are driven by each factor that the model is measuring. This variable captures the effect of Christmas
Step 2:Identify and model major trends in sales
-50
0
50
100
150
200
250
Wee
k 1
Wee
k 4
Wee
k 7
Wee
k 10
Wee
k 13
Wee
k 16
Wee
k 19
Wee
k 22
Wee
k 25
Wee
k 28
Wee
k 31
Wee
k 34
Wee
k 37
Wee
k 40
Wee
k 43
Wee
k 46
Wee
k 49
Wee
k 52
Wee
k 55
Wee
k 58
Wee
k 61
Wee
k 64
Wee
k 67
Wee
k 70
Wee
k 73
Wee
k 76
Wee
k 79
Wee
k 82
Wee
k 85
Wee
k 88
Wee
k 91
Wee
k 94
Wee
k 97
Wee
k 10
0W
eek
103
Wee
k 10
6W
eek
109
Wee
k 11
2W
eek
115
Wee
k 11
8W
eek
121
Wee
k 12
4W
eek
127
Wee
k 13
0W
eek
133
Wee
k 13
6W
eek
139
Wee
k 14
2W
eek
145
Wee
k 14
8W
eek
151
Wee
k 15
4
Sale
s (£
'000
s)
Store OpeningsSeasonalityActualModel
Step 3:Once the basic model is built, we can get a first estimate for larger marketing spends
-50
0
50
100
150
200
250
Wee
k 1
Wee
k 4
Wee
k 7
Wee
k 10
Wee
k 13
Wee
k 16
Wee
k 19
Wee
k 22
Wee
k 25
Wee
k 28
Wee
k 31
Wee
k 34
Wee
k 37
Wee
k 40
Wee
k 43
Wee
k 46
Wee
k 49
Wee
k 52
Wee
k 55
Wee
k 58
Wee
k 61
Wee
k 64
Wee
k 67
Wee
k 70
Wee
k 73
Wee
k 76
Wee
k 79
Wee
k 82
Wee
k 85
Wee
k 88
Wee
k 91
Wee
k 94
Wee
k 97
Wee
k 10
0W
eek
103
Wee
k 10
6W
eek
109
Wee
k 11
2W
eek
115
Wee
k 11
8W
eek
121
Wee
k 12
4W
eek
127
Wee
k 13
0W
eek
133
Wee
k 13
6W
eek
139
Wee
k 14
2W
eek
145
Wee
k 14
8W
eek
151
Wee
k 15
4
Sale
s (£
'000
s)
TV Store Openings
Seasonality Actual
Model
At every stage, diagnostic statistics tell us how well the model is working
• We get a lot of information from a model
1. The sales impact of each factor that we have included (ROI)
2. How sure we are that each individual measurement is accurate (confidence)
3. How sure we are that the overall model is robust
R2, t and F; diagnostic statistics that only econometricians find interesting
Step 4:The final model is a good fit for sales and includes all major marketing investments
-50
0
50
100
150
200
250
Wee
k 1
Wee
k 4
Wee
k 7
Wee
k 10
Wee
k 13
Wee
k 16
Wee
k 19
Wee
k 22
Wee
k 25
Wee
k 28
Wee
k 31
Wee
k 34
Wee
k 37
Wee
k 40
Wee
k 43
Wee
k 46
Wee
k 49
Wee
k 52
Wee
k 55
Wee
k 58
Wee
k 61
Wee
k 64
Wee
k 67
Wee
k 70
Wee
k 73
Wee
k 76
Wee
k 79
Wee
k 82
Wee
k 85
Wee
k 88
Wee
k 91
Wee
k 94
Wee
k 97
Wee
k 10
0W
eek
103
Wee
k 10
6W
eek
109
Wee
k 11
2W
eek
115
Wee
k 11
8W
eek
121
Wee
k 12
4W
eek
127
Wee
k 13
0W
eek
133
Wee
k 13
6W
eek
139
Wee
k 14
2W
eek
145
Wee
k 14
8W
eek
151
Wee
k 15
4
Sale
s (£
'000
s)
Press RadioTV Store OpeningsSeasonality ActualModel
It’s about asking the right questions
• There is a ‘standard’ econometric analysis, but modelling works much better, when we set it up from the start to answer specific questions
Can my media mix be made more efficient?
[this is a ‘standard’ analysis]
Should I transfer some of my ATL budget online? [High Street banking]
What will it cost to hit this sales target? [Automotive]
Finance are threatening to cut the ad budget; I need to prove that advertising is profitable.
[Online banking]Do I really need
‘brand’ TV, or can search and DM do
the job alone?[Car insurance]
How much budget do I need for a store re-launch?
[Supermarket retail]
Finding the most effective marketing mix is more than ROI measurement
• Response curves measured using econometrics forecast the effect of changing marketing budgets
0%
50%
100%
150%
200%
250%
300%
350%
100 200 300 400 500 600 700 800 900 1000
Add
ition
al S
ales
£'000s
TV
Press 1. The most effective marketing mix allocates budget first to TV…
2. …and then the remaining budget to press
Beyond a £400k campaign, additional TV spend generates few extra sales
Example response curves
Bringing together the three elements of marketing evaluation
1. Monitoring the market– Tracking competitor activity– Benchmarking (share of voice etc.)
2. Response tracking– Immediate indicators of consumer behaviour– Web traffic, Click through, Cost per click, Call volume, store footfall
and more…– Awareness & consideration tracking
3. Modelling– Filling in the gaps that can’t be measured by direct response
Direct response trackingCost per acquisition etc.Daily / Weekly updated
Market MonitoringShare of voice etc.Weekly / Monthly updated
The flow of campaign evaluation
Plan Campaign Post CampaignAnalysis
Long-termanalysis
Flow of resultsClicks (web response)Phone #sDirect sales
Econometrics
Budget setting, forecastingand optimisation
‘Multiplier’ adjustments
- Adjustments to directresponse Measures
- Click path analysis
In-campaign‘tuning’
0.00
200.00
400.00
600.00
800.00
1,000.00
1,200.00
1,400.00
01-O
ct-1
005
-Oct
-10
09-O
ct-1
013
-Oct
-10
17-O
ct-1
021
-Oct
-10
25-O
ct-1
029
-Oct
-10
02-N
ov-1
006
-Nov
-10
10-N
ov-1
014
-Nov
-10
18-N
ov-1
022
-Nov
-10
26-N
ov-1
030
-Nov
-10
04-D
ec-1
008
-Dec
-10
12-D
ec-1
016
-Dec
-10
20-D
ec-1
024
-Dec
-10
28-D
ec-1
001
-Jan
-11
05-J
an-1
109
-Jan
-11
13-J
an-1
117
-Jan
-11
21-J
an-1
125
-Jan
-11
29-J
an-1
102
-Feb
-11
06-F
eb-1
110
-Feb
-11
14-F
eb-1
118
-Feb
-11
22-F
eb-1
126
-Feb
-11
02-M
ar-1
106
-Mar
-11
10-M
ar-1
114
-Mar
-11
18-M
ar-1
122
-Mar
-11
26-M
ar-1
130
-Mar
-11
03-A
pr-1
107
-Apr
-11
11-A
pr-1
115
-Apr
-11
19-A
pr-1
123
-Apr
-11
27-A
pr-1
1
Sale
s
Sales minus Central TV burstActual sales (including Central TV burst)Projected Sales if Central burst had been run nationally
Sales Forecasts and Projections
National sales revenue projections
-15%
-10%
-5%
0%
5%
10%
15%
20%
25%
30%
35%
03-J
an-1
1
10-J
an-1
1
17-J
an-1
1
24-J
an-1
1
31-J
an-1
1
07-F
eb-1
1
14-F
eb-1
1
21-F
eb-1
1
28-F
eb-1
1
07-M
ar-1
1
14-M
ar-1
1
21-M
ar-1
1
28-M
ar-1
1
04-A
pr-1
1
11-A
pr-1
1
18-A
pr-1
1
25-A
pr-1
1
02-M
ay-1
1
09-M
ay-1
1
16-M
ay-1
1
23-M
ay-1
1
30-M
ay-1
1
06-J
un-1
1
13-J
un-1
1
20-J
un-1
1
27-J
un-1
1
04-J
ul-1
1
11-J
ul-1
1
18-J
ul-1
1
25-J
ul-1
1
01-A
ug-1
1
08-A
ug-1
1
15-A
ug-1
1
22-A
ug-1
1
29-A
ug-1
1
Year
on
Year
Cha
nge
A Brilliant client (with econometric models) asked why their recent performance had been so good
Strong year on year sales performance raised the question: What was going right? (Brilliant Media client example)
Year on year analysis provided strong evidence that the overall market was improving
• Year on year is good for evidence, but it doesn’t tell you what to track. We could only draw this chart because we already knew Google searches were important
-10%
-5%
0%
5%
10%
15%
20%
17-J
an-1
1
24-J
an-1
1
31-J
an-1
1
07-F
eb-1
1
14-F
eb-1
1
21-F
eb-1
1
28-F
eb-1
1
07-M
ar-1
1
14-M
ar-1
1
21-M
ar-1
1
28-M
ar-1
1
04-A
pr-1
1
11-A
pr-1
1
18-A
pr-1
1
25-A
pr-1
1
02-M
ay-1
1
09-M
ay-1
1
16-M
ay-1
1
23-M
ay-1
1
30-M
ay-1
1
06-J
un-1
1
13-J
un-1
1
20-J
un-1
1
27-J
un-1
1
04-J
ul-1
1
11-J
ul-1
1
18-J
ul-1
1
25-J
ul-1
1
01-A
ug-1
1
08-A
ug-1
1
15-A
ug-1
1
22-A
ug-1
1
Year
on
Year
Cha
nge
Year on Year Sales (5wk MA)
Google Searches for product term (non-brand)
Google search activity closely matched overall sales (Brilliant Media client example)
What about test and control?
What makes a ‘good’ media test?
1. Clear objectives
• What, exactly, are we trying to find out?
2. Designed to generate a measure• What uplift do we expect that the test
might generate …?
• … So what scale does the test need to be for this effect to be measurable?
3. Useful negative results
• If the test finds no significant uplifts, are we sure the activity doesn’t work?
Why is econometrics Important?
1. Controls for external factors
2. Lets us measure smaller effects
3. Helps to specify an appropriate scale for the test
A ‘bad’ media test… Real world example
• An advertiser wanted to find the effect of a combined TV and Radio campaign on various brand preference metrics
• The campaign was run over four weeks in the Central and Granada BARB regions
• Pre and Post survey dips in four cities provided theawareness data
– Three test cities: Manchester, Carlisle & Birmingham
– One control: Norwich
• 600+ respondents in the pre-campaign dip and 700+ in the post campaign
• What’s wrong with that… ?
What’s wrong with that?
1. No clear objective for the campaign
• ‘Run a campaign and see which metric moves’ is not an ideal starting point
2. No prior knowledge of whether the test is likely to be big enough to make a difference (and so be measured)
• In order to generate the 10% sales uplift that we will need to get a solid measure, should the test be run for longer? Or with more GRPs?
• One control city only is very, very risky
3. Post campaign measurement using a simple average vs. Control
• Leaves the test exposed to unforeseen events that have a different impact in the test and control regions
• Econometrics gives a much better chance of a useable result
What actually happened?The test was inconclusive…
Ove
rall
Te
st
Co
ntr
ol
Man
ch
este
r
Carl
isle
Bir
min
gh
am
53% 54%
63%
41%47%
57%61%
55%
68%
52%
Pre Post
Awareness Metric
Two test cities increased. One decreased.
The two that increased were in different BARB regions
Control also increased(by more than the test regions)
Inconclusive result.
Spontaneous Awareness
What actually happened?The test was inconclusive…
Ove
rall
Te
st
Co
ntr
ol
Man
ch
este
r
Carl
isle
Bir
min
gh
am
36%
15%
31%33%
43%
28%
8%
31% 31%
23%
Pre Post
Purchase Intent
Purchase intent fell in two test regions and fell very heavily in Birmingham
The control region declined
Manchester stayed at 31%, bucking the decline of the control region
Overall, were our test regions better? Or worse?
Purchase intent
Briefing an analysis
A variety of companies provide econometrics and bring different strengths to the analysis
Completely impartial
Little internal data
Smaller (riskier)
May not be mediameasurement specialist
Very weak ties to planning
Generally impartial
Strong internal data
Large analytical teams
Can be expensive
Weak ties to planning
Risk of conflict of interest
Strong internal data
Mid to large analytical teams
Media measurement specialist
Strong ties to planning
Independent Semi-Independent Media Agencies
Automated tools can help (and sometimes reduce analysis costs) but need a lot of care
• Automated tools are seductive, but we need to be aware of their limitations
• Analysis is only ever an aid to decision making
Briefing an econometric analysis
1. Why do you want the work done?
2. What data exists and who is responsible for it?
3. When is the decision deadline that the work informs?
4. How will the work inform future decisions?
5. Who are your project team?
6. Interim meetings
7. What was your marketing designed to achieve?
8. everyone who will use the results needs to be involved from the start
We haven’t mentioned consumer research, but it’s the final piece of the puzzle
EconometricsReturn on investment
Budget allocationBudget setting
Sales forecastingTest and learn
Direct response trackingOptimisation within a marketing
channel, including:Colour vs. B&W
Ad sizeNewspaper titles
Web display placement
Consumer ResearchBrand tracking
Brand perceptionCreative tuningTarget audienceSegmentation
Competitor benchmarks
Contact
Neil CharlesHead of Econometrics
Brilliant Media1 City SquareLeedsLS1 2FF
+44(0)113 394 0078+44 (0)7508 [email protected]