An Cova 2000

8/13/2019 An Cova 2000

1/81

Analytic Insights Group Slide 1 ancova2000.ppt (08/00)

Analyzing Test Results (ANCOVA)

Using this presentation: This is an internal trainingpresentation created for use at Testing Boot Camp.Select slides may be extracted for use with clients.Filename: ANCOVA2000.PPTPrepared by: Valerie Skala and Bob Bakshis based on

charts created by Bruce Richardson and Andy FisherLast Revised: August 8, 2000Other resources: Steve LowtherKey Words: Analysis of Covariance, ANCOVA, test

analysis, covariate

8/13/2019 An Cova 2000

2/81


Analyzing Test Results

using Analysis of Covariance

(ANCOVA)

8/13/2019 An Cova 2000

3/81


Signposts

Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis

How ANCOVA works Explanation of covariates Exercise: Non-Standard Designs

Part 2 -- For Statisticians More details on the ANCOVA model

8/13/2019 An Cova 2000

4/81


Case StudyPepsi is interested in determining the effect of a $40million ad campaign on sales of Aquafina bottledwater.

They need to see at least a +15% boost in brand

sales to make the ad campaign profitable.They run the ad campaign in Dallas.

They compare their sales in the test market pre-to-

post, using Houston as the control, and see a net+10% boost in sales.

Should they roll the ad campaign nationally?

8/13/2019 An Cova 2000

5/81


The Answer: It Depends!

Raw numbers do NOT tell the whole story!

What if one of these scenarios took place?

Test Cell Control Cell Net

Comp Dealing flat Comp Dealing UP

Aquafina Sales UP Aquafina Sales flat Advertis ing effect may

really be zero.Comp Dealing UP Comp Dealing flat

Aquafina Sales UP Aquafina Sales flat Advertis ing effect may bemuch greater.

8/13/2019 An Cova 2000

6/81


We need to make sure were not justseeing changes in sales caused by:

Pre existing differences between the test andcontrol cell -- no match is perfect E.g., category development

Differences that arose during the test period

between test & control cells -- the marketplaceis always changing E.g., price, feature, display -- of the test brand or

a key competitor

Unknown variables /normal random variation-- aka error variance

What we cant control by design,we attempt to control with statistics.

8/13/2019 An Cova 2000

7/81Analytic Insights Group Slide 7 ancova2000.ppt (08/00)

ANCOVA is an ex post facto statisticaltechnique that

Levels the playing field so we can isolate the true test effect Identify imbalances in non-treatment factors, pre vs. post and test vs.

control Use weekly data to estimate the effect of these imbalances on sales Adjust the 4 sales means for these imbalances Report a covariate-adjusted net test effect

Provides a statistical test of significance How confident are we that the reported test effect could not be due

strictly to chance? What is the likely error range around our estimate of the effect?

Analysis of Covariance (ANCOVA) answers the question:What would the test effect be if the test and control markets were held equal

in all respects except for the test treatment?

8/13/2019 An Cova 2000


We have weekly data on sales, price, & merchandisingfor the test brand, competitors, and total category

5/10 6/7 7/5 8/2 8/30 9/27 10/25 11/22 12/20 1/17 2/14 3/14 4/11 5/9 6/6 7/4 8/1 8/29 9/26

4 Weeks Ending

0

200

400

600

800

1000 Volume Sales Price

$2.00

$3.00

$4.00

$5.00

$6.00

Pre PostPrice

8/13/2019 An Cova 2000


We can summarize the weekly data into 4 means

Dummy variables are added to each store-week record toindicate which Group (test or control) and which Period(pre or post) the record belongs in.

CumPre-TestWeeks

CumPost-Test

WeeksChange

AggregatedTest Group A B (B - A)

AggregatedControl Group C D (D - C)

Diff of Diffs

% Effect

Measure (sales, price, merchandising)

D %

8/13/2019 An Cova 2000


But ANCOVA is only one piece of the process

A good analysis starts with a gooddesign

ANCOVA cannot

fix a bad design.

Keep your brain engagedand your team talking

There are many pieces to ananalysis, not just the ANCOVA

8/13/2019 An Cova 2000


Signposts

Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works Explanation of covariates Exercise: Non-Standard Designs


8/13/2019 An Cova 2000


What to discuss at project kick- off before its too late!

Review proposal, including test objectives anddesign . Any known issues or concerns with sample size,

sensitivity, unusual factors we may (or may not) beable to control for?

What client expectations have been set or shouldbe set?

Do match criteria line up with what we plan toanalyze?

8/13/2019 An Cova 2000


What to discuss at project kick- off before its too late!(continued)

Expand everyones knowledge base. Review marketing plans -- do we have full info? Review category and brand dynamics -- slow

moving? Heavily merchandised or couponed?

8/13/2019 An Cova 2000


What to discuss at project kick- off before its too late!(continued)

If were doing toplines, what do we need to doto ensure that final analysis data will line up? Calculate volume sales per $MM ACV adjusted

for distribution or not? Adjust and project trial and repeat figures?

8/13/2019 An Cova 2000


1. From ANALYZER get unadjusted data and calculate rawtest effect.

+0.03 - -0.66 = +0.69 / 2.83 = +24.4%

3.49 3.493.52 2.83

Test Markets Control Markets0.00

1.00

2.00

3.00

4.00

5.00

Volum e Sales per $MM ACV Pre Post

8/13/2019 An Cova 2000


2. Review unadjusted results. Any obvious covariates jump out: number UPCs moving, price, merchandising-- for test brand and key competitors?

5/10 6/7 7/5 8/2 8/30 9/2710/2511/2212/201/17 2/14 3/14 4/11 5/9 6/6 7/4 8/1 8/29 9/26

4 Weeks Ending

0

1

2

3

4

5Test Brand Volume Sales per $MM ACV Test Control

Pre-test match Test

8/13/2019 An Cova 2000


3. Determine the dependent variable and independentvariables to include in the analysis.

- Should we assume 100% distribution on test brand or not?- Is brand share large, such that we should adjust for All Other (category

minus test brand) development instead of Total Category development?

Adjustments Made to level the Playing Field

Causal Factors(covariates)

Diff of Diffs(Post - Pre,Test - Cntrl)

Impact on TestEffect

Test Brand % BVol on Disp Only _+4_

Test Brand % BVol on Feat Only _+5_

Test Brand Average Price _-3_

CompA Average Price _-2_

8/13/2019 An Cova 2000


4. Run ANCOVA with just one test brand covariate (e.g.,NUPC1), to make sure raw means match Analyzer #s.*

* This QC check helps to ensure that the right stores and weeks havebeen pulled in, the right projection system and scan rate adjustmentshave been used, and the stub is brand flagged correctly. ANCOVA #sshould match InfoView numbers within 1%. If were losing too manyobservations due to BIDs with no data; may need to re-BID or changethe Assume Distribution parameter to YES

3.49 3.493.522.83

Test Control0.00

1.00

2.00

3.00

4.00

5.00Volum e Sales per $MM ACV

Pre Post

3.49 3.493.522.83

Test Control0.001.00

2.00

3.00

4.00

5.00Volum e Sales per $MM ACV

Pre Post

ANCOVA Analyzer

8/13/2019 An Cova 2000


Test BrandRaw vs. Adjusted Results

+ 24.4%

+ 12.0%

Raw Test

Effect

Adjusted Test

Effect

How is each covariate impacting the test effect?

Causal Factors (covariates)

Incrs inCovariateCauses ?

Chg in Sales

Test Brand %BVol Disp Only + Test Brand %BVol Feat Only + Test Brand %BVol TPR Only + Test Brand %BVol Feat+Disp +

CompA %BVol Any Merch - Category Vol/$MM ACV +

Test Brand Price - Competitor Price +

Adjustments to Level the Playing Field

5. Run kitchen sink ANCOVA with all covariates

8/13/2019 An Cova 2000


Test BrandRaw vs. Adjusted Results

+ 24.4%

+ 12.0%

Raw Test

Effect

Adjusted Test

Effect

6. On the third and consecutive ANCOVA runs,add/remove/modify covariates to improve the model.

Causal Factors (covariates*)

Diff ofDiffs

Impact onTest Effect

Test Brand % BVol on Disp Only _+4_ _-6.4_

Test Brand % BVol on Feat Only _+5_ _-6.0_

Test Brand Average Price _-3_ _-2_

CompA Average Price _-2_ _+2_

Adjustments to Level the Playing Field

Focus on the covariates that represent 75% of theadjustment.

8/13/2019 An Cova 2000


7. Report adjusted test effect with confidence level and interval.

+12%

+20%

Raw Test

Effect

Covariate Adjusted

Test Effect*

+12%

+20%

+28%

Confidence Interval

Low End*

Best Estimate

of Test Effect

Confidence Interval

High End*

* 99% confidence level * 80% confidence interval

8/13/2019 An Cova 2000

22/81


Each outlet is analyzed individually, but only theCombined Outlet result is reported to the client.

Outlet(% of "Analyzed Product"

Volume by Outlet)

Unadjusted % Test Effect

Covariate Adjusted

% Test Effect

Confidence Level

Comb ined Out lets (100%) 18.6 8.0 98%

Drug (40%) 20.8 9.0 99%

Food (30%) 24.4 12.0 98%

Mass Merch (30%) 9.9 2.5 81%

8. After analyzing each outlet individually,calculate the combined-outlet effect.

8/13/2019 An Cova 2000

23/81


About three-fourths oftests had a statistically

significant effect

Of those that worked,almost one-fourth had aneffect greater than 20%.

73% 27% 22 8

Up to 5%

5.1 to 10%

10.1 to 20%

> 20% effect

32%

23%

23%

23%

n = 30

9. Compare the results of this testvs. benchmarks

8/13/2019 An Cova 2000

24/81


Are we confident that our figures are accurate? Do all of the data tie together?

Are we drawing the correct conclusions from the data? Are any of the findings going to be politically sensitive? What actionable recommendations can we make?

10. Analyst explains the model and the final results indetail. Client presentation is prepared.

8/13/2019 An Cova 2000

25/81


11. Project test market results to the roll-out universe

Translate test market results to Total U.S.Consider post period differences between testmarket and national in terms of:

- sales development and trends- distribution of key items- competitive environment- merchandising

Compare to action standard and calculate payout

8/13/2019 An Cova 2000

26/81


Expected national unit sales 95,000,000 in implementation year*

x % test effect x 0.12 ---------------

Incremental national unit sales expected from new TV advertising 11,400,000

11A. Translate test market results to national

* Brand sales have been declining from the 100 million units sold in the pre-test year.

+

8/13/2019 An Cova 2000

27/81


11B. Compare test effect to action standard.Calculate pay-out.

This program should pay for itself in approx. 3 years.

Incremental unit sales in 1 year 11,400,000x Profit per unit x $0.50Incremental profit in 1 year $ 5,700,000

+ 50%* of year 1 effect in year 2 $ 2,850,000+ 25%* of year 1 effect in year 3 $ 1,425,000Total incremental profit in 3 yrs $ 9,975,000Investment in TV advertising $10,000,000*

* These assumptions are loosely based on the 1989 IRI "How Advertising Worksstudy. Of 44 ad weight tests on established brands that showed a significantpositive effect in the test year, the average effects were +22% in year 1, +14%

in year 2, and +7% in year 3.

8/13/2019 An Cova 2000

28/81


Take-Aways from this section

8/13/2019 An Cova 2000

29/81


Signposts

Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works Explanation of covariates Exercise: Non-Standard Designs


8/13/2019 An Cova 2000

30/81


,

,,

,

0 1 2 3

X

02

4

6

8

10Equation for this line: y = 3 + 2x

Remember the linear regression equation? y = b + 1X

Y

y = Total Sales, b = Base Sales, X = Level of Display,1 = strength of display's influence on sales

Actual obs ervations

ANCOVA starts with linear regression, to measure theimpact of each marketing mix variable on sales

8/13/2019 An Cova 2000

31/81


In multiple linear regression, we fit a plane to the datapoints.

* **

***

*

_

_

_ _ _

_ _

1.8

2.0

2.2

2.4

2.6

2.8

3.0

S a

l e s

R a

t e

Group0.5 0

68

10 12

14

8/13/2019 An Cova 2000

32/81


In multiple linear regression, we fit a plane to the datapoints. This plane can be defined with an equation.

A simplified example is shown below.

.y = b + 1X1 + 2X2

Sales Rate = Base Sales Rate + (Effect of Variable1 * Value of Variable1)+ (Effect of Variable2 * Value of Variable2)

Sales Rate = 2.0043 + (0.1783 * 1) + (0.0174 * 11)

2.0043 = Sales rate when there is no display0.1783 = Effect of being in the test group

1 = 1 for test group, 0 for control group0.0174 = Increase in sales when display increases 1%11 = % of base volume on display

8/13/2019 An Cova 2000

33/81


ANCOVA combines two statistical techniques

Multiple Linear Regression Using all store-week observations (regardless of group or period), we

measure the degree to which the level of a non-treatment variable, orcovariate (X) influences sales.

The coefficient ( 1) represents the % difference in sales we wouldexpect from a 1% imbalance in the level of that variable.

The program partitions sales variability into components, representingsales attributable to:

Each covariate, as defined by the analyst The Group-by-Period term, or actual test effect Remaining unexplained variance, or error variance

When we have an equation that seems to predict sales well, the fourraw sales means are adjusted to represent what sales would havebeen, if each covariate were set to the average level across all groupsand periods

8/13/2019 An Cova 2000

34/81


ANCOVA combines two statistical techniques

continued

Analysis of Variance

Compares the 4 adjusted means to determine whetherthe difference is greater than the products normal sales

variation.

Tests for statistical significance: how sure are we thatthere is a difference?

Considers the degree of unexplained variability in thesales data to estimate a likely error range around theestimated test effect.

ANCOVA dj h 4 l h h

8/13/2019 An Cova 2000

35/81


ANCOVA adjusts the 4 sales means to what theywould be if display were at the average level

Pre-Test Post-Test Diff % Effect

RAW SALES

Ad Group 100 120 20 9.1%

No Ad Group 100 110 10DISPLAY

Ad Group 5.0 7.0 Avg = 6.125

No Ad Group 6.0 6.5

ADJ SALES

Ad Group 105.0 116.1 11.1 3.1%

No Ad Group 100.6 108.3 7.8

8/13/2019 An Cova 2000

36/81


Core Principles of Testing

Its better to be vaguely rightthan precisely wrong .

Draw conclusions only from properlyadjusted data.

We cant adjust for everything, butthe more we DO adjust for, thecloser we can get to the rightanswer.

2 TPRs1 Display

2 Features

2 TPRs1 Feature

Comparing Test vs.

Control is like using anold-fashioned scale:It needs a skillful touchto balance.

8/13/2019 An Cova 2000

37/81



Gut-check and reality-check . The covariate adjustments and the final

adjusted test effect must make sense whencompared to the unadjusted sales &merchandising data.

A red light should go off if we see a largeadjustment to volume for the price ofcompetitive brand A, but the raw data shows

only a small difference in brand As price. Test your hypotheses in more than one way

and look for converging evidence

8/13/2019 An Cova 2000

38/81



A test is successful when we learnsomething new and actionable which has adirect impact on our profitability.

Many tests do NOT return a positive result --after all, if a proposition was as sure thing, itwouldnt need to be tested.

Still every test IS a success when it yields sometype of actionable new information or gets usthinking in new directions

8/13/2019 An Cova 2000

39/81


How is Testings ANCOVA different from MM Models?

Testing analyses have a more narrow goal:

We are only interested in adjusting thegroup/period sales means in order to isolate thetest effect.

We are not interested in the ability of the equationto predict the sales impact of changes in a specificvariable.

Testing removes outlier* HHs for panel analyses.Testing also drops outlier store-weeks when

appropriate.* HHs whose category volume is >= 3 standard deviations from the mean

by market (test+control HHs combined), by period (pre and post)

8/13/2019 An Cova 2000

40/81


How is Testings ANCOVA different from MM Models?(continued)

Modeling analyses generally have a broader goal:

To create a marketing mix model that can correctlyestimate the sales impact of each variableindividually

Modeling drops imputed store-weeks; Testing keeps them.

Modeling mean-centers all records; Testing does not.

8/13/2019 An Cova 2000

41/81

Analytic Insights Group Slide 41 ancova2000.ppt (08/00)ancova00.pr4

Signposts

Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works

Explanation of covariates Exercise: Non-Standard Designs

Part 2 -- For Statisticians

More details on the ANCOVA model

8/13/2019 An Cova 2000

42/81


Typical Panel CovariatesFMOV - corrects for imbalances in the store environment

experienced by panelists. Brand share in store is assumed toreflect all in-store conditions . For each store-week-cell, VFMOV = Brand Share (store) times a Panelist

Store Trips. Alternatively, try SFMOV = Brand Share (STORE) times PanelCategory Volume. One benefit of this measure is that it should capture theeffect of Frequent Shopper Program discounts. If some element of the

store environment is part of the test treatment, use individual merch.covariates instead.

ALOYAL - corrects for imbalances in test vs. control panelistspre-test loyalty to the brand . For each week-cell ALOYAL = the average pre-test %loyalty to the brand,

among those panelists who bought the category this week. Dont use incats with long purchase cycle or where a HH may have dual loyalty (onesoap for women and one for men).

A variation on ALOYAL is ADJ_EV, which multiplies the %loyalty* categoryvolume purchased. Dont use with high share brands.

% of brand volume purchased with a coupon

S A V E $ 1

8/13/2019 An Cova 2000

43/81


Even in BehaviorScan, the cleanest test design, ANCOVAmakes an important difference about 40% of the time.

60 %

24 %16 %

Less than 5% 5 - 10% More than 10%

Difference Between Raw and Adjusted Test Effect

% of Tests

* 5% is usually the minimum "significant" effect seen.

8/13/2019 An Cova 2000

44/81


Typical Covariates in CST & MMT Analyses

Merchandising Covariates % of base volume on feature only, display only, feature+display,and/or price reduction only.

Price Covariates Average price per volume (or unit)

Or a combination of Base Price per volume (or unit) and AveragePercent Off base price, when on deal.

Other Covariates ACV Wtd average number of UPCs moving (variety) Category volume sales / $MM ACV (CDI)

If test brand has a large market share, use All Other Category (minustest brand).

Assumes brand volume is correlated with category volume, which maynot always be true.

Dummy variables to flag presence of an unusual event or out-ofdesign weeks

2 5 % O f f

$ 1 . 9 9

e v e r y d a y

V a r i e t y

8/13/2019 An Cova 2000

45/81


Merchandising Covariates

For the test brand, we usually break in-storemerchandising into 7 mutually exclusive conditionsFor competitive brands, we typically use a single AnyMerchandising covariate.

Feature Only

Feature andDisplay

Price Reductiononly

Display only

Feature and Display

Price Reduction &

Display

Feature, Display, & PriceReduction

8/13/2019 An Cova 2000

46/81


Price CovariatesFor the Test brand , price usually is segmented into components: Base (non-promoted)price -- although base price rately

changes Average % discount off base price, for UPCs or TRP (% of

base volume sold on any price reduction completes thepicture)

For competitors (and sometimes for the test brand, if thisapproach seems to work better), we use a single price covariate:

Average Price per Volume

WATCH OUT: IRI does not yet capture average price paid afterFrequent Shopper discountsWATCH OUT: Price is generally correlated with the promotioncovariates.

ANCOVA makes an important difference in

8/13/2019 An Cova 2000

47/81


ANCOVA makes an important difference inabout half of all Controlled Store Tests.

48 %

31 %20 %


Difference Between Raw and Adjus ted Test Effect

% of Tests (sample of 194 Controlled Store Tests)

* 3% is usually the minimum "signif icant" effect seen.

ANCOVA is most important in Matched Market Tests

8/13/2019 An Cova 2000

48/81


ANCOVA is most important in Matched Market Tests,making an important difference in two-thirds of cases.

34 % 34 % 32 %


Difference Between Raw and Adjus ted Test Effect

% of Tests (sample of 133 Matched Market Tests)

* 3% is typically the smallest "significant" eff ect seen.

8/13/2019 An Cova 2000

49/81


How do we decide which covariates to start with?

Test design (treatment)

Suggestions from client and category experts.

IRI analysts experience with similar tests.

Prior category studies such as cross-purchasing,brand switching, price elasticity, or marketing mix.

Each covariate must logically have an influence on sales,or act as a surrogate for other influencing factors.

8/13/2019 An Cova 2000

50/81

8/13/2019 An Cova 2000

51/81


Which covariates do we keep?

Covariates that are significantly correlated with thetest brands volume (p 80%) and have the logical sign . . .even if they arent making a large adjustment to thetest effect.

Covariates that are not significantly correlated withthe test brands volume, but do have the logical signand arent making a large impact on the test effect --doesnt hurt to leave them in.

8/13/2019 An Cova 2000

52/81


Signals that the model can still be improved

You have a large but insignificant test effect may be due to 1

unusual week (unusual promotion with big sales bump) in either baseor test period, causing unusual variability.One covariate represents more than 30% of the total adjustment, whenthe adjustment is significant.

A covariate is causing a large adjustment that does not seemreasonable given the raw data e.g., test cell experienced a $0.01advantage on average price, reducing raw effect of +10% to adjustedeffect of -2%.

A covariate has a counter- intuitive sign e.g., an increase incompetitor As merchandising having a positive effect on the testbrands volume.

A covariate is not significant, but is having a large impact on the testeffect.The test effect is significant, but the confidence interval is very wide Consider additional covariates to reduce the error variation.

8/13/2019 An Cova 2000

53/81


Signals that the model can still be improved continued

Common fixes to consider:Look for outlier observations in the data with unusually high levels offeature, display and/or TPR.If no unusual causal conditions are apparent, consider the possibility of anunadvertised Frequent Shopper discount. Solution to both of the above: if problem is in just 1-3 weeks, drop the

outlier week(s). If the problem seems to affect more weeks, considerusing a dummy variable.

If the problem is with the pricing variables on the test brand, reduce thenumber of covariates involving price.Eliminate Number of UPCs Moving, especially if there was no significantchange in the means for this covariate. If you know of a significant newproduct entry (e.g., test brand introduced a new flavor), see if it rolled intotest and control markets on different timing; you might create a dummyvariable to indicate weeks where the new item is present.Roll competitive products into larger aggregates.Make sure youve included all relevant competitors in the category

8/13/2019 An Cova 2000

54/81


Signals that the model can still be improved continued

Common fixes to consider continued Check for colinearity, or correlation (r close to |1|), betweencovariates; drop duplicative covariates.Look for category-driven explanations such as seasonality thattends to make all brands promote on the same timing and createcolinearity.Check if covariate is correlated with the test treatment, so thatby adjusting for it, you are washing out the treatment effect.Check for homogeneity of slopes (possibility that the variablemay affect sales differently across groups).Make sure the error term (the remaining variance in sales notexplained by the covariates) is both small and randomlydistributed.

Bottom line: Can you defend the design and output of your model?

8/13/2019 An Cova 2000

55/81

8/13/2019 An Cova 2000

56/81


Examples: when to use a custom covariate

In a Matched market test, you learn that the test marketreceived a coupon drop while the control did not. Create a covariate for couponing

Or a juice/iced tea MMT, the test markets experienced anunusually hot period that the control markets did not.

Create a covariate to adjust for weekly temperature.In a Controlled Store price test, you had an out-of designcondition in one or two weeks (I.e., price not at thedesignated level)

Add a field to flag the out of design weeks.In a CST of a complex display unit, you audited test storesto make sure the condition of display was up to spec. Create a covariate for quality of display

8/13/2019 An Cova 2000

57/81


Non-Standard Designs

We usually analyze VOLUME sales per $MM ACVWhere Available. In what situation (I.e., when testing what kind of variable)

would it be better to analyze UNIT sales rate ?

Dollar sales rate? Volume sales rate assuming 100% distribution?

Share?

Name two situations in which we would NOT useDISPLAY as a covariate

8/13/2019 An Cova 2000

58/81

Analytic Insights Group Slide 58 ancova2000.ppt (08/00)ancova00.pr4

Signposts

Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works

Explanation of covariates Exercise: Non-Standard Designs

Part 2 -- For Statisticians

More details on the ANCOVA model

W h e n y o u ' r

e r e a d y t o

t a k e t h e n e

x t s t e p u p t

h e

l e a r n i n g c u

r v e.

8/13/2019 An Cova 2000

59/81


There are many methods of test evaluation

t-tests (comparing 2 means) and F-tests (comparing 4means)

Analysis of Variance (ANOVA)

Baseline Burps (regressions) Analysis of Covariance (ANCOVA)

Marketing Mix (regressions)

Share / Attraction models (logits)

8/13/2019 An Cova 2000

60/81


ANCOVA is IRIs Method of Choice

ANCOVA is a statistical method specifically designedto evaluate the response to a treatment whileeliminating the influences of extraneous (non-test)factors.

ANCOVA provides: adjustments for non-test factors, also called nuisance

factors or noise

an estimate of the effect of the test treatment confidence intervals for that estimate

8/13/2019 An Cova 2000

61/81


Concepts of ANCOVAThe ANCOVA model uses a least squaresmethodology, like regression. We use Type 3 sum of squares, because it does not require

equal #s of store -week observations.

Statistical tests are performed by comparing theparameter estimates to unexplained variability.

In IRIs model, the Group * Period term representsthe increase in volume sales due to the test

treatment.

Our confidence level comes from testig whether theG*P term is zero.

8/13/2019 An Cova 2000

62/81


A typical model for test analysisY = + 1G + 2P + 3G*P + 4Pr + M + ... + e

where Y = Sales rate (the dependent variable we're trying to predict)

0 = The Y intercept, generally set to 0 in Tes tingx = The regres sion coefficient for covariate x (1, 2, 3, etc.)

G = 1 for test, 0 for control -- indicates the group affiliation of eachdata record

P = 1 for post, 0 for pre -- indicates the period affiliation of the dataG*P = 1 for post-period, test cell; 0 for all other observations. The group*period term = the effect of the test treatment

Pr = A price variable, like avg $/vol for each brandflag-store-weekM = A merchandising variable, like % of base vol on feat+disp... = Other covariatese = Error term -- the residual unexplained variance in sales

8/13/2019 An Cova 2000

63/81


Calculating the coefficient for each covariate

represents the strength of the relationshipbetween the covariate, x, and sales, y.

For each covariate x : (y = sales, n = # weeks) x = SPxy

SSx

Sum of the cross products of X and Y:SPxy = sum(X*Y) - sum(X) * sum(Y)

n

Sum of squares for X:SSx = sum (X^2) - sum(X)^2

n

Example of data records: for a test with 3 pre & 3 post

8/13/2019 An Cova 2000

64/81


Example of data records: for a test with 3 pre & 3 postweeks, and 3 stores per group

Average Price for the Test Brand is the only covariate

Vol/$MM ACV Group Period Store G*P Price

1.2 1 1 1 0 1.991.5 1 1 1 0 1.890.8 1 1 1 0 1.991.6 1 2 1 1 1.891.4 1 2 1 1 1.991.4 1 2 1 1 1.99

2.4 2 1 2 0 1.992.3 2 1 2 0 1.993.1 2 1 2 0 1.892.6 2 2 2 0 1.992.5 2 2 2 0 1.992.5 2 2 2 0 1.99

8/13/2019 An Cova 2000

65/81


The group, period, and group*period terms

The regression coefficient has to be interpreteddifferently for these terms than for a continuousvariable like price of merchandising.

Specifically, the coefficient is equal to the SALESmean for this group/period, when all otherindependent variables (like price or merchandising)are equal to zero.

8/13/2019 An Cova 2000

66/81


Basis for covariate adjustments

The size of the adjustment to sales depends on: the regression coefficient for each covariate the imbalance in the mean value of each covariate

(e.g., average price) pre-post, test-control.

The raw sales means are adjusted by setting eachcovariate to its overall average level across all groupsand periods.

l l h d d l

8/13/2019 An Cova 2000

67/81


Calculating the covariate-adjusted sales means

The adjusted sales means are the sum of: the average sales level as predicted by the sum ofcovariates at their overall average level plus

the test treatments estimated impact on sales

AS_11 = sum( c * c) + 3G*P

AS_11 = adjusted sales for brandflag 11, the test brand.c = the c th covariate, including the Group, period, and G*P terms

8/13/2019 An Cova 2000

68/81

8/13/2019 An Cova 2000

69/81


ANOVA -- testing for statistical significance

A statistical F-test is performed to determine if thereis a significant difference between groups, betweenperiods, and a group-by-period (G*P) interactionterm.

The value of F is calculated as follows:F = SS effect / degrees of freedom

SS error / degrees of freedom

The smaller the error term, the larger the F value,and the more likely that the effect will be significant.

8/13/2019 An Cova 2000

70/81


Assumptions in our use of ANCOVAThe independent variables are measured without error. pretty safe

The relationship between sales and each covariate is linear. We know this isnt true. For example, the relationship betweensales and price is not linear, but curvilinear. But we use linearregression because the outputs are more easily interpreted forevaluation of tests.

The variables are distributed normally.

We know this isnt true: weekly sales per store, for example, aredeal-driven. But we do make sure the residual error is randomlydistributed.

The model has been appropriately specified, including all covariatesthat are relevant and no incorrect covariates.The independent variables are not highly correlated with one another

These are both up to the analyst to ensure in the building of themodel .

The variance of errors at all values of X is constant (homoscedasticity) We can check for this by . . .

8/13/2019 An Cova 2000

71/81


Do we ever use ANOVA for test analysis? ANOVA is just ANCOVA without any covariates.There are almost always some covariates that we willwant to adjust for.In the rare instance where there are no covariates toadjust for, we would use ANOVA to determine whether the

pre-post, test-control difference in the raw sales means isstatistically significant. Example 1: A No Couponing test vs. Couponing control,

where the coupon $s in the test cell are being reallocated totrade spending.

Example 2: A category shelf reset test in a national retailerwhich has a high degree of control and standardization inpricing and merchandising by store, such that there wouldntbe an differences of that sort to adjust for.

8/13/2019 An Cova 2000

72/81


Example: Output of the final model -- Panel

Period Conf LevModel Name Group Name Pre Test Diff %Effect 1T 2TADJ PURVOL1 Test Cell 3.11 3.55 0.44

Ctrl Cell 3.62 3.12 -0.50 30% 93% 86% +0.94

80% CL 2T** 90% CL 2T Adjusted Lower Upper Lower Upper

Mean* Bound Bound Bound Bound ADJ PURVOL1 0.94 0.04 0.56 -0.03 0.63

read as 4% - 56%, -3% to 63%

* This is really the Diff of Diffs from the previous section.** The +/- 26 point confidence interval around the 30% es timate of the test effect sugges ts you might have been able to improve the model with more/different covariates. Also, try a 1T test and a lower CL.

(including all important covariates)

Means

8/13/2019 An Cova 2000

73/81


Example: Impact of Covariates -- PanelDependent Impact

Variable Covariate on Effect Coefficient P-ValueVOLA1 ALOYAL1 - 0.05 + 5.66 .02

FMOV1 +0.13 + 0.17 .02 PCTCOU1 +0.03 +11.39 .02

DOD T,Pre T,Post C,Pre C,Post ALOYAL1 +0.009 0.116 0.105 0.220 0.201 FMOV1 - 0.734 20.47 18.93 19.15 18.33 PCTCOU1 - 0.002 0.042 0.017 0.048 0.026 (numb ers have been rounded)

Interpretation: One minus the P-value = the significance of this covariate. Each of these covariates is

significant at the 92% CL. The coeff icient of each covariate is correctly signed. The test celhad an "unfair" advantage on ALOYAL (+0.009 Diff of Diffs); ALOYA L has a positivecoefficient, indicating that an increase in ALOYAL w ill increase sales; so the impact w as adow nw ard adjustment on the test effect.

l f h f l d l k

8/13/2019 An Cova 2000

74/81


Example: Output of the final model -- Store/Market

RAW MEANDependent Grp 1-Test Grp 2-Ctrl DiffVariable Base Test Base Test EffectVOLA1 0.84 0.79 1 .03 0.83 18

____ ____ _____

ADJUSTED MEAN Conf. LevelDependent Grp 1-Test Grp 2-Ctrl DiffVariable Base Test Base Test Effect* 1T 2TVOLA1 0.75 0.71 0.98 0.81 16 99 99

____ ____ _____

Dependent Adjusted 80% C.I. (2T)Variable Mean* Lower UpperVOLA1 0.16 0.10 0.22 (+/- 4 to 6 points)

* These are the same -- the adjus ted % tes t effect.

Summary of Raw and Adjusted Effects(including all important covariates)

8/13/2019 An Cova 2000

75/81


Example: Impact of Covariates -- Store/Market

Dependent Relative EstimateVariable Covariate Impact Impact CoefficientVOLA1 PBFE1 -1.05 -0.0048 0.0024

PBDI1 -0.09 -0.0004 0.0044 PBFD1 -0.26 -0.001 0.0126

G1,P1 G1,P2 G2,P1 G2,P2 DOD T,Pre T,Post C,Pre C,Post P-valuePBFE1 2.0 8.3 9.0 11.1 9.8 .0001

PBDI1 0.1 0.9 0.7 2.0 1.7 .0001 PBFD1 0.1 0.3 0.3 0.6 0.5 .0003

Inte r pr eta t ion : PBFE1 = Percent of Base volume on FEature only, Test Brand. PBDI1 - Disp lay only,

PBFD1 - Feature+Disp lay. An increase in Feature has a positive effect on sales (coefficient+0.0024). The test cell had an advantage on Feature, and thus there is a negative IMPACT.Relative impact = impact of this covariate as a % of the sum of all impacts.

8/13/2019 An Cova 2000

76/81


Example: Output of the vanilla model for an MMT

(including only one covariate)

Output of General Linear Models (PROCGLM) Procedure

Dependent Variable: VOLA1 - Brand volume per $MM ACVWeight: STRMKTWT (store market weight)

Source DF Type III SS Mean Square F Value Pr > FGroup *Period 1 167.5 167.5 2.90 0.0887

Pr > F = probability that the unadjusted test effect is significant (greaterthan the F). One minus Pr = the confidence level: 1 - .0887 = 91% CL

8/13/2019 An Cova 2000

77/81


QUESTIONS?

8/13/2019 An Cova 2000

78/81


The Bobs Bologna Case Bobs Bologna was considering doubling their adweight.Bob selected two representative markets in which totest his heavy-up media plan.

A pair of matching markets were selected to serve ascontrol

After twenty-four weeks, no difference between cellswas seen.Further investigation, in the form of an ANCOVA, wasnecessary

8/13/2019 An Cova 2000

79/81


Bobs Bologna Case -- continued

Pre-TestMean Post-TestMean Change Diff ofDiffs % TestEffect Conf.Level

Raw Test 18.7 21.2 2.5 0.1 0.5% 54%

Raw Ctrl 19.8 22.2 2.4 Adj Test 19.3 23.1 3.8 0.6 2.7 % 82%

Adj Ctrl 20.1 23.3 3.2

8/13/2019 An Cova 2000

80/81

8/13/2019 An Cova 2000

81/81

Bobs Bologna Case -- continued

In the raw data, the treatment was: advantaged by the imbalance in features disadvantaged by the imbalance in displays

Adjustments were made accordingly: a small negative impact for features a large positive impact for displays

Conclusion: After adjusting for promotionalimbalances, the additional advertising was shown tostatistically significantly provide incremental sales.

An Cova 2000

Documents

Transcript of An Cova 2000