Measuring the Impact of Measuring the Impact of Entertainment-Education Entertainment-Education

ProgramsPrograms

D. Lawrence Kincaid, Ph.D. Center for Communication ProgramsBloomberg School of Public Health

Johns Hopkins University

44thth International Entertainment-Education Conference International Entertainment-Education ConferenceSeptember 26-30, 2004 at the Lord Charles Hotel September 26-30, 2004 at the Lord Charles Hotel

in Somerset West, South Africa. in Somerset West, South Africa.

OverviewOverview

• Between a “rock and a hard place” . . . without a paddle• Association or cause and effect? • From experimental control by

means of research designs• To 8 conditions for causal

attribution

The biggest limitation to the The biggest limitation to the evaluation of communicationevaluation of communication

• Members of the population who watch the Members of the population who watch the program are program are alwaysalways different from those different from those who do not watch. who do not watch. (Self-selection bias)(Self-selection bias)

• These differences are usually related to These differences are usually related to the expected outcomes. the expected outcomes. (behavior, etc.)(behavior, etc.)

• So, a simple comparison of outcomes by So, a simple comparison of outcomes by exposure is exposure is biasedbiased by the differences by the differences between those exposed and not exposed.between those exposed and not exposed.

The Counter-Factual DilemmaThe Counter-Factual Dilemma

• How much does a communication program change the behavior of audience members who are exposed to the program compared to what they would have experienced if they had not been exposed?

• Problem: A perfect solution would require a parallel universe. Used only as an ideal.

• Randomized experimental design cannot be used for full-coverage programs, so the problem has to be solved with measurement and theory.

Solutions for Full-Coverage Solutions for Full-Coverage ProgramsPrograms

1. 1. Statistical ControlsStatistical Controls: Use multiple regression : Use multiple regression to control for all confounding variables that to control for all confounding variables that might affect exposure and the outcome. might affect exposure and the outcome.

2. 2. Simple MatchingSimple Matching: Limited to 1-3 variables. : Limited to 1-3 variables.

3. 3. Propensity Score MatchingPropensity Score Matching: Unlimited : Unlimited number of variables that might affect number of variables that might affect exposure and the expected outcomes. exposure and the expected outcomes.

4. 4. Theory-based evaluationTheory-based evaluation

• The goal is to estimate how much change The goal is to estimate how much change can be attributed to communication.can be attributed to communication.

• Establish the criteria for measuringEstablish the criteria for measuring how much impact can be causally how much impact can be causally attributed to your program. attributed to your program.

• Obtain empirical evidence to support Obtain empirical evidence to support each criterion.each criterion.

• Draw an appropriate conclusion.Draw an appropriate conclusion.

A Practical AlternativeA Practical Alternative

The South African TV Drama, Tsha TshaThe South African TV Drama, Tsha Tsha

1.1. No. of characters audience can No. of characters audience can correctly name from photographs (k=4)correctly name from photographs (k=4)

2.2. No. of correct items of knowledge about No. of correct items of knowledge about the drama (k=8)the drama (k=8)

3.3. No. of specific episodes they watched No. of specific episodes they watched that they can identifythat they can identify (k=26)(k=26)

First, how do you know who was really exposed? (implementation)

What is the name of the character What is the name of the character shown in each photo below?shown in each photo below?

1.1. Who is studying for a UNISA degree?Who is studying for a UNISA degree?2.2. Who steals money from Viwe’s father?Who steals money from Viwe’s father?3.3. Who is Joy’s father?Who is Joy’s father?4.4. What gift does Viwe give to Andile?What gift does Viwe give to Andile?5.5. Who gets a sexually transmitted infection?Who gets a sexually transmitted infection?6.6. What item does Unathi sell to get money for What item does Unathi sell to get money for

food? food? 7.7. Who gives Mimi R1,000 for her business?Who gives Mimi R1,000 for her business?8.8. Who does Cedric send to Johannesburg to Who does Cedric send to Johannesburg to

deliver dagga? deliver dagga?

Knowledge of the Drama Independent of Knowledge of the Drama Independent of the Expected Outcomes the Expected Outcomes (unaided)(unaided)

1. Prince opens a hair salon and interviews girls in 1. Prince opens a hair salon and interviews girls in the townthe town

2. Mimi and Andile have sex, and she is scared he 2. Mimi and Andile have sex, and she is scared he will wake her grandmother will wake her grandmother

..

..13. Funeral of Andile’s mother13. Funeral of Andile’s mother. . ..26. A party is organized for Andile to dance in Joburg 26. A party is organized for Andile to dance in Joburg

Number of the 26 episodes watchedNumber of the 26 episodes watched

Mark which episodes you have seen:

0

.1.2

.3D

ensi

ty

40 50 60 70Recall of Tsha Tsha Round 3 (episode3 + knowtsh3 std)

Distribution of the Index of Recall with Non-viewers

A Continuous Measure of Recall A Continuous Measure of Recall of Tsha Tshaof Tsha Tsha

0.0

2.0

4.0

6D

ensi

ty40 50 60 70

Recall of Tsha Tsha Round 3 (episode3 + knowtsh3 std)

Distribution of the Index of Recall

32

68

0

20

40

60

80

No Yes

Watched the Drama

Percent

Breaking recall into levels Breaking recall into levels for the analysis of impactfor the analysis of impact

32 34 34

0

20

40

60

80

None Low High

Level of Recall

Percent

1. Did the outcome change over time?1. Did the outcome change over time?

8 10

0

10

20

30

40

Wave 1 2003 Wave 3 2004

Abstinence for a Year or More

Percent

N = 754; not statistically significant2% of 10,000,000 = 200,000 youth

5. Was the observed change5. Was the observed change large and abrupt? large and abrupt?

8

28

910

33

17

0

10

20

30

40

Abstinence for aYear or more

Faithful to MyPartner *

Abstinence for aMonth or more *

PercentWave 1 Wave 3

Note: A one-year time interval* Statistically significant: p<0.01

2a. Was the change associated with 2a. Was the change associated with

exposure to communication? exposure to communication?

17

40

0

10

20

30

40

50

Not Exposed Exposed

Faithfulness to My Partner

Percent

* Statistically significant: p<0.01

2b. Was the change associated 2b. Was the change associated with exposure to communication?with exposure to communication?

Pe

rce

nt A

gre

em

en

t

Regression of AIDS attitude on level of recall (r=.32)Recall of Tsha Tsha, Round 3

39 50 72

.5

1 Mean Percent

3a. Did exposure occur before the 3a. Did exposure occur before the observed change? observed change?

Chi-square = 5.63; p<0.05

Decided to Remain Faithful to My Partner

No Yes Total

No 405 140 545 Yes 103 105 208

Total 508 245 753

Wave 3

Wave 1

Net Increase = 37

3b. Did exposure occur before the 3b. Did exposure occur before the observed change? observed change?

70

135

12

46

14 18 22

0

20

40

60

80

Stayed No Yes to No StayedYes

New Yes

PercentUnexposed Exposed

N=753; Chi-square = 46.8, p<.001

Decided to Remain Faithful to My Partner

-.01

.09

.13

.13

.11.26

.33

.12.09

.31

Learned about AIDS on TV

Frequency of TV viewing

Recall of the Drama

AIDS Attitude

Female

Lagged AIDS attitude (wave 1)

Education

Kwazulu provinceresidence

4a. Multiple Regression to Estimate the 4a. Multiple Regression to Estimate the IndependentIndependent Effect Effect of Exposure after Controlling for Other Influences *of Exposure after Controlling for Other Influences *

South Africa, 2004South Africa, 2004

* Including lagged attitude means that the impact of other variables is on change in attitude.

73

9

82

0

20

40

60

80

100

TreatmentGroup

MatchedControlGroup

NetDifference

Percent Variables Used to Construct thePropensity Score for Matching:

1. Age

2. Education

3. Gender

4. Income Level

6. Lagged Aids Attitude

7. Learned about AIDS on TV

8. Province (KwaZulu Natal)

4b. Propensity Score Matching to Estimate Effects 4b. Propensity Score Matching to Estimate Effects of Exposure of a Drama on AIDS Attitudes of Exposure of a Drama on AIDS Attitudes

Wave 3 South Africa, 2004Wave 3 South Africa, 2004

Positive AIDS Attitude

6. Is there evidence of a dose 6. Is there evidence of a dose response? response?

6873

78

60

70

80

90

100

None Low High

Level of Recall

Mean percent agreement

AIDS Attitude by Level of Recall

7. Is the causal inference justified theoretically?

What is a theory?

• How communication works.

• A tool for thinking and action.

Theory-Based EvaluationTheory-Based Evaluation

• Identify causal pathways Identify causal pathways (mediating variables for effect)(mediating variables for effect)

• Demonstrate:Demonstrate:• Impact of Impact of communicationcommunication on on

mediating variablesmediating variables• Impact of Impact of mediating variablesmediating variables on on

behavioral outcome behavioral outcome

COMMUNICATION

IDEATIONCOGNITIVE Beliefs Values Perceived Risk Subjective Norms Self-Image EMOTIONAL Emotional Response Empathy Self-EfficacySOCIAL Support & Influence Personal Advocacy

ENVIRONMENTALSUPPORTS & CONSTRAINTS

SKILLS & KNOWLEDGE INSTRUCTION

NONDIRECTIVE Dialogue Counseling Entertainment

Social Networks

PUBLIC Advocacy Regulation

reinforcement

enabling

Attitudes

INTENTION

BEHAVIOR

DIRECTIVE Dissemination Promotion Prescription

Source: Adapted from Kincaid (2000)

confirmation

IDEATION MODEL OF COMMUNICATION AND BEHAVIOR

A predictive model of communication & change

Knowledge

Attitudes

Self-Image

Perceived Risk

Self-Efficacy Norms

Emotions

Social Support & Influence

Personal Advocacy

BEHAVIOR

Implies simultaneous effect of all influences.

Implies communication can effect all influences.

Direct and Indirect Impact of Communication on Behavior

Behavioral Outcome

Mediating Effects

Communication Program

A Path Model of the Effects of Recall of A Path Model of the Effects of Recall of the Drama on Identification with Boniswa the Drama on Identification with Boniswa and AIDS Attitudesand AIDS Attitudes

Identification with a character in the

drama

Level of exposure to the

drama

Attitude towards

HIV/AIDS

direct effects

indirect effects

direct effects

7. Path Model of the Effects of Recall of the Drama on 7. Path Model of the Effects of Recall of the Drama on Identification with Boniswa and AIDS Attitudes at Wave 3Identification with Boniswa and AIDS Attitudes at Wave 3

South Africa, 2004South Africa, 2004

-.01

.09

.13

.13

.11.26

.12

.09

.06

.34.13

.33

.12.09

.31

Watches “Days of Our Lives”

Learned about AIDS on TV

Frequency of TV viewing

Identification with Boniswa

Recall of the Drama

AIDS Attitude

Abstained for a month or more

Female

Lagged AIDS attitude

Education

Kwazulu province

YESYES

7. Is the causal inference justified theoretically?

YESYES

8. Are the results consistent with previous program research?

1. Change over time in the expected 1. Change over time in the expected outcome is observed. outcome is observed.

2. An association between that change and 2. An association between that change and program exposure is observed. program exposure is observed. (correlation)(correlation)

3. Exposure occurs before the observed 3. Exposure occurs before the observed change is measured. (time-order)change is measured. (time-order)

Eight Criteria for Causal Attribution

Eight Criteria for Causal AttributionEight Criteria for Causal Attribution

4. No evidence of confounding variables that 4. No evidence of confounding variables that may have accounted for the change. may have accounted for the change.

5. The observed change is abrupt and large.5. The observed change is abrupt and large. (immediacy and magnitude) (immediacy and magnitude)

6. The impact increases in proportion to level 6. The impact increases in proportion to level

or duration of exposure.or duration of exposure. (Dose response)(Dose response)

7.7. A causal connection is justified.A causal connection is justified.

(causal (causal pathwayspathways and and theoreticaltheoretical coherence) coherence)

8. Consistency with previous program 8. Consistency with previous program research.research. (replication with variation)(replication with variation)

Eight Criteria for Causal AttributionEight Criteria for Causal Attribution

Unfinished WorkUnfinished Work

We can still not adequately measure many of the important effects of EE, especially drama.

1. Expanding the perspective and ways that people make decisions

2. Emotional Involvement (caring, fear) 3. Measuring effects of narrative

Measuring the effects of a storyMeasuring the effects of a story

The Eagle and the Cocks

• Two cocks in the farm yard fought to decide who should be master.

• The loser withdrew to a dark corner, while the victor flew to the roof top and crowed lustily.

• An eagle spied him from high up in the sky, swooped down, and carried him off.

• The other cock came out and ruled without a rival.

PRIDE COMES BEFORE THE FALL.

Aesop’s Fable (620 B.C.)

FIN

Thank you!

Why Programs Fail or Appear to Fail• Inappropriate theory

• Appropriate theory, but inadequate application of the theory to the program

• Appropriate theory, adequately designed, but poorly implemented

• Appropriate theory, adequately designed,

well implemented, but ineffective research

• Appropriate theory, design, implementation,

and research design, but inadequate measurement of the degree of impact

Sources of Program Success

• Appropriate theory

• Correct application of the theory to the design of the program

• Successful implementation

• Adequate evaluation research

• Adequate measurement of impact

Implications

• Five sources are interrelated.

• If not effective, it’s difficult to know why.

• Program may be effective, but the research cannot show it.

• You can only improve if you know what happened and why.

• There are realistic limits to what we can learn from program evaluation.