2017 Celebra,ng 10 Years of Shaping the Future of ......Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA...
Transcript of 2017 Celebra,ng 10 Years of Shaping the Future of ......Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA...
…thankstothegeneroussupportof:
2017Celebra,ng10YearsofShapingtheFutureofCorporateCommunica,oninNorway.
10./11.November2016,BI,Oslo
LatestDevelopmentsinEvalua,onofCommunica,onBreakthroughsatLast!
JimMacnamaraPhD,FAMI,CPM,FAMEC,FPRIAProfessorofPublicCommunica@on,UniversityofTechnologySydneyVisi@ngProfessor,LondonSchoolofEconomicsandPoli@calScience
Thispresenta,on
§ Recapsomefundamentals• Keyprinciples• Thebasisofevalua@onframeworksandmodels–wheredotheycomefrom?
§ Newframeworksandmodelsforevalua@on• EuropeanCommissionevalua@onframework• USTaskForceonstandards• UKCabinetOfficeevalua@onframework• AMECIntegratedEvalua@onFramework• NSWGovernmentEvalua@onFramework
§ Ataxonomyofevalua@on–stages,steps,metrics,andmethodstoapplyinprac,ce§ Casestudies–bestprac@ceevalua@onapplied
• GoalseRngandmeasurementarefundamentalforcommunica,onandPR
• Measuringcommunica,onoutcomesisrecommendedversusonlymeasuringoutputs
• Theeffectonorganiza,onalperformancecanandshouldbemeasuredwherepossible
• Measurementandevalua,onrequirebothqualita,veandquan,ta,vemethods
• AVEsarenotthevalueofcommunica,ons
• Socialmediacanandshouldbemeasured
• Measurementandevalua,onshouldbetransparent,consistentandvalid
Measurementvs.evalua,on
MEASUREMENT
• Thetakingofmeasures• Thecollec@onandanalysisofdatainrela@ontoa
par@cularobject,process,orcondi@on• SOWHAT?
EVALUATION
• “Makingajudgement”aboutthevalueorsignificanceofsomething(OxfordandMerriam-Websterdic@onaries)
• Evalua@onisthesystema@cacquisi@onandassessmentofinforma@ontoprovideusefulfeedbackaboutsomeobject”(Trochim,2006,para,3)
• “Thesystema@capplica@onofresearchprocedurestounderstandtheconceptualiza@on,design,implementa@on,andu@lityofinterven@ons(Valente,2001,p.106)
• Assessingresultsagainstobjec,ves
SMARTobjec,ves
§ Containnumbers,percentages,dates
§ Haveameasurementplanforeachstage
§ Berealis,c§ Linkedtoorganisa,onalobjec,vesandgoals§ Achievedbyaspecificdate
Threetypesofevalua,on
Forma@ve
Process
Summa@ve
• Baseline(e.g.,whatisthecurrentlevelofawareness)
• Precedents(haveothersimilarprogramsworkedinthepast?)
• Pre-tes,ng(willwhatweproposework?)
• CEA(isitthemostcost-effec@veapproach?)
• Insights/learning(whathasbeenlearnedthatcaninformfuturestrategy,policy,etc?)
• Impact(whathashappenedasaresultofthecommunica@on?)
• Outcomes–intermediateandlong-term(whatadtudeorbehaviourchangehasoccurred?)
• Monitoringandtrackingoutputs(e.g.,arewereachingtheaudience?)
• Monitoringandtrackingimmediateoutcomes(e.g.,whatresponsearewegedng?
Informa,onprocessing/communica,ontheory
§ Sixstagesofcommunica@on(W.J.McGuire,1968,1969)
ComprehensionPresentation Change/actionRetentionAttention Acceptance
Adver,singmodelsandthe‘marke,ng/salesfunnel’
§ TheAIDAmodelofadver@sing(Strong,1925) AWARENESS
INTEREST
DESIRE
ACTION
Informa,onprocessing/communica,on§ Exposure§ Aben,on§ Interestand/orliking§ Comprehension§ Cogni@on,par@cularlycogni@veelabora@on(thinkingaboutthemessage)§ Acquiringskillsorknowledgerequiredtodealwiththeissue(ifnecessary)§ ARtudechange,par@cularlytoagreement(whatMcGuirecalls“yielding”)§ Storinginforma@oninmemory(reten,on)§ Retrievinginforma@on(i.e.,recall)§ Decidingtoactinaccordancewithinforma@on(inten,on)§ Ac,on/behaviour§ Cogni@veintegra@onofbehaviour(e.g.,reinforcement)§ Encouragingotherstobehavesimilarly–whatMcGuirecalled“prosely@zing”andwhatiscommonly
regardedinmodernmarke@ngas‘advocacy’(McGuire,1999,2001)
ThePIImodelofevalua,on(Cutlip,Center,&Broom,1985)
PyramidmodelofPRresearch(Macnamara,2002,2005)
OUTCOMES(Functional & organisational evaluation)
OUTPUTS(Process & programevaluation)
INPUTS(Formativeresearch)
Number who . change ..…..……………………………………………….. Quantitative surveys (large scale structured)behaviour …...…………………………………………….. Sales; Voting results; Adoption rates; Observation
Number who ….. changeattitudes ……….…………………………………………… Focus groups; Surveys (targeted) (eg Customer, Employee
Number who understand messages..……………………………… Focus groups; Interviews; Complaint decline; ExperimentsNumber who retain messages …….………………………………… Interviews; Focus groups; Mini-surveys; Experiments
Number who consider messages ……....….. ………………………… Response mechanisms (1800, coupons); InquiriesNumber & type of messages reaching target audience ……………………… Media Content Analysis; Communication Audits
Number of messages in the media ……………..….`…………………… Media Monitoring (clippings, tapes, transcripts)Number who received messages ………………...……………………….. Circulations; Event attendances; Web visits & downloads
Number of messages sent …………………………………..……………….. Distribution statistics; Web pages posted
Quality of message presentation …………………………………………………. Expert analysis; Peer review; Feedback; Awards
Appropriateness of message content …………………………………….………….. Feedback; Readability tests (eg. Fog, Flesch); Pre-testing
Appropriateness of the medium selected ………………………………………….…….. Case studies; Feedback; Interviews; Pre-testing (eg. PDFs)
How does target audience prefer to receive information? …………….…………..……… Academic papers; Feedback; Interviews; Focus groupsWhat does target audience know, think, feel? What do they need/want? ………………… Observations; Secondary data; Advisory groups; Chat rooms
& online forums; Databases (eg. Customer complaints)
or Shareholder Satisfaction); Reputation studies
Measurement Methodologies: (formal & informal)
Key Steps/Stages in Communication:
OUT-TAKES(Proposed by some as a 4th stage)
PReffec,venessyards,ck(Lindenmann,1993)
Measuring:BehaviourchangeAttitudechangeOpinionchange
ADVANCED
Measuring:RetentionComprehensionAwarenessReception
INTERMEDIATE
Measuring:TargetaudiencesImpressionsMediaplacements
OUTPUT
LEVEL#3
LEVEL#2
LEVEL#1
PUBLICRLEATIONSEFFECT
IVEN
ESSYARD
STICK
OUTCOMESOUTGROWTHSOUTPUTS
Noble&Watson’sunitedmodel(Noble&Watson,1999)
INPUTSTAGEPlanning&Preparation
OUTPUTSTAGEMessages&Targets
IMPACTSTAGEAwareness&Information
EFFECTSTAGEMotivation&Behaviour
Tacticalfeedback
Managementfeedback
Communica,oncontrolling(DPRG/GPRA,2000;DPRG/IPV,2009)
ResultsofCommunicationProcesses
Output
Outcome
Outflow
InternalOutput
ProcessEfficiencyQualityofWorks
Product
BudgetComplianceThroughputTimesNo.ofShortcomings
Readability/Fogg-IndexSatisfactionofInternal
Clients…
ExternalOutput
MediaCoverageContent
ClippingsVisits
DownloadsImpactRatioShareofVoice
…
DirectOutcome
PerceptionUtil izationKnowledge
AwarenessUniqueVisitorsSessionLengthReaderperIssue
RecallRecognition
…
IndirectOutcome
OpinionAttitudesEmotion
BehavioralDispositionBehavior
ReputationIndexBrandImage
Strategic AwarenessofEmployees
PurchaseIntentionLeads
InnovativeIdeasProject Participation
…
RessourcesEmployeeAssignmentFinancialExpenses
PersonnelCostsOutsourcingCosts
…
Input
ValueCreation
ImpactonStrategicand/orFinancialTargets(ValueChain)Impacton
Tangibleand/orIntangibleAssets
(CapitalAccumulation)
SalesNo.ofProjectAgreements
CostReductionReputationCapital
BrandValueEmployeePerformance
…
Outflow
ORGANISATION
CommunicationProcessesInitiationofCommunicationProcesses
MEDIA/CHANNELS STAKEHOLDERS ORGANISATION
HighImpactonValueCreationWeakInfluenceofCommunicationsManagement
LowImpactonValueCreationStrongInfluenceofCommunicationsManagement
■ The European Commission’s Better Regulation Guidelines uses the terms: ― Inputs ― Outputs ― Results ― Impact (EC, 2015a)
EuropeanCommission(EC,2015a)
10politicalpriorities
Communicationobjectives
Activities Relevance Output Outtake Outcome
Recallindicatorsmeasuringtheextent
towhichthecommunicationactivityislikelytohavesatisfied,
capturedtheattentionofaudiences,orraised
knowledgeandawarenessfocussingontheaudiencedirectlyreached
PoliticalmatchOfthecommunicationactivitiesundertakenwiththe10political
priorities
• Organisingevents• Providing
information• Workingwith
informationnetworksanddesks
Reach indicatorsmeasuringtheextenttowhichextentthecommunicationactivityislikelyto
havereachedtherighttargetaudiencequantitativelyand
qualitatively
EngagementTrustindicatorsmeasuringtheextenttowhichthecommunicationactivityledtoeitheradiscernibleactionbeingtakenorthe
desiredchangeinthetargetaudience’s
perception
InputsResults
Didtheactivitymatchthe10priorities?
Didtheactivityreachthetarget?
Wasthemessagereceived?
Didtheactivitymeettheexpectationsofthetargetaudience?
Didtheactionleadtotheexpectedresult?
DidtheactionchangetheperceptionoftheEU?
Q.
EuropeanCommissionevalua,onmodel(EC,2015b)
PRevalua,onlandscape
§ Inputs/prepara,on§ Ac,vi,es/implementa,on
§ Outputs§ Oubakes§ Outcomes/ouelows/outgrowths
§ Results/effects/impact
“Icouldn’tunderstandwhyPRneededtostandapartfromthedominantapproachtoperformancealignmentandmanagementadoptedbythemajorityoffunc@onsonemightfindinatypicalorganiza@on”
PhilipSheldrakeTheMeasurementStandard,December8,2015,para.6
Programtheoryandprogramlogicmodels
§ Basicprogrammodelevolvedsincethe1970s(KelloggFounda@on,2004,p.1).
Programtheoryandprogramlogicmodels
§ ProgramlogicmodeldevelopedbytheUniversityofWisconsinUniversityCoopera@veExtensionProgram(UWEX)(Taylor-Power&Henert,2008,p.5)
ProgramandPRevalua,onmodels(1985–early2000s)
Basicprogramlogicmod
el–
UWEX
(Taylor-Po
wer&Hen
ert,
2008)
Expa
nded
program
logicmod
el
–UWEX
(Taylor-Po
wer&
Hene
rt,200
8)
ClassicProgramLogicM
odel
(e.g.,KelloggFou
nda,
on
(1998/2004)
Cutlip,Cen
ter&
Broom
(1985)
‘PIImod
el’
Macna
mara’sM
acro
mod
el’(1992)
Lind
enman
n’sE
ffec,vene
ss
Yards,ck(1
993,1997a)
IPRA
GoldPa
pero
nEvalua
,on
(1994,pp.10,18–19)
Fairc
hild(1
997,2001);Fairchild
&O’Con
nor,IPRTo
olkit(1999)
Lind
enman
nStan
dardsa
nd
Guide
lines,199
7b,199
7c)
Nob
le&W
atson’s‘Unifie
dMod
el’(1999)
Grunig&Hon
‘rela,
nships’
mod
el
DPRG
/GPR
A(2000),D
PRG/ICV
(2009)m
odels
LikelyPerform
ance
Measuremen
tFramew
ork
(2000)
Macna
mara’s‘Pryamid’m
odel
(2000,2002a,2002b
)
Lind
enman
n’s
‘guide
lines'(2002/2003)
Input Inputs Inputs Inputs Inputs Inputs(hinted)
Input Input Inputs
Ac@vi@es Planning
Output Outputs• Ac@vi@es• Par@cipa@on
Outputs Implement-a@on
Outputs Outputs(basic)
Outputs Output Outputs Output Outputs Output Outputs Outputs PROutputs
Ouqake Ouqakes Ouqakes PROuqakes
Outgrowths
(inter-mediate)
Outcomes Outcomes• Short-term• Intermediate• Long-term
Outcomes Outcomes(advanced)
Outcomes Outcome Outcomes Outcomes Outcome Outcomes Outcomes PROutcomes
Impact Impact Results Business/organiza@onoutcomes
Impact Rela@onships Ourlow Outgrowths Business/organiza@onoutcomes
Effect
9
144
112
51
(Macnamara,2016)
TheoryofChange
INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT
Communica@on• Objec@ves• Targetaudiences• Strategy
Ac@ons
Short|Medium|Longterm
TheOrganisa@onOrganisa@ongoalsandobjec@ves
Feedbackloopstomonitorandadjuststrategyandtac>csifrequired
ProgramTheory
INPUTSACTIVITIESOUTPUTSOUTCOMESOUTCOMESOUTCOMESShorttermIntermediateLongtermOUTTAKESOUTCOMESIMPACT
Communica@on• Objec@ves• Targetaudiences• Strategy
Feedbackloopstoplanning
Prepara@onProduc@onDistribu@on/Exposure/Recep@onResponseEffectsResultsOrganisa@on/StakeholderImpact
ORGANISATIONOrganisa@ongoalsandobjec@ves
TheoryofChangeProgramTheory
Whatyoudo(COSTCENTRE) Whatyouraudiencedoes(VALUECENTRE)
UKGovernmentCommunica,onServiceevalua,onframework(GCS,2015)
AMECIntegratedEvalua,onFramework(AMEC,2016)
NHSBloodandTransplantblooddona@oncampaign
INPUTSACTIVITIESOUTPUTSOUTCOMESOUTCOMESOUTCOMESShorttermIntermediateLongtermOUTTAKESOUTCOMESIMPACT
Communica@on• Objec@ves• Targetaudiences• Strategy
Feedbackloopstoplanning
Prepara@onProduc@onDistribu@onExposure/Recep@onResponseEffectsResultsOrganisa@on/StakeholderImpact
ORGANISATIONOrganisa@ongoalsandobjec@ves
TheoryofChangeProgramTheory
??????
Taxonomyofevalua,on–6-stagemodel
InputsAc@vi@esOutputsOuqakesOutcomesImpact
TheoryofChange
INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT
Communica@on• Objec@ves• Targetaudiences• Strategy
Ac@ons
Short|Medium|Longterm
TheOrganisa@onOrganisa@ongoalsandobjec@ves
Feedbackloopstomonitorandadjuststrategyandtac>csifrequired
ProgramTheory
INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT
TheOrganisa@onOrganisa@ongoalsandobjec@ves
• Budget• Personnel• Exper@se• Otherresources
• Paidadver@sing• Mediapublicity• Publica@ons(e.g.,
newsleEers,reports)• Websites• Socialmediaposts• Events• Sponsorships• Communityprojects
• Reputa@on• Rela@onships• Publicsupport• Sales/revenue• Dona@ons• Costsavings• Staffreten@on• Customerreten@on• Complyingbehaviour
(e.g.,giveblood;drivesafely,getfit)
• Qualityoflife/well-beingimprovements
Prepara@onProduc@onDistribu@onExposure/Recep@onResponseResults
Affec@veCogni@veCona@veAc@ons
Short|Medium|Longterm
• Recall• Awareness• Interest(e.g.,follows,
likes,retweets)• Engagement• Sa@sfac@on• Adtudechange• Trust• Preference• Inten@ons• Inquiries/leads• Advocacy
Feedbackloopstomonitorandadjuststrategyandtac>csifrequired
• Forma@veresearch• Crea@vedesign• Pre-tes@ng• Planning• Produc@on• Mediarela@ons
Communica@on• Objec@ves• Targetaudiences• Strategy TheoryofChange
ProgramTheory
INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT
TheOrganisa@onOrganisa@ongoalsandobjec@ves
• Budget• Personnel• Exper@se• Otherresources
• Forma@veresearch• Crea@vedesign• Pre-tes@ng• Planning• Produc@on• Mediarela@ons
• Paidadver@sing• Mediapublicity• Publica@ons(e.g.,
newsleEers,reports)• Websites• Socialmediaposts• Events• Sponsorships• Communityprojects
Stakeholders,Publics,Society
Feedbackloopstomonitorandadjuststrategyandtac>csifrequired
• Sales/revenuedata• Behaviourtracking• CRMdata• Surveys• Costbenefit/ROI• Well-beingmetrics
• Surveys(awareness,reputa>on,trust)
• Interviews• Socialmediaqualanalysis(likes,shares,comments)
• NetPromoterScore
• Mediametrics(reach,impressions,OTS,TARPs,CPM)
• Contentanalysis(e.g.,messages,tone/sen>ment)
• Websitedata• Socialmediastats• Abendancefigures• Readersurveys
• Pre-surveys• Focusgroups• Interviews• Baselinedatacollec,on(e.g.,databases)
• Pre-tes,ng• Peerreview/expertreview
EVALUATIONMETHODS
Prepara@onProduc@onDistribu@onExposure/Recep@onResponseResults
Short|Medium|Longterm
Contextualfactors(e.g.,resourcelimita@ons)
Contextualfactors(e.g.,compe@@vem
essages)
Contextualfactors(e.g.,cogni@vedissonance)
Contextualfactors(e.g.,financialrecession)
• Recall• Awareness• Interest(e.g.,follows,
likes,retweets)• Engagement• Sa@sfac@on• Adtudechange• Trust• Preference• Inten@ons• Inquiries/leads• Advocacy
• Reputa@on• Rela@onships• Publicsupport• Sales/revenue• Dona@ons• Costsavings• Staffreten@on• Customerreten@on• Complyingbehaviour
(e.g.,giveblood;drivesafely,getfit)
• Qualityoflife/well-beingimprovements
Communica@on• Objec@ves• Targetaudiences• Strategy TheoryofChange
ProgramTheory
Measurementandevalua,on(tradi@onalmodel)
Measurement
Evalua@on
Early findings used to fine-tune campaign
Data collection, data analysis Identifying value
Findings primarily used for reporting
TheMAIEModel
Measurement
Early findings used to fine-tune campaign
Inform strategic planning of next stage
INSIGHTS Future business/organisation strategy
Reporting and performance management and review
Data collection, data analysis
Identifying value Evalua@on
1
2 Analysis
Casestudy:Youthan,-cannabiscampaign
Ques,ons&Discussion