Measurement 101 A Presentation to the IPR Measurement SummitOctober 2009Katie Delahaye PaineMember, IPR Measurement Commissionwww.instituteforpr.comCEOkdpaine@kdpaine.comwww.kdpaine.com

Why Measure?

“The main reason to measure objectives is not so much to reward or punish

individual communications manager for success or failure as it is to learn from the

research whether a program should be continued as is, revised, or dropped in favor of another

approach ”

James E. Grunig, Professor Emeritus, University of Maryland “If we can put a man in orbit, why can’t we

determine the effectiveness of our communications? The reason is simple and

perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice.

Unpredictable, cantankerous,capricious, motivated by innumerable conflicting

interests, and conflicting desires.”Ralph Delahaye Paine,

Publisher, Fortune Magazine , 1960 speech to the Ad Club of St. Louis

The Ah-Ha moment, the Lotus PR Report

3

0% 20% 40% 60% 80% 100%

TAC

Manuscript

One Source

HAL

Positive Messages No Messages Negative Messages

Percent of impressions containing messages by product

The Ah-Ha moment, the Lotus PR Report

4

Cost per message communicated

$0.00 $0.50 $1.00 $1.50 $2.00 $2.50

TAC

Manuscript

One Source

HAL

Victory Over Martians Confirmed!

Share of exposure over time

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Jan

MarMay Ju

lSep Nov

Jan

MarMay Ju

lSep Nov

Jan

IN

TI

MO

NS

Interviews and media advisories generated best coverage

0% 20% 40% 60% 80% 100%

Application articles

Contract wins

Exec Interview

Media advisory

Release + conference

Press release plus VNR

Product review

Industry issue

Trade show/event

No Message

Negative Message

Positive Message

Comparing the effectiveness of different tactics

Cost per message communicated

$10.00

$50.00

$5.00

$33.00

$50.00

$9.00

$8.00

0 10 20 30 40 50 60

Earned Media (PR)

WSJ

NPR

Direct Mail

Trade Show

Blog

Web Site

Cost per messagecommunicated

Correlation exists between traffic to the ASPCA web site and the organization’s overall media

exposure

-

100,000

200,000

300,000

400,000

500,000

600,000

700,000

0

50,000,000

100,000,000

150,000,000

200,000,000

250,000,000

300,000,000

350,000,000

Web

Sit

e Vi

sito

rs

Expo

sure

Overall Exposure

Web Traffic

Correlations also exist between online donations to the ASPCA and the organization’s overall media

exposure

-

100,000

200,000

300,000

400,000

500,000

600,000

700,000

0

50,000,000

100,000,000

150,000,000

200,000,000

250,000,000

300,000,000

350,000,000

400,000,000

450,000,000

500,000,000

Don

ation

s

Expo

sure

Overall Exposure

Donations

Facebook: Correlating MSM, CGM and signups

Strong correlationNon-negative discussion only

0

100

200

300

400

500

600

700

800

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

8,000,000

9,000,000

10,000,000

Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

2006 2007

User Registrations and Media CoverageDecember 2006-November 2007

New Accounts

FB Mentions

A measurement timeline

10 signs that it’s the end of measurement as we know it

1. BMC Software measures communications effectiveness based on contribution to EPS

2. Procter & Gamble is now paying for engagement, not eyeballs

3. Sodexo measures success of its Twitter program by showing $300K reduction in recruitment costs

4. USO compares PR and PSAs to determine effectiveness of fund raising

5. HSUS measures Social media success by calculating $650,000 in new donations

6. Immunize BC counts calculates awareness + shots delivered

7. The Red Cross measures the effectiveness of Twitter via lives saved and harm avoided

8. IBM can calculate more sales from a $500 podcast than a $40K ad program

9. BestBuy measures 85% lower turnover as a result of its Blue Shirt community

10. Advertisers are starting to admit that all their measures are flawed

What do you need to measure to get to the Holy Grail of ROI measurement? Data.

Outputs?

Did you get the volume of coverage & visibility you wanted?

Did you produce the promised materials on time and on budget?

Outtakes?

Did your target audience see the messages?

Did they believe the messages?

Outcomes?

Did audience behavior change?

Did the right people show up?

Did your relationship change?

Did sales increase?

Goals, Actions and Metrics Goal Action Output Metric Outtake Metric Outcome Metric

Increased on-line donations

Revamp website

Amount of content on web site

% perceiving ASPCA as a reliable source

% increase in web traffi and donations

#1 most trusted source for information on companion animals

Increase staffing and resources for communications

Increased exposure of “trusted source” message

Increased perception of ASPCA as trustworthy and comprehensive source

% increase in agreement with the statement

Website is preferred site for information

Add content, features to web site, keep up to date

% increase in traffic % agreeing with the statement

# 1 rankings, and time spent on site

Integration of department

Reorganize department

More integrated materials produced on time and on budget

Internal perception is “one department”

Consistent messaging throughout

14

The 7 steps to PR ROI

1. Define the “R” – Define the expected results?

2. Define the “I” -- What’s the investment?

3. Understand your audiences and what motivates them

4. Define the metrics (what you want to become)

5. Determine what you are benchmarking against

6. Pick a tool and undertake research7. Analyze results and glean insight,

take action, measure again

Step 1: Define the “R” what return do you expect? s

What were you hired to change? If you are celebrating complete

100% success a year from now, what is different about the organization?If you eliminated your department

or failed utterly, what would be different?

Generates Revenue, Sales, ProfitMarketing Public Relations drives

salesInvestor Public Relations drives

investmentPublic Relations drives donations &

membership for relevant organizations

Drives EfficiencyBetter audience targetingReaching more people with a

credible message for less money

Avoids Catastrophic CostQuality counsel helps to mitigate

impacts of crises

17

How PR Impacts Financial Performance

Increases likelihood to purchase / consider your brand(s)Minimizes the effects of a crisis Reinforces communication of organizational

valuesRebuilds trust after a crisisEstablishes credibility of new products /

companies; ease of market entryCommanding higher prices, lower costs,

premium on stock priceEnhances recommendations / word of mouth

leading to faster adoptionIncreases customer loyalty / renewals /

satisfactionImproves the attracting / retaining of talentLowers legal costs

18

How PR Positively Affects Reputation / Brand Equity

Increases employee satisfaction and engagement, leading to greater efficiency, increased retention, lower turnover rates, lower recruitment costs, and higher productivityLowers legal costsChange employee behaviors such as greater

levels of focus on key areas such as safety, quality, call response timesProvides greater transparency and commitment

to and from employeesCreates a platform should it be necessary to

communicate bad news at some stage in the future

* Note items here can also refer to other internal

publics such as trade association members.19

How PR Impacts Employees

Creates public awareness, understanding, and support for legislation, regulation, and political candidatesAffects voter behaviorHelps pass legislation, regulation,

and initiativesAffects specific companies and

industries through appropriations, tax impacts, and regulatory changes that can affect any and all aspects of a businessInstigates and perpetuates

grassroots or grasstops campaigns

20

How PR Affects Public Policy

Step 2: Define your investment

You can’t divide by $0People timeOpportunity CostExecutive time/goodwill

Step 3: Understand what motivates your stakeholders and priorities the stakeholders

What motivates customers to purchase, members to join, students to apply, etc. What is your role in the

outcome?Prioritize based on benefits

Step 4: Define your Key Performance Indicators (KPIs)

23

The Perfect KPIGets you where you want to

go (achieves corporate goals)

Is actionableContinuously improves your

processesIs there when you need it

Step 4: Define your Key Performance Indicators (KPIs) carefully because you become what you measure

Cost savingsEfficiency

Cost per message communicatedCost per new

lead/customer acquired

Productivity: Increase in employee

engagement/moraleLower

turnover/recruitment costs

Engagement: Ratio of posts to

comments% of repeat visitors% of 5+min visitors% of registrations

Trust:Improvement in

relationship /reputation scores with customers and communities (Loyalty/Retention)

Thought leadership: Share of quotesShare of opportunities

Message penetrationPositioning on key issuesImprovement in

favorable/unfavorable ratioImprovement in Optimal

Content Score (OCS)

24

Step 4: Potential KPIs for Media Relations

Cost Per Message CommunicatedIncrease in % of discussion containing key messages% increase in Optimal Content Score Share of discussion vs. the competitionShare of brand visibility vs. the competitionShare of spokesperson visibility vs. the competitionShare of desirable coverage (positive+neutral coverage) vs. the competitionShare of negativesShare of favorable positioning on key topics/battles Share of mentions by key media (including A-list bloggers)

Potential KPIs for thought leader visibility

Our share of quotes in mass mediaOur share of papers in scientific/academic/trade journals Our leaders appointed to key professional boards over time Our share of papers at key conferences% awareness among “influentials”Total opportunities to see our message via speaking engagements

Potential KPIs for Community Relations

Ratio between positive and negative press in local media % of articles in local media that contain our key messages% improvement in relationships scores between us and local community and those that influence the local community.

Step 5: Define your benchmarks

Existing benchmarksPast PerformancePeer companiesWhatever keeps the C-suite up

at nightThink 3-5:

A stretch goalThe underdog who’s nipping at your heelsPeer organizationsAnyone that you compete with for share of mind or share of wallet

Past performance: tonality of blog content

Tonality of Coverage Over Time

4 9 5 9

27

37 43

91

17

914

12

0

25

50

75

100

125

150

Oct Nov Dec Jan

2006 2007

Men

tions

Positive

Neutral

Negative

Tonality of all blog postings

Total, 10%

Total, 71%

Total, 19%

The competitive landscape

Technorati mentions with high authority

Cingular7%

Sprint7%

Verizon10%

T-Mobile75%

US Cellular1%

Company "sucks" mentions in Technorati with high authority

US Cellular2% Cingular

16%

Sprint12%

Verizon19%

T-Mobile51%

Non-Profit industry benchmarks in social media % of Desirable Coverage 96%

% of Key Message Communication

37%

Number of Messages tracked 11.5

Words per Key Message 14

Number of Key Messages tracked

11.5

Most frequent conversation types

Express support (69%) Making an observation (28%)

% rallying support 3%

% asking a question 4%

% of exclusive mentions 17%

% mentioning brand in title 8%

% of discussions in Blogs 51%

% of discussion on Twitter 30%

% of visibility from YouTube & Flickr

10%31

Overview of Key Metrics for Higher Ed

Bookmark.

Facebook

Ext. Blogs

Inst. Blogs

YouTube

MSM

SOV 2% — 8% 9% 11% 7%

Popularity

230 bkmks

500/mo.

— 20 links150k views

—

Engagement

59 cmts 1 day 13 cmts2-12 cmts

2 cmts —

% Positive

20% 32% 54% 50% 15% 15%

% Negative

0% 0% 4% 0% 1% 2%

Strat. Mess.

40%† 18%† 42% 42%† 18% 38%

Step 6: Conduct research (if necessary)

First: find out what already exists

Web trafficCustomer Satisfaction dataCustomer Loyalty data

Second: Decide what research is needed to give you the information you need:

Step 6: Selecting a measurement tool

Objective KPI Tool

Increase inquiries, web traffic, recruitment

% increase in traffic#s of clickthrus or downloads

Omniture, Google Analytics, Web Trends

Increase awareness/preference

% of audience preferring your brand to the competition

SurveyMonkey, Zoomerang

Engage marketplace Conversation index greater than .8Rankings

TypePad, Technorati , Radian6,

Communicate messages

% of articles containing key messagesTotal opportunities to see key messagesCost per opportunity to see key messages

Media content analysis –Dashboards

% aware of or believing in key message

Survey

Your tool box needs:

1.A content source: Google News/Google Blogs, RSS feedsTechnorati, Social Mention, Twazzup, Cyberalert, CustomScoop, e-WatchRadian 6, Techrigy, Sysymos, Visible Technologies, Scout LabsSurvey Monkey/Zoomerang

35

Your tool box also needs to include:

2. A way to analyze that contentAutomated vs. Manual Census vs random sampleThe 80/20 rule – Measure what

matters because 20% of the content influences 80% of the decisions

Dashboards to aggregate data

36

The Content Analysis ProcessTraining

Are we all on the same page?

Content Collection Did we get everything?

Coding Start analyzing (computers/humans)

Final Reliability(Resource Permitting)

ReportingWhat does it all mean?

Adapted with edits from The Content Analysis Guidebook (Neuendorf, 2002).

Why an Optimal Content Score?

You decide what’s important:Benchmark against peers and/or

competitorsTrack activities against OCS over

time

Optimal content score for media coverage

Positive: Mentions of the brandPositive brand

mentionsKey messagesCustomer quoted

positiveAnalyst quotedPositioned as trusted

partnerPositioned as

trustworthy, ethical Positioned as vendor of

choicePositioned as global Positioned as one

company

Negative OmittedNegative toneNo key messageNo quoteNo analyst quote

How to calculate Optimal ContentQuality score +1 0 -1

Score Score ScoreTonality Positive 3 Neutral 0 Negative -3

Positioning Contains 2 Doesn't contain 0

Positions the competition favorably or positions Sargento negatively -2

Messaging Contains 3 partially contains 0

Does not contain or miscommunicates key message (neg mess) -1

Quotes Contains 1 Does not contain -1Competitive mention

Does not mention Competition 1

Competition mentioned prominently -3

Total Score 10 0 -10

Visibility Score+1 0 -1

Score Score Score

Brand Photo Contains 3 Doesn't contain 0Contains competitive photo -5

Dominance Focal point 3 Not a focal point -1Visibility Headline mention 2 Top -20 % of story 0 Minor mention -2Target publication Top Tier 2 2nd tier 0 Not on target list -2

Total Score 10 0 -10

Optimal Content Score

Charting OCS over time between divisions

Trend against competition with OCS

Standard classifications of discussion• Acknowledging receipt of

information• Advertising something• Answering a question• Asking a question• Augmenting a previous

post• Calling for action• Disclosing personal

information• Distributing media• Expressing agreement• Expressing criticism• Expressing support• Expressing surprise• Giving a heads up

• Responding to criticism• Giving a shout-out• Making a joke• Making a suggestion• Making an observation• Offering a greeting• Offering an opinion• Putting out a wanted ad• Rallying support• Recruiting people• Showing dismay• Soliciting comments• Soliciting help• Starting a poll• Validating a position

Standard classifications of videos

AdvertisementAnimationDemonstrationEvent/

PerformanceFictionFilmHome VideoInstructional VideoInterviewLecture

MontageMusic VideoNews BroadcastPromotional VideoSightseeing/TourSlideshowSpeechTelevision ShowVideo Log

Your tool box also needs to include:

3. A way to measure engagementThe conversation indexLinks/trackbacksTime spentRelationship studiesQuestionnaires

45

Share of conversation vs share of engagement

Page 46

2

2

1

2

1

6

5

3

1

1

1

1

1

1

1

1

1

1

2

2

2

1

2

1

2

1

4

2

1

4

2

1

1

4

1

6

7

6

2

2

2

2

1

3

2

3

1

0 2 4 6 8 10 12 14 16 18 20

Faculty

Students

Research, Physical Sciences

Courses

Research, Earth Sciences

Projects, Non -Research

Financials

Alumni Topics

Research, Life Sciences

Staff

Admissions

Legal News

Other

Research, Agriculture

Policies

Institution, Overall

Campus Life

Research, Social Sciences

Share of Subject

Peer 1

Michigan State

Peer 2

Peer 3

Peer 4

15.3%

68.7%

100.0%

4.4%

33.3%

96.8%

28.6%

34.9%

12.5%

43.3%

28.6%

13.0%

38.3%

100.0%

23.6%

66.7%

6.3%

28.6%

20.8%

2.3%

95.6%

33.2%

5.8%

28.6%

100.0%

86.8%

13.0%

31.0%

22.1%

3.2%

71.4%

43.5%

18.8%

94.2%

56.7%

14.2%

13.2%

53.2%

28.4%

21.1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Admissions

Alumni Topics

Campus Life

Community Relations

Courses

Events

Faculty

Financials

Institution, Overall

Inventions

Legal News

Other

Partnerships

Policies

Projects, Non - Research

Research, Agriculture

Research, Earth Sciences

Research, Life Sciences

Research, Other

Research, Physical Sciences

Research, Social Sciences

Staff

Students

Share of Engagement by Subject - ,External Blogs

Peer 1

Michigan State

Peer 2

Peer 3

Peer 4

The vast majority of discussion in external blogs is neutral.

Page 47

23

29

12

14

20

5

8

4

1

4

0

5

10

15

20

25

30

University of Michigan Purdue University Penn State Michigan State Arizona State

Share of Tone

Negative

Neutral

Positive

71%

3%

29%

94%

83%

42%

58%

6%

14%

58%

42%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Arizona State Michigan State Penn State Purdue University University of Michigan

Share of Engagement by Tone - External Blogs

Negative

Neutral

Positive

One definition of engagement

% increase or decrease in unique visits In the past  month,  what % of all sessions represent more than 5 page views % of sessions that are greater than 5 minutes in duration % of visitors that come back for more than 5 sessions % of sessions that arrive at your site from a Google search, or a direct link from your web site or other site that is related to your brand % of visitors that become a subscriber % of visitors that download something from the site % of visitors that provide an email address

Courtesy of Eric Peterson

Your tool box also needs to include:

5. Measuring ROI MembershipRegistrationsWebSide story/Web trends/ClicktraksSales trackingMarketing Mix Modeling

49

Research without insight is just trivia

What works, what doesn’tWhat needs to be done? What are you communicating?What tools work best?

Step 7: Analysis

50

Data mining the numbers you have

Look for failures firstThen look for exceptional successCompare to last month, last

quarter, last yearFigure out what worked and what

didn’t work

51

Ask for money Get Commitment Manage Timing Influence decisions Get Outside help Just Say No

Actionable Conclusions

52

Measuring doesn’t have to be complex

High Impact

Low Cost High Costs

Low Impact

BloggingIntentional Leaks

SEO-optimized press releases

e-newsletter Advertising

Press conferenceParty

53

Take action and measure again

Make sure data is ready when you need it

Work around regular reporting schedules

Keep questions and criteria consistent

54

Mistakes made most often in the measurement process1.Lack of agreed upon goals 2.Questionable methodology3.Unclear definition of

tone/sentiment 4.Incorrect search strings5.Incomplete or out-of-date media

list (no CGM) 6.Incorrect circulation figures7.New products/spokespeople not

added to analysis parameters8.Promising a Jaguar on a bicycle

budget9.Not allowing enough time to do

measurement right10.Insufficient so what – looking at

the trees not the forest

55

Thank You!

For more information on measurement, read my blog: http://kdpaine.blogs.com or give me your card and we’ll send you The Measurement Standard, www.themeasurementstandard.comTo start developing your own dashboard

or for a copy of this presentation go to: http://www.kdpaine.comOr call me at 1-603-868-1550Follow me on Twitter: @kdpaineFriend me on Facebook: Katie PaineSkype: KDPaine

56