Forget Big Data - Measure What Matters

Katie Delahaye Paine

CEO

Paine Publishing

www.painepublishing.com

@queenofmetrics

measurementqueen@gmail.com

Paine Publishing: Providing communications

professionals the knowledge and information they need to navigate the journey to good measurement

Newsletters

Training Courses

Consulting painepublishing.com

Katie Delahaye Paine: Helping communications professionals define and measure success for 25 years.

Founder of: The Delahaye Group KDPaine & Partners Paine Publishing

Author of: Measuring the Networked

Non-Profit Measure What Matters Measuring Public

Relationships measurementqueen@gmail.com

Don’t Measure What’s Easy

3

Cartoon by Rob Cottingham

Measure What Matters

10/11/2014 News Group International 4

Cartoon by Rob Cottingham

Welcome to the, “I want what

I want, not what you send

me,” era

aka: “The Relationship Era”

®

It’s the end of the world as we knew it

The average audience for a MyDrunkKitchen video (http://www.youtube.com/watch?v=uSXQNred3is

1,000,000

CNN average nightly audience 179,000

The number of times per hour Digital Natives switch media—every 2.2

minutes. 27

The percent of conversation that happens OFF LINE 90%

The amount of conversations generated by bots, spammers and pay-

per-click sites 40%

The percent of on-line conversations that are public 10%

The percent of Facebook & Twitter posts that are actually seen < 5%

We need a new Attribution Model

ROI

Other Paid

Marketing

Digital/On-line

Media Buy

Print/TV Media

Buy

ROI

Media Relations

/Social Media/PR

Digital/ Online Media

Buy

Print/TV

Media Buy

Media

Relations/

Social

Media/PR

What’s Changed?

Collapse of mass media

Growth of media

everywhere

Intolerance for messaging

The Barcelona Principles

8

“Viewers are more likely to stop watching commercials at the moment in which brand logos appear on the screen” - ARF Study

The Barcelona Principles, The Conclave &

Industry Standards

1. Importance of Goal Setting and Measurement

2. Measuring the Effect on Outcomes is Preferred to Measuring

Outputs

3. The Effect on Business Results Can and Should Be Measured

Where Possible

4. Media Measurement Requires Quantity and Quality

5. Earned Media Value/AVEs are not the value of Public Relations

6. Social Media Can and Should be Measured

7. Transparency and Replicability are Paramount to Sound

Measurement

All standards are available on

http://www.painepublishing.com/st

andards-central

The Conclave

Social Media Measurement Standards have been published for:

Content Sourcing & Methods 1

Reach and Impressions 2

Engagement 3

Influence & Relevance 4

Opinion & Advocacy 5

Impact & Value 6

11/10/2014

Eyeball counting

HITS Outcomes

MSM Online Social Media

Impressions are not awareness. Where’s the “So What”?

11

Likes Are Not Engagement

12

Advocacy Commitment Trial/Consideration

Followers Likes Impressions

11/10/2014

The Engagement Process

Listen

Purchase

Run Achieve Goal

Donate

Measuring the Engagement Process

Web Analytics

eCommerce

iCloud &

Google+

Map My Run/Social Sharing

CRM/Convio

#3: Standards for Engagement

Engagement is an action that happens in response to content – i.e. when someone

engages with you. (not “about you”)

Engagement manifests differently by channel, but is typically measurable based on effort

required and how it is shared with others.

Engagement can be desirable or undesirable

Any measure of engagement must be tied to the goals and objectives for your program

Engagement occurs both off and online, and both must be considered if you intend to

integrate your metrics with other marketing or communications efforts.

Engagement includes such actions as likes, +1, shares, votes, comments, links, links,

retweets, Facebook's “Talking about you” etc.

Engagement actions should be counted by % of the audience that responds.

968 1027

914

208

297 261 269

215 222 145

328

204

510

188 222 234

41 71 50 36 25 56 30 71 45

107

Share & Likes Over Time

Average of PostLikes Average of PShares

GWA’s, Photos Drive Shares & Likes

Mia Farrow

David Beckham & Dierks Bentley

Video

• Influence happens when you are persuaded to change behavior or opinion that

would otherwise not have changed.

• Influence can be attributed to either an outlet or an individual.

• Influence happens online and off line and both should be considered

• Should include some combination of the following five elements:

• Reach

• Engagement around individual

• Relevance to topic

• Frequency of posts around the topic

• Audience impact as measured by the ability to get the target audience to

change behavior or opinion.

• If an individual scores a 0 on one element, they have no influence

• Influence cannot be expressed in a single score or algorithm

#4: Influence & Relevance

Big Numbers Influence

All influence is relative

Who is influential depends on your audience

A computer cannot tell you who matters most

To be influential requires relevance, frequency & reach

#5 Standards for Measuring Tone/Sentiment

Whatever process is defined and applied, it must be used

consistently throughout any analysis.

Sentiment reliability varies by vendor and approach – therefore

coding definitions, consistency and transparency are critical.

Opinions, recommendations and other qualitative measures are

typically more valuable than raw sentiment and increasingly

measurable:

Opinions (“it’s a good product”)

Recommendations (“try it” or “avoid it”)

Feeling/Emotions (“That product makes me feel happy”)

Intended action (“I’m going to buy that product tomorrow”)

Testing the accuracy of coding

48.94%

68.31%

88.64%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

SDL

UberVu

NetBase

% Agreement with human coding

SDL UberVu NetBase

33.12%

58.00%

7.84%

11.95%

26.53%

13.00%

0.32%

9.46%

40.35%

30.00%

92.11%

78.00%

SDL

Beyond

NetBase

UberVu

% positive %negative %neutral

Everyone agrees on Neutral

Not so with Negative and Positive

100.00%

62.32%

14.82%

72.36%

0.00%

37.68%

85.18%

27.64%

Mixed

Negative

Neutral

Positive

Degree to which agreement occurred

No Yes

• Impact and value will always be dependent on client objectives

• Need to define outcomes in advance – will likely span multiple business

goals, especially for social (crosses disciplines)

• “ROI” should be strictly limited to measurable financial impact; “total

value” can be used for financial and non-financial impact combination

• Value can be calculated in positive returns (sales, reputation, etc.) or

avoided negative returns (risk mitigated, costs avoided)

• Key performance indicators and balanced scorecards are helpful to

connect social media impact to business results/language

#6: Impact & Value

ROI

ROI is:

Cost savings

Greater efficiency

Lower legal costs

Faster time to implementation

Lower cost per impact vs. other

marketing channels

Lower cost per message

exposure vs. other channels

ROI is not:

Impressions

Earned media value or AVE

Number of shares, views or

likes

Conversations

$$$$$$$$$$$$$$$$$$$$$$$$$$

Savings, shorter cycles, more

renewals, better ideas,

Product

Support HR IR Sales CI R&D Mktg

Research

Prod.

Mktg

PR//Mkt

24

Social Media isn’t media, it’s your business

11/10/2014

Good Relationships Are More Cost Effective

Type I love Zappos into Google, and you find

1.19 million references

Type Citibank and you get 21,000 references.

Citibank spends 100 times more a year on

advertising than Zappos.

Cost per delegate acquired:

Obama: $6,024

Clinton: $147,058

Romney: $2,389,464

So What?

It’s not about the media, it’s about

your business and your customer

It’s not how loud you’re shouting, it’s

what your stakeholders do after

they’ve heard your message and your

relationships.

There are no boundaries

It’s not about how big your data is, it’s

about how you use it

6 steps to standards-compliant

measurement

Step 1: Define your goal(s).

What outcomes is this strategy or tactic going to achieve?

What are your measurable objectives?

Step 2: Define the parameters

Who are you are trying to reach? How do your efforts

connect with those audiences to achieve the goal.

Step 3: Define your benchmarks.

Who or what are you going to compare your results to?

Step 4: Define your metrics.

What are the indicators to judge your progress?

Step 5: Select your data collection tool(s).

Step 6: Analyze your data.

Turn it into action, measure again

27

Six Steps

to Success

1

2

3

4

5

6

Definitions

Monitoring – process by which data are systematically and regularly collected about a program over time.

Measurement – a way of giving an activity a precise dimension, generally by comparison to some standard; usually done in a quantifiable or numerical manner; see also: data, scale

Metrics – an agreed upon number or percentage that indicates progress towards a goal

Evaluation –a form of research that determines the relative effectiveness of a public relations campaign or program by measuring program*

28

*IPR Dictionary of Measurement Terms, Dr. Donald Stacks www..instituteforpr.org

Why do we communicate?

11/10/2014 29

Outtakes

(Intermediary Effects)

• Awareness

• Knowledge/Education

• Understanding

Outcomes

(Target Audience Action)

• Revenue

• Leads

• Engagement

• Advocacy

Activities How does what you do

contribute to the bottom line?

Step 1: Define the goals

What return is expected? – Define in terms of the

mission.

Define your champagne moment. If you are

celebrating complete 100% success a year from

now, what is different about the organization?

30

Goals & Suggested Metrics

Goals Goal 1: Meet Sales Targets

Goal 2: Reduce Risk/Threats

Goal 3: Increase market share in

new market

Communications Contribution

• Expand the marketable universe

• Reduce sales cycle

• Increase Trust • Increase

advocacy

• Expand the marketable

universe

Metrics

• % Increase in desirable share of voice

• % in awareness

• % increase in undesirable voice

• % increase in trust scores

• % increase in desirable voice in new market

• % increase in awareness of brand in new market

Step 2: Understand the parameters.

What management’s priorities?

Who are you are trying to reach?

How do your efforts connect with

those audiences to achieve the

goal?

What influences their decisions?

What’s important to them?

What makes them act?

32

Goal: Get the cat to stop howling

Options: Local? Cheap? Convenient?

Strategy: Buy cat food

Step 3: Establish benchmarks

Past Performance Over Time

Think 3

Whatever keeps your

C-suite up at night

33

Step 4: Pick your Kick-Butt Index

The Perfect KBI

Is actionable

Is there when you need it

Continuously improves your processes

& gets you where you want to go

You become what you measure, so

pick your KBI carefully

34

The Path to Purchase

Social Media Program

Awareness/Consideration

Preference

Purchase, Retention, Renewal

Kick Butt Quality Score

36

Desirable Criteria Score Undesirable Criteria Score

Contains a key message 3.50

No Key Message -2.0

Contains a desirable visual 1.00

Negative message -3.0

Contains a quote from your spokesperson 2.50

Contains a competitor or opposition quote

-2.0

Positions you as a leader 1.00

A story or a headline that leaves the reader less likely to do business with you

-3.0

The story or headline leaves a reader more likely to do business with you

2.00

Total 10.00

-10.0

Typical Social Engagement Index

Action Score

Like/Follow/Opens/+1/Favorite .5

Share content (including retweets, forwards, Shares etc.) or shares a link to an ENZ site or page. 2

Signs up to receive email or other content or follows a link to content. 2

Contains one or more of the bank’s key messages .5

Positive Comment (leaves a reader more likely to support ) 1

Contains a quote from an alumni or expert 1.5

Views an video 1

Is from a Tier 1 source 1.5

Total 10

37

Step 5: Pick the right measurement tools

If you want to measure messaging, positioning,

themes, sentiment:

Content analysis

If you want to measure awareness, perception,

relationships, preference:

Survey research

If you want to measure engagement, action,

purchase:

Web analytics

If you want predictions and correlations

you need two out of three

All the tools you really need

Google Analytics

Ignore page views

Set up conversions

Excel

11/10/2014

Step 5: Selecting a measurement tool

40

Objective KBI Tool

Increase message

communications

Increase percent of items containing one or

more messages

Media Content Analysis

Increase

awareness/preference

% of audience preferring your brand

to the competition

Survey Monkey, Survata

Engage

marketplace

% increase in engagement on website and/or

social sites

Simply Measured, Google

Analytics, Site Catalyst,

Network Analysis

Step 6: Be Data Informed, not Data Driven

Rank order results from worst to best

Ask “So What?” at least three times

Put your data into an overall framework consistent with C-Suite expectations

Find your “Data Geek”

Compare to last month, last quarter, 13-month average

41 Page 41

Photo Event

High Message Content

Resource Use

Low

Hig

h

Med

ium

Ver

y h

igh

Level of Engagement

Ver

y H

igh

Med

ium

Hig

h

Low

High Resources

No Message

Low Resources

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

Advocacy vs. resource use

Resource Use

Low

Hig

h

Me

diu

m

Ve

ry h

igh

Ver

y H

igh

Med

ium

Hig

h

Low

High Resources Low Resources

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

Real-Time Metrics focus on Outcomes

43

PUBLIC AFFAIRS & BRAND DASHBOARD Priority 1: Brand Awareness Priority 2: Advocacy Priority 3: New Membership Culture

Bench-mark

Current Variance Bench-mark

Current Variance Bench-mark

Current Variance Bench-mark

Current Variance

% increase in share of desirable voice

8.9% 2.17% ↓6.73 pts

% increase in PRINZ share of Desirable Conversation around Digital privacy

10% 15 ↑6.73 pts

% Increase in lawyers who are extremely or very familiar with PRINZ’s positioning

38% 38% Annual Data Only

% increase in ratings on communications trustworthiness (from culture survey)

54% New data

available July 14

N/A

% decrease in share of undesirable voice

1% 1% Un-changed

% decrease in Government officials unaware of PRINZs position on Digital Privacy

10% 10% ↑6.73 pts

% Increase in online engagement on issue

25% 0 1% % increase in employee knowledge of PRINZ “Way We Work” (From Knowledge Quizzes)

23% New data

available Q3

N/A

% increase in awareness among lawyers (from 2014 Brand Metrics Survey)

46% 46% Next data

available Q3 14

% increase in Inbound info requests for info

516(Apr) 427 (May)

↓17%

% employees strongly understanding strategic objectives (From Knowledge Quizzes)

32% New data

available Q3

N/A

% increase in perception of PRINZ as a leader in Digital Privacy

5% 5% Un-changed

% increase in engagement on PRINZ.org.nz

202K

197K

↓2%s

% reduction in Say/Do gap on agility metrics (average of agile & make decisions faster)

36.5% New data

available Q4

N/A

Below Target

At Risk

On Target

Connecting data yield greater insights

106 38 24

73 101 49 25

1087

1331

699 637

848

1331

699

10.25

35.03

29.13

8.73 8.40

27.16 27.96

0

5

10

15

20

25

30

35

40

0

200

400

600

800

1000

1200

1400

Jan Feb Mar April May June July

Relationship between interviews and placements

Number of Interviews Number of placements Interview per placement

ACA has made a demonstrable difference in the quality of

media coverage

If an ACA or one of its

programs was mentioned

in a media story, it was:

More likely to contain

an endorsement

More likely to be

positive

More likely to contain

key messages

Less likely to be

negative

0.22

0.00

0.62

1.42

4.61

0.08

0.08

0.01

-0.61

-0.66

Endorsements

Positive Visual

Negative Visual

Tone

MessageCommunication

Average of OCS

Differences in Quality of Coverage with and without

ACA

Atlantic City Atlantic City Alliance

Without ACA events, OCS Scores for Atlantic City would

have been significantly lower

4

3.25

2.75 2.99

3.65

2.96 3.36 3.24

2.34 2.37 2.43

1.30

-1.24

0.37 -0.05

0.28 0.28

-1.56

4.91

3.92

2.99

3.58

4.14 4.1 4.27 4.12 4.29

2.78 2.56

1.53

-0.29

0.61 0.20

1.44 1.77

0.63

-2

-1

0

1

2

3

4

5

6

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June

Ave

rage

OC

S Sc

ore

Average OCS Score Over Time

Atlantic City OCS without ACA Atlantic City OCS with ACA

The red line represents coverage of Atlantic City minus all mentions of ACA and its programs

Free entertainment generated the highest OCS Scores in

Q2

5.26

5.73

6.36

6.54

6.71

7.15

7.31

7.36

7.50

7.67

8.00

8.10

8.68

9.30

Miss America

Meet AC

DO AC

July 4th Fireworks

Air Show

Blake Shelton

Miss'd America Pageant

Sand Blast

Hello Summer

Boardwalk Hall Light Show

Challenge Triathlon

Sand Sculpting World Cup

Lady Antebellum Concert

Free Entertainment

Top Programs By OCS

When ACA programs received media coverage, traffic

followed

0

50

100

150

200

250

300

0

20000

40000

60000

80000

100000

120000

140000

Relationship between ACA Program Mentions and Site Visits

Site Visits Program Mentions

ACA programs drive higher OCS scores, which correlate

highly with web visits

0.41

0.44

0.47

AC items

ACA Items

ACA OCS Scores

Correlations between Web Visits and PR Metrics

Pearson r. value

White House Volunteerism Office (CNCS) is able to connect

specific social outreach to registrations on serve.gov

51

The ROI of Emily – So What?

52

Red line

indicates

media

impressions

35,152,789 OTS

So What = Revenue

53

10 Ways to find the Money for Measurement

1. Don’t call it measurement – It’s Research

2. Consolidate - Social? IR? Agencies? Research?

3. Crowdsource – Who else has tools or surveys?

4. Spread the cost over 2 fiscal years

5. Reach out to universities

6. Take advantage of free stuff

7. Monitor only what matters

8. Narrow your Top Tier media list to what you CAN afford

9. Google analytics

10. Facebook Insights

11. Twitter Analytics

Remember These Points

It’s not about the media, it’s about the business and the customer 1

It’s not about Big Data, but about how you use it. 2

You need to be data informed, not data-driven 3

It’s not how loud you’re shouting it’s about relationships. 4

Standards are a reality not an excuse to hide behind 5

Thank You!

For more information on measurement, read my blog:

http://www.painepublishing.com/blog

For a copy of this presentation or to subscribe to our newsletter, give me

your card or email me at measurementqueen@gmail.com

Follow me on Twitter: @queenofmetrics

Friend me on Facebook: Katie Paine

Or call me at 1-603-682-0735

56