Developing and Pretesting Concepts, Messages, and...

Developing and Pretesting Concepts, Messages, and Materials

of 33

Developing and Pretesting

Concepts, Messages, and Materials

Ms. Williams: Today we are going to talk

about step two, which is developing and

pretesting concepts, messages, and materials.

We are also going to talk about implementing and

managing health communication programs. So,

you will have step one and step two under your

belt, and then you can work on your plan. We

will use the communication plan template that

comes out of the pink book to do that.

But, first, before we get into that, we

wanted to come back to what we talked about just

briefly yesterday towards the end of yesterday.

That was, once you develop this profile and you

have your indicators in place, and you kind of

have a sense of probably where your population

is, how do you make decisions about what

approach you are going to take?

The example we showed yesterday was that

you have southern small state, central small


of 33

state, and northern small state. They all have

sort of a different demographic profile. How

are you going to make a decision about we are

going to use health communication approaches in

this community and we are going to use policy

approaches in this community, and we are going

to do something else in that community?

It is a process of looking at a number of

different factors. So, you might say southern

small state has high unemployment, low income,

high uninsured, a few barriers, a few providers.

There are a lot of barriers to people making

behavior change.

If you were doing a social marketing

campaign and there is sort of this idea that you

go for the low-hanging fruit, southern small

state is not low-hanging fruit. However,

central small state or northern small state

might be considered more low-hanging fruit.

The challenge for us who work in the

government and use public money is that is not

always how we are expected to spend the tax


of 33

dollars, right, on people who are ready to

change? We are expected to work on the tough

problems. That is why they have us. Maybe some

of our partners can work on some of the more

low-hanging fruit.

What do you do in these cases of the

barriers are so high and there’s multiple

problems going on? How do you make a decision

about the appropriate approaches?

One place that you might start is back to

your literature review and your environmental

scan. What is that telling you about different

approaches you might take in communities that

face similar challenges?

There are a number of organizations out

there that have put out best practices and

guidelines. One place you can look is the

Community Guide to Preventive Services has a set

of recommendations around health communication

and marketing. You can look at what they have

recommended. Robert Wood Johnson has put out

some work on policy. I think Kellogg has some


of 33

information about strategies. The Public

Health Institute has strategies.

You can look at Healthy People 2012 now.

What are we on? Whatever the next one, 2020,

you can look at Healthy People, those

recommendations, and see what they are

suggesting in terms of how you approach some of

the issues that you are looking at.

The Community Guide also has

recommendations for diabetes, heart disease,

physical activity. You can look and see across

what all of these sort of people who have been

looking at the literature are saying are what

may be best practices right now.

You can also work with your stakeholders.

If you say this is the picture of where things

are, these are the different strategies we might

take. We might take a policy approach. We

might take a worksite approach. We might take

a communication approach. Discuss amongst

your stakeholders what seems like it is going

to be something that is going to be meaningful


of 33

and impactful.

Frankly, your own priorities within your

organization are going to dictate a lot of what

you do. You might have categorical funding that

says you must do health communication, and you

are looking at southern small state and saying,

“Well, what can we tell them when they can’t see

a doctor or they don’t have access to healthy

food or access to resources, but we have to do

communication?”

You might look at, then, what are your

options, given the conditions. You might

consider can you apply communication in another

way. Instead of reaching the people who don’t

have access to resources, can you use your

communication funding to reach decisionmakers,

to reach communities that might influence

access, to address some of the policy and

environmental barriers? Can you use your

communication funding to do some community

collaboration or some kind of community

mobilization kind of work? There are a lot of


of 33

different approaches that you could take, even

in these seemingly difficult areas for doing

communication.

But it is multiple approaches to making

a decision about, given all of these conditions,

what would be our best approach to take? You

can always, also, ask us at NDEP. I mean, you

can run by, “This is what we are thinking. These

are the strategies that we are thinking.” We

are happy to provide some technical assistance

and some advice in terms of how you might

approach any given situation.

Mr. Petty: Another thought that I had is,

looking at these three different profiles, you

are going to have three very different types of

campaigns that you would want to roll out in each

area. So, if I am looking at southern small

state, you are probably looking at more

environmental things, dealing with access,

maybe building parks and walking trails so that

people have the opportunity to do more physical

activity, maybe working a little more with


of 33

trying to get different policies put in place

that allow for some of these things to improve.

It looks like they have got some of the

highest smoking rates, but their obesity isn’t

too bad. Maybe there is more of an emphasis on

tobacco in that area.

Whereas, if you are looking over at the

northern small state, that is probably more

where you are trying to get down to individuals

and help work on behavior change for those that

aren’t quite there. I mean, just looking at the

different characteristics says to me that you

would have very different types of approaches

in each different area that you would just want

to think through.

Ms. Williams: Another thing you might

find is maybe central small state has tons of

campaigns already going on. They are sort of

in the middle, and they are sort of what people

like that do health communication or do

interventions. There may be lots of other

programs already happening. Maybe the place


of 33

for you, then, is southern small state because

there is less happening. So, there are more

opportunities to get things done.

There are a lot of environmental factors,

evidence factors. There is a lot that goes into

that decisionmaking process of deciding we are

going to take these approaches in this area with

these people. It is a process that you have to

go through, and you bring a lot of different

factors to bear in that decisionmaking.

Let’s talk about step two, which is

developing and pretesting concepts, messages,

and materials. I was at a presentation once,

and this young woman was developing a breast

cancer awareness campaign for African-American

women. She had done all the things you are

supposed to do in step one. She had done a

series of focus groups with African-American

women. She asked about what their preferred

communication channels were.

One of the channels they mentioned was the

links. She thought that was kind of weird,


of 33

links being another word for golf course. But

she had developed this whole golf campaign where

they were going to ride around in golf carts.

This is a true story. They were going to ride

around in golf carts and distribute breast

cancer information awareness. She hadn’t

launched the campaign yet, but she was in the

process of putting this all together.

I was like, “I’m sorry, but are you sure

they meant a golf course? Because you know

there is an organization of African-American

women called The Links?” And she just kind of

looked at me. She was like, “Oh, no, I didn’t

know that.”

She missed that, too, which is pretesting

your messages, materials, and concepts. Once

she had put together her golf, her links, and

her focus groups, and once she had decided on

the concept of a golf course intervention, she

should have pretested that. She could have just

run it by three African-American women, and they

would have said, “Well, they probably didn’t


of 33

mean golf course. They probably meant The Links

organization.” If she had done the pretesting,

she would have saved herself a lot of work.

Especially when you are not as familiar

with the community or you are an outsider to a

community, it is important to do pretesting and

to keep in mind that focus groups, they are good

for a certain amount of information, but you

have to check what you hear in a focus group.

Sometimes the facilitator will latch onto a

novel idea. That doesn’t mean that your

audience is going to like that. But if there

is something that you heard that you didn’t hear

anywhere else, and you thought it was a good idea

and it rises to the top for some reason, you have

to go back and check that.

When I was doing a secondary analysis of

the focus group data for the Surgeon General’s

Report on Physical Activity, they had made a

recommendation that one of the partners we

should work with to disseminate the Surgeon

General’s report for African-Americans was the


of 33

Nation of Islam. That was in the report from

the focus group, the contractor.

I went back and reviewed the tapes. It was

one person out of – they did probably nine focus

groups with African-Americans – one person in

one focus group and nobody else in the room.

There was no non-verbal agreement with that one

person. But it was something that the

facilitator had not heard before, and so she put

it in as a novel idea.

Can you imagine CDC or the Surgeon General

partnering with the Nation of Islam because they

heard one thing from one person in a focus group

that no one else -- I mean, there was not even

people nodding their heads, “Oh, yes.” There

was none of that.

You have to check what you hear from your

audience. That is why pretesting is so

important. Pretesting allows you to assess the

cultural appropriateness of what you are

developing. It helps you assess whether people

will pay attention to your message or your


of 33

strategy, whether they understand it.

You can assess how motivating your message

is or your concept, whether people can remember,

whether people are remembering accurately.

You put out a message and people interpreted it

in an entirely different way. You need to know

that before you move forward with your campaign.

Is it meaningful?

It can also help you identify

controversial areas. In one case, there was a

tobacco campaign that had done these focus

groups with teenagers that were smokers. One

of the stories that was told was that, before

you start smoking when you are 14, you think that

being addicted to cigarettes is like really

wanting a piece of chocolate. Once you become

addicted to cigarettes, you realize that it is

more like you are drowning and you need a next

breath of air. That is what it feels like to

be addicted.

And so, they came up with this whole

campaign about drowning, right? Smoking was


of 33

like drowning. When they launched the

campaign, they got blowback from the people who

had children who drowned, right? They had to

pull the campaign.

They didn’t know that that was going to

happen. That wasn’t their intent. They didn’t

mean to upset people. But you need to check.

Especially when you do something that is

potentially controversial, that is potentially

edgy, that is potentially out there, you just

need to run it by some people to make sure you

are going to be okay, and be prepared if there

is going to be any kind of blowback or outcry

over your message.

Pretesting allows you to assess the

strengths and weaknesses before the campaign

launches. Now keep in mind that there are

formal ways of doing pretesting that are

expensive and complex. You can also just run

things by people. You can do a check with a few

people that represent your target audience,

that represent people in your organization.


of 33

You can do it with your stakeholders. You can

go to opinion leaders, gatekeepers, and run your

message by them. So, there are also less-formal

ways to do pretesting.

It is not objective. It is not predictive

or statistically-precise. It is not a

substitute for a sound model or theory. You

still have to have your messages based on

something. They still need to come out of some

kind of framework.

Pretesting is not going to make up for poor

planning or flawed execution and

implementation. You still have to do all of the

steps, even if you pretest.

If you go to the pink book, what it is going

to explain to you about pretesting is the steps

that you take are you review the existing

material. So, you go back to your literature

review and review other campaigns. Or you go

back to your environmental scan. You review

what has already been done. You look at

national organizations. You look at what


of 33

messages are already out there.

You develop and test your message

concepts, which are ways of presenting

information. Are we going to take a funny

approach or a serious approach, a scary

approach? Are we going to use this kind of

person or that kind of person? Are we going to

use photos or illustrations or comics?

There are all different kinds of concepts

that come out, and you test these concepts to

find out which resonate with your audience. You

decide what materials to adopt or tailor. You

develop those messages and materials, and then

you pretest them.

It is a complex process. It takes people

who are really skilled at developing concepts

to do this kind of work. It is oftentimes people

that we don’t have. A lot of times, if you don’t

have a contractor that can do this work for you,

it can be very difficult to do it on your own

because it does take a certain amount of skill

to develop the concepts and then test the


of 33

concepts in a way that is going to give you

meaningful information.

So, that is kind of a red flag. It can be

an expensive and complex process. However, you

can work with materials that are already out

there where some of this work has already been

done, and it will save you some steps.

You can work with NDEP. NDEP has already

developed and tested the message concepts

around the diabetes prevention and control

materials that we put out. So, review NDEP

materials.

Decide what materials to adopt and tailor.

Based on what you have learned about your

audience and the approaches you are going to

take, and the strategy you want to take, you can

modify and adapt those messages and materials.

Then, you want to pretest. Pretest the

adaptations you made to make sure they are still

on point, to make sure people still are getting

the message. It saves you the step of doing that

concept testing, which can be very complicated


of 33

and expensive.

So, leverage the materials that are

already out there. Start with NDEP and go on

from there. Okay?

You review our existing materials. You

can search by risk status, ethnicity, language,

what group you are trying to reach. You can also

call us or send us an email because we do that.

People are thinking, “We are trying to do this

and we were thinking these materials you have.”

We will say, “Well, these materials, this much

material, or that material might work better for

what you are trying to accomplish.”

There is a lot of stuff on the website,

and sometimes it can be difficult to navigate.

We are working on that, but we are happy to help

you work through what is there, to identify what

might work.

What you want to look for is, are the

messages relevant? We learned all of this in

our formative research. We think this is the

message that is going to resonate. This is what


of 33

we are going to try to accomplish in our

community. Which of these materials? Are the

messages relevant in these materials to what you

are trying to accomplish?

Some are prevention messages. Some are

control messages. If you are trying to focus

on control, you may not want to use the

prevention materials. You may want to focus on

the control material. Okay? So, make sure the

messages are relevant.

Are the materials appropriate, the

format, the style? Are there cultural

considerations? The language, the literacy

level, you want to take a look at that. Those

are things that can be adapted, too. Just

because you see something like our “Paso a

Paso,” which is “step by step,” or you might see

one of our American Indian materials, it doesn’t

mean it can’t be adapted for your population if

the message seems to be on target.

Are the materials likely to meet your

communication objectives? Whittle down. Ask


of 33

yourselves these questions. Ask your

stakeholders these questions, and whittle down

your choices to the ones that are most likely

to be yes.

Decide which materials to adopt. You may

want to test existing materials. You may get

three or four NDEP resources and you may say,

“I’m not sure which of these is going to be the

best fit.” So, you might want to run that by

your target audience just to be sure. “What do

you think about this?” “Oh, this is fine, but

it’s wordy.” Or “This is fine, but I don’t want

to read it on paper. I want to read it on my

phone,” or something like that. So, you might

at this stage want to run something by your

target audience.

For program activities, you might want to

test the concept. If you are deciding to do like

Road to Health, you might want to run that by.

How will we do Road to Health in this community?

What is going to be the best way to make this

happen? Or power to prevent? Or, if we are


of 33

going to do some kind of provider outreach with

one of our provider tools, how is that going to

work?

You might want to test, actually, that

concept for making that activity work. Does

that make sense? Not testing necessarily the

material itself, but how you are thinking you

are going to implement or disseminate the

activity. Okay? Then, you can decide if any

tailoring is needed or if you are going to need

to add anything to make this really work.

It doesn’t necessarily have to be a whole

big thing, right? You don’t necessarily have

to convene another round of 18 focus groups to

do this. It can be a matter of sitting down with

your stakeholders and going through the

material, sitting down with members of your

target audience, running it by people who are

knowledgeable or who have done this work before,

and working through this.

So, it can be done very effectively. It

can be done without using a lot of resources.


of 33

It can be done in a very timely fashion.

Step three will be modify or adapt

messages and materials. The first place to

start is the principles established in the NDEP

Guidelines for Partner Collaborations. That

is on our website. There is a link to it from

the phConnect site. You can also find it on our

website.

But, basically, the materials are in the

public domain. There are no copyright

restrictions on them. You can brand them with

your logo or your partner logos. You can add

your contact information. You can tweak the

examples that are used. You can change the

pictures.

I think in Utah they were using “Managing

Diabetes: It Is Not Easy, But It Is Worth It.”

They used local people because they were trying

to tell local stories about diabetes. They just

took the pictures we put on there off and put

the pictures of their local people on.

You can alter the format. We have a


of 33

brochure. You convert that to HTML for your

website. Or we have a PSA canned. You can have

someone else read the script. That’s fine.

You can adjust for programs the delivery

and the timeline. We have the Power to Prevent

curriculum, which I think is 12 weeks. You can

make that shorter; you can make that longer.

You could add Road to Health to that. You can

do Road to Health in one sitting. You can do

Road to Health in three sittings. You can do

Road to Health in six sittings. If you have a

question, ask. Okay?

You can complement and supplement

existing materials. You might want to tailor

for cultural or local relevance. We have, say,

the Fat and Activity Counter. You might have

some restaurant or something in your community

that everybody goes to, and you want to add that

menu or you want to provide that menu as well,

those nutrition facts for those folks. You can

provide that as a supplement to our existing

materials.


of 33

We can work with you on literacy if

literacy level of the materials is an issue. We

can work with you on language if the language

of the materials is an issue.

Things that should not be modified: the

health topic. If we have something that is

prevention, it is prevention. Those are

scientifically-approved. We go through a

process of clearance, clearing our messages,

that people look at and say, okay, it is okay

for you to say this. So, if it is a prevention

message, those messages have been crafted for

prevention.

If it is a management message, those

messages have been tailored for management.

You have to be very careful about switching

them, about saying, “Okay, well, we are going

to use these prevention tools in our

self-management class.” You have to be careful

about that because there are differences

between the messages for the two audiences. Ask

us if you have a question or a concern.


of 33

You can’t really make substantial

alterations to the text that alter the meaning.

Like I said, that has been

scientifically-approved. You shouldn’t be

making substantial changes that really alter

the meaning of what is being said. Really,

attaching and inserting new text is not allowed,

although there is an exception for adding

culturally-relevant examples, for the most

part.

And you can’t imply that NDEP endorses

commercial products. There is language in the

Partner Guidelines about, if you are using a

commercial, you have Walgreens or Pfizer or

something like that who is local you are adding,

what the language is that needs to go on there

to make it clear that NDEP doesn’t endorse that

particular company.

If you adapt the material to the extent

that the NDEP logo has to be removed, generally,

our test is, if you have made changes to the

extent that we would have to get it cleared


of 33

again. If you run it by us and we say, “We would

have to run this through clearance,” then you

would have to take the logo off. It doesn’t mean

you can’t do it and you can’t use it. It just

means that the logo has to come off because it

is not what has been cleared by CDC or NIH, and

you put this message on it. This is also in the

Partner Guidelines.

I think Utah also changed some of the text

on some of the posters to be preventing and

managing diabetes. They added prevention and

management messages together. So, they took

the NDEP logo off because it was sort of pulling

messages from different places and putting them

into one place. You can always check with us,

always ask.

Let’s do an example. This campaign’s

messages are “Make a plan and manage your ABCs.

Make an appointment to visit your healthcare

provider and keep it. And make a referral to

the small state quit line.”

So, they are using our Managing Diabetes


of 33

posters. They added the state’s website, and

they used a local couple. Can they keep the NDEP

logo? How many people say yes? How many people

say no? They can keep the logo in that case.

So, it is okay to add your contact information,

add your local agency information. You can

switch out the pictures and still keep the logo.

Okay?

What about they pick this heart disease

flyer. It was developed for American Indians.

They changed the photographs and converted it

to HTML. Can they keep the logo? Yes. Good.

They took Four Steps for Life, and they

reprinted it with information about smoking

cessation in the back. Can they keep the NDEP

logo? No. For one thing, it is not four steps.

No.

(Laughter.)

Participant: Yes, and they changed it to

five steps.

Ms. Williams: And they changed it to five

steps.


of 33

You can’t attach new information that

hasn’t been cleared. You could add the smoking

cessation information as a supplement,

something that is offered. They get the Four

Steps brochure and they get another thing about

smoking cessation. That’s fine. But you can’t

physically attach new content to one of our

materials without us getting clearance for it.

Or you could go ahead and do it and just take

our logo off. You could change it to five steps,

add the smoking, take the NDEP logo off, add that

it was adapted on the back cover, and go on.

Okay?

And then, this is our poster about working

with your pharmacist and your optometrist and

your dentist and your podiatrist. They added

the phone number for the quit line to this

poster. Can they keep the NDEP logo? Yes.

Yes, you can because all you did was add local

contact information, and that is fine. Does

that make sense?

At the end of the day, just call us, send


of 33

us an email. Tell us what you are thinking. We

are happy to let you know.

Step four is pretesting the message and

materials. This happens after you have

substantially adapted a material. We took

this, we changed the format, we tweaked the

intervention, we did whatever, and it is

different from what it originally was. You want

to pretest that.

Or you have come up with sort of how we

think this campaign is going to run or this

intervention is going to run, or whatever the

strategy is. You might want to do a small pilot,

just to make sure that it is working the way that

you intended, that those messages are

resonating the way that you intended.

Again, it doesn’t have to be a whole big

thing. It is just running it by people who know

to make sure that you are still on target, you

are still on a message, your messages are making

sense.

It helps you to avoid missteps in


of 33

execution. So, you are not driving around on

a golf course looking for Black women and not

finding them, and wondering why the people in

your focus group lied to you.

It helps you confirm that messages and

materials are consistent with the audience

research and the testing. It also helps you

gain buy-in. So, we have all these people we

want to carry our message for us. They need to

have some kind of input into the development of

those messages.

This pretesting gives you a chance for

them to say, “Hey, this looks really good. We

can’t wait to start using these materials.” Or

“It would be good if you would tweak this a little

bit for us.” Or “This is on point.” Or “This

could change.”

Pretesting is a way to gain buy-in from

your stakeholders. Again, it is always helpful

to run your messages, your materials, by someone

who knows.

Mr. Petty: Just another quick example on


of 33

this. I am not sure exactly where this came

from, but I was at a social marketing conference

a year or two ago and heard about an example of

some folks that got some money to do an HIV

campaign. They decided that, through their

environmental scan, they really wanted to

target prostitutes, street workers.

They appropriately discovered that the

government should not be the right message

deliverers for this program because it is an

illegal activity. But, nonetheless, they

recognized that it was a problem.

And so, very creatively, they went out and

got several of them in a room. Basically, they

created this program for themselves. Talk

about getting buy-in and avoiding all these

potential problems, because they recognized we

know nothing about their life and what they do.

But they really went out and got the people who

are in that situation to craft messages to help

them make more healthy life choices and had a

lot of really good success with it.


of 33

It is kind of out of the box, but a great

example of really pulling in the right people

at the right time to make sure you don’t make

some of these mistakes.

Ms. McDivitt: And that is recommended,

also, for working with kids and youth, is

getting them involved in the development.

Mr. Petty: Early, yes.

Ms. Williams: When you are looking at

developing and pretesting messages and

materials, this step is key to ensuring the

effectiveness of your program materials and

activities. Sometimes you go through all your

formative research and you come up with a great

concept, and you just want to get started. You

just want to run out and implement it. You think

you are okay because you did a pretty good job

with your formative research.

But don’t shortcut step two. Even if it

is just a matter of running your ideas by some

people, it is worth the time and effort to make

sure that you avoid some missteps in the


of 33

execution of your campaign or your activities.

NDEP materials can serve as the basis for

most campaigns. We are willing to work with you

to make those work. You can look at other

materials, and we are willing to work with you

on that as well.

Test the materials and activities for

relevance to your intended audience. Test them

to make sure they are going to be okay in your

agency, that your stakeholders are okay with

them.

If you adapt materials significantly or

you adapt an approach significantly, pretest or

pilot before launching the program, just to make

sure that everything is working the way you

intended, and not just the messages, but your

800 number, your website, whatever systems you

put in place to make this work. Make sure those

things work.

I worked on a campaign that crashed NCI’s

Call Center. That is not something you want to

do. That didn’t make them happy. But it was


of 33

because we didn’t have all the right pieces in

place when we launched our campaign. We had no

idea how many people were going to call. We had

no idea that the Call Center was going to be

understaffed the day that we launched our

campaign because half the people were in

training. And so, we crashed the Call Center.

Running through your plan, not just your

message, but how you are going to execute your

plan, is important, too. It is not helpful to

make people in your agency upset with you about

crashing the Call Center. So, this is a really

important step.

Developing and Pretesting Concepts, Messages, and...

Documents

Transcript of Developing and Pretesting Concepts, Messages, and...