P R E S E N TAT I O N A D A P T E D f r o m o n e d e v e l o p e d b y

E m i l y R o t h m a n , Sc DB o s t o n U n i v e r s i t y

Program Evaluation

Agenda

Check in One reflection on PPI thus far One expectation for the second half of PPI

Process Evaluation

Outcome Evaluation

Check out Homework +/-/Change

Process Evaluation

Pre-reading Review

What is process evaluation?

How can process evaluation be useful?

Are you already collecting process evaluation data? How? What types of information?

Small Group Activity

Look at your logic modelBrainstorm the type of process evaluation

data you will need to collect for you programJot down how you will track the information

Process Indicator How will you track it?

Who will track it?

Attendance Weekly sign-in sheet Teen Center Director

Participant satisfaction

Monthly surveys Annual focus groups

Teen Center Director

Session Implementation

Implementation report forms & session outlines

Program staff

WHAT ARE MEASURES?

Designing Pre and Post-Tests

Lingo

Concept / word Explanation

Survey This is the whole package—typically comprises multiple measures

Measure Bunch of questions (or “items”) that together assess a construct; may comprise scales

Questionnaire same thing as measure

Scale Sub-section of a measure or instrument

Inventory Same thing as measure; Measure of typical performance

Test Same thing as measure; Measure of best performance

Instrument Same thing as measure

Item Single question on a scale

What does a measure do?

Operationalizes a concept

Gets people to answer in the best possible way

“Best” means most reliable and valid

Gives you something to compare with other research studies

Ensures your response options can be analyzed

Operationalizing a concept

Example: Fear of falling cannot be directly measured. It is a concept.

Operationalizing a concept

Are we truly trying to figure out…: How worried might fall? How likely might fall? How often think about falling? How often hold stair railings? How confident can perform without falling?

All slightly different takes on the same thing

Operationalize…

“Bullying”

Being physically assaulted by a peerBeing called names, or harassed repeatedly

by one or more peersBeing intentionally excluded from activities

by peersBeing harassed via email, text, or social

media repeatedly by one or more peers

Why use a “measure” instead of a question?

The set of questions do a better job that one single item would alone

Why? People try to outsmart surveys People are inconsistent in answers More is better

Example: self-esteem

Do you have high self-esteem?

Example: self-esteem

1. On the whole, I am satisfied with myself

2. At times I think I am no good at all. 3. I feel that I have a number of good qualities.

4. I am able to do things as well as most other people.

5. I feel I do not have much to be proud of.

6. I certainly feel useless at times.

7. I feel that I'm a person of worth, at least on an equal plane with others.

8. I wish I could have more respect for myself

9. All in all, I am inclined to feel that I am a failure

10. I take a positive attitude toward myself.

Psychometrics of scales/measures

Social scientists develop scalesThey test and re-test them with groups of

peopleThey determine how reliable and valid the

measures are

Wording changes responses

Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year? 65% said companies could avoid layoffs 22% said couldn’t avoid layoffs 15% had no opinion

Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year, or do you think layoffs are unavoidable? 35% said companies could avoid layoffs 41% said couldn’t avoid layoffs 24% had no opinion

Payne, 1951

What is the connection to logic model?

Right now…

Look at your logic modelIdentify an outcomeHow will you measure it?

Using which existing measure? … OR Write your own “measure”? Write your own single-item question?

Where to get measures

Where do you find measures?

Lit search for measuresLit search for articles on same topic, see

what measures they usedAfter you know the name of a measure, you

can sometimes Google itUsually you have to find it printed in a peer

review journalIt is hard to find measures!

Don’t make up measures unless…

Golden rule:

Do not make up survey questions out of your own head, ever, unless you really, really, really have to

Acceptable reasons for writing own question:Searched literature, none existSearched literature, all measures are inappropriate

for some reasonYou are never going to publish your results or

compare with another existing source

Types of Instruments

Likert-type scaleYes/no questionsFree response or qualitative sections

Creating an entire survey

What goes into a survey?

Standard things: Version (pre- or post-test) Today’s date Subject ID number?

Demographics Race/ethnicity? Sex? Age?

Measures Measure short-term, intermediate and long-term

outcomes?Nothing extra!

Don’t ask any questions that you don’t have to—long surveys are bad.

What makes unreliable and invalid questions?

Respondent (R)…1. Doesn’t know the answer2. Cannot recall an accurate answer3. Doesn’t understand the question4. Doesn’t want to report an accurate

answer

Question design pitfalls

Putting two questions in the same question:

Have you ever broken your arm or been too sick to go to class?

Question design pitfalls?

Using technical jargon without defining it:

How frequently do you advocate for prophylactic use during sexual encounters?

Question design pitfalls

Leading respondents:

Would you agree that this program is awful?

Question design

Be wary of socially desirable or undesirable questions:

“Do you like pancakes?” vs.“Have you ever been diagnosed with a STI?”“How much money did you earn last year?”

Question design

Keep it simple

Place the burden on yourself to make things clear, not the respondents to figure things out “What was your pattern of enrollment?” vs. “Were you a part-time or full-time student?”

Keep questions short

Question design

No double negatives!

If you have not had sex in the past year, do not answer the next question

Formatting design

Neat, clean appearanceEasy to readLarge enough fontNo fancy serifs or font styleNo grammar or spelling mistakes

Work Session!

Begin designing your survey…

Homework

Develop your outcome tool

Administer with at least 10 people

Wrap-up

+/-/Change

Be in touch for technical assistance!