EvaluationEvaluationKrista S. Schumacher

Schumacher Consulting.org918-284-7276

krista@schumacherconsulting.orgwww.schumacherconsulting.org

Prepared for the Oklahoma State Regents for Higher Education

2009 Summer Grant Writing Institute

July 21, 2009

Why Evaluate? Why Evaluate? How will you know your project is

progressing adequately to achieve objectives?

How will funders know your project was successful? ◦ Increasing emphasis placed on evaluation,

i.e., U.S. Department of Education National Science Foundation Substance Abuse and Mental Health Services

Administration (SAMHSA)

2

Why Evaluate?Why Evaluate?Improve the program –

◦ “Balancing the call to prove with the need to improve.” (W.K. Kellogg Foundation

Determine program effectiveness – ◦ Evaluation supports “accountability and quality

control” (Kellogg Foundation)

◦ Significant influence on program’s futureGenerate new knowledge –

◦ Not just research knowledge◦ Determines not just that a program works, but

analyzes how and why it works With whom is the program most successful? Under what circumstances?

3

Why Evaluate? Why Evaluate? WHAT WILL BE DONE WITH THE

RESULTS?????

“Evaluation results will be reviewed (quarterly, semi-annually,

annually) by the project advisory board and staff. Results will be

used to make program (research method) adjustments as needed.”

4

What to Evaluate?What to Evaluate?◦Objectives

Use baseline data from need section

◦Activities◦Program/research fidelity

How well program implementation or actual research matched established protocol

◦Attitudes◦Consider sorting data by

demographics, e.g., location, gender, age, race/ethnicity,

income level, first-generation status

5

Types of EvaluationTypes of EvaluationProcess evaluation:

◦ What are we doing?◦ How closely did implementation match the

plan (program fidelity)?◦ What types of deviation from the plan occurred?◦ What led to the deviations?◦ What effect did the deviations have on the project

and evaluation?◦ What types of services were provided, to

whom, in what context, and at what cost? services (modality, type, intensity, duration) recipients (individual demographics and

characteristics) context (institution, community, classroom) cost (did the project stay within budget?)

6

Types of EvaluationTypes of EvaluationOutcome evaluation:

◦ What effect are we having on participants?◦ What program/contextual factors were

associated with outcomes? ◦ What individual factors were associated

with outcomes? ◦ How durable were the effects?

What correlations can be drawn between outcomes and program? How do you know that the program was the cause of the

effect?

How long did outcomes last?

7

Who will Evaluate?Who will Evaluate?External evaluators increasingly required or

strongly recommended◦ Partners for effective and efficient programs

Collaborators in recommending and facilitating program adjustments

◦ Methodological orientations Mixed-methods? Quantitative only? Qualitative only?

◦ Philosophical orientations Purpose to use evaluation to boost personal research

publication record or to help organization/effort?◦ Experience and qualifications

Years conducting evaluations for types of organizations and types of projects (e.g., education, health, technical research?)

Master’s degree required, PhD preferred (or working toward PhD)

Social science background: Sociology, Psychology, Political Science

8

How much will it cost?How much will it cost?External evaluations cost money…

period.Some evaluators charge for pre-award

proposal development phaseKellogg Foundation recommends

5% to 10% of total budgetCheck funder limits on evaluationEnsure cost is reasonable but

sufficient to conduct in-depth evaluation and detailed reports

9

Two Types of DataTwo Types of DataQuantitative

◦ Numbers based on objectives and research design

◦ What data do you need? e.g., Number of participants Grade point averages Retention rates Survey data Research outcomes

Qualitative◦ Interviews◦ Focus groups ◦ Observation

10

Methods/InstrumentsMethods/InstrumentsHow are you going to get your

data? ◦ Establish baseline data◦ Pre- and post-assessments (knowledge, skills)

◦ Pre- and post-surveys (attitudinal)◦ Enrollment rosters◦ Meeting minutes◦ Database reports◦ Institutional Research Office (I.R.)

11

Data Analysis: Data Analysis: So you have your So you have your data, now what?data, now what?

Quantitative dataData analysis programs:

◦ SPSS (Statistical Program for the Social Sciences), Stata, etc...

◦ Descriptive and statistical data: # and % of respondents who strongly agree that flying pigs

are fabulous compared to those who strongly disagree with this statement. Likert scale

On a scale of 1 to 5, with 1 being “strongly disagree” and 5 being “strongly agree,” rank your level of agreement with the following statement…..

t-ratio, F test, chi-square…..

◦ “Quantitative data will be assessed using Stata statistical analysis software to report on descriptive and statistical outcomes for key objectives (e.g., increase in GPA, retention, enrollment, etc.).”

12

Data AnalysisData AnalysisQualitative Data

◦ Data analysis programs NVivo (formerly NUD*IST), ATLAS.ti, etc…

◦ “Qualitative data will be analyzed for overarching themes by reviewing notes and transcripts using a process of open coding. The codes will be condensed into a series of contextualized categories to reveal similarities across the data.” Contextualization – how things fit together

◦ More than pithy anecdotes “May explain – and provide evidence of – those hard-to-

measure outcomes that cannot be defined quantitatively.” – W.K. Kellogg Foundation

Provides a degree of insight into how and why a program is successful that quantitative data simply cannot provide

13

Two Types of TimeframesTwo Types of TimeframesFormative

◦Ongoing throughout life of grant◦Measures activities and objectives

Summative◦At conclusion of grant program or

research

NEED BOTH!

14

TimelinesTimelinesWhen will evaluation occur?

◦Monthly? ◦Quarterly? ◦Semi-annually? ◦Annually? ◦At the end of each training session? ◦At the end of each cycle?

How does evaluation timeline fit with formative and summative plans?

15

Origin of the Evaluation: Origin of the Evaluation: Need and ObjectivesNeed and Objectives

Need: For 2005-06, the fall-to-fall retention rate of first-time degree-seeking students was 55% for the College’s full-time students, compared to national average retention rates of 65% for full-time students at comparable institutions (IPEDS, 2006).

Objective: The fall-to-fall retention rate of full-time undergraduate students will increase by 3% each year from a baseline of 55% to 61% by Sept. 30, 2010.

16

Evaluation Data Collection Evaluation Data Collection and Reporting Planand Reporting PlanObjectives

Data collected and timeline

Methods for data collection and timeline

Instruments to be developed and timeline

Reports/outcomes timeline

Increase fall-to-fall retention by 3% per year to 61%

Student enrollment in first fall and second fall within one month of start of second fall

Enrollment entered by gender and race/ethnicity into Stata within first four weeks of each semester

Enrollment rosters separated by gender and race/ethnicity by Jan. 15, 2009

At mid-point of each semester

17

BEWARE THE LAYERED BEWARE THE LAYERED OBJECTIVE!OBJECTIVE!

By the end of year five, five (5) full-time developmental education instructors will conduct 10 workshops on student retention strategies for 200 adjunct instructors.

18

Origin of the Evaluation: Origin of the Evaluation: Research HypothesisResearch HypothesisGood:

◦ Analogs to chemokine receptors can inhibit HIV infection.

Not so good:◦ Analogs to chemokine receptors can be

biologically useful.◦ A wide range of molecules can inhibit HIV

infection.

*Waxman, Frank. Ph.D. (2005, July 13). How to Write a Successful Grant Application. Oklahoma State Regents for Higher Education.

19

Logic ModelsLogic Models

From: University of Wisconsin-Extension, Program Development and Evaluation

http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

A Logic Model is……◦ A depiction of a program showing what the program will do and

what it is to accomplish.

◦ A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes

◦ The core of program planning and evaluation

20

Inputs Outputs Outcomes

Get food Eat food Feel betterHungry

Situation

Evaluation ResourcesEvaluation Resources W.K. Kellogg Foundation – “Evaluation Toolkit”◦ http://www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&L

anguageID=0 Newsletter resource – The PEN (Program Evaluation

News)◦ http://www.the-aps.org/education/promote/content/newslttr3.2.pdf

NSF-sponsored program◦ www.evaluatorsinstitute.com American Evaluation Association◦ www.eval.org Western Michigan University, The Evaluation Center◦ http://ec.wmich.edu/evaldir/index.html (directory of evaluators) OSRHE list of evaluators and other resources◦ http://www.okhighered.org/grant%2Dopps/writing.shtml “Evaluation for the Unevaluated” course◦ http://pathwayscourses.samhsa.gov/eval101/eval101_toc.htm

21

Evaluation ResourcesEvaluation Resources The Research Methods Knowledge Base

◦ http://www.socialresearchmethods.net/ The What Works Clearinghouse

◦ http://www.w-w-c.org/ The Promising Practices Network

◦ http://www.promisingpractices.net/ The International Campbell Collaboration

◦ http://www.campbellcollaboration.org/ Social Programs That Work

◦ http://www.excelgov.org/ Planning an Effective Program Evaluation short

course◦ http://www.the-aps.org/education/promote/pen.htm

22

Schumacher Schumacher Consulting.orgConsulting.org

In DIRE need of grant assistance?

We provide All Things Grants!™

Grant Development | Instruction | Research | Evaluation

Established in 2007, Schumacher Consulting.org is an Oklahoman-owned firm that works in partnership with organizations to navigate transformation and

achieve change by securing funding and increasing program success.

23