Reflect and Revise: Evaluative Thinking for Program Success

Tom DeCaigny andLeah Goldstein Moses

Topics to guide our discussion

• Benefits of evaluation and assessment to staff, program participants and the organization as a whole

• Building on existing knowledge and resources for evaluation

• Launching evaluation and assessment efforts • Choosing the right methods of assessment and

evaluation • Learning from other organizations to integrate

evaluative thinking and use evaluation to support programs

Let’s get to know each other

• Your presenters: Leah Goldstein Moses and Tom DeCaigny

• How comfortable are you with evaluation? • What comes to mind when someone asks

you to evaluate and/or assess your work?• What are your challenges with evaluation?

First, some definitions

• Assessment: The act of determining the standing of an object on some variable of interest, for example, testing students and reporting scores.

• Evaluation: Systematic investigation of the worth or merit of an object; e.g., a program, project, or instructional material. – Source Joint Committee on Standards for Educational

Evaluation. (1994). The Program Evaluation Standards, 2nd ed. Thousand Oaks, CA: Sage. (used with permission of publisher)

Benefits of evaluation and assessment

• Be accountable to important stakeholders• Professional and organizational

development – learn how you are doing• Program management – see where your

programs need continued support or improvement

• Investigation and learning• Feeds curiosity and fosters innovation

Stories of evaluation benefits

• Utilizing evaluation results to identify a need and develop a new project (special needs - ARISE case study)

• Utilizing evaluation results to improve program quality and design (teaching artist training case study)

• Evaluation and assessment as part of reflective artistic practice (A Cycle of Artistic Inquiry case study)

A Cycle of Artistic InquiryPerforming Arts Workshop and Dr. Richard Siegesmund (2000)

Ensuring evaluation benefits are shared

• Evaluation, at its best, is “engaged in”, not “done to”.

• When developing or improving evaluation systems, think about who will be doing the work for the evaluation (distributing surveys, gathering information, analyzing the data):– Is there a way to decrease the burden? – Is there a way to provide benefits?

• Examples of evaluation efficiency and incentives

Build on existing knowledge and resources

• Internal insights can really support a new evaluation effort. Determine:– What do we collect already? – What does the information we already have tell

us?– What can we report on just from our own internal

record keeping or observations?– What can we gather in the course of our work –

during existing programs, contact, etc.– Example in ARISE: student achievement – test

scores..

Build on existing knowledge and resources

• Use external information, such as reports done by organizations you admire:– What did they study?– How did they gather information?– Can you apply any of their tools or methods?– Can you infer/generalize anything from their

findings so you don’t have to replicate their effort?

Getting started in evaluation and assessment

• Logic models are incredibly useful. They help you:– Determine how your efforts are related to

your expected impact– Map out what you want to measure, and why

• You can determine what data you already have and what you are lacking during the logic model process

Getting started in evaluation and assessment

• After you have created a logic model and/or identified data gaps, you can determine what you are going to collect, when, and in what format

• Surveys are great; but in the arts, you might want to use an artistic process or other valid alternative assessments– Illustrative rubrics– Observation– Portfolios

Learning from others’ experiences

• Notes from our discussion:– What evaluation approaches have worked well for you?

Electronic portfolios in classroom. Reflects project-based learning. Can see progression over course of year. Parents, administrators can also see.

Time to reflect can be challenging but is important. “Level Best” is a good resource. Tried to find things that existed and could

be used in the organization. Festin“Theater Communications Group” is a good resource. Anecdotal information, journals can have a bigger impact on Boards and

other audiences that don’t care much for quantitative. Site visits for Board members are required as part of their responsibilities.

Having tools at your fingertips – did anything good happen today? Did any challenges happen? Right at time they are needed.

Using incentives – crayola pencils were good for parents. – Where have you struggled?Finding time for reflection. Having right tool for evaluation. Logic models- can be

cumbersome or difficult to use. Having a way to capture, understand and communicate unexpected outcomes. Avoiding bias through body language, tone – need to make sure to encourage honesty.

Board can ignore quantitative. Finding sophistication/depth in questions has been hard when they are in a survey.

Our contact information

Tom DeCaigny, Executive Director, Performing Arts Workshop

T:  (415) 673-2634 x207 / F:  (415) 776-3644E: tom@PerformingArtsWorkshop.org www.PerformingArtsWorkshop.org

Leah Goldstein Moses, President, The Improve GroupT:  (877) 467-7847 x11 / F:  (612) 656-1731 E: leah@theimprovegroup.com www.theimprovegroup.com