Introduction to Performance Measurement for AmeriCorps*VISTA 1-800-548-3646 Washington Dulles...

Introduction toIntroduction toPerformance MeasurementPerformance Measurement

for AmeriCorps*VISTAfor AmeriCorps*VISTA

1-800-548-3646 www.projectstar.org1-800-548-3646 www.projectstar.org

Washington Dulles Hilton Washington Dulles Hilton

September 12-14, 2005September 12-14, 2005

Day One - Afternoon Day One - Afternoon SessionSession

This session will cover: Six Steps of Performance

Measurement (PM) Types of Results: Outputs,

Intermediate and End Outcomes Use of the Logic Model as a

Planning Tool

Performance measurementPerformance measurementis a six-step process foris a six-step process for

regularly measuringregularly measuringyour project results.your project results.

Performance Measurement:The Six Steps

Performance Measurement Six Steps

The logic model describes The logic model describes your AmeriCorps*VISTA your AmeriCorps*VISTA

project.project.Use it to identify your Use it to identify your

project activities and what project activities and what you will measure to gauge you will measure to gauge the success of your project. the success of your project.

1. Develop Logic Model

& Project Plan



& Project Plan

2. DevelopPM Plan

Identifying your data Identifying your data sources and data sources and data

collectors.collectors.Determine a timeline Determine a timeline for data collection, for data collection,

analysis, and analysis, and reporting reporting



& Project Plan

2. DevelopPM Plan

3. Developand Test

Instruments

Adapt existing Adapt existing instruments or instruments or

develop your own; develop your own; pilot test to make sure pilot test to make sure

instruments collect instruments collect valid, appropriate valid, appropriate

informationinformation



& Project Plan

2. DevelopPM Plan

3. Developand Test

Instruments4. Collect

Data

Go out and collect Go out and collect your data (e.g., fill your data (e.g., fill

out attendance out attendance sheets, administer sheets, administer surveys, conduct surveys, conduct

interviews)interviews)



& Project Plan

2. DevelopPM Plan

3. Developand Test


Data

5. AnalyzeData

Aggregate and Aggregate and analyze data; analyze data;

generate generate summary info summary info

(totals, averages, (totals, averages, etc)etc)



& Project Plan

2. DevelopPM Plan

3. Developand Test


Data

5. AnalyzeData

6. ReportResults

Write progress Write progress report; include report; include

your performance your performance measurement measurement

resultsresults



& Project Plan

2. DevelopPM Plan

3. Developand Test


Data

5. AnalyzeData

6. ReportResults

ContinuousProgram

Improvement

What Are Results?What Are Results?Outputs and outcomes your AmeriCorps*VISTA project hopes to achieve by engaging in service activitiesOutputs: Services delivered and products completed by VISTA membersOutcomes: Changes or benefits that occur as a result of services delivered

Intermediate Outcomes: Changes that are short of the “final” result you hope to achieveeach yearEnd Outcomes: The most significantchanges that your AmeriCorps*VISTAproject hopes to achieve each year

Sample ScenarioSample Scenario

Community-based organization (CBO)Community-based organization (CBO)has existing mentoring program, buthas existing mentoring program, butfaces challenges in recruiting and faces challenges in recruiting and retaining volunteer mentors.retaining volunteer mentors.

CBO unable to deliver high-quality service CBO unable to deliver high-quality service to youth.to youth.

VISTA members will serve with CBO to VISTA members will serve with CBO to develop system to recruit, train and develop system to recruit, train and support volunteer mentoring program.support volunteer mentoring program.

Position descriptions for volunteer mentors Written procedures for screening volunteer

mentors Tracking database Training curriculum for new volunteer mentors Written procedures to provide ongoing support

to volunteer mentors

Outputs answer the question, “How much service did we do?” They do NOT answer the question, “What changed as a result of the service?”

OutputsOutputs

Trained agency personnel demonstrate

understanding of how to use new system

and procedures (new knowledge and skills).

Agency personnel use new systems and

procedures to recruit, train, and support

volunteer mentors (changes in behavior).

Intermediate outcomes are important milestones on the way to achieving end outcomes. Achieving intermediate outcomes can lead to attaining end outcomes.

Intermediate OutcomesIntermediate Outcomes

More youth are served by volunteer mentors under the new system than were served under the previous system (increased beneficiaries served).

More youth experience a close and stable relationship with a caring adult, because a greater proportion of volunteer mentors complete their service commitments (improved quality of service).

End outcomes are changes that reflect the overarching goals of your AmeriCorps*VISTA project.

End OutcomesEnd Outcomes

1616

Examples of ResultsExamples of Results

OUTPUTS INTERMEDIATE OUTCOME(S)

END OUTCOMES

Individual Development Account trainings providedParticipants

IDA program participants accumulate savings to use for purchase of assets

IDA program participants use accumulated savings to purchase assets they had planned to purchase

Training sessions providedParent Network Collaborative meetings

Network referral system developed

Network agencies expand quality and range of services

1717

ExerciseExercise

Identifying Types of ResultsIdentifying Types of Results

1818

Benefits of Developing a Logic ModelBenefits of Developing a Logic Model Helps to build consensus about

program goals and desired results

Clarifies relationship between keyprogram elements (inputs, activities, outputs, and outcomes)

Brings “fuzzy goals” down to earth

Begins to identify things to measure

Provides solid foundation for developing project plan

1919

AmeriCorps*VISTA Logic ModelAmeriCorps*VISTA Logic Model

A Logic Model = a planning tool that diagrams how you intend to achieve your AmeriCorps*VISTA project goals by

combining resources (inputs) and service activities to produce specific results (outputs and

outcomes) that address poverty-related community needs.

2020


Community Need

Community Need = Unresolved poverty-related problem or

issue in the community that your VISTA project will

address

2121


Community Need

Goals Inputs

Activities

Outputs

Intermediate Outcomes

EndOutcomes

End Outcomes=The most significant changes achieved

by your projectAnswer the questions:

“So what”? “What changed?”

2222


Goals = Broad purposes toward which your

VISTA project is directed

Community Need

Goals

2323


Inputs= Resources your VISTA project will need

to engage in the identified activities and produce outputs and outcomes

(human, financial, organizational, community)

Community Need

Goals Inputs

2424


Activities=What VISTA members do to address the community need and achieve desired

results

Community Need

Goals Inputs

Activities

2525


Outputs:Reflect the amount of service completed

by VISTA members.Outputs do not answer the questions:

“what changed as a result of the work?” or “how did people or the organization

benefit?”

Community Need

Goals Inputs

Activities

Outputs

2626


Intermediate Outcomes=Initial changes achieved by your project; milestones on the way to achieving end

outcomes

Community Need

Goals Inputs

Activities

Outputs


2727


Community Need

Goals Inputs

Activities

Outputs


EndOutcomes

End Outcomes=The most significant changes achieved

by your projectAnswer the questions:

“So what”? “What changed?”

2828


Community Need

Goals Inputs

Activities

Outputs


EndOutcomes

Program Planning

Intended Results

2929

Community Need

Goals Inputs

Activities

Outputs


EndOutcomes

Logic Model for a Mentoring Program Logic Model for a Mentoring Program (Year 3)(Year 3)

Children of incarcerated parents lack social and emotional support for positive youth development that can help to break the cycle of poverty.

Develop, enhance systemsto recruit, train, and support volunteer mentors.

2 VISTAs

Supervisor

Program staff

Money

Partners

Assist with implementa-tion, assess performance of new systems, identify promising practices.

Policiesand guidance revised for volunteer recruitment and sustain-ability systems.

Improved recruitment: more youth are served by more volunteer mentors.

Improved retention: more volunteer mentors complete service.

Improved program quality: more volunteer mentors form closer bonds with youth; mentoring relationships last longer.

3030

ExerciseExercise

The Logic Model PuzzleThe Logic Model Puzzle

3131

ExerciseExercise

Develop a Logic Model Develop a Logic Model

for Your Projectfor Your Project

3232

End of Day ReviewEnd of Day Review

++What Worked Well What Worked Well

About Today?About Today?

∆∆How Could Tomorrow How Could Tomorrow

Be Made Even Be Made Even Better?Better?

3333

Day Two – Morning SessionDay Two – Morning Session

This session will cover:This session will cover:

1.1. Moving from a Logic Model to the Moving from a Logic Model to the Project PlanProject Plan

2.2. CNCS Requirements for VISTACNCS Requirements for VISTA

3.3. Components of a Project PlanComponents of a Project Plan

4.4. Indicators, Targets, InstrumentsIndicators, Targets, Instruments

3434

AmeriCorps*VISTA Performance Measurement Requirements

One project plan containing one or more One project plan containing one or more goalsgoals

Identify activities and results for each goalIdentify activities and results for each goal Results are written as outputs, intermediate Results are written as outputs, intermediate

outcomes, or end outcomesoutcomes, or end outcomes Label 3-5 of these results as “performance Label 3-5 of these results as “performance

measures”measures”– One OutputOne Output– One Intermediate OutcomeOne Intermediate Outcome– One End OutcomeOne End Outcome

Identify indicators, targets, and instruments Identify indicators, targets, and instruments for each performance measurefor each performance measure

3535

VISTA Project Plan VISTA Project Plan ComponentsComponents

Goals (Same) Activities (formerly “Objectives”) Results (New)

Output(s)Intermediate Outcome(s)End Outcome(s)

3636

AmeriCorps*VISTA Project PlanAmeriCorps*VISTA Project Plan

Column AGoals, Activities, and Results

Column BDate(s)

Column CProject Reporting

GOAL 1

Activities

Results Outputs Intermediate Outcomes End Outcomes

GOAL 2

Activities


GOAL 3 Etc.

3737



Column BDate(s)


GOAL 1

Activities


GOAL 2

Activities


GOAL 3 Etc.

3838



Column BDate(s)


GOAL 1

Activities


GOAL 2

Activities


GOAL 3 Etc.

3939

From Logic Model to Project PlanFrom Logic Model to Project Plan


Column BDate(s)


Goal(s)

Activities

Results

Outputs


End Outcomes

Community Need

Goals Inputs

Activities

Outputs


EndOutcomes

4040

Multi-Year Goals,Multi-Year Goals,Annual Activities and Annual Activities and ResultsResults

GoalsGoals are are multi-yearmulti-year..

Activities and results may vary from one Activities and results may vary from one

year to the next.year to the next.

Each year the project builds on the previous Each year the project builds on the previous

year’s activities and results.year’s activities and results.

Only list Only list activitiesactivities and and resultsresults for for current current

yearyear..

4141

ExerciseExercise

Identify One Goal for Your ProjectIdentify One Goal for Your Project For that Goal, identify the Activities For that Goal, identify the Activities

for Year One and Possible Results:for Year One and Possible Results:– OutputsOutputs– Intermediate outcome(s)Intermediate outcome(s)– End outcome(s)End outcome(s)

4242

AmeriCorps*VISTA Project PlanAmeriCorps*VISTA Project PlanColumn AGoals, Activities, and Results

Column BDate(s)


GOAL 1

GOAL 2

Activities

Results Output “Performance Measure” Indicator Target Instrument Intermediate Outcome “Performance Measure” Indicator Target Instrument End Outcome “Performance Measure” Indicator Target Instrument

4343

AmeriCorps*VISTA Project PlanAmeriCorps*VISTA Project PlanColumn AGoals, Activities, and Results

Column BDate(s)


GOAL 1

GOAL 2

Activities

Results Output “Performance Measure” Indicator Target Instrument Intermediate Outcome “Performance Measure” Indicator Target Instrument End Outcome “Performance Measure” Indicator Target Instrument

4444

What is an Indicator?What is an Indicator?

Information you collect to determineInformation you collect to determineprogress toward achieving resultsprogress toward achieving results

Concrete and measurableConcrete and measurable

Answers the question, “What is myAnswers the question, “What is my evidence that something has occurred?”that something has occurred?”

Output indicator usually begins with Output indicator usually begins with “Number of...”“Number of...”

Outcome indicator usually begins with Outcome indicator usually begins with “Percent of...”“Percent of...”

4545

Example of Result and IndicatorsExample of Result and Indicators

Desired Result: Desired Result: Increase service to clientsIncrease service to clients

Possible Indicators:Possible Indicators:

- Greater number of clients served Greater number of clients served (number of clients served, compared to last (number of clients served, compared to last year)year)

- Clients receive more servicesClients receive more services(number of services provided, per client, (number of services provided, per client, compared to last year)compared to last year)

- Clients receive services of greater value Clients receive services of greater value (dollar value of services provided, per client, (dollar value of services provided, per client, compared to last year)compared to last year)

4646

What is a Target?What is a Target? Describes amount of change youDescribes amount of change you

anticipate achieving each year dueanticipate achieving each year dueto efforts of VISTA members.to efforts of VISTA members.

Based on indicator – specific, measurable Based on indicator – specific, measurable amount of change that will signify amount of change that will signify success

Outcome Target:Outcome Target:– Who will experience change? will experience change?– How many people will experience change? people will experience change?– How much change will occur (compared to change will occur (compared to

baseline)?baseline)?– Over Over what time period will change occur (one will change occur (one

year)?year)?

4747

TargetTarget describes amount/level of describes amount/level ofchange you anticipate achieving inchange you anticipate achieving inone year due to efforts of VISTA members.one year due to efforts of VISTA members.

Based on Based on indicatorindicator (concrete, measurable) (concrete, measurable)

Rely on Rely on experienceexperience and knowledge of and knowledge of community community to judge how much change to to judge how much change to expect in one year.expect in one year.

Keep in mind your Keep in mind your resources.resources.

Be realistic!Be realistic!

Formulating Annual TargetsFormulating Annual Targets

4848

Indicator and Target Indicator and Target ExamplesExamplesOutput Outcome

Result Volunteer Volunteer database database developed.developed.

Improve Improve volunteer base.volunteer base.

Indicator Volunteer Volunteer database database developed.developed.

Number ofNumber of new new volunteers volunteers recruited.recruited.

Target OneOne volunteer volunteer database database developed.developed.

300300 new new volunteers volunteers recruited.recruited.

4949

What is an Instrument?What is an Instrument?

Document or form used to collectDocument or form used to collectinformation from data sourceinformation from data source

Data source: Person, place or group that Data source: Person, place or group that provides you with information to measure provides you with information to measure indicatorsindicators

Examples: Attendance rosters, tally sheets, Examples: Attendance rosters, tally sheets, questionnaires, checklistsquestionnaires, checklists

Adapt existing instruments or develop your Adapt existing instruments or develop your ownown

5050

Advice about InstrumentsAdvice about Instruments

Make sure instrument will measure Make sure instrument will measure

indicatorsindicators

Consider who will Consider who will administeradminister instrument instrument

Consider who will Consider who will complete complete instrumentinstrument

Consider Consider whenwhen to administer instrument to administer instrument

Consider data Consider data accessibilityaccessibility

5151

Project Plan: Performance MeasuresProject Plan: Performance MeasuresColumn AGoals, Activities, and Results

Column BDate(s)


GOAL

ACTIVITIES (Year 1)

RESULTS (Year 1)Outputs:1. Volunteer mentor recruitment systems are developed.2. Agency personnel receive orientation in new systems

“Performance Measure”Indicator: # agency personnel oriented Target: 2 F/T and 4 P/T agency personnel oriented Instrument: Orientation Log

Qtr 1–3Qtr 1–3

Qtr 4

Qtr 4

5252


Column BDate(s)


GOAL

ACTIVITIES (Year 1)



Intermediate Outcome: Agency personnel understand how to use new systems. “Performance Measure” Indicator: # personnel who correctly identify staff roles and who can find info in new databaseTarget: 5 personnel can recruit, screen and train mentorsInstrument: Mentoring System User Checklist

Qtr 1–3Qtr 1–3

Qtr 4

Qtr 4

5353


Column BDate(s)


GOAL

ACTIVITIES (Year 1)



Intermediate Outcome: Agency personnel understand how to use new systems. “Performance Measure” Indicator: # personnel who correctly identify staff roles and who can find info in new databaseTarget: 5 personnel can recruit, screen and train mentorsInstrument: Mentoring System User Checklist

End Outcome: Agency personnel use new systems. “Performance Measure”Indicator: # agency personnel using # new systems Target: Four agency personnel, two systemsInstrument: Year-end Agency Interview

Qtr 1–3Qtr 1–3

Qtr 4

Qtr 4

5454

ExerciseExercise

Identifying Indicators, Identifying Indicators, Targets and How Measured Targets and How Measured

for Outcomesfor Outcomes

5555

ExerciseExercise

Add Enhancements to Your Add Enhancements to Your Project Plan ResultsProject Plan Results

5656

Day Two – Afternoon Day Two – Afternoon SessionSession


1.1. Selecting InstrumentsSelecting Instruments

2.2. Data Collection StrategiesData Collection Strategies

3.3. Data Aggregation and AnalysisData Aggregation and Analysis

4.4. ReportingReporting

5757

Strategies to Ensure Successful Strategies to Ensure Successful ReportingReporting

From the beginning…

Align work plan and instruments Plan for collecting the information Understand how to analyze data Anticipate how you will report

5858

What Do I Report On?What Do I Report On? Actual Activities

Actual Output– Actual target as compared to anticipated target– How data collected

Actual Intermediate Outcome– Actual target as compared to anticipated target– How data collected

Actual End Outcome– Actual target as compared to anticipated target– How data collected

5959

Data Collection Issues

Can you access the data source (beneficiaries, organizations)?

Does an instrument (site or project) exist? Are they willing to give you data? (burden,

capacity) Can you get the data when you need it? Are your helpers (e.g. VISTAs) trained? Do you have a mechanism to collect

summarized data from sites?

6060

Methods

Logs, Tally Sheets, Attendance Rosters Survey, Questionnaire (Check list, Index, Scale) Interview (Interview Guide)

Service Recipients Interview Expert/Provider/Key Informant

Observation (Observation Guide) Case Study (Measurable change of a case over time) Focus Group (Focus Group Guide) Secondary Data (Existing records and information) Test / Experimental Designs

6161

Instrument Considerations Connect to Project Plan results

– Measures the identified PM Aim for appropriate method

– Measures the correct population (data source) Lower burden of use

– e.g. Captive audience, (part of project)– Balance between “best” method and

“manageable” method Assure that sites have instruments or

summary tool

6262

ExerciseExercise

Instrument and Data Instrument and Data Collection Collection Review and DiscussionReview and Discussion

6363

Data AnalysisData Analysis

Quantitative FrequenciesFrequencies RangesRanges PercentsPercents MeansMeans Pre-post comparisonPre-post comparison

Qualitative Content analysisContent analysis

– Pre-existing Pre-existing themesthemes

– Emergent themesEmergent themes

6464

ExerciseExercise

Data Aggregation and Data Aggregation and Analysis Analysis

6565

Reporting Tips

Provide evidence. Avoid hearsay and generalizations.

Be candid about your efforts. Don’t claim too little. Don’t claim too much.

Tie report to Project Plan. State results in same format as original target in Project Plan. Pay attention to enhancements: Pay attention to enhancements:

indicator, target, and instrumentsindicator, target, and instruments

6666

Report TemplateReport TemplateReporting Format for Each Reporting Format for Each

Performance MeasurePerformance Measure

Actual TargetActual Target::

1) 1) Compare the actual to the anticipated target stated in Compare the actual to the anticipated target stated in Column A. Column A.

2) 2) Met target? If target not met, explain. Met target? If target not met, explain.

3) 3) Include any additional information about your result.Include any additional information about your result.

How Data Collected: How Data Collected:

1) 1) Describe how the result was measured. Describe how the result was measured.

6767

Reporting Each ResultReporting Each ResultActual Target:

1) Compare the actual target to the anticipated target stated in Column A. Interpret results; explain what it means

2) Met target? If target not met, explain. Include numbers not just percentages.

Explicitly state that you did (not) meet target

3) Include any additional information about your result.Describes challenges, next steps

6868

Describing Your ResultsDescribing Your Results

Actual Target: Actual Target: 1) Compare the actual target to the anticipated target 1) Compare the actual target to the anticipated target stated in Column A. [Always include an actual number] stated in Column A. [Always include an actual number]

“10 interview respondents (86% of collaborative) identified at least three beneficial practices resulting from the partner collaboration. This exceeds our target of one project improvement. The top two included an increase in clients served and…”

6969

Reporting Format for Each Reporting Format for Each ResultResult

How Data Collected:

1) Describe how the result was measured. What was used to track the result? If applicable, how was the instrument administered?

(When was it administered? How often?) If applicable, include total number approached as well

as those who responded.

7070

Describing Your ResultsDescribing Your ResultsHow Data Collected:

1) Describe how the result was measured

“During the 2nd and 3rd weeks of April, the Project Supervisor, and two VISTAs conducted telephone interviews during the afternoon and evening…”

“...Of the 12 collaborative partners called, 10 completed the interview that assessed their practices resulting from the collaborative…”

7171

Describing Your Work Before Describing Your Work Before Results Are AvailableResults Are Available

“...We don’t have any results yet.”

7272

What do I Report When I Have What do I Report When I Have “Nothing to Report” Yet? “Nothing to Report” Yet?

We are…..Development/piloting of instruments

Introducing instrument to sites

Gaining access to data

Training data collectors

Challenges and proposed solutions

Administering pre-survey data. (If you have baseline data, report it early!)

Say something! Don’t wait.

7373

Describing Your Work Before Describing Your Work Before Results Are Available (Improved)Results Are Available (Improved)

How Data Collected: 1) Describe how the result was measured

“...In preparation for implementing performance measurement, we drafted a telephone interview protocol containing six questions designed to assess partner collaborative practices. During the first week of March, two collaborative members reviewed the draft and six VISTAs were trained to conduct interviews...”

7474

Day Three – Morning Day Three – Morning SessionSession


1.1. ReportingReporting

2.2. Using the Results of Performance Using the Results of Performance Measurement Measurement

3.3. eGrants and VISTAeGrants and VISTA

4.4. Closing - Receive box lunch and Closing - Receive box lunch and departure departure

7575

ExerciseExercise

Progress Report Critique Progress Report Critique Using ChecklistUsing Checklist

7676

ExerciseExercise

Communicating SuccessCommunicating SuccessWays to Use PM Results to Strengthen Ways to Use PM Results to Strengthen

and Promote Your Projectand Promote Your Project

Additional Resources:Additional Resources: Contact your CNCS State OfficeContact your CNCS State Office Contact Project STARContact Project STAR

• www.projectstar.orgwww.projectstar.org• 1-800-548-36561-800-548-3656• [email protected]@aiweb.com

http://www.projectstar.org/

mailto:[email protected]

Introduction to Performance Measurement for AmeriCorps*VISTA 1-800-548-3646 Washington Dulles...

Documents

Transcript of Introduction to Performance Measurement for AmeriCorps*VISTA 1-800-548-3646 Washington Dulles...