If we build a partnership will they come?

Integrating Needs Assessment, Process Evaluation, and Impact Evaluation

Ronald Jester and Robert Wilson, University of Delaware

rcjester@udel.edu, rwilson@udel.edu

A Program Evaluation may include

• Process Indicators

or• Impact Indicators

or• A combination of Process and Impact

Indicators

A Process Evaluation Analyzes Program Inputs, such as

•Outreach Activity•Services Delivered•Adherence to Grant

Proposal•Day-by-day Staff Activity•Costs •Materials

An Impact Evaluation Analyzes Program Outputs, such as

Productivity•Farm output•Individual farmer or waterman output•Efficiency•Benefits/Costs

Client Satisfaction with services

Improvements in Quality of Life

AgrAbility evaluations may include both process and impact indicators

The Delaware Maryland Program (DMAP) employs a systems model for program evaluation

This model starts with the target population - -2400 farmers with disabilities

The model tracks the target population through a service delivery system

The system includes a number of filters that regulate the flow of AgrAbility clients

The model includes a number of efficiency measures that identify impediments to efficient client flow

The AgrAbility Process includes 5 steps:

1. Identify farmers with disabilities.

2. Diagnose their problems.

3. Identify the services that will alleviate the problems.

4. Find resources to support for the services.

5. Deliver the services.

The Delaware-Maryland project relies on “partnerships” to deliver services.

The efficiency of the project depends on

1. Locating the target population 2. Engaging the target population 3. Linking the target population with the

partners 4. Routing the clients to the partners in the

network 5. Delivering appropriate services6. Producing specific outcomes (impacts)

as a result of the services

Overall efficiency is gauged by theproportion of the target population

• Located• Diagnosed • Referred to services • Provided services • Helped by services

Areas of Evaluation Process Evaluation Impact Evaluation

Locating Clients Diagnosing Clients Referring Clients to

services Providing Services

Measuring the Impact of Services

The Model (Figure 1) portrays the flow of a target population through a series of filters.

As the target population of 2400 farmers flowsthrough the service delivery system an increasing numberdrop out at each stage.

How to interpret Figure 1 The target group is the estimated 2400 disabled farmers who reside in

Delaware and Maryland. This estimate was generated from a random survey.

A farmer may seek help the AgrAbility program (columns B and C) or a program can reach out to a farmer (columns D,E, and F).

Line 1 shows that 280 of the estimated 2400 disabled farmers were contacted either by AgrAbility or another program.

Line 2 shows that 168 farmers out of the 280 farmers who were contacted were subsequently evaluated.

Line 3 shows that 120 of the farmers who were evaluated were subsequently referred for service.

Line 4 shows that 105 of those who were referred for service subsequently received a service.

Line 5 shows that 90 out of the 105 farmers who received a service subsequently experienced an improvement in their quality of life.

Figure 1

Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers

Sought Help From Did Not Seek Help Contacted by

DMAP

110Other

110DMAP

60Other

?None

?

Assessed

110Assessed

?

Referred

80Referred

?

Service Provided

60

Service Provided

?

Quality of Life Improved

50

Quality of LifeImproved?

Assessed

58Assessed

?Not Assessed

Referred

40Referred

?

Not Referred

Service Provided?

No Service Provided

Quality of Life Improved ?

Quality of Life Improved

40

Quality of Life Improved

?

B C D E F

1

2

3

4

5

Service Provided

45

Six Process Measures are employed

Each measure indicates the efficiency of a different part of the

system

1. Market Penetration

The percentage of the target population that contacts social service agencies for help.

Include all agencies- AgrAbility and other agencies that provide services to farmers with

disabilities

Market Penetration Measurementis based on Target Population: the estimated 2400 disabled farmers residing in MD and DE

The farmers who were contacted are indicated in line 1 of Figure 1

Figure 1

Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers

Sought Help From Did Not Seek Help Contacted by

DMAP

110Other

110DMAP

60Other

?None

?

Assessed

110Assessed

?

Referred

80Referred

?

Service Provided

60

Service Provided

?

Quality of Life Improved

50

Quality of LifeImproved?

Assessed

58Assessed

?Not Assessed

Referred

40Referred

?

Not Referred

Service Provided?

No Service Provided

Quality of Life Improved ?

Quality of Life Improved

40

Quality of Life Improved

?

B C D E F

1

2

3

4

5

Service Provided

45

1. Market Penetration(220)/(2400)

The percentage of the target population that contacts social

service agencies for help

1. Market Penetration(B1+ C1)/(2400)=.09

The percentage of the target population that contacts social

service agencies for help

Market Penetration is a crude measure of public service advertising reach.

2. Outreach

(B1 + C1+DI)/TP

% of the target population (TP)

identified and contacted by

a specific program

The target population that was reached is shown in cells B1, C1, and D1 of Figure 1.

Figure 1

Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers

Sought Help From Did Not Seek Help Contacted by

DMAP

110Other

110DMAP

60Other

?None

?

Assessed

110Assessed

?

Referred

80Referred

?

Service Provided

60

Service Provided

?

Quality of Life Improved

50

Quality of LifeImproved?

Assessed

58Assessed

?Not Assessed

Referred

40Referred

?

Not Referred

Service Provided?

No Service Provided

Quality of Life Improved ?

Quality of Life Improved

40

Quality of Life Improved

?

B C D E F

1

2

3

4

5

Service Provided

45

3. Assessment Efficiency.

(B2+D2)/(B1+D1)

Assessment Efficiency is the percentage of potential clients eligible for assessment that was assessed

The clients who were assessed are show in cells B2 and D2 of Figure 1.

Clients eligible for assessment are shown in cells B1 and D1 of Figure 1

3. Assessment Efficiency. (100 +58)/

(110 +60)=.93

The percentage of potential clients eligible for for

assessment that was assessed

4. Referral Efficiency. (B3+D3)/(B2 with positive screen + D2 with positive

screen)

Referral efficiency is the percentage of clients with a positive screen that was referred for

service.

Clients who were referred for services are shown in cells B3 and D3.

Those with a positive screen are in cells B2 and D2.

Figure 1

Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers

Sought Help From Did Not Seek Help Contacted by

DMAP

110Other

110DMAP

60Other

?None

?

Assessed

110Assessed

?

Referred

80Referred

?

Service Provided

60

Service Provided

?

Quality of Life Improved

50

Quality of LifeImproved?

Assessed

58Assessed

?Not Assessed

Referred

40Referred

?

Not Referred

Service Provided?

No Service Provided

Quality of Life Improved ?

Quality of Life Improved

40

Quality of Life Improved

?

B C D E F

1

2

3

4

5

Service Provided

45

5. Service Delivery Efficiency. (B4 + D4)/B3 +D3)=

(60+ 45)/(80 + 40)=.88

Service delivery efficiency is the percentage of clients referred for service

that received service.Clients who received a service are show

in cells B4 and D4.Clients who were referred for service are

shown in cells B3 and D3.

The Delaware-Maryland Project employs the SF-36 Quality of Life Instrument as an Impact Measure

Additional impact measures that are specific to farmers with disabilities

and assistive technology will be incorporated in the evaluation

These measures are being developed by a committee drawn

from AgrAbility programs and national program staff

The SF-36 is the most widely used quality of life instrument in health and social services

The SF-36 is calculated for 8 health scales (0-100)1. Physical limitations

2. Limitations in social activities

3. Limitations in work and home roles

4. Bodily pain

5. Psychological distress and wellbeing

6. Limitations because of emotional problems

7. Energy and fatigue

8. General health perceptions

The SF-36 has been validated through a

numerous studies

Several thousand scientific articles employ the SF-36 as a

quality of life measure for measuring program impact

All active Delaware and Maryland clients are currently completing SF-36 interviews

The SF-36 serves two purposes

The SF-36 is used in the initial assessment to develop a service plan.

The SF-36 is also used in a 1-year follow-up evaluation to assess changes in the quality of life subsequent to the initial assessment.

6. Improved quality of Life: (# of Clients with improved quality of life) /

(# of Clients that Received AgrAbility Service)

Clients with improved quality of life are shown in cells B4 and D4. Those who received a

service from AgrAbility are shown in cells B3 and D3.

(B4 + D4)/(B3 +D3)=

(50 + 40)/(60 + 45)=

90/105=

.86

Finally, 2 questions for the audience:

Which of the measures focus on the operation of partnerships in AgrAbility programs?

What other techniques should be used to evaluate partnerships?