Assessing CustomerSatisfaction with

Vocational Rehabilitation Programs

Florida State University Survey Research LaboratoryCollege of Social Sciences

www.fsu.edu/~survey

Dr. Mary Stutzman, DirectorFSU Survey Research Laboratory

Presentation at the Quarterly Meeting

West Palm Beach , September 14 – 16 2009

2

Why Should the FRC Survey Customers?Provides an “outside” look of customer service.

Customers are sometimes uncomfortable to give feedback directly to the agency giving them services.

Gives customers an opportunity to “speak” to someone outside the Department.

Open-ended comments800 number calls and referrals to FRC

Provides longitudinal data base to assess trends.

Data can be “packaged” for to address specific concerns.

3

Evolution of the Survey Effort

Survey of Customers with Cases ClosedLaunched 2001Questions developed by FRC Evaluation Committee and DVR staffDesigned survey instrument one-page legal front and backSample Design—all cases closed previous month

Customers in Program after 6 monthsLaunched 2002 Questions mirrored Closed Cases SurveySample Design—random sample of 700 per monthCustomers in program after 6 months

4

Customer Surveys OverviewGather information from customers:

Program OutcomesServicesStaff

CLOSED CASES--Surveyed Cases Closed Successful (Status 26)Unsuccessful (Status 80 – 97, 89)

ACTIVE CASES—Surveyed Customers in Program after 6 months

Random sample of 700 per month; 800 beginning July 2008

5

Survey ApproachClosed Cases—Survey all cases closed at the end of month

LetterInstrument• 12 Closed items• 2 Open-ended

Active Cases—Survey sample of 800 cases still active (opened 6 months earlier)

LetterInstrument• 12 Closed items• 2 Open-ended

6

Staffing the Survey Effort

FSU Survey Research LaboratoryAll staff paidStaffed with trained interviewers 7 days a week• can handle PR issues on 800 line and conduct call-

in interviewsMail within 1-week of sample receiptData and open-ended responses entered Analysis of dataProvide reports and presentations on a quarterly and ad hoc basis

7

Design BenefitsSampling Strategy

Power of numbers to analyze by various segments (Areas, Disabilities, etc).Monthly—helps eliminate “seasonal” bias

Response CategoriesAllow “ordinary language” responses for respondentsHelps make reports understandable

OperationsMode choice—mail , Internet, phone800 number and open comments—public relations for Council

ReportingSufficient numbers for Quarterly TrendsSufficient numbers Annual Trends and sub-analyses

8

Respondents are King/Queen

Make convenient2-pages legalKeep short—12 Questions

Accommodate Respondent PreferencesWeb access800 number for phone responseTTY line

9

Privacy Issues: Called ”Project”

10

Closed Cases: Survey Instrument

11

Closed Cass: Survey Instrument

12

Survey Areas

Program OutcomesProgram Choice

ServicesStaff

13

Program Outcomes

Closed Cases

Active Cases

Program OutcomesOverall Satisfaction with Services X XServices Made Life Better X XWould Recommend to A Friend X XEmployment Outcomes and Satisfaction X

14

Program ChoiceProgram Choice Closed

CasesActive Cases

Informed of Alternative Service Providers and Vocational Goals X

Able to Choose Service Providers and Vocational Goals X

Informed of Alternative Service Providers XAble to Choose Service Providers XInformed of Alternative Vocational Goals XAble to Choose Alternative Vocational Goals X

15

Program ServicesProgram Services Closed

CasesActive Cases

Accessibility to VR Offices X XAppropriateness of Services Considering Rehabilitation Goals X

Promptness of Services Provided X XNeed Other Services but Did Not Receive X

Open-Ended Closed Cases

Active Cases

Especially helpful VR associates or services X XWhat VR can do to improve services X XWhy did not get a job X

16

Program Staff

Program Staff Closed Cases

Active Cases

Understand Needs and Feelings X X

Treated with Courtesy and Respect X X

Explained Responsibilities as Client X

Do What Say X

17

CLOSED CASES: APPROACH AND RESPONSE RATE

CUSTOMERS WITH CASES CLOSED: 2007-08 Surveys mailed monthlySurveys mailed to all customers whose cases were closed in the prior month October 2007 to September 2008 mailings18,100 surveys mailedTwo Mailings Analysis based on 5,176 returns

Response Rate = 29%

18

CURRENT CASES: APPROACH AND RESPONSE RATE

CUSTOMERS WITH CASES ACTIVE CASES: 2007-08Surveys mailed monthlySurveys mailed to random sample of 700-800 customers whose case is active 6 months after plan8,700 surveys mailedTwo Mailings Analysis based on 2,441 returnsResponse Rate = 28%

19

Adaptation and Change

Refined and revisited over timeInstrument—”already have a job”Letters—explanation of council and repeated surveysReports—content and format

20

FFY 2007-08

PROGRAM OUTCOMES

Overall Satisfaction --Very or Mostly SatisfiedClosed Cases (Q9) -- 85%Active Cases (Q11) – 79%

Services Made Life Better –Yes, Definitely Helped or Somewhat

Closed Cases (Q8) -- 86%Active Cases (Q10) -- 81%

Recommend to a Friend Yes, Absolutely or Probably Closed Cases (Q10) -- 90%Active Cases (Q12) -- 86%

21

FFY 2007-08

PROGRAM OUTCOMES[Closed Cases - Status 26]

Job Status of Customers (Q12)Nearly one-third of the customers (31%) obtained a job since becoming a clientNearly one-half (60%) already had jobs

Customers (Status 26) who Obtained New Jobs were Satisfied (Q12A)

83% were Very or Mostly Satisfied with their jobs

22

FFY 2007-08

PROGRAM SERVICES

Accessibility Very or Mostly SatisfiedClosed Cases (Q1) – 91%Active Cases (Q1) – 88%

Appropriate ServicesHighly Appropriate or Appropriate

Closed Cases (Q4) -- 88%Active Cases (Q8) -- 86%

Need Other ServicesYes--Definitely or Think So

Closed Cases (Q5)-- 29%

23

FFY 2007-08

PROGRAM STAFF

Understand Needs & FeelingsVery Clearly or Clearly

Closed Cases (Q3) – 87% Active Cases (Q3) – 81%

Promptness Very Promptly or PromptlyClosed Cases (Q7) – 76%Active Cases (Q9) – 67%

Courtesy & Respect Yes (Definitely or Think So)Closed Cases (Q2) – 95%Active Cases (Q2) – 94%

24

FFY 2007-08

PROGRAM STAFF

Do what They Say Will DoYes, definitely-Yes, I think so

Active Cases (Q4) – 84%Explained responsibilities as a clientYes, definitely-Yes, I think so

Active Cases (Q5) – 93%

25

Reporting the Results

26

“Packaging” Results

Design allows for detailed analysesEvolves and changesDifferent products to address different needs

Quarterly Statewide ComparisonsDistrict analysesOpen-ended summariesSpecial Reports

27

Quarterly Comparison

28

District Comparison

29

Quarterly Comparisonover Time

30

Open-Ended Summary

31

Special Report: Open-Ended Detail by Category and Satisfaction

32

Special Report:

Results by Primary Disability by Fiscal Year(2003 through 2005)

33

Major Challenges for Customer Survey Efforts

Participation in SurveysBalancing informational needs of FRC and Agency

34

Response Rates are Declining in the New Millennium

70%

84 03

35

Respondent Behavior Then

36

Respondent Behavior Now

37

What part of “NO” don’t you understand?

COPING WITH REFUSALS

38

Balancing Informational Needs

Monitoring dataOperational informationData Packaging—detail vs. overview