Post on 09-Mar-2018
Assessing CustomerSatisfaction with
Vocational Rehabilitation Programs
Florida State University Survey Research LaboratoryCollege of Social Sciences
www.fsu.edu/~survey
Dr. Mary Stutzman, DirectorFSU Survey Research Laboratory
Presentation at the Quarterly Meeting
West Palm Beach , September 14 – 16 2009
2
Why Should the FRC Survey Customers?Provides an “outside” look of customer service.
Customers are sometimes uncomfortable to give feedback directly to the agency giving them services.
Gives customers an opportunity to “speak” to someone outside the Department.
Open-ended comments800 number calls and referrals to FRC
Provides longitudinal data base to assess trends.
Data can be “packaged” for to address specific concerns.
3
Evolution of the Survey Effort
Survey of Customers with Cases ClosedLaunched 2001Questions developed by FRC Evaluation Committee and DVR staffDesigned survey instrument one-page legal front and backSample Design—all cases closed previous month
Customers in Program after 6 monthsLaunched 2002 Questions mirrored Closed Cases SurveySample Design—random sample of 700 per monthCustomers in program after 6 months
4
Customer Surveys OverviewGather information from customers:
Program OutcomesServicesStaff
CLOSED CASES--Surveyed Cases Closed Successful (Status 26)Unsuccessful (Status 80 – 97, 89)
ACTIVE CASES—Surveyed Customers in Program after 6 months
Random sample of 700 per month; 800 beginning July 2008
5
Survey ApproachClosed Cases—Survey all cases closed at the end of month
LetterInstrument• 12 Closed items• 2 Open-ended
Active Cases—Survey sample of 800 cases still active (opened 6 months earlier)
LetterInstrument• 12 Closed items• 2 Open-ended
6
Staffing the Survey Effort
FSU Survey Research LaboratoryAll staff paidStaffed with trained interviewers 7 days a week• can handle PR issues on 800 line and conduct call-
in interviewsMail within 1-week of sample receiptData and open-ended responses entered Analysis of dataProvide reports and presentations on a quarterly and ad hoc basis
7
Design BenefitsSampling Strategy
Power of numbers to analyze by various segments (Areas, Disabilities, etc).Monthly—helps eliminate “seasonal” bias
Response CategoriesAllow “ordinary language” responses for respondentsHelps make reports understandable
OperationsMode choice—mail , Internet, phone800 number and open comments—public relations for Council
ReportingSufficient numbers for Quarterly TrendsSufficient numbers Annual Trends and sub-analyses
8
Respondents are King/Queen
Make convenient2-pages legalKeep short—12 Questions
Accommodate Respondent PreferencesWeb access800 number for phone responseTTY line
9
Privacy Issues: Called ”Project”
10
Closed Cases: Survey Instrument
11
Closed Cass: Survey Instrument
12
Survey Areas
Program OutcomesProgram Choice
ServicesStaff
13
Program Outcomes
Closed Cases
Active Cases
Program OutcomesOverall Satisfaction with Services X XServices Made Life Better X XWould Recommend to A Friend X XEmployment Outcomes and Satisfaction X
14
Program ChoiceProgram Choice Closed
CasesActive Cases
Informed of Alternative Service Providers and Vocational Goals X
Able to Choose Service Providers and Vocational Goals X
Informed of Alternative Service Providers XAble to Choose Service Providers XInformed of Alternative Vocational Goals XAble to Choose Alternative Vocational Goals X
15
Program ServicesProgram Services Closed
CasesActive Cases
Accessibility to VR Offices X XAppropriateness of Services Considering Rehabilitation Goals X
Promptness of Services Provided X XNeed Other Services but Did Not Receive X
Open-Ended Closed Cases
Active Cases
Especially helpful VR associates or services X XWhat VR can do to improve services X XWhy did not get a job X
16
Program Staff
Program Staff Closed Cases
Active Cases
Understand Needs and Feelings X X
Treated with Courtesy and Respect X X
Explained Responsibilities as Client X
Do What Say X
17
CLOSED CASES: APPROACH AND RESPONSE RATE
CUSTOMERS WITH CASES CLOSED: 2007-08 Surveys mailed monthlySurveys mailed to all customers whose cases were closed in the prior month October 2007 to September 2008 mailings18,100 surveys mailedTwo Mailings Analysis based on 5,176 returns
Response Rate = 29%
18
CURRENT CASES: APPROACH AND RESPONSE RATE
CUSTOMERS WITH CASES ACTIVE CASES: 2007-08Surveys mailed monthlySurveys mailed to random sample of 700-800 customers whose case is active 6 months after plan8,700 surveys mailedTwo Mailings Analysis based on 2,441 returnsResponse Rate = 28%
19
Adaptation and Change
Refined and revisited over timeInstrument—”already have a job”Letters—explanation of council and repeated surveysReports—content and format
20
FFY 2007-08
PROGRAM OUTCOMES
Overall Satisfaction --Very or Mostly SatisfiedClosed Cases (Q9) -- 85%Active Cases (Q11) – 79%
Services Made Life Better –Yes, Definitely Helped or Somewhat
Closed Cases (Q8) -- 86%Active Cases (Q10) -- 81%
Recommend to a Friend Yes, Absolutely or Probably Closed Cases (Q10) -- 90%Active Cases (Q12) -- 86%
21
FFY 2007-08
PROGRAM OUTCOMES[Closed Cases - Status 26]
Job Status of Customers (Q12)Nearly one-third of the customers (31%) obtained a job since becoming a clientNearly one-half (60%) already had jobs
Customers (Status 26) who Obtained New Jobs were Satisfied (Q12A)
83% were Very or Mostly Satisfied with their jobs
22
FFY 2007-08
PROGRAM SERVICES
Accessibility Very or Mostly SatisfiedClosed Cases (Q1) – 91%Active Cases (Q1) – 88%
Appropriate ServicesHighly Appropriate or Appropriate
Closed Cases (Q4) -- 88%Active Cases (Q8) -- 86%
Need Other ServicesYes--Definitely or Think So
Closed Cases (Q5)-- 29%
23
FFY 2007-08
PROGRAM STAFF
Understand Needs & FeelingsVery Clearly or Clearly
Closed Cases (Q3) – 87% Active Cases (Q3) – 81%
Promptness Very Promptly or PromptlyClosed Cases (Q7) – 76%Active Cases (Q9) – 67%
Courtesy & Respect Yes (Definitely or Think So)Closed Cases (Q2) – 95%Active Cases (Q2) – 94%
24
FFY 2007-08
PROGRAM STAFF
Do what They Say Will DoYes, definitely-Yes, I think so
Active Cases (Q4) – 84%Explained responsibilities as a clientYes, definitely-Yes, I think so
Active Cases (Q5) – 93%
25
Reporting the Results
26
“Packaging” Results
Design allows for detailed analysesEvolves and changesDifferent products to address different needs
Quarterly Statewide ComparisonsDistrict analysesOpen-ended summariesSpecial Reports
27
Quarterly Comparison
28
District Comparison
29
Quarterly Comparisonover Time
30
Open-Ended Summary
31
Special Report: Open-Ended Detail by Category and Satisfaction
32
Special Report:
Results by Primary Disability by Fiscal Year(2003 through 2005)
33
Major Challenges for Customer Survey Efforts
Participation in SurveysBalancing informational needs of FRC and Agency
34
Response Rates are Declining in the New Millennium
70%
84 03
35
Respondent Behavior Then
36
Respondent Behavior Now
37
What part of “NO” don’t you understand?
COPING WITH REFUSALS
38
Balancing Informational Needs
Monitoring dataOperational informationData Packaging—detail vs. overview