Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes...
Transcript of Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes...
![Page 1: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/1.jpg)
© 2017 Materials may not be reproduced without permission of the author.
Developing an Effective Notes Review Process for CAPI Studies
Nick SchultzUW Survey Center
University of Wisconsin Madison
May 2017
![Page 2: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/2.jpg)
Overview
• What is Notes Review?
• History of CAPI Notes Review
• Evolution of Notes Review system
• Current system & effectiveness
• Lessons learned & next steps
2
![Page 3: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/3.jpg)
Example open ended response notes
3
“Gas prices went up”
“They talked about it on the news”
![Page 4: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/4.jpg)
Example open ended response notes
4
“There wasn’t a specific event that made me aware”
“Resp has confirmed DOB as XX-XX-XXXX”
![Page 5: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/5.jpg)
What is Notes Review?
5
• Open-ended responses
• Interviewer comments within interview
• Interviewer comments during contact attempts
• UW Survey Center committed to 100% Notes Review
![Page 6: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/6.jpg)
What is Notes Review?
6
• Why review notes?
• Improve data quality
• Improve interview quality through feedback
• Who reviews notes?
• When?
• How?
![Page 7: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/7.jpg)
Overview
• What is Notes Review?
• History of CAPI Notes Review
• Evolution of Notes Review system
• Current system & effectiveness
• Lessons learned & next steps
7
![Page 8: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/8.jpg)
History of UWSC CAPI Notes Review
8
• Fewer, smaller scale CAPI projects
• Supervision & case management conducted by Project Directors
Pre-2009
2009 -2012
2012 -2013
2013-2016
2016 -Today
![Page 9: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/9.jpg)
History of CAPI Notes Review
9
• Wisconsin Longitudinal Study begins
• Over 12,000 respondents, 50+ CAPI interviewers
• Respondents scattered across the country
• Transition of CAPI supervision & case management to field
Pre-2009
2009 -2012
2012 -2013
2013-2016
2016 -Today
![Page 10: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/10.jpg)
History of CAPI Notes Review
10
• Notes Review handled by Programming staff
• Cost effective staff? Yes
• Timely feedback to interviewers? No
• Project-specific knowledge? No
Pre-2009
2009 -2012
2012 -2013
2013-2016
2016 -Today
![Page 11: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/11.jpg)
History of CAPI Notes Review
11
• Notes Review handled by Project Directors
• Cost effective staff? No
• Timely feedback to interviewers? Sorta
• Project-specific knowledge? Yes
Pre-2009Pre-
20092009 -2012
2012 -2013
2013-2016
2016 -Today
![Page 12: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/12.jpg)
History of CAPI Notes Review
12
• Notes Review handled by CAPI Field Team
• Cost effective staff? Yes
• Timely feedback to interviewers? Better
• Project-specific knowledge? Yes
Pre-2009
2009 -2012
2012 -2013
2013-2016
2016 -Today
![Page 13: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/13.jpg)
Overview
• What is Notes Review?
• History of CAPI Notes Review
• Evolution of Notes Review system
• Current system & effectiveness
• Lessons learned & next steps
13
![Page 14: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/14.jpg)
Evolution of Notes Review system
14
What do we need to do to make our Notes Review system the most beneficial?
• Make Notes Review efficient
• Make Notes Review timely
• Make Notes Review scalable
![Page 15: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/15.jpg)
Making Notes Review efficient
15
• Need a place to accumulate notes
• Find ways to automate certain types of note review
• Allow the ability to review note outcomes
• Per project
• Per item
• Per interviewer
![Page 16: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/16.jpg)
Making Notes Review timely
16
• Want to prevent systematic, recurring errors
• Want to give interviewers relevant notes feedback they can incorporate immediately
![Page 17: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/17.jpg)
Making Notes Review scalable
17
• UWSC has over 40 CAPI interviewers working simultaneously across 8 field projects
• Each project has its own timeline
• Need a system that easily could add new projects
![Page 18: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/18.jpg)
Overview
• What is Notes Review?
• History of CAPI Notes Review
• Evolution of Notes Review system
• Current system & effectiveness
• Lessons learned & next steps
18
![Page 19: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/19.jpg)
Current system & effectiveness
19
• Field, Project Directors, & Programming collaborated to request a tool to address Notes Review needs
• UWSC programmer Kate Krueger developed the Notes Review Database
![Page 20: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/20.jpg)
Current system & effectiveness
20
![Page 21: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/21.jpg)
Making Notes Review efficient
21
• Provides a space to review all notes
• Automatically codes common note strings
• Can view overall status, and allows for sorting by interviewer, item # and case ID
![Page 22: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/22.jpg)
Making Notes Review timely
22
• Notes Review Database updates notes when you connect – up to date note review
• Notes have date stamps to allow CAPI Supervisors to track interviewer improvements
![Page 23: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/23.jpg)
Making Notes Review scalable
23
• Notes Review Database can easily add new projects
• Allows for customization on date range review & interviewer review
![Page 24: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/24.jpg)
Current system & effectiveness
24
• Is it effective?
• Wisconsin Families Study wave comparison
• Wave 1 (2016) review performed periodically
• Wave 2 (2017) review started immediately
![Page 25: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/25.jpg)
Current system & effectiveness
25
0.79
0.19
0.44
0.05
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
FeedbackPer
Complete
Recodes PerComplete
Wave 1
Wave 2
![Page 26: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/26.jpg)
Current system & effectiveness
26
• Is it effective?
• Lots of outside factors (veteran interviewers, past feedback, ongoing project)
• Notes Review trending in the right direction
![Page 27: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/27.jpg)
Overview
• What is Notes Review?
• History of CAPI Notes Review
• Evolution of Notes Review system
• Current system & effectiveness
• Lessons learned & next steps
27
![Page 28: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/28.jpg)
Lessons Learned
• Notes Review conducted by Field has the most advantages
• Timely and efficient interviewer feedback
• Cost-effective
• Proactive
• Catches interviewer errors early on to prevent repetition of these same errors throughout the field period
• Ultimately provides better quality data to client
28
![Page 29: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/29.jpg)
Lessons Learned
• Development of the Notes Review Database allowed for more efficient & effective Notes Review
• Reduction in systematic errors
• Quicker response time for feedback
• Flexibility to expand to accommodate new projects
29
![Page 30: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/30.jpg)
Next Steps
• Further integration of the Notes Review Database with existing UWSC QC tools and technologies
• Review the reviewers
• Improve automated Notes Review functionality
30
![Page 31: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/31.jpg)
Acknowledgments
• Kate Krueger
• Carol Wintheiser
• Rob Schultz
• Tom Francis
• Kyle Krueger
• Steve Coombs
• Kerryann Diloreto
• Karen Jaques
• Jaime Faus
• Grete Wichmann
• Jacek Kraszewski
31
![Page 32: Developing an Effective Notes Review Process for CAPI Studies€¦ · Developing an Effective Notes Review Process for CAPI Studies Nick Schultz UW Survey Center. University of Wisconsin](https://reader033.fdocuments.in/reader033/viewer/2022060402/5f0e5e977e708231d43ee9ce/html5/thumbnails/32.jpg)
Please visit us at:www.uwsc.wisc.edu
Thank You!
For copies of this presentation or more information, contact:
Nick [email protected]