1 The RRFSS Evaluation Framework 2005 Background Information and Progress to Date Presenters Amira...
-
Upload
anissa-lester -
Category
Documents
-
view
214 -
download
0
Transcript of 1 The RRFSS Evaluation Framework 2005 Background Information and Progress to Date Presenters Amira...
1
The RRFSS Evaluation The RRFSS Evaluation Framework 2005Framework 2005 Background Background
Information Information and Progress to Dateand Progress to Date
PresentersPresenters Amira AliAmira Ali Jane HohenadelJane Hohenadel Kate O’ConnorKate O’Connor Deborah Radcliffe Deborah Radcliffe
June 22, 2005June 22, 2005
2
Group MembersGroup Members
RRFSS Evaluation Group: RRFSS Evaluation Group: Amira Ali, Elaine Hector, Jane Amira Ali, Elaine Hector, Jane Hohenadel, Irene Koren, Kathy Hohenadel, Irene Koren, Kathy Moran, Karen Moynagh, Lynne Moran, Karen Moynagh, Lynne Russell, Louisa WongRussell, Louisa Wong
PHRED RRFSS Evaluation Group: PHRED RRFSS Evaluation Group: Jane Hohenadel, Adriana Jane Hohenadel, Adriana Newbury, Kate O’Connor, Louise Newbury, Kate O’Connor, Louise Picard, Deborah RadcliffePicard, Deborah Radcliffe
3
Outline of PresentationOutline of Presentation
Link to strategic planning sessionLink to strategic planning session Link to previous evaluation (2002)Link to previous evaluation (2002) Evaluation Framework – RRFSS Evaluation Framework – RRFSS
Evaluation 2005Evaluation 2005 Tasks and TimelinesTasks and Timelines
4
HistoryHistory Project was identified in March 2004Project was identified in March 2004 Proposal developed in May 2004 and an Proposal developed in May 2004 and an
Evaluation committee was formed with Evaluation committee was formed with representation from each of the RRFSS representation from each of the RRFSS regional working groupsregional working groups
The PHREDs were identified as partners in The PHREDs were identified as partners in the project – proposal submitted to them in the project – proposal submitted to them in October 2004October 2004
Terms of reference for the RRFSS group and Terms of reference for the RRFSS group and the PHRED group were finalized in December the PHRED group were finalized in December 20042004
5
Link to Strategic Planning Link to Strategic Planning SessionSession
6
Strategic Planning SessionStrategic Planning Session Strategic Planning Session in December Strategic Planning Session in December
20042004• To develop strategic statements To develop strategic statements
(vision, mission, values, (vision, mission, values, ggoals, oals, strategies) that will guide the collective strategies) that will guide the collective RRFSS work over the next 3 yearsRRFSS work over the next 3 years
• To determine the specific objectives To determine the specific objectives and activities for the next yearand activities for the next year
• To create an action document To create an action document summarizing the directions setsummarizing the directions set
7
Strategic Planning SessionStrategic Planning Session Vision:Vision: RRFSS envisions that all RRFSS envisions that all
decisions within the public health decisions within the public health system to promote and protect health system to promote and protect health and wellbeing and prevent adverse and wellbeing and prevent adverse health events are informed by valid, health events are informed by valid, timely and relevant health intelligencetimely and relevant health intelligence
Mission:Mission: Ontario’s RRFSS is a flexible, Ontario’s RRFSS is a flexible, timely and responsive surveillance timely and responsive surveillance system designed to meet local Public system designed to meet local Public Health intelligence needs and address Health intelligence needs and address information, geographic and time-information, geographic and time-related data gapsrelated data gaps
8
Strategic Planning SessionStrategic Planning Session Four strategies were identified:Four strategies were identified:
1.1. AdvocacyAdvocacy
2.2. CommunicationCommunication
3.3. GovernanceGovernance
4.4. Quality Improvement and Quality Improvement and DevelopmentDevelopment
9
Strategic Planning SessionStrategic Planning Session
Strategy: Quality Improvement and Strategy: Quality Improvement and DevelopmentDevelopment
Priority ActivityPriority Activity: “Conduct the RRFSS : “Conduct the RRFSS 2005 Evaluation to identify areas of the 2005 Evaluation to identify areas of the surveillance system in need of surveillance system in need of improvement”improvement”
The full report “Strategic Plan for The full report “Strategic Plan for Action” can be viewed at: Action” can be viewed at: www.rrfss.on.cawww.rrfss.on.ca
10
Link to RRFSS Evaluation Link to RRFSS Evaluation 2002 2002
11
RRFSS Evaluation 2002 RRFSS Evaluation 2002 Covered first year of operation (2001)Covered first year of operation (2001) Number of health units ranged from 6 to 12 Number of health units ranged from 6 to 12
during the evaluation periodduring the evaluation period Purpose:Purpose:
Inform change to system proceduresInform change to system procedures Contribute to continuous quality improvementContribute to continuous quality improvement Act as a baseline and template for future Act as a baseline and template for future
evaluationsevaluations
The full evaluation report can be viewed at: The full evaluation report can be viewed at: www.rrfss.on.cawww.rrfss.on.ca
12
RRFSS Evaluation 2002RRFSS Evaluation 2002
Evaluation Framework covered Evaluation Framework covered 4 areas:4 areas: Process EffectivenessProcess Effectiveness Collaboration EffectivenessCollaboration Effectiveness UsefulnessUsefulness Cost EffectivenessCost Effectiveness
13
RRFSS Evaluation 2002RRFSS Evaluation 2002
Performance criteria linked to Performance criteria linked to following outcomes expected from following outcomes expected from RRFSS:RRFSS: Increased local awareness of Increased local awareness of
surveillance informationsurveillance information Increased involvement in generating Increased involvement in generating
appropriate, accurate informationappropriate, accurate information Increased practice of evidence-Increased practice of evidence-
based program planningbased program planning Demonstrated or potential cost Demonstrated or potential cost
benefit due to RRFSSbenefit due to RRFSS
14
RRFSS Evaluation 2002 RRFSS Evaluation 2002 Fourteen Key Issues Fourteen Key Issues
IdentifiedIdentified(grouped as below for 2005 (grouped as below for 2005
evaluation)evaluation) Local Resources (2)Local Resources (2) Centralization IssuesCentralization Issues
• Provincial Implications (3)Provincial Implications (3)• Implications for ISR (4)Implications for ISR (4)
Process Considerations (5)Process Considerations (5)
15
Key Issues from 2002 Key Issues from 2002 EvaluationEvaluation
Local ResourcesLocal Resources Adequate staff/resources for data Adequate staff/resources for data
analysis and making data useableanalysis and making data useable
RRFSS coordinators overwhelmed RRFSS coordinators overwhelmed by work at the local levelby work at the local level
16
Key Issues from 2002 Key Issues from 2002 EvaluationEvaluation
Centralization – Provincial Centralization – Provincial ImplicationsImplications Need for stability and support for the Need for stability and support for the
central work of the RRFSS central work of the RRFSS partnership partnership
Sustaining funding for existing and Sustaining funding for existing and new participants (smaller health new participants (smaller health units in particular); includes funding units in particular); includes funding for survey administration and for survey administration and analysis analysis
Provincial representation of dataProvincial representation of data
17
Key Issues from 2002 Key Issues from 2002 Evaluation Evaluation
Centralization – Implications Centralization – Implications for ISRfor ISR
Increasing the sample size for a Increasing the sample size for a health unithealth unit
Access to ISR, decision-making, Access to ISR, decision-making, workload and resourcesworkload and resources
Improved timeliness of data from ISRImproved timeliness of data from ISR ISR Ability to handle increase in HU ISR Ability to handle increase in HU
participationparticipation
18
Key Issues from 2002 Key Issues from 2002 EvaluationEvaluation
Process Consideration IssuesProcess Consideration Issues Increasing use of data in program Increasing use of data in program
planning and evaluation in the health planning and evaluation in the health unitsunits
Limited new question/module capacity Limited new question/module capacity under current questionnaire under current questionnaire
Process for revising old modules and Process for revising old modules and developing new modulesdeveloping new modules
Upgrading RRFSS websiteUpgrading RRFSS website Sharing data among organizationsSharing data among organizations
19
RRFSS Evaluation 2005 RRFSS Evaluation 2005
20
2005 Evaluation 2005 Evaluation Framework ObjectivesFramework Objectives
1.1. To document progress made in the To document progress made in the key areas identified in the 2002 key areas identified in the 2002 EvaluationEvaluation
2.2. To document some of the “best To document some of the “best practices” used by Health Unitspractices” used by Health Units
3.3. To envisage what it would take to To envisage what it would take to get RRFSS on board for all Ontario get RRFSS on board for all Ontario Health Units and how it could work Health Units and how it could work as a provincial systemas a provincial system
21
2005 Evaluation 2005 Evaluation FrameworkFramework
Refinement of the Key Refinement of the Key IssuesIssues To address the evaluation objectives, we To address the evaluation objectives, we
began with what were identified as began with what were identified as Key Key Issues Issues in the 2002 Evaluationin the 2002 Evaluation
Evaluation group members were asked to Evaluation group members were asked to review each of those 14 Key issues, and:review each of those 14 Key issues, and:• report on progressreport on progress• whether it was still an issue nowwhether it was still an issue now• if so, how important it was for the 2005 if so, how important it was for the 2005
evaluationevaluation
22
2005 Evaluation 2005 Evaluation Framework Framework
Refinement of the Key Refinement of the Key IssuesIssues It was felt that seven issues had been It was felt that seven issues had been
largely addressed, and that the 2005 largely addressed, and that the 2005 evaluation needed only to document their evaluation needed only to document their evolution since 2002evolution since 2002
The issues that we felt were addressed: The issues that we felt were addressed: • increasing the sample size for a increasing the sample size for a
health unithealth unit• improved timeliness of data from ISRimproved timeliness of data from ISR• access to ISR, decision-making, access to ISR, decision-making,
workload and resources, andworkload and resources, and• limited new question/module capacitylimited new question/module capacity
23
2005 Evaluation 2005 Evaluation Framework Framework
Refinement of the Key Refinement of the Key IssuesIssues
The issues that we felt were addressed The issues that we felt were addressed (cont’d): (cont’d):
• limited new questionnaire/module limited new questionnaire/module capacity under current capacity under current questionnairequestionnaire
• process for revising old modules process for revising old modules and developing new modulesand developing new modules
• upgrading RRFSS websiteupgrading RRFSS website
24
2005 Evaluation 2005 Evaluation Framework Framework
Refinement of the Key Refinement of the Key IssuesIssues
The remainder were felt to still be The remainder were felt to still be issues for the RRFSS partnership and issues for the RRFSS partnership and should remain in the 2005 evaluationshould remain in the 2005 evaluation
The outstanding issues were refined, The outstanding issues were refined, defined and expressed as evaluation defined and expressed as evaluation questionsquestions
25
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue Local ResourcesLocal ResourcesQuestionQuestion Do HUs have adequate staff and Do HUs have adequate staff and
resources for regional and local resources for regional and local administrative tasks of RRFSS?administrative tasks of RRFSS?
Data Collection ToolData Collection ToolQuestionnaireQuestionnaire
Source of DataSource of DataRRFSS reps, MOH of participating HURRFSS reps, MOH of participating HU
26
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue
Local ResourcesLocal ResourcesQuestionQuestionDo HUs have adequate staff and Do HUs have adequate staff and
resources locally for data analysis and resources locally for data analysis and making data useable?making data useable?
Data Collection ToolData Collection ToolQuestionnaireQuestionnaire
Source of DataSource of DataRRFSS reps, MOH of participating HURRFSS reps, MOH of participating HU
27
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue Local ResourcesLocal ResourcesQuestionQuestionAre local RRFSS reps able to meet the Are local RRFSS reps able to meet the
local work demand for RRFSS module local work demand for RRFSS module development?development?
Data Collection ToolData Collection ToolQuestionnaireQuestionnaire
Source of DataSource of DataRRFSS reps, MOH of participating HU, RRFSS reps, MOH of participating HU, program and research staff, RRFSS program and research staff, RRFSS coordinator, RRFSS Evaluation Group, ISRcoordinator, RRFSS Evaluation Group, ISR
28
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue Local ResourcesLocal ResourcesQuestionQuestionAre local RRFSS reps able to meet the Are local RRFSS reps able to meet the
local work demand for RRFSS module local work demand for RRFSS module development?development?
Data Collection ToolData Collection ToolQuestionnaireQuestionnaire
Source of DataSource of DataRRFSS reps, MOH of participating HU, RRFSS reps, MOH of participating HU, program and research staff, RRFSS program and research staff, RRFSS coordinator, RRFSS Analysis Group, ISRcoordinator, RRFSS Analysis Group, ISR
29
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssue Issue Local ResourcesLocal ResourcesQuestionQuestionAre local RRFSS reps able to meet the Are local RRFSS reps able to meet the
local work demand for RRFSS module local work demand for RRFSS module review?review?
Data Collection ToolData Collection ToolQuestionnaireQuestionnaire
Source of DataSource of DataRRFSS reps, MOH of participating HU, RRFSS reps, MOH of participating HU, program and research staff, RRFSS program and research staff, RRFSS coordinator, RRFSS Analysis Group, ISRcoordinator, RRFSS Analysis Group, ISR
30
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue Centralization Issues: Provincial ImplicationsCentralization Issues: Provincial ImplicationsQuestionQuestionWhat resources are needed currently to What resources are needed currently to
support and to ensure the stability of the support and to ensure the stability of the provincial and administrative work of the provincial and administrative work of the RRFSS Partnership?RRFSS Partnership?
Data Collection ToolData Collection ToolQuestionnaireQuestionnaire
Source of DataSource of DataRRFSS reps, MOH of participating HU, ISR, RRFSS reps, MOH of participating HU, ISR, RRFSS CoordinatorRRFSS Coordinator
31
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssueCentralization Issues: Provincial ImplicationsCentralization Issues: Provincial ImplicationsQuestionQuestion
What funds are required to enable What funds are required to enable continued participation of HUs continued participation of HUs currently participating in RRFSS and currently participating in RRFSS and allow non-participating HUs to join?allow non-participating HUs to join?
Data Collection ToolData Collection ToolKey Informant InterviewsKey Informant Interviews
Source of DataSource of DataRRFSS reps, MOH of all HUs, ISR, RRFSS reps, MOH of all HUs, ISR, MOHLTCMOHLTC
32
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue
Centralization Issues: Provincial Centralization Issues: Provincial ImplicationsImplications
QuestionQuestionIs provincial representation of data a Is provincial representation of data a desired product of RRFSS?desired product of RRFSS?
Data Collection ToolData Collection ToolKey Informant InterviewsKey Informant Interviews
Source of DataSource of DataRRFSS reps, MOH of all HUs, MOHLTC, RRFSS reps, MOH of all HUs, MOHLTC, PHRED, Chief MOH, external partnersPHRED, Chief MOH, external partners
33
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue Centralization Issues: Provincial ImplicationsCentralization Issues: Provincial ImplicationsQuestionQuestion
What would it take to extend RRFSS to What would it take to extend RRFSS to all Ontario HUs?all Ontario HUs?
Data Collection ToolData Collection ToolKey Informant InterviewsKey Informant Interviews
Source of DataSource of DataRRFSS reps, epidemiologists in non-RRFSS reps, epidemiologists in non-participating HUs, MOHLTC, ISR, Chief participating HUs, MOHLTC, ISR, Chief MOH, MOH of all HUsMOH, MOH of all HUs
34
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue
Process ConsiderationsProcess Considerations
QuestionQuestion
To what extent are RRFSS data being To what extent are RRFSS data being used in PP&E in the health units?used in PP&E in the health units?
Data Collection ToolData Collection Tool
Questionnaire, Case ExamplesQuestionnaire, Case Examples
Source of DataSource of Data
RRFSS reps, program and research staffRRFSS reps, program and research staff
35
2005 Evaluation Framework 2005 Evaluation Framework Evaluation Questions for Evaluation Questions for
Outstanding IssuesOutstanding IssuesIssueIssue
Process ConsiderationsProcess Considerations
QuestionQuestion
To what extent are provincial RRFSS To what extent are provincial RRFSS data being shared among agencies data being shared among agencies and organizations?and organizations?
Data Collection ToolData Collection Tool
Questionnaire, Key Informant InterviewQuestionnaire, Key Informant Interview
Source of DataSource of Data
RRFSS Coordinator, ISR, RRFSS repsRRFSS Coordinator, ISR, RRFSS reps
36
Evaluation Framework for Evaluation Framework for RRFSSRRFSS
Evaluation Questions for Evaluation Questions for Outstanding IssuesOutstanding IssuesIssueIssue
Process ConsiderationsProcess ConsiderationsQuestionQuestion
To what extent are health unit To what extent are health unit specific RRFSS data being shared specific RRFSS data being shared among agencies and organizations?among agencies and organizations?
Data Collection ToolData Collection ToolQuestionnaireQuestionnaire
Source of DataSource of DataRRFSS repsRRFSS reps
37
2005 Evaluation Framework2005 Evaluation Framework Evaluation Question for Objective Evaluation Question for Objective
22IssueIssue
Process ConsiderationsProcess Considerations
QuestionQuestion
What Success Stories can be What Success Stories can be identified for RRFSS? identified for RRFSS?
Data Collection ToolData Collection Tool
Workshop, QuestionnaireWorkshop, Questionnaire
Source of DataSource of Data
RRFSS reps, program and research staffRRFSS reps, program and research staff
38
2005 Evaluation 2005 Evaluation Framework Evaluation Framework Evaluation
QuestionQuestionIssueIssue
Process Considerations (Parked Issue)Process Considerations (Parked Issue)
QuestionQuestion
What is the quality of RRFSS data? What is the quality of RRFSS data?
Data Collection ToolData Collection Tool
QuestionnaireQuestionnaire
Source of DataSource of Data
RRFSS reps, ISR, partnersRRFSS reps, ISR, partners
39
Tasks and TimelinesTasks and Timelines
40
RRFSS Evaluation TasksRRFSS Evaluation Tasks Finalize development of research tools. Finalize development of research tools.
• QuestionnaireQuestionnaire• Focus groupsFocus groups• Key informant interviewsKey informant interviews• Document reviewsDocument reviews
Data collection:Data collection:• RRFSS Workshop-- “Success stories”RRFSS Workshop-- “Success stories”• Primarily August and SeptemberPrimarily August and September
Data AnalysisData Analysis Draft and finalize the reportDraft and finalize the report
41
RRFSS Evaluation RRFSS Evaluation TimelinesTimelines
RRFSS Evaluation Project Tasks and Timelines, 2005
Month
June
July
August
Sept
Oct
Nov
Dec
Task
Develop Research Tools
Data Collection
Data Analysis
Draft Report
Finalize Report
42