Metrics that matter: Assessing workflow and staffing · NCH 001. 7. NCH 002. 8. Determine . Base...
55
METRICS THAT MATTER: ASSESSING WORKFLOW AND STAFFING Michele Vaughan, BS, CCRP Research Strategy and Compliance Specialist Nationwide Children’s Hospital, Columbus, OH Susan Sedwick, PhD, CRA, CSM Senior Consultant Attain LLC
Transcript of Metrics that matter: Assessing workflow and staffing · NCH 001. 7. NCH 002. 8. Determine . Base...
METRICS THAT MATTER: ASSESSING WORKFLOW AND STAFFING
Michele Vaughan, BS, CCRP Research Strategy and Compliance Specialist Nationwide Children’s Hospital, Columbus, OH
Susan Sedwick, PhD, CRA, CSMSenior Consultant Attain LLC
Learning Objectives
Develop a rating matrix for proposals, awards, and clinical studies applicable to their institution
Understand how to use workflow matrix to justify additional positions and increase salaries to retain experienced staff
Metrics that Matter
The ability to quantify associated workloads can have a significant impact on the efficiency and success of a research organization.
Why do we need to assess workload?• Balance workloads among staff• Staff Satisfaction• Efficiency in workloads• Justification in grant/clinical trial budgets• Strategic planning for budgeting • Help anticipate growth in staffing • Staff retention • Staff competencies for workflows
Scoring Workloads
Workload Complexity
Which Tool??
a. Validated and Published Tool b. Fits the needs of your practicec. Buy In from your staff d. Leadership team values the data e. Reproduce the data f. Compare your results to other departmentsg. Metrics that you can make decisions from
Tools are Available
• OPAL: Ontario Protocol Assessment Level developed by Ontario Institute for Cancer Research
• Measuring Clinical Trail-Associated Workload in a Community Clinical Oncology Program
• Advantage Clinical-Protocol Scoring and Determining Study Load per Clinical Research Coordinator
• American Society of Clinical Oncology• NCI Trial Complexity and Elements Scoring Model• University of Michigan Research Effort Tracking Application (RETA)• Ontario Protocol Assessment Tool (OPAL)• Wichita Community Clinical Onc Program Protocol Tool (WPAT)
What Metrics Matter?
# Active Patients
Frequency of Visits
#Special Procedures
# Central Processing
Treatment vs Non
Treatment
Multiple Contact Events
Phase of Study
Monitoring Source
Documentation
ACUITY
OPAL- Scoring System Workloads
Metrics What MattersComplexity of Trial Multiple Treatments, complex data
forms, special procedures, central processes
Number of patient visits Frequency of staff to visit patientsNumber of patients How many patients are enrolled Type of Trial Early Phase to registry trialsStatus of Patient on trial On study, off study, on follow up
How to Get Started
1
2
3
OPAL-Protocol Scoring (Base Score)12345678
Single ContactNon Treatment
Multi ContactNon Treatment
Phase II, III, IV Non Drug
Interventional
Treatment Phase II, III, IV Single
SP/ CP
Treatment Phase II, III, IV
Multiple SP or CP-more than 2 occurrences
Treatment Phase II, III, IV
Single SP + Multiple CP or Single CP and Multiple SP
Treatment Phase II,III,IV
2 or more SP2 or more CP
Any Phase 1 trial
SP-Special Procedure CP- Central Processing
OPAL-Protocol Scoring
Protocol BaseScore
Inpatient(max .5)
Monitoring or 100% Source
# or Duration TX
FUP Visits
IndustrySponsor
CROIIT
MultipleSurveys
TOTALScore
NCH001 7
NCH 002 8
Determine Base OPAL Score
Each Protocol
Calculate OPAL Total Score
Each Protocol
Base Score + Optional Elements = Total Score for Protocol
OPAL-Protocol Scoring Optional Elements
Scoring Add ValueInpatient needed Maximum value .5 On site Monitoring or 100% Source Documents Submission
Maximum value .5
# or Duration of treatment/Follow Up Visits
Maximum value .5
Industry sponsor/CRO/IIT
Maximum value .5
Multiple Survey’s (3+) . Questionnaires
Maximum value .5
TOTAL OPAL SCOREEach
Protocol
Determine the Total Score for Each Protocol determineIf there are Optional Elements to the base score:
OPAL-Protocol Scoring
Protocol BaseScore
Inpatient(max .5)
Monitoring or 100% Source
# or Duration TX
FUP Visits
IndustrySponsor
CROIIT
MultipleSurveys
TOTALScore
NCH001 7 .5 .5 .5 .5 0 9
NCH 002 8 .5 .5 .5 .5 0 10
Determine Base OPAL Score Each Protocol
Calculate Total OPAL Score Each Protocol
Base Score + Optional Elements = Total OPAL Score for Protocol
OPAL- Protocol Scoring Metric Higher Protocol Score – More Complexity Involved
02468
1012141618
2 3 4 5 6 7 8 9 10
# Tr
ials
Ope
n
Acuity Score Per Trial
# TrialsEnrollments
OPAL Scores Per Protocol • Acuity Scores Per Protocol Open in Each Service Line
0
5
10
15
20
25
30
BMTONCHEM
OPAL Scoring for Staffing
Workload Complexity OPAL Protocol Scores
OPAL Scoring for Staff
OPAL- Add Staff VolumeTotal OPAL Staff Workload Calculation:(# Active Patients X Total Score) + (# FUP X Total Score/2)+ Total Score
Protocol OPAL Score
# ActivePatients
# FUPPatient
Total Score
NCH001 9 2 0 27NCH002 10 3 3 55NCH005 4 8 10 56NCH009 6 1 5 27TOTALS 14 18 165
Protocol OPALScore
# ActivePatients
# FUPPatient
Total Score
NCH010 6NCH020 3NCH030 2NCH040 3TOTALS
Susan Workload Michele Workload
NCH002= (10X3) + (3 X (10/2) ) + 10= 55
OPAL- Add Staff VolumeTotal Staff Workload Calculation:(# Active Patients X Total Score) + (# FUP X Total Score/2)+ Total Score
Protocol OPALScore
# ActivePatients
# FUPPatient
Total Score
NCH001 9 2 0 27NCH002 10 3 3 55NCH005 4 8 10 56NCH009 6 1 5 27TOTALS 14 18 165
Protocol OPAL Score
# ActivePatients
# FUPPatient
Total Score
NCH010 6 4 1 33NCH020 3 15 0 48NCH030 2 15 0 32NCH040 3 5 0 18TOTALS 39 1 131
Susan Workload Michele Workload
Total Active Patients=14Acuity Total=165
Total Active Patients=39Acuity Total=131
Staffing MetricsDetermine workloads per staff member:
Employee #Active # FUPTotal
Acuity Score
# Different Protocols
Sue 22 0 154 6Amy 14 0 111 2Mike 15 0 193 8Sally 13 0 110.5 6
Aaron 13 0 105 7Kelly 17 0 39 3Heidi 18 25 120.75 6
Veronica 12 0 117.5 6Melinda 7 0 68 3
Joe 12 0 133.5 8
Staffing Metrics
Jan-17 Feb-17 Mar-17 Apr-17 May-17 Jun-17 Jul-17 Aug-17 Sep-17 Oct-17 Nov-17 Dec-172017
Avg/mth
Hem 139 136 138 131 98 74 91 128 94 92 112 103 111
Onc 134 152 139 174 178 151 148 137 143 145 96 90 141
Neuro Onc 130 136 134 155 142 101 87 72 60 57 53 131 104
BMT 99 99 112 131 174 181 165 165 178 212 111 113 145
Host Def 99 99 112 131 174 181 165 165 178 212 111 154 148
2017 OPAL SCORES by Division
Determine Workload per Service Line Teams
Add Acuity scores all employees / # of employees=Avg Acuity Per Month
Average Staff Acuity Per Month
139 136 138 131 98 74 91 128 94 92 112 103 111
134 152 139 174178
151 148137
143 145 96 90141
130 136 134155
142
101 87 7260 57
53131
104
9999 112
131174
181 165 165178 212
111
113
145
9999 112
131174
181165 165
178
212
111
154
148
0
100
200
300
400
500
600
700
800
900
Jan-17 Feb-17 Mar-17 Apr-17 May-17 Jun-17 Jul-17 Aug-17 Sep-17 Oct-17 Nov-17 Dec-17 2017Avg/mth
Average Acuity Score Per Month
Host Def
BMT
Neuro Onc
Onc
Hem
Metrics that Matter
Learning Objectives Protocol and Staffing:• Choose a Rating Matrix for your team • Apply a scoring system to assess workload • Justify staffing needs with your matrix data• Workloads assessment includes acuity• Workloads per team • Acuity per protocol
OPAL Scoring for Feasibility • OPAL Scoring to determines Staff Time and Effort for
Research Budgets
How to Get Started OPAL & Feasibility
1
2
3
OPAL Scoring for Feasibility
Protocol Score
% of Effort Per Month
RN
% of Effort Per Month
CRC
Hours Per Month
RN
Hours Per MonthCRC
<2 10% 0% 17 03-4 15% 0% 26 04-5 20% 5% 35 95-6 25% 10% 43-52 176-7 30% 15% 52-69 267-8 40% 25% 52-69 43-61
9-10 40% 40% 52-69 69-87
Monthly hours assumes 2080 hours per 12months (173.334 per month)
OPAL Scoring for Feasibility
Protocol Score
% of EffortPer Month
RN
% Effort Per Month
CRC
Hours Per Month
RN
Hours Per Month CRC
5-6 25% 10% 43-52 17
DetermineProtocol Score
Grant Application Time and Effort
Needed for Research Staffing
Clinical Trial Budget
Use HoursMultiply by #
Months
OPAL Scoring for Feasibility
Protocol Score
% of EffortPer Month
RN
% Effort Per Month
CRC
Hours Per Month
RN
Hours Per Month CRC
5-6 25% 10% 43-52 17
Protocol Score $ Per MonthRN
$ Per Month CRC
5-6 $2,256-$2,704 $676
RN hourly rate=$52CRC hourly rate=$39
OPAL Scoring for Feasibility Budget Request: Sponsor: XXXOpal Score: 5-6Estimated Number of Months: 6Phase of Study: 3
Research Time $ Per Month
# of Month
On Study
TOTAL$$
% Effort
RN $2,256 6 $13,536 25%CRC $676 6 $4,056 10%
TOTAL Per Patient $2,932 6 months $17,592 35%
Protocol Score
$ Per MonthRN
$ Per Month CRC
5-6 $2,256-$2,704
$676
Estimated cost per patient for 6 months
Metrics that Matter
Learning Objectives Feasibility:• Choose a Rating Matrix can use for feasibility• Collect team’s data for time and efforts per protocol • Update $$ rates yearly with employee raises• Decrease staff needs by streamline workflow• Budget estimates Buy In from Staff and PI’s• Costs were Reproducible across Protocol Scores
COMPLEXITY SCORING
Daily Download
Tracking Progress
Criticality
Where is it?
Who is it with?Initial Review
Awaiting C&G Action Legal Review Sponsor Review Coordinator
6 2 6 22 A
5 8 3 12 B
7 9 3 10 C
3 2 6 17 D
9 3 1 8 E
1 8 3 6 F
0 2 2 13 G
5 0 1 8 H
0 1 2 8 I
0 0 0 5 J
0 1 1 3 K
0 0 0 0 L
0 0 0 0 M
0 0 0 0 N
0 0 0 0 N
0 0 0 0 O
0 0 0 0 P
36 36 28 112 Daily Total
Numbers Don’t Equal Workload
Type of Agreement
Sponsor
Scope
Budget
Intellectual Property
Subawards
Risk
Numbers Don’t Equal Workload
Type of Agreement• Contract• Grant or Cooperative Agreement• Material Transfer Agreement• Use Agreement• Non-disclosure Agreement• Clinical Trials Agreement• Incoming Subcontract• Purchase Order/Service• Teaming Agreement• Memorandum of Understanding• Other Agreements
Numbers Don’t Equal Workload
Sponsor• Federal• State• Industry• Foundation• Foreign• New Sponsor• Problem Sponsor
Numbers Don’t Equal Workload
Scope• Vague or Specific• Place of Performance• Criticality of Performance• Subawards• Compliance
• IRB• IACUC• IBC• Export
Numbers Don’t Equal Workload
Budget• Cost Share• Significant
Dollars• US Dollars• Contingency
Funds
Numbers Don’t Equal Workload
Intellectual Property• Ownership• BIP• Licensed
Technology Involved• Principal
Investigator
Numbers Don’t Equal Workload
Subawards• Multiple• FDP or Institutional
Template• FAR Flow Down• Past experience with
subawardee• Dependence• High risk• Large dollar amount
Numbers Don’t Equal Workload
Risk• Place of performance• Principal Investigator• Warranty on
deliverables• Export controlled or
NIST 800-181 requirements
• Foregin sponsor
Complexity ScoringTYPE SPONSOR BASE SCOPE BUDGET IP SUBS RISK C-SCORE
Grant NSF 1 .5 .5 0 0 .5 2.5
Coop NSF 2 1 1 0 .5 2 6.5
Contract DoD P/T 2 1 1 .5 1 2 7.5
Other State 2 .5 .5 0 0 .5 3.5
Other State 1 .5 .5 1 1 1 5
SBIR DoD 2 1 .5 .5 0 2 6
TOTAL 31
ior Negotiator A
NUMBER ROLE
36 Sr. Coordinator28 Sr. Coordinator29 Sr. Coordinator28 Coordinator21 Sr. Coordinator18 Sr. Coordinator17 Coordinator14 Coordinator11 Coordinator5 Coordinator5 Asst. Director0 Proposal0 Proposal0 Proposal0 Proposal0 Proposal0 Proposal
212
Sheer Numbers
WORKLOAD ROLE
72 Sr. Coordinator56 Sr. Coordinator44 Sr. Coordinator28 Coordinator21 Sr. Coordinator18 Sr. Coordinator51 Coordinator21 Coordinator18 Coordinator5 Coordinator
25 Asst. Director0 Proposal0 Proposal0 Proposal0 Proposal0 Proposal0 Proposal
269
WORKLOAD
WORKLOAD ROLE
72 Sr. Coordinator56 Sr. Coordinator44 Sr. Coordinator28 Coordinator21 Sr. Coordinator18 Sr. Coordinator51 Coordinator21 Coordinator18 Coordinator5 Coordinator25 Asst. Director0 Proposal0 Proposal0 Proposal0 Proposal0 Proposal0 Proposal
269
WORKLOAD(Hint: It is not just numbers or weighted workload).
Over time this will more accurately predict • Capacity• Trends• Staffing• Compensation• Promotion
Taking into consideration• FTEs• Events (ARRA, natural
disasters)
Proposal Productivity
0
100
200
300
400
500
600
700
800
900
Proposals per Year
SPS/25 SPS/8 SPS/5 PS/3 PS/2 PS/1 PS/<1
Proposal Productivity
0
200
400
600
800
1000
1200
1400
Proposals per Year Workload per Year
SPS/25 SPS/8 SPS/5 PS/3 PS/2 PS/1 PS/<1
Multiple Applications
Proposals
Post-award(Billing)
Invention Disclosures
INPUT AND QUESTIONSMichele Vaughan, BS, CCRP Research Strategy and Compliance Specialist Nationwide Children’s HospitalEmail: [email protected]
Susan Sedwick, PhD, CRA, CSMSenior Consultant Attain LLCEmail: [email protected]
