CTSA Program Common Metrics Initiative Maximizing CTSA ...

Maximizing CTSA Program Impact

Module Two

Philip L. LeeResults Leadership Groupwww.resultsleadership.org

CTSA Program Common Metrics Initiative

Based on the principles of Results-Based Accountability from Trying Hard is Not Good Enough: How to Produce Measurable Improvements for Customers and Communities by Mark Friedman (Trafford 2005)

Module Two

1. Review of Interim Learning Activities

2. Maximizing the Impact of the Whole (as well as the parts)

3. Selecting Headline Performance Measures

4. Interim Learning Activities

Review of Interim Learning Activities

1. Frequently Asked Questions

2. Observations

3. Recap

Module Two


2. Maximizing the Impact of the Whole (as well as the

parts)



Population Health

Health Care

Biomedical Research

Whole Versus PartsMaximizing the Impact of the Whole (as well as the parts)

Environment

Public Safety

Education

Economy

Socio-economicStatus

Biomedical Research

NIH

Whole Versus Parts

IndustryPatients/Communities

Universities/Medical Centers

CTSAProgram

CTSAProgram

Biomedical Research

CTSA ProgramHub

Hub

Hub

Hub

HubHub

Hub

TICS RICs

Trial Innovation Network

Whole Versus Parts

Hub

Hub

Hub Hub

Hub

Hub

HubHubHub

Hub

Hub

Hub Hub

HubHub

Hub

Workforce Development

Hub

BMIRegulatory Knowledge

PilotFunding

Community Engagement

Whole Versus Parts

BERDBERD

BERD

Whole Versus Parts

10

How much work did an

individual partdeliver?

Maximizing the Impact of the Whole (as well as the parts)

Distinguishing Parts vs. Whole; Effort vs. Impact

What was the impact of the

individual part?

What was the impact of the

whole?

Parts Whole

Impa

ct

Effo

rt How much work did the

whole deliver?

FPSI/RLG11

Effort versus impact?

Benchmark

Target

Indicator Goal

Result

Objective

Outcome

Measure

ModifiersMeasurable CoreUrgent QualitativePriority ProgrammaticTargeted PerformanceIncremental Strategic

Systemic

Whole Versus Parts

Definitions

Result

Indicator

Performance Measure

Babies Born Healthy, Safe Communities, Clean Environment, Free from Death and Suffering Due to [disease]

Rate of low birth weight babies, Crime rate, Air quality index, Mortality and morbidity rates for [disease]

1. How much did we do? 2. How well did we do it? 3. Is anyone better off?

A condition of well-being for children, adults, families or communities.

A measure which helps quantify the achievement of a result.

A measure of how well a program, agency, or service system is working.

= Customer Results

POPU

LATI

ON

C

UST

OM

ERS

Three types:

(Language Discipline)

Performance Measures1. How much did we do?

Who are our customers and what services do we provide to them?

2. How well did we do it? How well do we provide those services?

3. Is anyone better off?What is the desired impact of those services for/on our customers (a.k.a. “customer results”)?

Indicator orPerformance

Measure?

What kind ofPerformance

Measure?

Provide Training

Mastered New Job Skills

Employed

Employed @ skill level

Employment@ Skill Level & > year

CityEmployment

Rate

Job Training Program

How much?

Population = Indicator

Certified Instructors

1. Safe Community

2. Crime Rate

3. Average Police Dept response time

4. Free from death and suffering due to hypertension

5. Mortality and morbidity rates for hypertension

6. Average blood pressure of pulmonary hypertension clinic patients

7. Youth Enter College or Good Jobs

8. % of youth who graduate from high school

RESULT, INDICATOR OR PERFORMANCE MEASURE?RESULT

INDICATOR

PERF. MEASURE

RESULT

INDICATOR

RESULT

INDICATOR

PERF. MEASURE

HOW WELL?

BETTER OFF?

PERF. MEASUREBETTER OFF?

Indicator or Performance

Measure?

What kind ofPerformance

Measure?

Provide CT resources & services and CT

education/training & early career support

Improved translation & translation processes

(speed, cost, quality, etc.)& successful careers in

CT Research

Improved Mortality & Morbidity

Rates


CTSAPROGRAM

Develop, Demonstrate,& Disseminate

How much?

How much?

Whatactivities, services,

and/orresources

do we perform/provi

de?

Who are our customers?

What is the desired impact

of those activities/services/resourceson/for those customers?

How are we doing at

achieving that desired impact?

Common Metrics = Data/Feedback on ImpactEffort/How much? Impact/Better off?

What PIs/NCATS want to track

(“Requirements”)

Common Metrics(Data)

1. Free from death and suffering due to [disease]

2. Mortality and morbidity rates for [disease]

3. Median IRB duration

4. % CT scholars with grants/publications

5. Median study start up time

4. Rates of Pilot Funding publications and subsequent funding

4. Accrual rates

RESULT

INDICATOR

PERF. MEASURE

PERF. MEASURE

PERF. MEASURE

CTSA Program

BETTER OFF?

BETTER OFF?

BETTER OFF?



RESULT, INDICATOR ORPERFORMANCE MEASURE?

The Whole, the Parts & Ends-to-Means Decision Making

(a.k.a., Turn-the-Curve Thinking)

1. Aligning our activities in service of our ends.

2. The matter of contribution.

Population HealthLung Cancer

NIH

Biomedical Research

Industry

Patients/Communities

Universities/Medical Centers

CTSAProgra

m

Whole Versus Parts

Population HealthLung Cancer

Health Care

Biomedical Research

Environment

Public SafetyEducation

Economy

Socio-economicStatus

Strategic Priorities?

Whole Versus Parts

Story behind the curve

Partners (with a role to play in turning the curve)

What would work to turn the curve

Strategy

How are we doing?

Why?

Help?

Options?

Propose to do?

(c) Results Leadership Group, LLC 23

Mortality & Morbidity Rates (the “Curve”)

Free from Death and Suffering Due to Lung Cancer

1. Cigarette smoking2. Socio–economic status3. Environmental hazards4. Inadequate knowledge of the causes

of lung cancer

What is the story behind the curve?

What is our strategy to turn the curve?1. Smoking Cessation Campaign2. Behavioral research on smoking3. Targeted education campaign4. Lung cancer research

Whole

Part

Priority

Doing the

right things?

Ends

Means

Lung Cancer Mortality and Morbidity Rates

Smoking Cessation Program

Whole Versus Parts


Partners (with a role to play in turning the curve)


Strategy


ProgramPerformance

Measures

Whole

Smoking Cessation Program

Doing things

right?

Start withthis End

Mortality and Morbidity rates for ...

National/Community-wide Strategy/Partners

Smoking Cessation Program Performance Measures (baselines)

WH

OLE

PAR

TS

Customer Impact END

X

1. Doing the right things?

2. Doing those things right?

Population Accountability

Performance Accountability

?

Joint Accountability

Sole Accountability

END

M

EAN

S

People are Healthy

The Whole, the Parts & Ends-to-Means Decision Making

People are Healthy

The Matter of Contribution (Joint vs. Sole Accountability)

People are HealthyMortality and morbidity rates

Health Care Patients are healthy/not suffering

Biomedical Research New biomedical knowledgeNew treatments

Translational Research/Innovation • New knowledge about translation• Improvements to translation processes &

translation rates

What Is Your “End”?

Population HealthIndicators:

Story:Partners:Strategy (including the role of the CTSA Program):

Population Accountability

CTSA ProgramCTSA Program Common Metrics:

Story:Partners:Strategy to “turn the curve” of a CTSA Program Common Metric:

Performance Accountability

31

The CTSA Program shares joint accountability with multiple stakeholders to “turn the curve” of an indicator (e.g., mortality and morbidity rates). The CTSAProgram has a role within the larger strategy.

The CTSA Program is solely accountable to “turn the curve” of the CTSA Common Metrics –(which contributes to the larger strategy to turn the curve of the indicator).

!

32

I sure am glad we don’t have that problem!

I sure am glad we don’t have that problem!

!

33

!

34

Joint Accountability

!

35

Sole AccountabilitySole Accountability

Right Strategy Tools

Right Strategy Roles & Partnerships

Right Implementation?

Right Data for Useful Feedback?

CTSA ProgramHub

Hub

Hub

Hub

HubHub

Hub

TICS RICs

Trial Innovation Network

Hub

Hub

Hub Hub

Hub

Hub

HubHubHub

Hub

Hub

Hub Hub

HubHub

Hub

Whole Versus Parts

Workforce Development

Hub

BMIRegulatory Knowledge

PilotFunding

Community Engagement

BERD

Whole Versus Parts

Hub Accrual Rates


What is our strategy to turn the curve?

Whole

Doing the right things?

End

1. Non-investigator clinicians not screening patients for potential study participants during clinic consultations.

2. Lack of systems for data-driven cohort discovery in planning clinical studies

3. Inadequate knowledge/skills in recruitment planning/implementation

4. Study designs that impede participation5. Sites inconveniently located.

Priority

1. Research, develop, and institute targeted strategies to motivate non-investigator clinicians to screen patients.

2. Develop and implement social media strategies.

PartsMeans

“People and their managers are working so hard

to be sure things are done right, that they hardly have time

to decide if they are doing the right things.”

Stephen R. Covey

Common Metrics for the CTSA Program

Hub Strategy to Turn-the-Curve on Accrual Rates

Performance Measures (baselines)

WH

OLE

PAR

TS

Customer Impact END

X



?

END

M

EAN

SHub Accrual Rates

Whole or Part?End or Means?

Clinician PatientScreening Program

Hub Accrual Rates

Hub Strategy to Turn-the-Curve on Accrual Rates

Performance Measures (baselines)

Customer Impact END

X



?

Whole or Part?End or Means?

Clinician Patient Screening Program

PM


• Non-investigator clinicians not screening patients for potential study participants during clinic consultations. (See: CTSI report)

• Lack of systems for data-driven cohort discovery in planning clinical studies

• Inadequate knowledge/skills in recruitment planning/implementation

• Study designs that impede participation

• Sites inconveniently located.

% of studies that achieved accrual goal within time specified in study designPM

50 %Current Value:Q2 2016

PM

What are we going to do?

% of studies that achieved accrual goal within time specified in study designPM

Research, develop, and institute targeted strategies to motivate non-investigator clinicians to screen patients. Engage institutional leadership.

Develop and implement social media strategies.Partner with the communications school.

50 %Current Value:Q2 2016

PerformanceMeasures? Performance

Measures?


Whole Versus Parts


Partners with a role to play in turning the curve


Strategy to turn the curve


Performance Measures?

Whole


Doing things

right?

Start withthis End

Accrual Rates:

Story:

Strategy (including the role of the Clinician Patient Screening Program):

Hub Common Metric

Headline Performance Measures:

Story:

Strategy to “turn the curve” of a program headline performance measure:


50

The manager of the Clinician Patient Screening Program shares joint accountability with multiple stakeholders to “turn the curve” of the accrual rates for the Hub. The Clinician Patient Screening Program has a role within the larger strategy.

The manager of the ClinicianPatient Screening Program is solely accountable to“turn the curve” of the HeadlinePerformance Measures for the Clinician Patient Screening Program.

Module Two





52

How much service did we deliver?

Selecting Headline Performance Measures

How well did we

deliver it?

How much change / effect

did we produce?

What quality of change / effect

did we produce?

Quantity Quality

Effe

ct

E

ffort

How muchdid we do?

Three Kinds of Program Performance Measures

How welldid we do it?

Is anyonebetter off?

Quantity QualityE

ffect

E

ffort

# %

Quantity Quality

Efficiency, Admin overhead, Unit costStaffing ratios, Staff turnoverStaff morale, Access, Waiting time, Waiting lists, Worker safetyCustomer Satisfaction1. quality service delivery2. customer benefit

Cost / Benefit ratioReturn on investmentClient results or client outcomes

EffectivenessValue addedProductivity

Benefit value

ProcessInput

Effe

ctEf

fort

CostTQM

1. Did we treat you well?

2. Did we help you with your problems?

ProductOutput Impact

Performance Measure Categories

# of students

Education

Student-teacher ratio

#/% of high school graduates

Quantity QualityEf

fect

Effo

rt

# of persons treated

Drug/Alcohol Treatment

% of staff with training certification

#/% of clients off of drugs/alcohol

> at exit> 12 months after exit

Quantity QualityEf

fect

Effo

rt

# of responses

Fire Department

Response time

#/% of fires kept to room

of origin

Quantity QualityEf

fect

Effo

rt

Criteria for SelectingHeadline Performance Measures

Communication Power

Proxy Power

Data Power

Does the performance measure communicate to a broad range of audiences?

Does the performance measure say something of central importance about the result/program?

Does the performance measure bring along the data HERD?

Quality data available on a timely basis.

(c) Results Leadership Group, LLC59

Selecting Headline Performance Measures WorksheetUnit/Program _______________________________

H = High; M = Medium; L = Low

Candidate Performance Measure

CommunicationPower

ProxyPower

DataPower

H M L

H

Measure 1Measure 2Measure 3Measure 4Measure 5Measure 6Measure 7Measure 8

HDataDevelopmentAgenda

H M L H M L

H H

H L

Provide Training

Mastered New Job Skills

Employed

Employed @ skill level

Employment@ Skill Level & > year

CityEmployment

Rate

How much?


Certified Instructors

Right Question

Data Development Agenda

Feasible “Proxy”

Job Training Program

Selecting a HeadlinePerformance

Measure

Attendance

Reading on Grade Level

Graduation Rate

College Success Rate

Career SuccessRate

Right Question

Feasible “Proxy”

Data DevelopmentAgenda

Schools

Selecting a HeadlinePerformance

Measure

Module Two





Interim Learning Activities1. Revise and enter into Results Scorecard the headline performance

measure and turn-the-curve plan for the component you selected for Module Two.

2. Complete the Asthma Sorting Exercise.

3. Practice facilitating a turn-the-curve process with a group of colleagues. Have your learning partner observe and provide you with feedback.

4. Using the RBA Turn-the-Curve Plan Rubric, assess the sample Turn-the-Curve Plan.

5. Read the sample performance report and reorganize its elements into a turn-the-curve plan in Results Scorecard.

CTSA Program Common Metrics Initiative Maximizing CTSA ...

Documents

Transcript of CTSA Program Common Metrics Initiative Maximizing CTSA ...