Performance Feedback: Use It or Lose It Randy Keyworth Ronnie Detrich Jack States.

Performance Feedback:

Use It or Lose It

Randy Keyworth

Ronnie Detrich

Jack States

To be sure of hitting the target,

shoot first and whatever you hit,

call it the target.

Ashleigh Brilliant

Performance Feedback SystemsPerformance Feedback Systems

relyrely on Staff Behavior on Staff Behavior

In order for a performance feedback system to be effective, staff must:

implement it

compliancecompliance

correctly over time

competencecompetence sustainabilitysustainability


compliance / competencecompliance / competence

• ask the right questions

• identify appropriate data to collect (validity)

• implement interventions according to plan (treatment integrity)

• collect data accurately (reliability)

• display / analyze data

• interpret data / draw correct conclusions

• give / receive feedback based on the data

• modify interventions based on data

Performance Feedback ObstaclesPerformance Feedback Obstacles

staff resistance to a performance feedback:staff resistance to a performance feedback:

long standing mistrust of the purpose of data

educator autonomy, implicit power relationships

cynicism about fads, new ideas, education reform

resistance to performance feedback

data collection is too difficult

data collection causes too much change

desired outcomes take too long to materialize

perceived costs exceed perceived benefits


sustainability sustainability

• implemented with procedural fidelity and desired outcomes (effectiveness) at the consumer level

• maintains over time

• maintains over generations of practitioners and decision-makers

• operates within existing resources (financial, staff, materials) and existing mandates

• becomes institutionalized, routine…

National Implementation Research Network (NIRN)

“the way we do business”

SystemSystem’’s Change Track Records Change Track Record

average life of an education innovation is 18-48 months (Latham, 1988)

initial data on comprehensive school reform models initiated in 2000:

1 in 5 maintained reforms through 2002

1 in 10 maintained reforms through 2004

(American Institute for Research)

Performance Feedback SolutionsPerformance Feedback Solutions

Sustainable implementation requires a Sustainable implementation requires a CULTURE CHANGECULTURE CHANGE across all levels of an organization:across all levels of an organization:

changes in adult professional behavior (all stakeholders)

changes in organizational structures and cultures, both formal and informal (systems, policies, contingencies, values, procedures)

changes in relationships to consumers, stakeholders, and systems partners (metacontingencies)

National Implementation Research Network (NIRN)

2004 - present2004 - present

independent, non-profit operating foundation

promote evidence-based education policies and practices

act as a catalyst to facilitate communication, cooperation and collaboration between individuals and organizations currently engaged in evidence based education

1978 - 20041978 - 2004

Operated a large non-profit organization in SF Bay AreaOperated a large non-profit organization in SF Bay Areasix spec. ed schools adult programs

residential programs employment supportive services

public school consultation teacher training campus

Operated comprehensive performance feedback Operated comprehensive performance feedback systemssystems

student performance

staff performance

organizational performance

METRICS

Program Classrooms Students

Schools (6) 60 450

Satellites (2) 2 16

Public School Consultation 300 300

Contracting school districts 95

Total Staff 450

Total New Staff Each Year 150

“real world” challenges

Operating within direct service funding (no grants, research, university students)

Perpetual growth mode (services, programs, technology)

Serving extremely “high risk” kids w/ challenging behaviors

(requiring high treatment integrity implementing sophisticated programs)

High profile (regulatory oversight, parents, districts, community)

Constant shortage of trained staff (staff turnover, failure of Universities to train in effective teaching strategies)

A Buddhist View of CultureA Buddhist View of Culture

Kōan

A Kōan consists of a story, dialogue, question, or statement, the meaning of which cannot be understood by rational thinking but may be accessible through intuition.

VERBAL BEHAVIOR

AUTOMATIC BEHAVIOR

Kōan of the DayKōan of the Day

To implement and sustain a performance feedback culture….

you need a performance feedback culture.

A Behavioral View of CultureA Behavioral View of Culture

“As a set of contingencies of reinforcement maintained by a group…

it has…

a continuing existence beyond the lives of members of the group,

a changing pattern as practices are added, discarded, or modified…

A culture so defined controls the behavior of the members of the group that practice it.”

B.F. Skinner

cognitive dissonance

Culture change requires an expanded unit of analysis

organizations, systems, cultures…in addition to individual behavior

new analytic tools

utilization of expanded forms of research (group designs, quasi-experimental, qualitative)

the uncomfortable feeling caused by holding conflicting ideas simultaneously

Stakeholderspolicy makersparentsschool administratorsclassroom staffstudents

What is an Organizational Culture?What is an Organizational Culture? The complex interaction of formal and informal contingencies governing the behavior of all stakeholders, embodied in:

External Contingencieslaws & regulationsfundingjob markettraining and ideology

Internal Contingenciespoliciespracticesvaluesresource allocationsdata systemsfeedback systemsreporting requirements

program evaluationrecruitment & hiringinitiativesjob expectationscompensationstaff trainingstaff coachingstaff feedback

Overcoming Baseline Cultural Obstacles:Overcoming Baseline Cultural Obstacles: Calibration, Process and EngagementCalibration, Process and Engagement

a “learner centered” culture (calibration)focus on student learning and educational practicesestablishing consensus on standards, definitions, goals

a culture of “inquiry” rather than “compliance” (process)use of data to answer questions, problem solveuse of data-based decision making at all levels of the organization not having all of the answers

a culture of “universal participation” (engagement)wide-spread involvement (ownership, pride, participation)collaboration across disciplinesgiving, receiving, and using feedback data analysis as positive, non-threatening experience

Overcoming Baseline Cultural Obstacles:Overcoming Baseline Cultural Obstacles: AlignmentAlignment

Alignment of all organizational cultural components so that contingencies consistently support performance feedback

policiespracticesvaluesresource allocationsdata systemsfeedback systemsreporting requirements

program evaluationrecruitment & hiringinitiativesjob expectationscompensationstaff trainingstaff coachingstaff feedback

Using Using ““cultural alignmentcultural alignment”” to increase performance feedbackto increase performance feedback

Goals:

Definitions:

Outcomes:

Increase the number of staff using effective performance feedback

staff share common values about data, accountability, feedback and problem solving

staff have technical skills in instruction, data analysis, problem solving

staff positions filled by qualified staffstaff retentionperformance feedback systems implemented

Using Using ““cultural alignmentcultural alignment”” to increase performance feedbackto increase performance feedback

X X Xstaff training

Process Measures

Outcome MeasuresStrategies

X X Xstaff feedbackX X XhiringX X Xjob expectationsX X XrecruitmentX X XselectionX X X

X X X

X X X

X X X

X X X

X X X

policies

practices

resource allocations

staff evaluation

compensation

initiatives

Evolution is chaos with feedback.

John Ford, In Clifford A. Pickover, Computers, Pattern, Chaos, and Beauty (), 203.

Aubrey Daniels International© 2008

Five-Step Behavior Management Process

5. Evaluate

4. Reinforce

3. Feedback

2. Measure

1. Pinpoint

Technical skills: Technical skills: specialized skills required to be successful specialized skills required to be successful in a in a

specific positionspecific position

teaching skills organization formative assessment

data collection reinforcement direct instruction

behavior analysis reading data writing reports

Core skills:Core skills: foundation skills required to be successful in foundation skills required to be successful in thethe organizational cultureorganizational culture

assertiveness outcome oriented optimism

effective communication takes initiative motivation / buy-in

conflict resolution established priorities data driven

feedback (give & receive) leadership scientific

flexibility high tolerance for ambiguity nice

1. Pinpoint

Keeper of the Organizational Culture: HUMAN RESOURCES

job descriptions orientation

recruitment employee contracts

hiring orientation

offer sheets

employee contracts

1. Pinpoint

SupervisorsTeachers

Process

Outcomes

treatmentfidelity

proceduralfidelity

studentteacher

performance

2. Measure

direct observation, behavior products, checklists, etc.


“incidental” feedback

“formative” feedback

“formal” evaluations

3. Feedback


compensation

recognition

bonuses

promotions

n

4. Reinforce


consumers measures

staff measures

organizational measures

n

5. Evaluate


relyrely on Staff Behavior on Staff Behavior

antecedent

need to know need to know what to dowhat to do

behavior consequence

need to know need to know how to do ithow to do it

need to be need to be motivated motivated to do itto do it

The secret of success is sincerity.

Once you can fake that

you've got it made.

Daniel Schorr

Building a Building a Performance Feedback CulturePerformance Feedback Culture

All organizations have cultures…

most don’t support performance feedback

Performance Feedback: Use It or Lose It Randy Keyworth Ronnie Detrich Jack States.

Documents

Transcript of Performance Feedback: Use It or Lose It Randy Keyworth Ronnie Detrich Jack States.