1© 2011

®1

®®

New ExaminersPreparation & Workshop

The Secretary’sRobert W. Carey

Performance Excellence Awards

May 24-27, 2011Washington, D.C.

2© 2011

®222

• Safety–fire exits, emergencies, etc. • Purpose• Agenda and general information • Code of Conduct• Expectations• Roles

S.P.A.C.E.R.

3© 2011

®333

• Any medical professionals in the room?• Others that are first aid / CPR certified?• AED locations?• Fire exits• Rally point• You are responsible to account for the

person to your right and to your left

Safety SAFETY

4© 2011

®444

After completing this program, participants will be able to:

• Apply the 6-Step Evaluation process to the application review process

• Prepare feedback comments that add value to the applicant and meet the Comment Guidelines

• Complete the Independent Review process

• Adequately prepare for the Consensus Review Process

Purpose – Examiner Preparation Learning Objectives

5© 2011

®555

After completing this pre-session, participants will:• Understand the history and purpose of the

Baldrige National Performance Excellence Program & Carey Performance Excellence Awards Program

• Understand the examiner’s role in the Carey Program

• Be familiar with essential Baldrige-based examination terminology

• Be prepared for the next level of examiner training

Purpose – New Examiner Pre-session Learning Objectives

6© 2011

®666

• Tuesday Introductions Pre-test Program Overview & History

• Baldrige • Carey

Examiner’s Role Examination Cycle Overview Application Overview Criteria Book Overview

Agenda – New Examiner Pre-session

7© 2011

®777

Introductions Examination Process Overview Requirements & Expectations of Key Stakeholders Key Terms Criteria Overview Lessons Learned

Agenda – Joint Session

8© 2011

®888

• Wednesday – Full Day Organizational Profile & Key Factors

• KF Worksheet tool Practice one – Process item Lessons Learned

• What went well?• Even more effective tomorrow, if only …

Agenda –Day 1All Examiners

Generic Type Applicant Specific Info Here 1.1 1.2 2.1 2.2 3.1 3.2 4.1 4.2 5.1 5.2 6.1 6.2 7.1 7.2 7.3 7.4 7.5

ContextMain health care service offerings X X X X X X X X X X X X

Relative importance X X X X X X

Delivery mechanisms to provide health care services X X X X X X X X X X

Organizational culture X X X X X X X X

Purpose X X X

Vision X X X X

Values X X X

Mission X X X

Core competencies and relation to mission X X X X X X X

Workforce profile X X X X X X X

Workforce groups and segments and their requirements / expectations X X X X

Workforce educational levels X X X X X X

Key engagement elements X

Workforce and job diversity X X X X X X X

Organized bargaining units X

Key Benefits X X X

Special health and safety requirements X X X

Major facilities, technologies, and equipment X X X X X

Legal and regulatory environment X X X X

Occupational health and safety regulations X X X X

Accreditation, certification, registration requirements X X X X X X X X X X

Health care industry standards, environmental, financial, health care service delivery regulations X X X X X X X X X X

Organizational structure and governance system X X X

Reporting relationships among governance board, senior leaders, and parent organization X X

Key market segments, patient and stakeholder groups X X X X X X

Key requirements and expectations for health care services, support services, and operations X X X X X X

Differences in requirements and expectations among market segments and patient and stakeholder groups X X X

Key types of suppliers, partners, and collaborators X X X X

Role of suppliers, partners, and collaborators in delivery of key health care services, and patient and stakeholder support services X X X X X

Key mechanisms for communicating with supplier, partners, and collaborators X X X X X

Role of suppliers, partners, and collaborators in implementing innovations X

Key supply chain requirements X X X

Competitive position X X X X

Relative size and growth in the health care industry or markets served X X X

Numbers and types of competitors and key collaborators X X X

Key changes that affect competitive situation, including opportunities for innovation and collaboration X X X

Key available sources of comparative and competitive data from within the health care industry X X X X X X X

Key available sources of comparative data from outside the health care industry X X X X

Limitations affecting ability to obtain comparative / competitive data X X X X X X

Key health care service, operational, societal responsibility, and human resource strategic challenges and advantages X X X X X X

Key elements of performance improvement system X X X X

Evaluation, organizational learning, and innovation processes X X X X X X X X

9© 2011

®999

Agenda – Day 2

• Thursday – Full Day Practice Two – Process Item Practice Three – Result Item Independent Review Workbook Tool Using the Scorebook Lessons Learned

10© 2011

®101010

• Friday Morning Practice Four – Results Item Key Themes Consensus Process Post-test Lessons Learned

• What went well?• Even more effective for the next group, if only …

Team meetings

Agenda – Day 3

11© 2011

®111111

• Respect opinions of others• One speaker at a time – avoid sidebar conversations• Listen and have an open mind• Value the diversity of group members• All questions are good questions• Share experiences and lessons learned• What is said in class – stays in class• Cellphone on vibrate• Refrain from texting, emailing, web-surfing , etc.• Others?

Code of Conduct (Ground Rules)

12© 2011

®121212

• Our expectations: Timely return from breaks LEARN something of value to you, your team,

your organization, and …have FUN in the process of doing

so!!• Your expectations?

Expectations

13© 2011

®131313

• Carey Program Office (008B3) Scott Holliday, Director, Management Systems Improvement

Service (202-349-9832, x1927) Gwendolyn Young (gwendolyn.young@va.gov 202-349-9829) Diane Burton (diane.burton@va.gov 202-461-5756)

• Quiet Excellence Denise Haynes (330-573-4025) Doug Serrano (703-869-6658) Glenn Bodinson (972-489-5430) Kay Kendall (978-930-4545) Email Carey@quietexcellence.com

Roles

14© 2011

®141414

• First and only public-private partnership aimed at improving the performance of U.S. organizations

• Manages the Malcolm Baldrige National Quality Award• Educates organizations through identifying and

sharing performance excellence management practices, principles, and strategies

Baldrige Web site Hard copy criteria Conference hosting Examiner training

Program Overview – Baldrige

1461 Applicants 87 Recipients (92 awards)

= 6.3%

15© 2011

®151515

• The Malcolm Baldrige National Quality Improvement Act of 1987, Public Law 100-107

• Created Award Program to identify/recognize role-model

businesses establish criteria for evaluating

improvement efforts disseminate/share best practices

• Expanded to health care and education (1998)• Expanded to nonprofit sector (2005)

Baldrige Program History

16© 2011

®161616

• 200+ really good questions• No answers• Not easy to answer• Define and focus upon performance excellence• Not prescriptive … although …• Designed to facilitate serious reflection• “Leading edge of validated management practice”• “Universally” applicable to all organizations

All sizes All sectors

The Baldrige Criteria

There are no answers in the Criteria - only really good questions.

17© 2011

®171717

• Are considered a validated organizational performance assessment tool

• Are used to identify Award recipients• Are used by diverse organizations in all sectors of the U.S.

economy• Comprised of an Organizational Profile and seven Categories—

an integrated management framework • Are in the “public domain” – no copyright restrictions• Are updated regularly

The Baldrige Criteria

The Baldrige Criteria are updated every two years.

18© 2011

®181818

• Secretary’s program Award recipients selected by the Secretary Based on the Baldrige Criteria Program office in Washington, D.C.

• Examiners include VA and non-VA Examine the application Prepare initial feedback and recommended score to judge Conduct site visit and/or modify feedback as appropriate

• Judges are non-VA and are all Baldrige experts Validate or revise applicant score as appropriate Modify feedback comments as appropriate Provide comments and questions to the examiners for site visits Evaluate the work of the examiner teams

The VA Carey Performance Excellence Program

Carey Award recipients are selected by the Secretary.

19© 2011

®191919

• Certificate of Commitment• Achievement Awards

Score range 340-430 All Item scores ≥ 20%

• Excellence Awards Score range > 430 All Item scores ≥ 30%

• Trophy Awards Limited to two in any given award cycle

• Circle of Excellence Awards Recent trophy level winners that are not eligible to compete again No limit on number of awards

The VA Carey Awards

Scores are the principal basis for determining recognition levels

20© 2011

®202020

• # 1 – Provide objective, actionable feedback to the applicant Primary target audience is the CEO / Executive Director Secondary target audience is the work system / work

process owners

• Provide input to the Judge Accurate scoring recommendation is a must

• Be a member of a team We can all read the same thing and interpret differently Fulfill obligations to the team for timeliness, adequacy, &

accuracy

The Examiner’s Role

21© 2011

®212121

Examination Cycle Overview

(8) Array by score

(3)Examiner team

consensus

(5)Site Visit?

(9) Recommendations sent to Secretary

(6)Clarify/verifyInformation

(1) MSIS receives

application

(2)Examiner team

independent review

(4) Judge review &

Calibration

(7)Other checks

(8a)Technical Edit by MSISFeedback to applicant

Applicant Improvement

IR Workbook

TeamScorebook

TeamScorebook

Awards

(8a)Technical Edit by MSISFeedback to applicant

22© 2011

®222222

• Glossary of acronyms• Organizational Profile• Responses to the 2011-2012 Baldrige Performance

Criteria for Performance Excellence Processes Results

Overview of an ApplicationWho are we?

What do we do?

How do we do what we do?

How well do we do what we do?How do we know how well we’re doing?

Carey applications contain an Organizational Profile (maximum of 5 pages) and responses to Multiple

Requirements (maximum of 50 pages).

23© 2011

®232323

• No page limit• Includes only terms and abbreviations used in the

application, with very brief definitions• Does not include descriptions of processes, tools,

methods, or techniques• The key to “all those acronyms”

Some are “common usage” Some are “organization specific” Some may not mean what you think

Glossary of Terms

24© 2011

®242424

• Not scored• 5 page maximum• Applicant perspective = “get to know us”

Who are we, and what do we do?• Examiners & Judges perspective = “key factors”

What is important / vital to the organization Context of the organization Sets expectations for processes and results

Organizational Profile

No points! Used by examiners and judges to understand the organization and what they consider important.

25© 2011

®252525

• The “meat” of the application 50 page limit

• 55% of the score is from “Process” items– How do we do what we do?

• 45% of the score is from “Results” items– How well are we doing?– How do we know how well we are doing?

Application itself is not scored• Only the content• “Examiner-friendly” doesn’t hurt• Résumé to get the interview (site visit)

Responses to Items

26© 2011

®262626

• Criteria pg 4/4 Criteria Notes

• Category and Item Descriptions pg 34/33• Core Values and Concepts pg 49/49• Glossary of Key Terms pg 56/57• Scoring System pg 66/67

Overview of the Criteria Book

27© 2011

®27

®®

What are your Questions?info@quietexcellence.com

Denise Haynes330.573.4025

Doug Serrano703.869.6658

28© 2011

®28

®®

The Secretary’sRobert W. Carey

Performance Excellence Awards

Joint Session

29© 2011

®292929

• First person tells us a bit about themself• When you hear a “connection,” join up front and tell us a bit

about yourself• Repeat process until we’re all connected

Connections

30© 2011

®303030

Begin with the end in mind

SIPOCCOPIS

31© 2011

®313131

S-I-P-O-C / C-O-P-I-S

IR Workbook

Site Visit

Scorebook &

Feedback Report

ConsensusReview

Scorebook

Applicant Application Independent Review by Examiners Consensus

Team

Independent Review Team Members Consensus

Process

IR Workbook

Judges

Site Visit Team

ConsensusReview

Scorebook

Site Visit Team ApplicantConsensus

Team

R&E?

R&E?

R&E?

R&E?

S I P O C

32© 2011

®32

Key Terms

33© 2011

®333333

• The Core Values are embedded beliefs and behaviors found in high-performing organizations

• They are the foundation for integrating key performance and operational requirements within a results oriented framework that creates a basis for action and feedback

Core Values – Definition

34© 2011

®343434

• Commonalities among high-performing organizations• Always “in the back of the mind” of the examiners• May be used in comments, particularly as the “so

what” of an OFI• May impact the scoring by making the strength / OFI

particularly important• Link to the “key themes”

Core Values – Why they are important

35© 2011

®353535

Core Values – List

● Visionary leadership● Customer-driven excellence● Organizational and personal learning● Valuing workforce members and partners● Agility● Focus on the future● Managing for innovation● Management by fact● Societal responsibility● Focus on results and creating value● Systems perspective

36© 2011

®363636

• Key factors are significant attributes of an organization that influence the way the organization operates

• Examiners will use key factors to focus their assessment on what is important to the applicant

• A critical consideration in evaluation is the importance of reported process and results to key business factors

Key Factors – Definition

37© 2011

®373737

• Mission, Vision, Values• Employee / Staff profile• Customer Segments• Strategic Challenges• Critical Success Factors

These are facts or attributes that affect the way the organization operates

Key Factors – Examples

38© 2011

®383838

Key Factors – non-examples

• Complaint Management Process• Strategic Planning approach

These are processes that the applicant uses to manage work ~ not key factors

39© 2011

®393939

• Describe the systems and processes used to accomplish mission requirements

• Responses to “how” questions should address all four evaluation factors: Approach Deployment Learning Integration

How … ?

A-D-L-

I

40© 2011

®404040

• Basic information on key processes “What are the steps … ?”

• Information on key findings, plans, objectives, goals, or measures “What are your plans … ?” “What are your key measures … ?”

• Set the context for showing alignment and integration

• Tables provide clear, concise information (more info in less space)

What … ?

SO’s Measure Goal Figure

May contain columns showing links to strategic objectives or key challenges/advantages to provide

evidence of Deployment.

41© 2011

®414141

• 2011/2012 Criteria has 33 questions to address in the Organizational Profile

• All of them begin with the word “what”Therefore…

• The description of “how” will come later• Remember, it’s not the application itself being

examined

• Sets the examiner understanding of who the applicant is / what they do

Profile = What

42© 2011

®424242

• The Process Categories (1 through 6) 167 questions 23 are “what” 1 is “who”

• That leaves … 143 asking the question “how”

• That does not mean 143 separate processes!

Processes = How (plus a few what’s and a who)

43© 2011

®434343

Anatomy of the Criteria Pg 29

44© 2011

®44

Evaluation factors for processes

45© 2011

®454545

• Methods used by an organization to address the Baldrige Criteria item requirements

• Methods include processes, techniques, facilities, & equipment• Includes the appropriateness of the methods to the item

requirements and the organization’s operating environment; and the effectiveness of their use

Approach

How do the examiners evaluate appropriateness?

How do the examiners evaluate effectiveness?

46© 2011

®464646

• Linked activities with the purpose of producing a product or service for patients and other customers within or outside of the organization

• Generally, processes involve combinations of people, machines, tools, techniques, materials, and improvements in a defined series of steps or actions

Process

Step 1Step 2Step 3

Step-by-step description

47© 2011

®474747

• Approaches that are well-ordered, repeatable, and use data and information so learning is possible

• Approaches are systematic if they build in the opportunity for evaluation, improvement, and sharing, thereby permitting a gain in maturity

Systematic Approach

Flow Diagram

“We have a Policy/Procedure”

“Key Measures”

48© 2011

®484848

• The extent to which an approach is applied in addressing the requirements of a Baldrige Criteria item

• Deployment is evaluated on the basis of the breadth and depth of the use of the approach to relevant departments and work units throughout the organization

Deployment

49© 2011

®494949

• New knowledge or skills acquired through evaluation, study, experience, and innovation Personal learning is achieved through education, training,

and developmental opportunities that further individual growth

Organizational learning is achieved through research and development; evaluation and improvement cycles; workforce, patients, other customers, and other stakeholder ideas and input; best practice sharing; and benchmarking

Learning

50© 2011

®505050

• Has only happened when there is a resulting change made: to the process itself, and/or … to inputs within the process

• “Learning” is expected to drive improvement• And …

the change is either sustained or is subsequently changed again for the better

Learning …

Learning can be demonstrated through examples of both – process improvements and sharing

51© 2011

®515151

• Harmonization of plans, processes, information, resource decisions, actions, results, and analyses to support key organization-wide goals

• Effective integration is achieved when the individual components of a performance management system operate as a fully interconnected unit

Integration

52© 2011

®525252

• Consistency of plans, processes, information, resource decisions, actions, results, and analyses to support key organization-wide goals Requires a common understanding of purposes and goals Requires the use of complementary measures and

information for planning, tracking, analysis, and improvement

Alignment

53© 2011

®535353

Aligned, or Integrated?

54© 2011

®545454 Or …

55© 2011

®555555

As the organization gains “maturity”

56© 2011

®56

Evaluation factors for results

57© 2011

®575757

• Numerical information that places or positions an organization’s results and performance on a meaningful measurement scale

• Performance levels permit evaluation relative to past performance, projections, goals, and appropriate comparisons

Levels

58© 2011

®585858

• Numerical information that shows the direction and rate of change for an organization’s results Trends provide a time sequence of organizational

performance A minimum of three historical (not projected)

data points generally are needed to begin to ascertain a trend. More data points are needed to define a statistically valid trend.

The time period for a trend is determined by the cycle time of the process being measured, e.g., bi-annually (not daily)

Trends

59© 2011

®595959

Who did they claim as “comparison” data resources in

the Organizational Profile – P.2a(3)?

• All Results Items call for “appropriate comparative data”• Data on competitors’ performance, and comparisons

with other organizations providing similar services Information may be obtained from:

• sharing or contributing to external databases in order to obtain feedback for reference

• information obtained from open literature (e.g., internet)

• data gathering and evaluation by independent organizations

Comparisons

60© 2011

®606060

• Benchmarks are a form of comparative data (n) Processes and results that represent best practices and

performance for similar activities, inside or outside an organization’s industry

(v) Organizations “benchmark” to understand the current dimensions of world-class performance and to achieve discontinuous (non-incremental) or “breakthrough” improvement

Benchmark

Any instances to “spotlight” where the applicant is the benchmark?

61© 2011

®616161

Better than average?or

Best among the average?or

Average among the best?or

Best of the best?

Do they want to be …

What did they say in theirMission / Vision statements

in the Organizational Profile – P.2a(2)?

62© 2011

®626262

• Required to have a systems perspective to goal alignment, particularly when strategy and goals change over time

• Action-oriented cycles of learning take place via feedback linkages between processes and results

• Great sources for key linkages are the item notes at the end of Criteria item 2.2 (p. 12), various item descriptions (i.e., 3.2, 5.1, 5.2, 6.1) and definitions (e.g., “systems perspective”)

Dynamic Linkages

63© 2011

®636363

• “Deployment” gaps in approaches can prevent higher scoring• Missing information will be interpreted as a gap in the

performance management system• We expected to see something, but it was not there

Key factor not addressed by a process Key factor without results Process without results Results that don’t pertain to a

described process

Gaps

Results should pertain to what the applicant said is important, not “just” to the things that are performing well at the moment

64© 2011

®646464

• A part of an organization’s overall patient, other customer, market, health care service, or workforce base Typically have common characteristics that can be

grouped logically Critical to identifying the distinct needs and expectations

of different customer, market, and workforce groups – and to tailoring offerings to meet their needs & expectations

In Results items, the term “segmentation” refers to disaggregating results data in a way that allows for meaningful analysis of an organization’s performance

Segments & Groups

65© 2011

®656565

• How did the applicant define the “groups and segments” … In the Organization Chart?

• Organizational units In the Organizational Profile?

• Product offerings [P.1a(1)]• Workforce groups & segments [P.1a(3)]• Workforce and job diversity [P.1a(3)]• Locations or facilities [P.1a(4)]• Market segments, customer groups, & stakeholder groups [P.1b(2)]

In the process descriptions?• Key work systems (6.1)• Key work process types and locations (6.2)

Segments & Groups

66© 2011

®666666

• Category 7 call outs for segmentation, “as appropriate” 7.1 – product offerings, customer groups, and market

segments; and by process types and location 7.2 – product offerings, customer groups, and market

segments 7.3 – address the diversity of your workforce and address

your workforce groups and segments 7.4 – organizational units 7.5 – market segments or customer groups

Segments & Groups

67© 2011

®676767

• The term “core competencies” refers to the organization’s areas of greatest expertise

• Those strategically important capabilities that are central to fulfilling the mission or provide an advantage in the marketplace or service environment

• Frequently are challenging for competitors or suppliers and partners to imitate

• May provide a sustainable competitive advantage• Absence of a needed organizational core competency may result in a

significant strategic challenge or disadvantage in the marketplace• May involve technology expertise, unique service offerings, a

marketplace niche, or a particular business acumen (e.g., health care delivery start-ups)

Core Competency

68© 2011

®686868

• Customer engagement Patients’ and stakeholders’ investment in or commitment to

your organization and health care service offerings. Based on your ongoing ability to serve their needs and build

relationships so they will continue using your services. Characteristics include loyalty, willingness to make an effort to

obtain services from your organization, and willingness to actively advocate for and recommend your organization and service offerings.

Satisfaction / EngagementCustomer

69© 2011

®696969

• Workforce engagement Extent of workforce commitment, both emotional and

intellectual, to accomplishing the work, mission, and vision of the organization.

Often characterized by high-performing work environments in which people are motivated to do their utmost for the benefit of their patients and stakeholders and for the success of the organization.

Workforce engagement also depends on building and sustaining relationships between your administrative / operational leadership and your independent practitioners.

Engagement / SatisfactionWorkforce

70© 2011

®707070

• Meaningful change to improve health care services, processes, or organizational effectiveness and to create new value for stakeholders. Involves the adoption of an idea, process, technology, product, or business

model that is either new or new to its proposed application. Discontinuous or breakthrough change in results, services, or processes. Multistep process that involves development and knowledge sharing, a

decision to implement, implementation, evaluation, and learning. Applicable to all key organizational processes that would benefit from

change, whether through breakthrough improvement or a change in approach or outputs.

Could include fundamental changes in organizational structure or the business model to more effectively accomplish the organization’s work and to improve critical pathways and practice guidelines, facility design, the administration of medications, the organization of work, or alternative therapies.

Innovation / Improvement

71© 2011

®717171

• Performance projections = estimates of future performance Based on understanding of past performance, rates of

improvement, and assumptions about future internal changes and innovations, as well as assumptions about changes in the external environment that result in internal changes.

Can serve as a key tool in both management of operations and strategy development and implementation.

• Projections – Statement of expected future performance • Goals – Statement of desired future performance

Projection / Goal / Target

72© 2011

®727272

• All groups that are or might be affected by an organization’s services, actions, and success

• Examples might include: Patients and patients’ families, the community, insurers

and other third-party payors, employers, health care providers, patient advocacy groups, departments of health, students, the workforce, partners, collaborators, governing boards, stockholders, investors, charitable contributors, suppliers, taxpayers, regulatory bodies, policy makers, funders, and local and professional communities.

Stakeholder

73© 2011

®737373

• How the work of your organization is accomplished Involves workforce, key suppliers and partners, contractors,

collaborators, and other components of the supply chain Coordinate internal work processes and external resources

necessary to develop, produce, and deliver your services to patients and stakeholders and to succeed in your marketplace.

• Decisions about work systems are strategic Involve protecting and capitalizing on core competencies Deciding what should be procured or produced outside

your organization in order to be efficient and sustainable in your marketplace.

Work System

74© 2011

®747474

• Most important internal value creation processes Might include health care service design and delivery, patient

support, supply-chain management, business, and support processes

Processes that involve the majority of workforce and produce patient and stakeholder value

• Frequently relate to: Core competencies Factors that determine your success relative to competitors and

organizations offering similar health care services Factors considered important for business growth by your

senior leaders

Work Processes

75© 2011

®75

®®

What are your Questions?info@quietexcellence.com

Denise Haynes330.573.4025

Doug Serrano703.869.6658