2010 tqa assessor workshop by thomas schamberger day one

Baldrige Foundation – Thailand Quality Assessor Workshop

2010

Welcome


2010

Enhancing the Effectiveness of Enhancing the Effectiveness of the TQA Assessment Process the TQA Assessment Process for Assessors and Senior for Assessors and Senior AssessorsAssessors

Thomas SchambergerExecutive Director

Foundation for Malcolm Baldrige National Quality Award, Inc.


2010Objectives:

– Identify the competencies and skills required as a assessor & senior assessor/team leader to include some advanced skills

– Senior Assessor Roles – Team Leader and Team Member

– Review Assessment Process

– Critical Success Factors

– Tools and Techniques for Conducting a Successful Site Visit

– Feedback Report Writing Skills and Score Analysis based on findings

– Conflict management & Crisis management

– How to build teamwork

– Identify the competencies needed as a mentor and coach

– What to Do and What Not to Do – Lessons Learned


2010

Introduction Activity—Questions


2010

Today and Tomorrow let’s agree to Establish a Mentoring Relationship

Therefore…….in order for it to be the most effective let’s agree to build on mutual trust and openness by both parties by asking questions and sharing information so that we can grow and learn


2010

Day 1 Key Outcomes and Learning

Assessor Skills and CompetenciesSenior Assessor Skills and CompetenciesSenior Assessor RolesAdvanced Assessor SkillsIndependent Review and Key FactorsProcess and Results Item - Comment Writing using ADLI and LeTCI


2010

– Oral/Written Communication – Ethical Behavior– Action Orientation– Peer Relationships– Time Management

(Individual)– Self-Development– Technical Learning

(Computer Skills)

General Assessor Skills are….


2010Assessment Knowledge & Skills

– Analytical -•• Use of statistical methodsUse of statistical methods•• Evaluating financial data and resultsEvaluating financial data and results•• Ability to analyze levels, trends, comparisons, Ability to analyze levels, trends, comparisons,

segmentation, breadth of results, and linkages segmentation, breadth of results, and linkages between Process and Results Itemsbetween Process and Results Items

– Probing Skills•• Isolate the real issues to uncover most appropriate Isolate the real issues to uncover most appropriate

findingsfindings•• Guide the team on the best questions to askGuide the team on the best questions to ask

and know how you can simplify themand know how you can simplify them– Use of technology tools and resources to

•• manage and communicate informationmanage and communicate information•• Interpersonal skillsInterpersonal skills


2010Assessor Competencies

1. Writing skills (i.e. ability to produce/give clear, accurate and useful feedback) and asking relevant questions

2. Ability to comprehend organizational context based on Criteria Framework

3. Field/site visit requirements4. Better understanding of Criteria Framework and

score consistency


2010

Assessor Competencies5. Ability to gather, assimilate and analyze evidence 6. Ability to produce and deliver clear, concise, and

accurate feedback7. Ability to understand organization and review the

organization in terms of structure, processes, performance and results

8. Ability to interact positively with other members of team, organizers, and representatives of the organization being assessed

9. Adherence to requirements of rules of conduct, ethics, computer practices and confidentiality considerations


2010

Assessor Competencies

10. Leadership Skills11. Better understanding of evolving contemporary

management philosophies/methodologies (i.e. revisions to criteria require understanding of new terminology and how it impacts an organization’s ability to respond)

12. Systematic process for conducting an assessment13. Methodology for sharing knowledge in conducting

assessments more effectively


2010

Team Leader Skills as an Assessor

1. Organizing Skills2. Planning Skills

– Managing the Process– Managing People

3. Time Management (Team)– Managing Scorebook Production

4. Team Building


2010

– General Assessor Skills– Assessment Process

Knowledge & Skills– Team Leader Skills

Senior Assessor Basic Competencies


2010

5 Considerations for a Senior Assessor5 Considerations for a Senior Assessor……....

The Organization must be able to Tell their

Story

The Organization must be able to Tell their

Story

The more engaged the organization is the more you can fully assess them

and strengthen confidence in them

that you are capturing their story.

The more engaged the organization is the more you can fully assess them

and strengthen confidence in them

that you are capturing their story.

Engage Organization

Use the strength of your team and make assignments. Don’t carry the full load.

Use the strength of your team and make assignments. Don’t carry the full load.

Strengths + Quantity =

Quality

Strengths + Quantity =

Quality

Delegate

Work to make the process Fluid vs. Rigid

Know where you’re going so that you know who, what, when, where, and

how you are going to be able to get there

Know where you’re going so that you know who, what, when, where, and

how you are going to be able to get there

Plan with the End in Mind

Plan with the End in Mind

Start at the End

Apply Murphy’s Law to the

Assessment

Apply Murphy’s Law to the

Assessment

Crisis Management

Plan for a crisis by exploring all options

before hand. Be prepared to be

prepared.

Plan for a crisis by exploring all options

before hand. Be prepared to be

prepared.

Keep the Team together at all times

and don’t allow individuals to work alone. One person can’t win it for you.

Keep the Team together at all times

and don’t allow individuals to work alone. One person can’t win it for you.

Individuals are not as

Strong as a Team

Individuals are not as

Strong as a Team

Team Strength


2010

? ? ?

Questions ??

? ??

?


2010

–

Team Leader

External Coach (if required or asked)

Senior Team Member

Internal Coach

Senior Assessor Roles


2010

Senior Assessor as Team Member

Review of Assessor WorkProvide Guidance to Team and Lead when requiredProvide Constructive FeedbackBack-up team leader


2010

Internal/External Coach’s responsibilities

Provide feedback to team member – planning documents– team assignments– correspondence– call and meeting agendasProvide support to Team LeaderMentor/coach team lead and team membersCommunicate frequently


2010

–– Empathy Empathy –– NonNon--JudgmentalJudgmental–– BelievableBelievable–– Reflective Instructor/TeacherReflective Instructor/Teacher–– Supporter of the learning processSupporter of the learning process–– Observational Feedback Observational Feedback –– Experiential (Shares Experience)Experiential (Shares Experience)–– Effective in different interpersonal Effective in different interpersonal

contexts; adjusting mentoring to contexts; adjusting mentoring to meet the needs of the menteemeet the needs of the mentee

Coaching Competencies


2010

“Based on some of the obstacles listed on the next slide as a leader/member

of a Business Excellence assessment team?”

Let’s Discuss –

“What can we do to prevent/ or minimize the impact in encountering these obstacles for a team? Four Groups

Group Discussion


2010

1. Unclear expectations – Assessor (s) ask repeated questions for understanding,

restates instructions incorrectly, or describes how it was done in the state program.

2. Lack of team member knowledge, skills, and ability – Assessor misuses terms,

submits a poorly written bio (e.g., grammar or organization is poor), or prefaces

comments with “I’m new”; Assessor is quieter and participates less than other team

members.

3. Technology breakdowns (e.g.. computer, Internet) – Assessor has repeated ISP or

server shutdown for maintenance; repeated computer malfunctions.

4. Lack of responsiveness – Team members are slow to respond to e-mail or phone

messages, miss initial deadlines, misplaces team notices, assessor gets behind in

responding to e-mails or voice mails, etc.

Early Warning SignalsEarly Warning Signals……..


2010

Senior Assessor Competencies

Former skills and competencies areessential …but those skills alone will not but those skills alone will not make you a competentmake you a competentSenior AssessorSenior Assessor


2010

Advanced Skill Set for a Senior Assessor….Someone who CARESCARES

–– ability toability to CCoach Effectively –– isis AAgile and/or flexible in

face of change–– hashas RReading Comprehension–– capablecapable ofof EEnergizing a Team–– reducesreduces SStress Management


2010

Listening Skills…– Must be able to carefully listen to the assessment

team and people within the organization being assessed each time they talk as this is the key to understanding where they are coming from.

– Maintain good eye contact - show genuine interest in hearing them out. • Enables you to uncover more information. The

more you obtain, the easier it is for you to help identify their strengths and areas for improvement.

Providing Effective feedback1. Scorebook Comments2. Consensus3. Site Visit

Coaching includes..


2010

Group Discussion

What would you say (or write) to checka team member’s rationale for acomment that you don’t agree with orthat you think could be written better?

Discuss in small group for five minutes –


2010

Example Coaching Responses….to maintain self-esteem of team member

“Help me understand ....”“I see that you did a good job (specifics) .... one additional thing to consider would be ....”“Your OFI comment addresses the specific approach, but do you think it would add value to add a ‘so what’? If so, what would that sound like?“You’ve been working hard on these comments. One comment that I need to check with you on is …. I’m looking for a phrase that ....”


2010

Senior Team Member

• Review• Guidance• Constructive Feedback


2010Agility Means… Leading MoreLeading More and Managing Less and Managing Less by by ……

Making sure the right things are being discussed, evaluated, prioritized and delivered. Making sure the big picture is always kept in sight. Understanding the importance of communication, conflict resolution and influencewhen it comes to getting things done. Changing direction whenthings change around them ... as they do constantly ... to make sure they are still heading for success.


2010

Planning Skills…– Ability to plan ahead– Create a blueprint for the assessment that

contains every piece of information that you need to gather and validate/verify and all the activities that the team needs to do to promote easy, fun learning. Doing this, you can make the whole thing organized.

Managing Less Requires..Requires..


2010

Team Leader Responsibilities

PlanningManaging the ProcessManaging ScorebookProduction

Managing People


2010

Planning

Tools1. Wave Timeline2. Proposed Timeline for Independent &

Consensus Review (visual & link)3. Timeline: Stakeholder Roles and

Responsibilities (visual & link)


2010Assessment Timeline - Example


2010

Reading and Comprehension1. “Cover to Cover”- not beginning to

end –(i.e. - develop a reading technique & order that works effectively)

2. Based on information ability to link process to results

3. Identify cross-cutting themes and Key Business Factor’s

4. Validate responses to Core Values and Concepts


2010

EEnergizing a Teamnergizing a Team1.1. Keep your wordKeep your word2.2. Operate with integrity and respectOperate with integrity and respect3.3. Make appropriate team assignmentsMake appropriate team assignments4.4. DonDon’’t mislead or misrepresent the t mislead or misrepresent the

Award processAward process5.5. Make the experienceMake the experience

Fun!!!!Fun!!!!6.6. Create a learningCreate a learning

environment environment


2010

Obstacles to Team SuccessObstacles to Team Success

Lack of team member knowledge, skills, and abilityTechnology breakdownsWeather-related emergenciesPersonal emergenciesInterpersonal conflictCompeting prioritiesChanges in requirementsTeam member addedUnclear expectationsTeam member drops


2010

How to Build an Effective TeamHow to Build an Effective Team

Stage 1 Stage 2 Stage 3 Stage 4

EmpoweringEmpowering

LeadingLeading

Team Team Team FormationBuildingBuilding

Team Formation CollaborationCollaboration SelfSelf--DirectionDirection


2010

Managing the Process

Organizing People and ActivitiesEfficient Work FlowMeasurementEffective Decision MakingCommunications


2010

Managing the PeopleResources

Shared goalsShared languageShared processShared toolsTeam diversity (sector, expertise, process

experience)Established roles and responsibilities


2010

1. Generate possibilities (Plan B and C)2. Make choices decisively3. Communicate clearly and regularly to maintain

focus4. Manage your time first5. Solve problems promptly6. Improve team relationships

don’t let conflict fester/grow

Stress reducers….


2010

? ? ?

Questions ??

? ??

?


2010

Group DiscussionGroup DiscussionUsing examples from your experience as Assessors, in terms of a team member not pulling their weight, what has worked well for you as a Team Leader, or what have you experienced as a team member, that has worked well to prevent conflict on the team arising from this kind of issue?”

Discuss in small group for five minutes –


2010

Steps Required in the Award Process- Independent Review Writing Process/Result Comments- Identifying Key Themes, and Key Factors


2010


2010

Independent Review and Key Factors


2010


2010

Scorebook Development Process


2010

Analysis

“To analyze” is toDivide a whole into its component parts.Identify or separate ingredients of a substance.State the constituents of a mixture.

Examine, Study, Investigate, Break Down


2010

Item Evaluation Process

Step 1 – Read the Criteria ItemStep 2 – Identify the 4-6 most important key

factors for the ItemStep 3 – Read the appropriate section of the

applicationStep 4 – Generate a list of summary

notations


2010

Worksheet GuidelinesDo– Include 4-6 key factors based on the Criteria

requirements for the Item. These will differ depending on the Item.

– Include the 6-10 comments per Item that are most relevant and important to the applicant based on its key factors.

– Ensure that the Item’s score is supported by the 6-10 comments.


2010

Process and Result Item

Comment Writing and Item Backup


2010

What comments should do:

Let the applicant know what it does well—and what it needs to improve—specific to

equitable evaluation1. Criteria requirements

2. The particular organization meaningful feedback

3. Evaluation factors (ADLI and LeTCI) insights onorganizational maturity


2010

Comment Guidelines


2010

Summary Notation EssentialsSubject from the Criteria or Application

+

Verb & Descriptive Language from the Criteria

+Transition: by/through /using

+

Example from application methods/ activities

+ Scoring Guidelines language+ so-what

+ grammar and punctuation check


2010

Content GuidelinesDo– Address central requirements of the Criteria.– Use a single, complete thought to specify the

strength (using specific examples from the application) or OFI (using specific omissions or concerns identified from the application) clearly.

– Write process comments so they contain a subject identified from the criteria, the application, or the scoring guidelines, verb(s) and requirements from the Criteria, examples from the application, and figure numbers, as appropriate.


2010

Content GuidelinesDo• Write results comments so they contain a

subject identified from the results or Criteria requirement being addressed, from/to timeframes, from/to performance levels, linkages to KFs, so whats, and figure numbers, as appropriate.

• State observations in a factual manner, e.g., “Customer satisfaction rates have increased from 75% in 2002 to 94% in 2004 and now exceed best-in-class levels.


2010

Content GuidelinesDo• Draw linkages across Items or

between an Item and the applicant’s Organizational Profile.

• Place the comment on the correct Item Worksheet based on the Criteria, not on where the information appears in the application.

• Ensure that the comment does not contradict other comments in the same or other Items or in the Key Themes Worksheet.


2010

Content GuidelinesDo not– Go beyond the requirements of the Criteria or

assert your personal opinions.– Be prescriptive by using “could,” “should,”

and “would.”– Be judgmental by using terms such as

“good,” “bad,” or “inadequate.”– Comment on the applicant’s style of writing or

data


2010

Style Guidelines

Do– Use a polite, professional, and positive tone.– Use active voice and present tense (e.g., ‘completes’

rather than ‘is completed’).– Use vocabulary/phraseology from the Criteria and the

Scoring Guidelines.– For Stage 1 and 2 scorebooks, tell what is missing if

something “is not clear.”– For Stage 3, clarify all “not clear” statements.


2010

Style GuidelinesDo

• Use such words as “the applicant” or “the organization” to refer to the applicant in the Stages 1 and 2 scorebooks.

• Use the applicant’s name in Stage 3 scorebooks.

• Use the applicant’s terminology when appropriate.

Do not• Use jargon or acronyms unless they

are used by the applicant.


2010

Comments - General

Item-Specific Key Factors Should Be!Hunting for OFI’s? Score High!Don’t Get Stuck in “Microview” (Question by Question)Use Criteria and Scoring Guidelines for Comments


2010

? ? ?

Questions ??

? ??

?


2010

Process


2010

Evaluating Process Items

ApproachDeploymentLearningIntegration

ADLI


2010

Using ADLI to Evaluate Process Items–Module Objectives

1. Use the approach, deployment, learning, and integration (ADLI) evaluation factors while assessing an applicant’s responses to Process Items.

2. Use the ADLI evaluation factors throughout the Independent Review phase of the Award process.


2010

A-D-L-I Evaluation Factors– Approach

• The methods used to accomplish the process

• The appropriateness of the methods to the Item requirements

• The effectiveness of your use of the methods

• The degree to which the approach is repeatable and based on reliable data and information (i.e., systematic)


2010

A-D-L-I Evaluation Factors

– Deployment• Your approach is applied in

addressing Item requirement relative and important to your organization

• Your approach is applied consistently• Your approach is used by all

appropriate work units


2010

A-D-L-I Evaluation Factors– Learning

• Refining your approach through cycles of evaluation and improvement

• Encouraging breakthrough change to your approach through innovation

• Sharing of refinements and innovation with other relevant work units and processes in your organization


2010

A-D-L-I Evaluation Factors– Integration

• Your approach is aligned with your organizational needs identified in other Criteria Item requirements

• Your measures, information and improvement systems are complementary across processes and work units

• Your plans, processes, results, analyses, learning, and actions are harmonized across processes and work units to support organization-wide goals


2010

Process Comments - GeneralUse single complete thought1-3 complete sentencesAddress central Criteria requirementRelevant and important to applicant based on the Key FactorsNon-prescriptiveNon-judgmentalDraws linkages to the Organizational Profile or Other ItemsDoes not conflict with other comments


2010

Process Comment General Guidelines

Comments should be– Criteria-based– Relevant– Clear– Concise– Actionable


2010

Process Comment General Guidelines

Comments should not– “Parrot” the application– Be prescriptive– Be judgmental– Conflict with one another


2010

Basic Format for Process Item Comments OFI

Intro. Statement Subject Verb and Implied Descriptive or explicit

It is not clear from the language "so what"how/apparent Criteria,

application, from the related to The applicant does or scoring Criteria KFs ornot report/address guidelines Criteriahow

Adhering to content and style guidelines

+ + +


2010

Process Comment Examplel Applicant determines the student segments and markets that its education programs

will address through an effective, systematic, 4-step Segmentation Process. That

process, managed by the Research and Knowledge Management Department,

involves: 1) gathering, analyzing, and integrating information and data using a variety

of listening/learning methods (Fig. 3.1-3); 2) validating existing segments and

identifying emerging segments and market opportunities; 3) communication final

segmentation throughout the district; and 4) organizing information, data and

knowledge by segment. The segmented information is used to meet organization,

information, and NCLB requirements and is integrated with the district’s

organizational needs identified with other Criteria categories, e.g., strategic

planning (Figure 2.1-1), relationship management, and curriculum design (Figure 6.1-

1).


2010

Basic Format Example: Strength

Senior leaders use a variety of methods (Figure 1.1-2) to communicate values, directions, and expectations to faculty, staff, partners, and stakeholders. These methods, as well as a five-step communication process, were developed through a PDSA cycle by the district Communication Team. In addition, as a result of Baldrige self-assessments and benchmarking best-in-class communication methods, the team has implemented multiple improvements, including adding two-way communication methods, developing a Communication Methods Manual, integrating “communication effectiveness”into the PMP, and, most recently, integrating communication liaisons within the SICs.


2010

Basic Format Example: OFI

o While the applicant has a Culture Change Process to nurture an environment conducive to student and district excellence, it is not clear how this process creates an environment that fosters and requires legal and ethical behavior. This may be particularly important to the district given the highly regulated environment described in its Organization Profile.


2010

Process Comment

To carry out its strategic planning, the organization uses a systematic, biennial 12-step Strategic Planning Process (Figure 2.1-1) that involves community leaders, volunteers, member agencies, and donors. The organization determines its strategic challenges in the “Current State” step, and it includes inputs from the SWOT Analysis and Environmental Scan. The SPP was initiated in 1997 and restructured in 2004 as a result of benchmarking analyses of food banks and other nonprofit organizations.


2010

Comment Diagnosis ActivityStep Time

1. As a triad, review the comment assigned and identify the criteria elements contained within the comment and determine whether there are any problems with the comment. Ask

a. What is wrong?b. What is the evidence?

Hint: Use the Comment Guidelines and the Criteria as tools.

5 min.

2. Review some of your triad’s findings. 10 min.


2010

Find the Elements of a Comment

XYZ uses the Comprehensive Complaint Management System (CCMS) (Figure 3.2-3) to

log and track the resolution of customer complaints received through personal

conversations, phone calls, e-mails, or survey responses. Data entered into the system

are analyzed for root causes to determine a solution and are communicated to the

appropriate departments, with additional training provided as necessary. These data are

reviewed monthly at corporate performance meetings and again during year-end strategic

planning. As the result of these reviews, XYZ improved the process in 2007 with the

addition of the Handle It Now process to allow all frontline customer service staff members

to resolve complaints on the spot. This process has been implemented in all 25 XYZ

branches.


2010

Find the Elements of a CommentXYZ uses the Comprehensive Complaint Management System (CCMS) (Figure 3.2-3)

[Approach] to log and track the resolution of customer complaints [Criteria] received through personal conversations, phone calls, e-mails, or survey responses.

Data entered into the system are analyzed for root causes to determine a solution and

are communicated to the appropriate departments, with additional training provided as

necessary. These data are reviewed monthly at corporate performance meetings and

again during year-end strategic planning. As the result of these reviews, XYZ improved

the process in 2007 with the addition of the Handle It Now process [Learning] to

allow all frontline customer service staff members to resolve complaints on the spot.

This process has been implemented in all 25 XYZ branches. [Deployment]


2010

Completing the Item Worksheet

Identify each comment by its Area to AddressIdentify the key factors that link to the commentRate comments with a +, ++, -, or –Review the comments as a final check


2010

? ? ?

Questions ??

? ??

?


2010

Results


2010

Results Evaluation EssentialsFactors to consider as you evaluate results:– Gaps – missing information– Performance levels– Segmentation– Trends– Comparisons– Breadth of performance improvements– Linkage of results measures to important

organizational performance results


2010

Evaluating Results Items

LevelsTrendsComparisonsIntegration

LeTCI


2010

Results: Performance Analysis – Performance Levels (Le)

• Refer to numerical information that places or positions an organization’s results and performance on a meaningful measurement scale. Performance levels permit evaluation relative to past performance, projections, goals, and appropriate comparisons

• Are key results missing?• What levels are provided?• Is the measurement scale meaningful?


2010

Results: Performance AnalysisTrends (T)– Refer to numerical information that shows the direction and

rate (slope of trend data) and breadth (how widely deployed and shared) of performance improvements. A Minimum of three data points generally is needed to begin to ascertain a trend. More data points are needed to define a statistically valid trend.

– Are trends provided?

– Is the interval between measures or frequency appropriate?

– Are the trends positive, negative, or flat?

– What is the rate of change (slope of the trend)?


2010

Results: Performance AnalysisComparisons– Refer to establishing the value of results by their relationship

to similar or equivalent measures. Comparisons can be made to results of competitors, industry averages, or best-in-class organizations. The maturity of the organization should help determine what comparisons are most relevant.

– Are comparisons provided?

– Are the comparisons to an industry sector average, key competitors, or best-in-class organizations?

– How does the applicant compare against other organizations?


2010

Results: Performance AnalysisLinkage (Li)– Refers to a connection to important customer, product

and service, market, process, and action plan performance requirements identified in the Organizational Profile and in Process Items

– To what extent do results link to key factors and Process Items (e.g., important customers/patients/ students, product and service, market, process, and action plan performance requirements)?

– Are results segmented to help the applicant improve (e.g., by customer, patient, or student segment; employee type; process/education program or service)?


2010

Using LeTCI to Evaluate Results Items—Module Objectives

1. Identify the levels, trends, comparisons, and integration (LeTCI) evaluation factors for Results Items.

2. Use the LeTCI evaluation factors to effectively assess an applicant’s responses to Results Item requirements.


2010

Comment Diagnosis ActivityStep Time

1. As a triad, review the comment assigned and identify the criteria elements contained within the comment and determine whether there are any problems with the comment. Ask

a. What is wrong?b. What is the evidence?

Hint: Use the Comment Guidelines and the Criteria as tools.

5 min.

2. Review some of your triad’s findings. 10 min.


2010

Results Comment: A Sample

While Figure 7.3-1 (Average Tuition and Fees) indicates that NCON tuition costs have risen from an average of $4,000 in 2005 to $5,800 in 2008, the level of increase has remained below that of peers 1 and 2 since 2005. This aligns with the fiscal Health point of LIGHT and the principal success factor of moderate tuition rates.


2010


ABC provides no comparative data for workforce results relative to workforce engagement and satisfaction. Without appropriate comparative data, ABC may have difficulty addressing its strategic challenges of a shortage of qualified licensed professionals andthe influx of private urgent-care centers.


2010


ABC provides no comparative data for workforce results relative to workforce engagement and satisfaction [Criteria]. Without appropriate comparative data [comparisons], ABC may have difficulty addressing its strategic challenges of a shortage of qualified licensed professionals and the influx of private urgent-care centers [applicant-specific].


2010

Activity: Comment Writing Practice

1 As a table, select the core idea that will be the subject of the comment.

1 min.

2 Individually write the comment. 14 min.

3 In a round-robin, read your comments to your tablemates and identify the elements you used.

12 min.

4 Select a comment to scribe on chart paper and share. 3 min.

30 minutes


2010

? ? ?

Questions ??

? ??

?


2010

Scoring: Process Item


2010

Scoring

Key Scoring Terms– Systematic approach– Deploy/deployment– Evaluation and improvement– Align/alignment– Learning– Integration


2010

Steps in the Scoring Process1. Review Criteria requirements and summary notations2. Note balance and importance of strengths and OFIs3. Read the description of the scoring ranges in the

Scoring guidelines4. Determine the scoring range most descriptive of the

organization’s achievement level5. Read range descriptions above and below to

determine where, within the range, the score would fall6. Verify that the balance and substance of strengths and

OFIs correspond to the score7. Record the score


2010

Issues Leading to Greatest Scoring Variability

Scores not adequately related to the key factors or Scoring GuidelinesAcceptance of applicant statementsFeed of scoring above 50%Using the Areas to Address and Item Notes as a checklistTreatment of missing information


2010

“Best Fit” Considerations

Not all comments will fit into one scoring bandSome comments are more important – remember Key FactorsWhat is the balance of A-D-L-I comments?Being on the border between 2 scoring bands is typical – and okaySelect a score (5% point increments) that most suitably reflects the applicant’s performance to the criteria for that item ONLY


2010

More “Best Fit” Considerations

Don’t AverageA good approach can get you to 50% but lack of deployment could stop you from moving any furtherApproach has more influence in the lower bands and deployment has more influence in the mid-bandsLearning and Integration have more influence in the upper bands


2010

Rating CommentsStrengths+ Baseline observation++ Particularly important observation that has

significant impact on applicant’s performance management system, potential benchmark, or key theme significance.

Opportunities for Improvement

Baseline observation

Particularly important observation that has significant impact on applicant’s performance management system or key theme significance


2010Baldrige National Quality Program

Comment-Writing Activity

Aligning Scoring Ranges with Comment Language


2010

Itches and Scratches Performance Projections

“If you don’t know where you are going, any road will take you there.”

Lewis Carroll


2010

Itches and ScratchesPerformance Projections

l Now Mentioned in Every Scoring Range

l Not a Significant Factor until 50–65% Range

l “Clear” Rationale in Top Two Ranges


2010

Itches and Scratches Performance Projections

Are They Present?

Do They Make Sense?

Are They Related to Strategic Objectives, Competitive Performance, and Customer and Stakeholder Needs?

We Are Not Examining “Absolute”Goodness.

We Are Examining Relevance.


2010

Itches and Scratches Evaluating Financial Performance

l Most Trends and Actual Performance May Be Negative This Year.

l This Alone Does Not Result in a Low Results Item Score.

l Review Results in Context of Overall Economy and Competitors’ Performance.—What Is the Organization’s Comparative

Performance?—How Is It Adjusting?


2010

? ? ?

Questions ??

? ??

?


2010

What have we learned?

Day 1 Recap and What’s Next


2010

Day 1 Key Outcomes and Learning

l Assessor Skills and Competenciesl Senior Assessor Skills and Competenciesl Senior Assessor Rolesl Advanced Assessor Skillsl Independent Review and Key Factorsl Key Themes: First Lookl Comment Writing using ADLI and LeTCI


2010

ReflectionTake a few moments to reflect on your personal learning from today’s session, and make a few notes on your next steps.

1. What have you learned regarding your strengths?

2. What did you discover that may be opportunities for your development?

3. How will you address your opportunities for development?

2010 tqa assessor workshop by thomas schamberger day one

Leadership & Management

Transcript of 2010 tqa assessor workshop by thomas schamberger day one