Rebecca L. Renshaw, PhD, MSL, COMS

AER International Conference July 2010

Agenda Background Statement of the Problem Relevance for the Study Overview of the Study

Creation of the O&M CCEMMethodology and Results

Limitations Conclusions Discussion

Background Since the 1940s, the competency requirements

for O&M specialists have evolved:Initial certification standards required only graduation

from a university program, membership in a professional organization, and letters of recommendation in order to become certified.

Later standards focused on the visual abilities of O&M specialists to monitor clients from various distances.

Finally, in 1990s, standards addressed all the essential job responsibilities of O&M specialists.

This evolution was based on a handful of studies

that were done over the years.

Statement of the Problem Prior research has focused primarily on identifying

academic and clinical competencies and determining how essential those competencies are in the training of O&M specialists. (Eichorn & Vigoroso, 1975; Crouse & Kappan, 1975; Kimbrough, 1980; Uslan, Hill, & Peck, 1989; Weiner & Siffermann, 2000)

Very little research focused on methods for evaluating clinical competencies. Those studies that have focused on evaluation, only examined one aspect, observation skills. (Geruschat & De L’Aune, 1989; Zebehazy, Zimmerman, & Fox, 2005; Zebehazy, Renshaw, Zimmerman, Fox, & D’Andrea; 2008; Renshaw, Zimmerman, & Zebehazy, 2009)

Statement of the Problem No research has investigated a valid and reliable

means to assessing all of the clinical competencies of O&M specialists and the various performance levels within them.

The current method for evaluating has several flaws. Never examined for reliability and validity evidenceSubjective evaluationCompetency statements are vagueSpecific competency skills not identifiedPerformance levels not provided

Relevance of the Study Important to have a standard

Safety of clients at riskCompetencies should be consistently measured

across internship placements and O&M university programs and between supervisors.

A new tool that provides valid and reliable scores could be used by:ACVREP for national certification purposesO&M university programs for curriculum decisionsClinical internship supervisors for evaluating

competencies throughout the internship

Overview of the Study Creation of the O&M Clinical Competency Evaluation

Matrix (CCEM) Competency Skills Rating Scales Performance Levels

Methodology Surveyed O&M university faculty Revisions made the to O&M CCEM Surveyed O&M clinical internship supervisors Analyzed validity and reliability evidence

Creation of the O&M CCEM Competency Domains: ACVREP clinical competency

domains were used as the framework Communication O&M Assessment Instructional Planning Instruction Monitoring and Safety Facilitating Independence Professionalism

Competency Skills Determination of specific competency skills within

each domain was based on: competency statements by Kimbrough (1980)

direct and in-direct teaching competencies by Uslan, Hill, & Peck (1989)

job task responsibilities identified by Weiner & Sifferman (2000)

domain statements on ACVREP clinical competency form (ACVREP, 2001)

Competency skills under the monitoring and safety domain were based on the studies pertaining to observation skills and instructor positioning. (Zebehazy, Zimmerman, & Fox, 2005; Zebehazy, Renshaw, Zimmerman, Fox, & D’Andrea; 2008; Renshaw, Zimmerman, & Zebehazy, 2009)

Competency Skill ExerciseDomain 1: Communication and Professional Relationships -

“Candidate is able to establish and maintain effective communication and professional relationships with students, families, colleagues, and supervisors, including individuals from culturally and linguistically diverse backgrounds.” (ACVREP, 2001)

“Orally communicate with students at their level of verbal communication” (Kimbrough, 1980)

“Communicate information about students’ mobility that can be understood by other mobility specialists, students’ families, and other professionals” (Kimbrough, 1980)

“Communicate a learner’s O&M program, including goals and objectives, to significant others (Uslan, Hill, & Peck, 1989)

“Assisting family members and significant others in understanding the impact of the student’s disability or disabilities” (Uslan, Hill, & Peck, 1989)

“Communication activities” (Uslan, Hill, & Peck, 1989) “Establishing rapport with clients” (Weiner & Siffermann, 2000)

Domain 1 Competency Skills Based on this exercise, there appeared to be 5

competency skills under the Communication and Professional Relationships Domain: Establishing Rapport Communicating with Clients Communicating with Families Communicating with Colleagues Communicating with Supervisors

List of Competency SkillsDomain 1: Communication

• Establishing Rapport• Communication with Clients• Communication with Families• Communicating with Supervisors• Communicating with Colleagues

Domain 2: O&M Assessment

• Planning O&M Assessments• Conducting O&M Assessments• Synthesizing Findings in a Report• Communicating Results

Domain 3: Instructional Planning

• Reviewing and Interpreting Relevant Records• Previewing and Selecting Potential Training Areas• Considering Instructional Materials and Appropriate Devices• Knowing Options for Mobility Systems• Choosing Appropriate Goals• Writing Appropriate Behavioral Objectives• Developing Individualized Lessons• Sequencing of Lesson

List of Competency SkillsDomain 4: Instruction

• Introducing Lessons• Teaching Use of Low Vision/Other Senses• Teaching Human Guide Techniques• Teaching Indoor Mobility Techniques• Teaching Outdoor Mobility Techniques• Teaching Indoor Orientation Skills• Teaching Outdoor Orientation Skills• Managing Lessons• Using Instructional Strategies• Gauging Acquisition of Skills• Providing Feedback to Clients• Reflecting on Teaching

Domain 5: Monitoring and Safety

• Monitoring O&M Skills• Recognizing Potentially Dangerous Situations• Intervening

Domain 6: Facilitating Independence

• Fostering High Expectations• Facilitating Self-Advocacy Skills• Facilitating Problem Solving Skills• Facilitating Decision Making Skills

Domain 7: Professionalism

• Maintaining Professional Conduct• Utilizing Resources• Scheduling• Record Keeping and Reporting

Rating Scales Rating scales are recommended when evaluating a

behavior that exists on a continuum. (Borman, 1986; Jacobs, 1986; Manatt & Peterson, 1988; McGreal, 1990)

Behaviorally anchored rating scales are recommended if any rating scale is to be used for this purpose. (Aiken, 1996)

This type of rating scale defines performance in terms of explicit behaviors allowing for more control over errors in rating and less subjectivity in judgment. (Aiken, 1996)

The behavior summary scale (BSS) is one type of rating scale format that anchors performance with general benchmarks.

Performance LevelsPerformance

LevelsSkill 2: Communicating with

ClientsSkill 5: Communicating

with Supervisors

Unsatisfactory Intern’s interactions with clients are negative, demeaning, or inappropriate based on the ages, abilities, or cultural backgrounds of the clients.

Intern responds poorly to constructive suggestions and demonstrates no respect for authority.

Basic Intern’s interactions with clients are generally friendly and personable but may reflect inconsistencies, favoritism, or disregard for clients’ ages, abilities, or cultural backgrounds

Intern accepts constructive suggestions and is courteous and respectful of the supervisor’s opinions and advice.

Proficient Intern’s interactions are friendly and demonstrate general warmth, caring, and respect. Such interactions are appropriate to the clients’ ages, abilities and cultural backgrounds.

Intern views constructive suggestions as a learning opportunity and seeks to have in-depth conversations about those suggestions in order to learn more.

Distinguished Intern always demonstrates genuine caring and respect for all clients while maintaining appropriate interactions based on client’s ages, abilities, and cultural backgrounds.

Intern fully respects the constructive suggestions from the supervisor and incorporates those suggestions into lessons.

Methodology Phase 1:

Surveyed O&M university faculty Analyzed content validity evidence Made revisions to O&M CCEM

Phase 2: Surveyed O&M clinical internship supervisors Analyzed validity and reliability evidence

○ Content validity evidence○ Internal structure validity evidence○ External structure validity evidence○ Internal consistency reliability evidence○ Inter-rater reliability○ Practicality validity evidence

Results: Content Evidence Research Question:

Are the competency skills on the O&M CCEM representative of the content area it is designed to measure?

Are any competencies skills missing?

Answer: Yes. When examining the level of agreement across ALL competency

skills:○ O&M faculty had an 83.33% level of agreement ○ Clinical internship supervisors had a 92.8% level of agreement○ This implies that the O&M CCEM as a whole is representative of the

content area Concerning any missing competencies:

○ O&M faculty reported some missing skills○ Revision were made based on those comments○ Clinical internship supervisors made additional recommendations

Results: Internal Structure Evidence

Research Question: Are competency skills within the seven clinical competency

domains highly related to other competency skills within the same domain?

Answer: Yes, but…. Competency skills are not only related to competency skills

within the same domain but competency skills in OTHER domains as well.

O&M CCEM actually measures three different aspects of instruction: ○ Standard teaching skills○ O&M specific skills○ Advanced O&M instruction skills

Results: External Structure Evidence

Research Question: Is there a relationship between scores on the O&M CCEM and

the ACVREP clinical competency evaluation form?

Answer: Yes. ACVREP scores were calculated by summing the number to

“met” responses. O&M CCEM mean scores for the 3 dimensions were calculated. Spearman correlation coefficients revealed the relationship

between these scores. There is a positive, moderate relationship between the scores

on the ACVREP form and O&M CCEM.

Results: Internal Consistency Reliability

Research Question: Is there consistency in the rating for each item and total test

score? Do all items measure various aspects of clinical competency?

Answer: Yes. Consistency in ratings within each domain. Consistency in ratings within 3 dimensions. Implies the competency skills measured some aspect of clinical

competency consistently.

Results: Inter-rater Reliability

Research Question: Is there consistency in the rating between raters?

Answer: Yes and No Six pairs of supervisors Percentages of agreement obtained

○ Across ALL competency skills○ Within Domains

There was not consistency when 4 levels of performance were analyzed but there was consistency when levels 3 and 4 were collapsed.

Possible explanations include:○ Subjectivity of the performance level wording. ○ Role of the supervisors rating the intern.○ Lack of a standard perception in the field of what competence looks

like. ○ Lack of a training tool that provided examples of competence.

Results: Practicality Evidence

Research Question: Is the O&M CCEM a practical tool to use to evaluate the clinical

competencies of O&M interns?

Answer: Yes. Four-performance level format

○ All O&M faculty indicated the format was appropriate for assessing the clinical competency of O&M interns.

○ 15 of the 16 clinical internship supervisors indicated that the format was appropriate.

Usefulness○ 11 of the 12 O&M faculty indicated they would use the O&M CCEM.○ 13 of the 16 clinical supervisors indicated they would use the O&M

CCEM.

Limitations to the Study Small sample size Random sources of error:

administration method respondent effects instrument effects

Systematic sources of error: Competency skill rating could have been inflated

Subjectivity of the rating scale

Conclusions Validity Evidence

The combination of all the validity evidence supported the intended inferences, that scores on the O&M CCEM can be interpreted as measuring clinical competence.○ Content ○ Internal Structure○ External Structure○ Practicality

Reliability Evidence There was consistency of scores within domains and some

consistency between raters ○ Internal consistency○ Inter-rater

Dimension 1: Standard Teaching SkillsCommunication:• Establishing Rapport• Orally Communication with Clients• Orally Communication with Families• Orally Communicating with Supervisors• Orally Communicating with Colleagues Assessments: • Planning O&M Assessments• Conducting O&M Assessments• Synthesizing Findings in a Written Report Planning for Instruction: • Knowing Options for Mobility Systems • Reviewing and Interpreting Relevant Records• Choosing Appropriate Goals • Writing Appropriate Behavioral Objectives• Developing Individualized Lessons • Considering Instructional Materials and Appropriate Devices• Sequencing of Lessons

Instructional Strategies: • Using Instructional Strategies• Communicating During Lessons• Managing Lessons • Modifying Lessons• Providing Feedback• Gauging Acquisition of Skills• Reflecting on TeachingProfessionalism:• Maintaining Professional Conduct• Utilizing Resources• Scheduling• Record Keeping and Reporting Procedures

List of Competency Skills

Dimension 2: O&M Specific Skills

Dimension 3: Advanced O&M Instruction

Instruction in Techniques:• Teaching Use of Low Vision• Teaching Use of Other Senses• Human Guide and Protective Techniques• Orientation Skills• Cane Techniques• Complex Environments• Street Crossings• Public TransportationFacilitating Independence:• Promoting Participation• Fostering Self-Assessment Skills• Fostering Self-Advocacy Skills• Facilitating Problem Solving Skills• Facilitating Decision Making Skills

Monitoring and Safety:• Monitoring O&M Skills• Positioning During Lessons• Recognizing Potentially Dangerous Situations• Intervening

List of Competency Skills

Discussion Impact of the study on the field of O&M

ACVREP Universities Internship placements

Future Research Development of a protocol for the O&M CCEM Tool to measure competencies of practicing O&M specialists