University of Calgary

PRISM: University of Calgary's Digital Repository

Kinesiology Kinesiology Research & Publications

2007

Development of a content-valid standardized

orthopedic assessment tool (SOAT)

Lafave, Mark; Katz, Larry; Butterwick, Dale

Springer Verlag

Advances in Health Sciences Education, (2007) online

http://hdl.handle.net/1880/44402

journal article

Downloaded from PRISM: https://prism.ucalgary.ca

Adv Health Sci Educ Theory PractDOI 10.1007/sl 0459-006-9050-2

PAPER

Development of a content-valid standardized orthopedicassessment tool (SOAT)

Mark Lafave • Larry Katz • Dale Butterwick

Received: 31 July 20067 Accepted: 19 October 2006© Springer Science+Business Media B.V. 2006

Abstract Introduction Content validation of an instrument that measures stu-dent performance in OSCE-type practical examinations is a critical step in a tool'soverall validity and reliability [Hopkins (1998), Educational and PsychologicalMeasurement and Evaluation (8th ed.). Toronto: Allyn & Bacon]. Purpose Thepurpose of the paper is to outline the process employed to create a content-validStandardized Orthopedic Assessment Tool (SOAT). Orthopedic assessment skillsare employed by athletic therapists, physiotherapists and physicians. All followvery similar diagnostic algorithms and that system was used to develop the originalSOAT [Cyriax (1982). Textbook of Orthopaedic Medicine, (vol. 1) Bailliere Tin-dall]. Methods To validate the tool, the study followed procedures outlined byViolate, Salami, and Muiznieks (2002), Journal of Manipulative Physiological Ther-apeutics, 25, 111-115, and Butterwick, Paskevich, Vallevand and Lafave (2006),Journal of Allied Health: a modified Ebel procedure. An expert group of athletictherapists from across Canada were chosen to create the content valid tool. Repre-sentation from all accredited athletic therapy programs in Canada was sought. Ex-perts participated in three stages of content validation: Stage one consisted ofindividuals grading tasks on difficulty (hard, medium, easy) and importance (essen-tial, important, not important) for 8 body regions (cervical spine, lumbar spine,shoulder, elbow, wrist/hand/thumb, hip, knee and lower leg/foot/ankle) and threediagnoses for each body region (24 total). If 80% consensus was not achieved in thefirst stage, then in stage two, face to face discussion is meant to clarify positions and

M. Lafave (El)Department of Physical Education and Recreation Studies,Mount Royal College, 4825 Mount Royal Gate SW,Calgary, AB, Canada T3E 6K6e-mail: mlafave@mtroyal.ca

L. Katz • D. ButterwickFaculty of Kinesiology,University of Calgary, 2500 University Dr. NW,Calgary, AB, Canada T2N 1N4

«Q Springer

M. Lafave et al.

achieved consensus, if possible. Results The history component, the observationcomponent, scanning exams, clearing joints above and below the lesion site and activerange of motion, passive range of motion and isometric resisted testing all yielded80% consensus in the first two stages of development. A third stage was added to thisprocess because a new model of measurement was conceptualized during the secondstage due to a lack of consensus on two orthopedic assessment categories: specialtesting and palpation. The new model consisted of a "clinical reasoning" metric thattied each section of an orthopedic assessment (history; observation; scanning andclearing; examination, special testing; palpation; conclusion) together and wouldpermit the expert rater to evaluate the student performance based on the student'srationale for tests chosen rather than the traditionally rigid checklists. At least 80%consensus was achieved with the new "clinical reasoning" metric and the originallycontentious special testing and palpation categories. Conclusions The new SOATthat underwent content validation may be a hybrid between the traditional OSCE-type checklists and global scales that provide a more realistic and thus more validdepiction of orthopedic assessment clinical competence. Now that content validationhas been established, the next steps include inter-rater reliability testing.

Keywords Validity • Validation • Content Validation • Reliability •Orthopedic assessment • Orthopedic evaluation • Clinical competence •Assessing clinical competence • Ebel Procedure • Validation committee •Athletic therapy • Physiotherapy • Athletic therapist • Physiotherapist •Orthopedic • Assessment • Evaluation • Checklists • Global rating scales •Objective structured clinical examination • OSCE • Performance basedexaminations • Rating scales

Introduction

Clinical orthopedic assessment skills are employed by physicians (Cyriax, 1982;Hoppenfeld, 1976;), physiotherapists (Magee, 2002) and athletic therapists (Shultz,Houglum, & Perrin, 2005). The orthopedic assessment methods are based on themedical model of clinical diagnosis including a history and physical examination(Cyriax, 1982). There are slight differences in the order of the assessment categories(Shultz et al., 2005) and the level of detail within each of those categories, butconceptually, the goals are the same: to employ "a method of clinical examinationwhich leads to accurate diagnosis in locomotor disorders" (Cyriax, 1982). As Cyriax(1982) points out, there are many musculoskeletal conditions that do not requiresurgery. As a result, athletic therapists and physiotherapists work collaborativelywith physicians in the patient's diagnosis and conservative treatment plan. Clinicalcompetence in musculoskeletal assessment skills is paramount to physiotherapy andathletic therapy curricula as are measurement and evaluation of those same clinicalskills.

Assessing clinical competence is complex and even though it is highly unlikelythat any one measurement instrument will capture the full spectrum that would

4JJ Springer

Development of a content-valid standardized orthopedic assessment tool (SOAT)

define it (Neufeld & Norman, 1985), we continue the pursuit. Some common mea-sures of clinical competence include direct observations, oral examinations, writtenexaminations, global rating scales, medical record reviews, patient managementproblems, computer simulations and simulated patients. Written examinations lackthe validity to measure the construct of clinical competence and other high fidelitymeasures such as global rating scales, direct observation or oral exams lack ofreliability (Harden, Stevenson, Downie, & Wilson, 1975). Validity and reliabilitymust work together because they are at the heart of sound measurement principles(Hopkins, 1998). Hopkins (1998) identified content validity as the most importantstep to create a valid and reliable assessment tool. Based on the above consider-ations, the following study is presented as an important first step in the overallvalidation of a Standardized Orthopedic Assessment Tool (SOAT). Further, theultimate goal is for the tool to measure a clinical orthopedic competence construct.

The purpose of this study is to create a content-valid tool to in order to evaluateathletic therapy student performances in practical, Objective Structured ClinicalExamination (OSCE)-type examinations. Results from this study should facilitatethe creation of an instrument to measure clinical competence with musculoskeletaland orthopedic assessment skills.

Methods

Gorter et al. (2000) advocate for full and complete descriptions of the methods usedto validate performance based assessment tools. Consequently, extensive descrip-tions of the methods employed are provided here so that others can use/modify theprotocols for future analysis of orthopedic assessment skills. A modified Ebel pro-cedure was used to create the SOAT. A detailed description of the methodsemployed in the current study are outlined in Butterwick, Paskevich, Vallevand, andLafave (2006) and in Violate, Marini, and Lee (2003), Violate, Salami, andMuiznieks (2002). This method has been shown to be superior to other methods,particularly for performance based, practical examinations (Violate et al., 2003).

Creation of an instrument to measure student performance in practical OSCE-type examinations at the Mount Royal College-University of Calgary athletictherapy program began in 1999. The examination consisted of 7 stations, one ofwhich was a 30 min orthopedic assessment station. Practical internship clinicalsupervisors met once a month for 5 years and provided feedback on studentperformance in the clinic. This was an important step in the assessment tool's genesisand development. The tool consisted of 10 major categories commonly used inorthopedic assessments (history; observation; scanning examination; clearing jointsabove and below the lesion site; active range of motion; passive range of motion;isometric resisted testing; special testing; palpation; conclusion) with approximately200-250 tasks (depends on the body region) comprising the details under thosemajor categories. Each major category (i.e.; history, observation, etc..) measuresdifferent skills (i.e.; communication skills, clinical reasoning ability, observationalskills, psychomotor & technical skills) which collectively contribute to the ortho-pedic assessment clinical competence. Students are typically provided 30 min tocomplete such an examination.

In 2005, a 12 person expert panel of athletic therapists from across Canada wasstruck to participate in the content validation process. An email advertisement was

«£) Springer

M. Lafave et al.

sent to all Certified Athletic Therapists in Canada (approximately 1000). Selection ofthe twelve experts was based on the following basic criteria: (1) Must be a CertifiedAthletic Therapist for at least 5 years; (2) Expertise in teaching and/or examiningorthopedic assessment skills. Additionally, to ensure representation from all(Canadian Athletic Therapists' Association) CATA Accredited Programs, theprimary investigator (PI) sorted applications accordingly. There was no represen-tation from two schools (University of Manitoba and York) from the initial pool ofapplicants. Subsequently, the PI successfully sought experts from those institutionsby contacting the program director or the CATA certification committee.

A unique SOAT representing the tasks necessary to complete a clinical ortho-pedic assessment for 8 body regions (cervical spine, lumbar spine, shoulder, elbow,wrist/hand/thumb, hip, knee and lower leg/foot/ankle) were the foci of this study.Three diagnoses (scenarios) for each body region (8) were judged by the expertgroup (Table I). Experts were asked to grade each task for each body part andscenario on importance (essential, important, not important, not applicable) anddifficulty (hard, medium, easy, not applicable). Data collection and tabulation wasaccomplished through an on-line survey tool housed on a Mount Royal Collegeserver. Experts were given 6 weeks to complete the on-line surveys.

A modified Ebel procedure was employed to create a content valid instrument/tool(Butterwick et al., 2006; Violate et al., 2002). Traditional use of the Ebel procedurehas been employed to determine the minimum passing level in professional practicalexams (Cantor, 1989; Violate et al., 2003). Normally, this process consisted of twostages: Stage 1. Experts graded each SOAT based on the diagnosis provided using the

Table 1 Body regions and diagnoses

Body region Diagnosis

Cervical region

Shoulder region

Elbow region

Wrist & hand region

Lumbar region

Hip region

Knee region

Ankle region

C 5-6 Facet SprainStrained SternocleidomastoidDegenerated Disc with Radicular PainSubacromial BursitisSuperior Labral Tear/LesionInfraspinatus TendinitisOlecranon BursitisTennis ElbowUlnar Collateral Ligament SprainScaphoid FractureExtensor Pollicis TenosynovitisUlnar Collateral Ligament SprainLumbar 3-4 Facet SprainStrained Abdominal MuscleSpondylolis thesisStrained HamstringStrained IliopsoasAnterior Capsule StrainStrained Rectus FemorisAnterior Cruciate Ligament and Medial Collateral Ligament SprainPatellar TendinitisAnterior Talofibular Ligament SprainAchilles TendinitisDeltoid Ligament Strain

<gj Springer

Development of a content-valid standardized orthopedic assessment tool (SOAT)

Difficulty

0)uc€oaE

Easy &Essential

1

Easy &Important

4

Easy & Not asImportant

7

Medium &Essential

2

Medium &Important

5

Medium & Notas Important

8

Hard&Essential

3

Hard&Important

6

Hard & Not asImportant

9

Fig. 1 Ebel Grid

difficulty (hard, medium, easy, not applicable) and importance (essential, important,not important, not applicable) scale; Stage 2. Face to face discussion attempted tobuild consensus on items that did not achieve 80% consensus for each task/item onthe SOAT for each region on the Ebel grid (Fig. 1).

Butterwick et al. (2006) established the benchmark of 80% consensus to identifythe tasks or items that should be included in the final instrument. Any task listed onthe tool that did not achieve 80% consensus was considered for elimination from thetool. The current study employed the same benchmark (Butterwick et al., 2006) forboth the initial grading of importance and difficulty and the subsequent face to faceconsensus building process.

During the second stage of the validation process, twelve experts from acrossCanada were flown to Calgary, Alberta, Canada for a full day (10 h) of face to facediscussion. The primary focus of the second stage was to discuss and debate tasksthat had not achieved the initial 80% consensus target in the first stage. Discussionwas facilitated by the PI while each major category (patient history, observation,etc..) and each task therein was debated for inclusion and assignment to one of thethree importance scale categories (essential, important, not as important).

Results

In the first stage of the validation process, there was limited consensus for items andcategories that achieved 80% or higher in grid square number 1 (easy and essential):Active Range of Motion; Passive Range of Motion; Isometric Resisted Testing.Summarized task data is presented in Table 2 categorized on the Ebel grid squarepresented in Fig. 1. The forearm, wrist and hand body region was 79% andconsidered close enough to the 80% consensus target to accept.

In the second stage of the validation process (the face to face component), themajority of the discussion focused on those items and categories that did not achieveinitial consensus: History; Observation; Scanning Exam; Clearing Joints Above and

<W Springer

M. Lafave et al.

Table 2 Consensus from j{initial grading of theexamination component of theSOAT

HipActive ROMPassive ROMIsometric Resisted

KneeActive ROMPassive ROMIsometric ResistedLower leg, foot &Active ROMPassive ROMIsometric Resisted

ShoulderActive ROMPassive ROMIsometric ResistedElbowActive ROMPassive ROMIsometric Resisted

% Consensus

Essential(Grid #1)

958185

828482

ankle100100100

808080

838690

Important(Grid #2)

51915

181618

000

181818

171410

Less Important(Grid #3)

000

000

000

222

000

Forearm, wrist & handActive ROMPassive ROMIsometric Resisted

797979

191919

222

Below; Special Testing; Palpation. There was little consensus on any history items/tasks before or during the face to face meeting (i.e.; stage 1 or 2). This resulted ingreater consensus on individual items and ensured that each expert's preferenceswere addressed. As a result of the thematic clusters, there was 100% consensus on allthematic clusters and subsequent consensus on the vast majority of tasks under eachthematic cluster. Three tasks did not meet the 80% target of consensus (pain scale,sport and sleeping position), but the PI decided to keep these tasks in the finalSOAT nonetheless.

The observation component achieved 80% consensus for all tasks during the faceto face discussion. Experts felt that if an upper extremity lesion was being evaluated,then only the upper extremity had to be observed (and vice versa with the lowerextremity). Additionally, clearing joints above and below and the scanning examcomponents achieved consensus with relatively little discussion. Details of thesecomponents can be obtained from the PI and therefore, more important results arepresented herein.

The special testing and palpation components were extremely controversial areasand it was obvious after hours of discussion that achieving 80% consensus was notattainable. As a result, a third stage to the process was added due to time constraintsof the expert group (they had to fly out of town). The group discussed andconceptualized a new model in these components of the orthopedic assessment

fi Springer

Development of a content-valid standardized orthopedic assessment tool (SOAT)

which the PI later captured, articulated and summarized as "clinical reasoning."Discussion centred around the concept of deductive and inductive approaches toorthopedic assessment and which of these should be expected for athletic therapygraduates. In the end, a new model of the SOAT which included a clinical reasoningcomponent that tied each of the 10 components (history, observation, scanningexam, clearing joints, active range of motion, passive range of motion, isometricresisted testing, special testing, palpation and conclusion) together was sent (viaemail) to the expert group 2 weeks after the face to face evaluation for approval andvalidation. This new model achieved the 80% consensus target for all clinical rea-soning components, and the originally controversial special testing and palpationcomponents.

Discussion

Content validation is a crucial step in overall creation of a valid and reliable tool tomeasure orthopedic assessment clinical skills (Hopkins, 1998). Content validation ofthe SOAT followed a modified Ebel procedure (Butterwick et al., 2006; Ebel &Frisbie, 1986; Violate et al., 2002) but also added a third stage of consensus buildingso as to create a more valid tool that could measure clinical orthopedic skills atmultiple levels of expertise. There are many (Cox, 2000; Hodges, 2003; Hodges,Regehr, McNaughton, Tiberius, & Hanson, 1999; Joorabchi & Devries, 1996;Norman, Tugwell, Feightner, Muzzin, & Jacoby, 1985) who believe that attemptingto measure clinical competence through performance based examinations trivializesthe complexities of clinical competence. There has been debate on the appropri-ateness of using detailed checklists as opposed to global rating scales to measure theclinical competence construct (Norman, van der Vleuten, & De Graaff, 1991;Reznick et al., 1998; Regehr, MacRae, Reznick, & Szalay, 1998; Van Luijk & Vander Vleuten, 1990; Van der Vleuten, Norman, & De Graaff, 1991). This debate wasconsidered during the current study.

The SOAT evolved from a checklist that only measured a deductive reasoningdiagnostic approach to a tool that permitted greater flexibility in the choice andorder of the assessment protocol. Moreover, the result of this evolution was thecreation of a tool that is robust enough to permit either a deductive reasoning or aninductive reasoning approach while still measuring the same construct of clinicalcompetence along a novice-expertise continuum. The tool accomplishes this byadding a "clinical reasoning" metric (a global scale) that attempts to tie each of themajor components of an orthopedic assessment method together. By going throughthe rigorous content validation process, a tool has been created that should permitflexibility in student's decisions throughout their orthopedic evaluation thus leadingto the possibility of higher fidelity at multiple levels of expertise. This may addresssome of the issues and concerns that OSCE checklists have a tendency to trivializethe construct being measured (Norman et al., 1991; Van der Vleuten et al., 1991).

The next step to test the tool's validity is to measure the tool's reliability. It ispossible to have a tool that is highly reliable which was the original purpose behindthe objective checklists with OSCEs (Harden et al., 1975). However, a balance mustbe created between validity and reliability and this study was an important first stepin the design of valid and reliable tool to measure orthopedic assessment skills.

fi Springer

M. Lafave et al.

Limitations

The expert panel struggled with the lack of definition and clear boundariesbetween the terms essential, important and less important. This lack of clarity anddefinition seemed to prevent 100% consensus with all tasks, fn fact, one personquestioned their own reliability from one moment to the next. As a result, the finalSOAT task list used a binary scale for inclusion or exclusion and not a scale thatsplit the vote between essential, important and less important. This variation of theEbel procedure was justified by the researchers because the original intent of theEbel procedure was to establish a minimal passing score in performance basedexaminations and not to establish the inclusion or exclusion of tasks which was theoriginal goal of this study. In the future, if the SOAT was to be used as a tool tomeasure a minimal passing score for orthopedic assessment, then an expert panelwould need to measure each task for the level of importance and difficulty and amathematical equation which weighted each task would be assigned to provide apassing test score.

A third phase of content validation was added which breaks from the traditionalmethod described by Violato et al. (2002) and Butterwick et al. (2006). However,this seemed to be a necessity to make the tool truly content-valid. During thisprocess, two of the experts on the panel dropped out of the study due to timecommitment. It may not be evident from the consensus statistics, but in the thirdstage of the validation process, the consensus figures were reported based on 10rather than 12 experts.

Conclusions

There was consensus on all components and tasks for each body region that com-prise the SOAT thus allowing authors to say the SOAT has content validity tomeasure student performance in practical examinations. Hodges advocates forcontextualized validity rather than making blanket statements for a tool's overallvalidity (Hodges, 2003). Part of the validation process included unanimous expertvalidation committee support for the assumptions and guidelines for tool use, andthus its validity. Those assumptions and guidelines that contextualize this tool were:

1. This form is intended to measure a basic level of competence in AthleticTherapy students and/or recent graduates. The examinee should be consideredto be at a basic level of competence;

2. This form should be robust enough to be used for any scenario (or injury) forthe specific body region;

3. All tasks in all components of the tool will default to n/a if they are not gradedby the examiner;

4. Students will lose grades on tasks that they missed (or completed incorrectly)which were thought to be relevant by the evaluator for a specific scenario orcondition;

5. Students should be graded on tasks based on the index of suspicion theyidentified at the end of each section. If the student does not have the correctindex of suspicion and they fail to complete tasks that would permit them toknow more about the condition, the student will lose marks for those specific

Q Springer

Development of a content-valid standardized orthopedic assessment tool (SOAT)

tasks. If they perform tasks that relate to their index of suspicion, they shouldreceive grades for those tasks done correctly;

6. The model (standardized patient) will ask the student if the index of suspicionhas changed at the end of each component of the, assessment (i.e; history,observation, scanning exam, clearing joints above and below, examination,special tests, palpation);

7. All lower extremity forms will have the same tasks for history, observation,scanning examination and examination (AROM, PROM, IR);

8. Since the form will be used generically for all lower and upper extremityconditions, respectively, all motions will be listed for AROM, PROM and IR.The default will be n/a and should only be graded if the student did it or failedto do it when they should have;

9. All upper and lower extremity forms will have unique sections for the specialtests and palpation sections;

10. Students should not be required to complete a specific and discrete list of specialtests for any given condition. Rather, the specific special tests will be based onwhat the student/examinee completed throughout the preceding components ofthe orthopedic assessment (i.e.; history, observation, scanning exam, clearingjoints above and below and examination). All special tests will be defaulted as n/a. Special tests will only be marked if the student performed them (i.e.; correctlyor incorrectly) OR if they should have tested a structure, but failed to do so. Inthe case when a student/examine FAILS to complete a special test which wouldidentify or confirm the anatomical structure/lesion site, they will be given agrade of zero for all special tests which would have tested that structure.

It is yet to be determined if this tool can be generalized to the physician studentpopulation, particularly because of the specificity required in the athletic therapyprofession compared to undergraduate medicine (Coady, Kay, & Walker, 2003; Kay& Walker, 1998). The next step in studying the SOAT to measure studentperformance in practical exams will be to evaluate the tool's reliability with theathletic therapy student population. In addition, future studies should evaluate thevalidity and reliability of the SOAT within the medical education realm.

Acknowledgements This study was completed with the support of the Canadian Athletic Thera-pists Association (CATA) Research Grant. Thanks to the expert validation committee for all theirtime and energy to complete this research. The PI would like to thank the CATA CertificationCommittee for partnering to bring experts to the second stage of the validation process. Finally, aspecial thanks to the University of Calgary Athletic Therapy Education Group for their assistanceand guidance throughout this process.

References

Butterwick, D. J., Paskevich, D. M., Vallevand, A. L. C, & Lafave, M. R. (2006). The developmentof content valid technical skill assessment instruments for athletic taping skills. Journal of AlliedHealth, 35, 149-157.

Cantor, J. A. (1989). A validation of Ebel's method for performance standard setting through itsapplication with comparison approaches to a selected criterion-referenced test. Educational andPsychological Measurement, 49, 709-721.

Coady, D., Kay, L., & Walker, D. (2003). Regional musculoskeletal examination. Clinical Rheu-matology, 9, 67-71.

'£} Springer

M. Lafave et al.

Cox, K. (2000). Examining and recording clinical performance; a critique and some recommenda-tions. Education for Health, 13, 45-52.

Cyriax, J. (1982). Textbook of orthopedic medicine, (vols. One). Tindall: Bailliere.Ebel, R., & Frisbie, D. (1986). Essentials of Educational Measurement. (4 ed.) Englewood Cliffs, NJ:

Prentice Hall.Gorter, S., Rethans, J., Scherpbier, A., van der Heijde, D., Houben, H., & van der Vleuten, C. et al.

(2000). Developing case specific checklists for SP based assessments in internal medicine.Academic Medicine, 75, 1130-1137.

Harden, R., Stevenson, M., Downie, W., & Wilson, G. (1975). Assessment of clinical competenceusing objective structured examination. British Medical Journal, 1, 447-451.

Hodges, B. (2003). Validity and the OSCE. Medical Teacher, 25, 250-254.Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., & Hanson, M. (1999). OSCE checklists do

not capture increasing levels of expertise. Academic Medicine, 74, 1129-1134.Hopkins, K. (1998). Educational and psychological measurement and evaluation. (8th ed.). Toronto:

Allyn & Bacon.Hoppenfeld, S. (1976). Physical examination of the spine and extremities. Norwalk: Appleton &

Lange.Joorabchi, B., & Devries, J. (1996). Evaluation of clinical competence; the gap between expectation

and performance. Pediatrics, 97, 179-186.Kay, L., & Walker, D. (1998). Improving musculoskeletal clinical skills teaching. A regionwide audit

and intervention study. British Medical Journal, 57, 656-659.Magee, D. (2002). Orthopedic physical assessment. (4th ed.). Toronto: Saunders.Neufeld, V., & Norman, G. (1985). Assessing clinical competence, (vols. 7). New York: Springer.Norman, G., Tugwell, P., Feightner, W., Muzzin, L., & Jacoby, L. (1985). Knowledge and clinical

problem solving. Medical Teacher, 19, 344-356.Norman, G., van der Vleuten, C., & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity; issues

of validity, efficiency and acceptability. Medical Education, 25, 119-126.Regehr, G., MacRae, H., Reznick, R., & Szalay, D. (1998). Comparing the pschyometric properties

of checkliss and global rating scales for assessmeing performance on an OSCE-formatexamination. Academic Medicine, 75(9), 993-997.

Reznick, R., Regehr, G., Yee, G., Rothman, A., Blackmore, D., & Dauphinee, D. (1998). HighStakes Examinations; What do we know about measurement? Academic Medicine, 73, S97-S99.

Shultz, S., Houglum, P., & Perrin, D. (2005). Examinations of Mttsculosketetal Injuries, (2 ed.)Champaign, IL: Human Kinetics.

Van der Vleuten, C., Norman, G., & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity: issuesof reliability. Medical Education, 25, 110—118.

Van Luijk, S., & Van der Vleuten, C. (1990). A comparison of checklists and rating scales inperformance based testing. In I. Hart, R. Harden, & J. Des Marchais (Eds.), CurrentDevelopments in Assessing Clinical Competence, Montreal: Can-Heal Publications.

Violate, C, Salami, L., & Muiznieks, S. (2002). Certification Examinations for Massage Therapists:A Psychometric Analysis. Journal of Manipulative Physiological Therapeutics, 25, 111-115.

Violate, C., Marini, A., & Lee, C. (2003). A validity study of expert judgement procedures for settingcutoff scores on high stakes credentialing examinations using cluster analysis. Evaluation and theHealth Professions, 26, 59-72.

& Springer