“Putting a Luster on a Good Testing Program”

Barbara Showers, PhDWisconsin Department of Regulation and Licensing

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Low-Tech Indicators of Potential Exam Problems

• Frequent Score Adjustments

• Pass Rate Changes

• “I Can’t Find it on Your Web Site”

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Frequent Score AdjustmentsIndicators

• Regular grade adjustments on practical or oral exams before release or on appeal

• Increasing numbers of multiple-keyed or omitted multiple choice questions

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Frequent Score AdjustmentsPractical/Oral Issues

• Are examiners unclear on the criteria?

• Are examiners given too much flexibility?

• Are the criteria themselves solid (one correct answer)?

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Frequent Score AdjustmentsPractical/Oral Solutions

• Clarify scoring criteria

• Reduce flexibility – checklists

• Develop a bank of pre-approved questions with correct answers

• Train new examiners/board members

• Pair new examiners with experienced ones

• Use multiple examiners

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Frequent Score AdjustmentsMultiple Choice Issues

• Questions are out of date

• Questions are ambiguous

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Frequent Score AdjustmentsMultiple Choice Solutions

• Multiple question sources and writers

• Committee review before use

• Update on a regular schedule

• Allow candidate comments after exam

• If sufficient volume, run item analysis statistics – ID problems

• Don’t keep the multi-keyed or omits in the test – reflects poorly on quality

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Pass Rate ChangesIndicators

• Gradual pass rate increase over time, or

• Pass rates swing up or down in short time

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Pass Rate ChangesIssues

Reasonable Explanations for Pass Rate Change

Exam content change

Exam cut score change

Candidate qualifications change

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Pass Rate ChangesIssues

If no reasonable explanation, consider

• Security breach

• Standard setting issues – are the forms equivalent in difficulty?

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Pass Rate ChangeIssues

• What is a “reasonable” pass rate?

Example:

WI Hearing Instrument Specialist Exam

Audiologist Pass Rate = 70%

Apprentice Pass Rate = 38%

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Pass Rate ChangesSolutions

Security solutions

• Multiple forms

• New content as often as possible

• Avoid repeating highly memorable scenarios

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Pass Rate ChangesSolutions

Unexplained Variations

• Equate forms if possible

• If statistical equating is not possible, use at least subject matter expert ratings of difficulty to create similar forms

• Define and document the passing performance standard and use the definition over time for consistency

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

“I can’t find it on your web site.”

• Put the luster on your testing program by polishing your web site

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Polishing the Web Site

• Is the candidate bulletin prominent?

• How deeply buried is the fee information?

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Polishing the Web Site

• How to apply

• What to expect in content and administration

• Study resources

• Links to other relevant sites

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona

Speaker Contact Information

Barbara Showers, PhDWisconsin Department of Regulation and LicensingP.O. Box 8935Madison, WI 53708608-266-7703, fax 608-267-1809barbara.showers@drl.state.wi.ushttp://drl.wi.gov