Dr. Manny Straehle Dr. Liberty Munson · 2020-06-25 · Reach out to Questionmark at...

Dr. Manny Straehle

Dr. Liberty Munson

A S S E S S M E N T , E D U C A T I O N , A N D

R E S E A R C H E X P E R T S

2Copyright © 1995-2020 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark, and Assessment, Education and Research Experts (AERE). All rights

reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

To ask questions,use the “Questions”

feature

Watch for an email after the webinar:• Download slides (PDF)• View a recording• Answer a survey



• Founded in 1988

• Assessment solutions to measure knowledge, skills, abilities and attitudes securely for certification, regulatory compliance, workforce learning, sales-force readiness and higher education

• ISO/IEC 27001 Certified (Learn more: www.questionmark.com/trust)

Background

http://www.questionmark.com/trust



Collaborative, secure authoring of surveys, quizzes, tests, and

examsReporting to analyze results

and inform stakeholders

Blended delivery to support a broad range of devices and

security needs

Multilingual: 36 Languages

Global 24/7 Support



Manny Straehle

Liberty Munson



Ensure quality of the exam

Ensure fairness in all aspects

Ensure interpretations of test scores

are appropriate

Someone who is certified is proficient at

skills measured by exam



Fairness

Reliability

Validity



PopulationN=12

Blue =3 (25%)Red =6 (50%)Green=3 (25%)

SampleN=4

Blue=1/4 (25%)Red=2/4 (50%)Green=1/4 (25%)

Random Sampling

• Obtaining a certain percentage of a sample at random will lead to greater confidence that your sample represents the population

Stratified Random Sampling

• Population is divided into groups, called strata, based on some characteristic

• Within each strata, a sample, usually random, is selected

• How many are selected from each strata depends on the purpose for creating the strata initially





Identify Test Content

12 to 15 SMEs

JTA Survey

200 SMEs

Develop Test/Exam Specifications

10 to 12 SMEs

Write Items*

10 to 15 SMEs

Review Items*

5 to 10 SMEs

Review Exam/Form*

5 to 10 SMEs

Administer Exam (Beta)

50+ SMEs

Item Analysis

3+ SMEs

Standard Setting

5+ SMEs



“…the systematic process of discovery of the nature of a job by dividing it into smaller units, where the process results in one or more written products with the goal of describing what is done in the job or what capabilities are needed to effectively perform the job” (p. 8).

-Michael Brannick (2006) Test Specifications

KNOWLEDGE, SKILLS, ABILITIES, ETC.

TASKSNUMBER AND PERCENTAGE OF

EXAM QUESTIONS

ACROSS DOMAINS



“…the systematic process of discovery of the nature of a job by dividing it into smaller units, where the process results in one or more written products with the goal of describing what is done in the job or what capabilities are needed to effectively perform the job” (p. 8).

-Michael Brannick (2006)

1. KICKOFF MEETING2. IDENTIFY TASK AND

KNOWLEDGE STATEMENTS3. TASK FORCE MEETING

4. SURVEY DEVELOPMENT &

ADMINISTRATION

5. DATA ANALYSIS 6. TEST SPECIFICATIONS

MEETING



• Demographic responses

• Importance and frequency ratings of task and knowledge statements

o Test weight survey recommendations

• Comparative analysis (mean importance ratings by selected demographic questions)

o Are there differences based on demographics (e.g., region, gender, ethnicity, experience, etc.)

Used to Make Decisions that

Drive Test Specifications



Target Audience

Exam Blueprint

Assignments

Exam Eligibility Criteria

Minimally Qualified

Candidate

Item Writing Guidelines



In general, consider writing “application” level questions rather than “remembering” or “understanding” level questions to evaluate a deeper level



Formats

• Open

• The year that John Adams was elected President:

• Closed (Preferred)

• In what year was John Adams elected President?

Best Practices

• Succinct –remove unwanted language

• Relevant and important

• Non-trivial

• Stem is NOT teaching

• Avoid using “Not” and “Except”

• Avoid using definitions in stem

• Avoid two questions at once

Quality Check

• Can you cover the options and answer the question?

• What is the capital of France?

• A. Lyon

• B. Paris

• C. Normandy

• D. Orlean



Key: Best Practices

• Don’t be different from the other options such as the longest and technical

• Don’t use words that are in the stem and repeated only in the key = cueing

• Parallel in structure

Distractors: Best Practices

• Incorrect

• Plausible

• Common misconceptions

• No overlap

34

• Application of knowledge, skills, and abilities, not recall, understanding, etc.

• Present a scenario describing a business problem or situation that requires application of skills in real world scenarios

• Require the test taker to:

o Analyze and provide a solution for a complex problem o Combine several concepts, ideas, or skills to solve the

problemo Identify and analyze patterns based on a limited amount

of information or datao Design a solution by drawing on multiple skills learned

during the training, and/oro Evaluate outcomes that may result from actions taken

• Assess ability to apply knowledge to real-world problems, and/or

• Ask about concepts, actions, and outcomes that require extrapolating beyond what is specifically covered in the training or linking information from multiple sources

Copyright © 1995-2020 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark, and Assessment, Education and Research Experts (AERE). All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

35

• Justifies correct answer

• If available, use approved references (e.g., docs)

o If an approved list is unavailable, use the more common and often cited references

• When providing a rationale, most of your peers would agree with it




Cognitive Level of Item

# of Items Written in a Day

Estimated Time to Complete One Item

Understanding 12-15 30 minutes

Application 10-12 30-45 minutes

Problem Based Items 6-8 45-60 minutes



Item difficulty (p-value)

Item discrimination

• Biserial/Point-biserial correlation coefficients

Option analysis

• P-value

• Pt biserial

• QuintilesComments



Proportion of candidates who correctly answer a test item

Ranges from 0 – 1

• Polytomously scored = Recommend dividing average score by total number of points possible so on same scale regardless of number of points

Low values = “Difficult” items

High values = “Easy” items



General rule of thumb: 0.3 to 0.7

• p-value = .5 provides max information about candidates

• If p-value = 0.5 then variance = 0.5*(1 - 0.5) = 0.25 (max variance)

Avoid items with p-values near 0 or 1

• No information provided unless needed for content validity reasons



To what extent does an item “discriminate”

between candidates of low

and high ability levels?

Correlations range: -1 to +1; 0 = no

relationship; 1 = perfect

Large positive correlations ideal

Rule of thumb: >.2 or higher



Rpbis/bis range Interpretation

If rpbis/bis ≥ 0.30 Item is functioning very well

If rpbis/bis [0.20 – 0.29] Little or no revision required

If rpbis/bis [0.10 – 0.19] Item is marginal and needs to be revised

If rpbis/bis < 0.10 Item requires serious revision or should be

eliminated



Item name% answering

correctly CorrelationNumber of Responses

Q1 0.722 0.593 205

Q2 0.850 0.708 206

Q3 0.654 0.592 205

Key

Answer choice selected

% answering correctly correlation 2 to 25

26 to 34 35 to 36 37 to 38 39 to 47

A1,B3,C5 0.005 -0.003 1

A1,B3,C6 0.044 -0.207 6 3

A1,B4,C5 0.054 -0.492 10 1

A1,B4,C6 0.010 -0.185 1 1

A2,B3,C5 0.044 -0.358 8 1

CorrectA2,B4,C5 0.834 0.651 13 44 26 50 38

A2,B4,C6 0.010 0.031 1 1



Rasch Analysis

• More sophisticated look at item difficulty

• Estimates difficulty in relation to other items in item pool

2 Parameter Model

• Adds discrimination parameter

3 Parameter Model

• Adds guessing parameter

More responses needed to

estimate more parameters



NedelskyCompromiseBorderlineBookmarkAngoff



Most commonly used

Most widely accepted

Item based method

• SMEs answer and rate each item

Criteria referenced (rather than based on a norm group of testers)

51

• In general, longer exams are more reliable

• How many skills, abilities do you need to cover?

• How many items are in your item pool?

• How many test forms will you create?

• How much overlap is acceptable?

• Spearman-Brown Prophecy formula

o Estimate reliability changes as you add or remove itemso Estimate number of items to achieve desired level of reliability




• Same general rank order between administrations

Test-retest Reliability: Consistency across

time

• Each form is designed to measure same content areas in the same manner

Alternate forms reliability; interrater

reliability: Consistency across forms

• Cronbach’s alpha

• Determined by interrelatedness of the items and test length

Internal consistency; split-half reliability: Consistency among

items



• a ≥ 0.9 = Excellent

• 0.7 ≤ a < 0.9 = Good

• 0.6 ≤ a < 0.7 = Acceptable

• 0.5 ≤ a < 0.6 = Poor

• a < 0.5 = Unacceptable



Defined

• How well an exam measures what it is meant to measure

• A property of how the exam is used (scores are interpreted) rather than of the exam itself

Ensuring Validity

• Exam objectives must be derived from job role requirements and skills needed to use the product

• Exam must include items that cover all functional groups and major objectives

• Exam content must be representative of the appropriate domain of knowledge

• Subject matter experts (SMEs) should review the objectives and items; revisions should be incorporated as necessary

What this REALLY Means

• Appropriateness of inferences or judgments based on test scores, given supporting empirical evidence

57

An exam can be reliable without being valid, but a test cannot be valid without being reliable


Reliable but Invalid Reliable and ValidUnreliable and Invalid



Evaluation: Is exam scored accurately with respect to

constructs measured?

Generalization: Does performance on exam

generalize to the broader content domain?

Extrapolation: Do the performances generalize to practice and other outcomes

of interest?

Explanation: Do scores reflect performance on the constructs intended to be measured by the exam?

Decision making: Do scores reflect readiness to practice?

Validity of a test

Validity of a score

Validity of an argument



Don’t claim more than

what is supported by

evidence

What claims would your test NOT support?

Is it plausible given the empirical

evidence at hand?

Is it clear and coherent?

Clearly lay out the claim that you’d like to

make based on the candidate

test scores

60

• Test fairness

• Item exposure

• Equating

• Localization vs. Globalization

• Item Forensics

• And More…

• So, ask your psychometrician!




Founder and [email protected]

Chief Psychometrics [email protected], Education, and Research Experts (AERE)

www.aerexperts.com

mailto:[email protected]


62

AERA, APA, NCME. (1999). Standards for Educational and Psychological Testing.

Washington, DC: American Educational Research Association.

Brennan, R. L. (Ed.). (2006). Educational Measurement (4th ed.). Westport, CT: Praeger.

Francis, G. (Ed.). (2007). Behavior Research Methods. New York: Springer.

Linn, R. L. (Ed.). (1989). Educational Measurement. New York: Macmillan.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric Theory. New York: McGraw-Hill.

Whitley, B. E. (1996). Principles of Research in Behavioral Science. Mountain View, CA,

Mayfield.


63

• Inventor of the Swearing Chicken

• Ph.D. in Educational Psychology

• ABD in Counseling Psychology

• IT Certifications: GISF, Data Management Support

• Therapy Certifications: Social Therapy

• Testing Organizations Worked at:

o Psychometrics: NBME, Prometric, USGBC

• Organizations founded:

o International Credential Associates o Assessment, Education, and Research Associates (AERE)

• University Teaching Experiences: Temple, Penn State, Saint Joseph’s University, Johns Hopkins, USC, and George Washington University

• Number of Organizations Consulted: 100+

• Social Responsibility: TEDx, E-ATP, ATP, ACA, ALA, Special Olympics, Spark, ESI, Habitat for America

• Number of Presentations: 70+

• Interests: Pizza Making, Presidential Libraries, Healthcare Communications, Pro Bono, Family, Friends, and Good Laughs


64

• Principal Psychometrician for Microsoft’s Learning & Readiness organization

• Responsible for ensuring that the skills assessments in Microsoft Technical Certification and Professional Programs are valid and reliable

• Prior to Microsoft, worked at Boeing in their Employee Selection Group, assisted with the development of their internal certification exams, and acted as a co-project manager of Boeing’s Employee Survey

• BS in Psychology from Iowa State University and MA and PhD in Industrial/Organizational Psychology with minors in Quantitative Psychology and Human Resource Management from the University of Illinois at Urbana-Champaign


UpcomingWebinars

Request a one-on-one demoThe Questionmark team will contact you to arrange a demonstration tailored to your needs and questionswww.questionmark.com/request-demo



http://www.questionmark.com/request-demo

Reach out to Questionmark at [email protected]

or request a demo at https://www.questionmark.com/request-demo

If you would like to reach out to Manny - [email protected]

Or Liberty – [email protected]

www.aerexperts.com


https://www.questionmark.com/request-demo



http://www.aerexperts.com/

Dr. Manny Straehle Dr. Liberty Munson · 2020-06-25 · Reach out to Questionmark at...

Documents

Transcript of Dr. Manny Straehle Dr. Liberty Munson · 2020-06-25 · Reach out to Questionmark at...