1 Wallace Judd APA Testing Standards: RFC [email protected].
40
-
Upload
laurence-green -
Category
Documents
-
view
216 -
download
0
Transcript of 1 Wallace Judd APA Testing Standards: RFC [email protected].
2
The APA Testing Standards: Request for Comment
Wallace Judd, Ph.D.
President of Authentic Testing Corporation
President-Elect of the PTC
Webinar Sponsored by the PTC and byAuthentic Testing Corporation.
3
The APA Testing Standards: Request for Comment
Comments due October 15, 2007
http://www.TestStandards.org
4
TestStandards.org
5
Webinar is organized in 2 parts:
Part I. 20-minute intro to the Standards:
• How they were formulated.
• Why they’re important.
• How they’re likely to impact your program.
• Problems with the current Standards.
6
Webinar is organized in 2 parts:
Part II. Forty minutes of discussion…
• Are these issues relevant to you?
• What (if anything) do you want to do?
• What are Next Steps?
7
NCME Session Title
8
What The Standards Address:
Importance of the Standards –
• 30,000 copies sold.
• Represents testing practice worldwide.
• Basis for ANSI ISO 17024 enforcement
• Basis for NCCA Accreditation
9
Why Standards Are Important
• ANSI – required by DOD for security certifications
• NCCA – required by Maine for stateaccreditations
• More coming…
10
Who Created the Standards?
75 Organizations provided input, volunteer time, editing, ideas. They were from:
• Original Sponsors: AERA, APA, NCME
• Academic Institutions
• Non-Profit Associations
• State & Federal Government Agencies
• Test Publishers
11
Not a single sponsor or reviewer was from corporate America.
12
Current Standards
• Reviewed, revised 1996-1998.
• Adopted 1998.
• Published, disseminated 1999.
13
Contents of the Standards
• The Standards were written
• for the professional and
• for the educated layperson
14
The Standards
address professional and technical issues of test development and use in
• education,
• psychology and
• employment. (APA.org, home page)
15
The Standards
are the Bible of Testing in the U.S.
16
Standards = The Bible
17
Contents of the StandardsThis book is a vitally important reference for • professional test developers, • sponsors,sponsors, • publishers, • users, • policymakers, • employers,employers, and • students (http://www.APA.org, home page)
18
But the Standards clearly target educational & psychological
testing.
19
Few Job-Related Standards
• Educational – measuring and predicting student achievement
• Psychological – measuring and evaluating psychological traits
• Sections on Performance Testing and corporate / industrial testing – minimal.
20
Initial Issues
21
Problem: Bylaws
Standard 2. A. The certifying program must show that the governance structure, that the governance structure, policies, and procedures that have been policies, and procedures that have been established protect against undue established protect against undue influenceinfluence that could compromise the integrity of the certification process.
22
Problem: Governance Structure
Standard 2. B. The governance structureThe governance structure, policies, and procedures must provide formust provide for autonomy in decision makingautonomy in decision making regarding important aspects of the certification program, such as eligibility standards; the development, administration, and scoring of the assessment instruments; selection of personnel; and operational processes.
23
Problem: No Training & Testing In the Same Organization
• Standard 2. D. To avoid conflicts of interest between certification and education functions, the certification agency must the certification agency must not also be responsible for accreditation not also be responsible for accreditation of educational or training programsof educational or training programs or courses of study leading to the certification
24
Problem: Board Composition
• Standard 3. The certification board or governing committee of the certification governing committee of the certification program must include individuals from program must include individuals from the certified populationthe certified population, as well as voting representation from at least one consumer or public member.
25
Problem: No Place for Gating Items
Gating Items – Items Required to Pass Test
• Pilot’s Test – Land the plane
• Veterinarian’s Test – Spay a cat
• Crane Operator’s Test – Put ball in barrel
• Word Test – Save a file
• System Admin. Test – Add a new user
26
Problem: No Performance Items
• Standard 3. (p. 39) When [using IRT], the fit of the model is evaluated … by fit of the model is evaluated … by evaluating unidimensionality,evaluating unidimensionality, local independence, speededness, and equality and equality of slope parametersof slope parameters.
27
Problem: Expert Judges
• Standard 3.6 The qualifications, relevant experience, and demographic characteristics of demographic characteristics of expert judges should also be documentedexpert judges should also be documented.
28
Problem: Content Domain
• Standard 3.11 Test developers should document the extent to which the content document the extent to which the content domain of a test represents the defined domain of a test represents the defined domaindomain and test specifications.
29
Problem: Publish Scoring Criteria
Standard 3.20 When appropriate, …criteria criteria for scoring …should be provided to the for scoring …should be provided to the test takerstest takers prior to the administration of the test.
30
Why Do I Care?
• I’ve developed tests since 1981 at Xerox, Oracle, Honeywell, McGraw Hill, Kelly Services, and my tests were the foundation for Microsoft’s MOUS tests.
• I believe that skills testing and certification deserves as much recognition and validation as psychological or educational testing.
31
Why Does the PTC care?
• It has many members who have testing programs, and who want them to exemplify best practices in testing.
• PTC meets October 17-19 in Scottsdale.
• PTC will formulate its own response to the RFC.
• PTC is at PerformanceTest.org
32
How important is compliance with the Standards to your program?
1. Insignificant
2. A small factor
3. Significant
4. A large factor
5. Critical
33
How important is NCCA accreditation to your program?
1. Insignificant
2. A small factor
3. Significant
4. A large factor
5. Critical
34
How important is ANSI accreditation to your program?
1. Insignificant
2. A small factor
3. Significant
4. A large factor
5. Critical
35
We have two routes to go. What should we do?
1. Formulate separate standards for corporate-sponsored testing
2. Work with AERA, APA, NCME, to revise the Standards.
3. Both
36
Which would you participate in? (You can vote for more than 1.)
1. Individual corporate response
2. Industry-specific group response
3. Response representing corporations with testing programs
37
Plug: The PTC
• Where: Scottsdale, Arizona
• When: October 17 – 19, 2007
• Who: – Graham Brent – Crane Operators’ Exam– Ric Luecht – Engineering Assessments– Brian Rice – How Virtualization Works
http://www.PerformanceTest.org
38
Plug: Negotiating the Standards
• Workshop October 16th, PTC Summit in Phoenix.
39
Plug: AuthenticTesting.com
• Automated Job Task Analysis
• Automated Exam & Item Analysis
• Performance Test Design
