Testing Standards Testing Standards

Chris Gruber, Barbara Plake &Chris Gruber, Barbara Plake &Wayne CamaraWayne Camara

Session Part 1: ATP Experiences 1995-1999

Chris Gruber

ATP Liaison to the APA &

Member of the ATP Test Standards Committee

during the time covered.

What are “The Standards?”

• A book: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

• With a history:

APA, “Technical Standards…”, 1954

NEA, “Technical Standards…” 1955

APA, AERA & NCME, “Standards…”, 1966, 1974, 1985

What are the areas of concern?

• Part 1: Test Construction, Evaluation & Documentation—6 chapters, 113 standards

• Part 2: Fairness in Testing—4 chapters, 48 standards

• Part 3: Testing Applications—5 chapters, 93 standards

What is a standard?

Standard 1.4: If a test is used in a way that has not been validated, it is incumbent on the user to justify the new use, collecting new evidence, if necessary.

Comment: Professional judgment is required to evaluate the extent to which…

ATP experience on the 6th Edition

• Call for Comments—1995

ATP Proposed New Standard

Test users have the responsibility to respect copyright.

ATP experience on the 6th Edition

Draft 1—1996

• ATP Comments cover 14 single-spaced pages.

One comment reflected continuing concern that there was still no mention of Copyright

ATP experience on the 6th Edition

• Draft 2—1997, ATP comments extend to 25 single spaced pages.

• Cover memo cites involvement of 12 member organizations—virtually all of our largest members at the time—with comments from scores of staff research and development professionals.

• A standard on copyright appears, rewritten, restricted in application, and qualified by “...to the extent that developers enjoin users to do so...”

ATP experience on the 6th Edition

• Draft 3—1998, ATP comments extend to 26 single spaced pages.

• Cover memo cites involvement of 12 member organizations—virtually all of our largest members at the time—with comments from scores of staff research and development professionals.

• A standard on copyright? See next slide…

ATP experience on the 6th Edition

• Published version—1999

Standard 11.8

Test users have the responsibility to respect copyright.

What works?

• Clear, respectful disagreement where appropriate and necessary

• Great science!

• Teamwork

• Perseverance

Part 2: Revision of Test StandardsWayne Camara

•Management Cmte: – David Frisbie (NCME)– Suzanne Lane (AERA) – Wayne Camara (APA)

• Staff support AERA

•Joint Cmte on the Revision of the Testing Standards: – Co-Chairs: Lauress Wise and Barbara Plake– Cmte may number 12-18 members

• Staff support APA

Management Cmte• Develop agreement (SOW) for

Technical Cmte. • Approve budget and expenses

for the Standards and the Tech Cmte.

• Oversee finances and Development Fund

• Appoint members to Joint Cmte• Determine if/when revision is

required• Establish timeline for revision• Establish new agreement for 3

sponsoring assoc. 12-08

Technical Cmte• Follow separate agreement

developed by Management Cmte and approved by sponsoring associations

• Decide technical issues• Determine overall purpose of

standards and general guidelines on use

• Establish communication structures with sponsors for revision effort

• Develop draft for review and final document

COMMENTS RECEIVED ON STANDARDS 9-07 – 12/07

• There were 71 participants who provided responses to the web surveys– 30 from the Standards web page– 41 from the APA web page

• Of these 71, only 45 provided specific comments (1 or more) on the standards (the remaining 26 filled in some demographic items but not comments). – 21 from the Standards web page– 24 from the APA web page

• Note – individuals not submitting any comments are not included in remaining analyses or spreadsheets

Total Number of Participants

Type of Participant No. of Participant

Non APA Participants 30

APA Participants 41

Total Number of Participants 71

Total Number of Participants Who Submitted Comments

Participants Number

Non APA Participants Who Submitted Comments 21

APA Participants Who Submitted Comments 24

Total Number of Participants 45

• From the 45 participants who submitted comments:

– 265 comments were provided by respondents on the Standards web page

– 158 comments were provided by APA respondents

Total Number of Comments Submitted by Survey Group

Survey Group No. of Comments

Non APA Survey Group 265

APA Survey Group 158

Total 423

•The 45 submitted 423 comments for review•The comments were submitted under three main categories:

•Additions•Deletions•Revisions

•The largest percent of the comments were revisions (52%) and additions (41%)

Type of Comment Submitted

Type of Comment No. of

Comments% of Total Comments

Total Revisions 222 52%

Total Additions 175 41%

Total Deletions 23 5%

Total Other Submissions 3 1%

Total Number of Comments 423 100%

•The total number of comments was also analyzed by chapter number

•Chapters One, Three, Five, Seven and Ten had the largest percent of comments submitted

Number of Comments by Chapter Number

Chapter Total Comments % of Total Comments

Chapter One 41 10%

Chapter Two 26 6%

Chapter Three 35 8%

Chapter Four 22 5%

Chapter Five 46 11%

Chapter Six 13 3%

Chapter Seven 44 10%

Chapter Eight 11 3%

Chapter Nine 28 7%

Chapter Ten 36 9%

Chapter Eleven 15 4%

Chapter Twelve 22 5%

Chapter Thirteen 20 5%

Chapter Fourteen 20 5%

Chapter Fifteen 16 4%

Background 1 0%

Select Background or Standard... 8 2%

Select chapter... 7 2%

(blank) 12 3%

Total Comments 423 100%

Affiliation for APA Respondents (N=42)Text Affiliations Represented by the APA Participants No. of Participants

APA Individual Member (No Affiliation) 26

APA member + member of SIOP 1

Committee on Aging 1

Committee on Children, Youth, and Families 1

Committee on Ethnic Minority Affairs 1

District of Columbia Psychological Association 1

Division 12 - Society of Clinical Psychology 1

Division 14 - The Society for Industrial and Organizational Psychology 2

Division 17 - Society of Counseling Psychology 1

Division 5 - Evaluation, Measurement, and Statistics 1

Individual Member 1

Individual member (myself) 1

member 1

member of APA 1

Pearson VUE psychometricians 1

State Psychological Association 1

Total APA Participants (One Participant is a member of Two Organizations) 42

Affiliation Type By Number of Participants

Affiliation Type No. of Participants % of Participants

APA 41 58%

Testing Organization 10 14%

College or University 8 11%

Other 7 10%

R&D Organization 3 4%

State Agency 1 1%

Federal Agency 1 1%

School System 0 0%

Industry 0 0%

Total 71 100%

•The participants were asked to select a type of affiliation•58% of the total group were affiliated with the APA•35% of the total group were affiliated with either a Testing Organization (14%), a College or University (11%) or “other” type of affiliation (10%)

Comments and Next Steps• Major Presentation at NCME – AERA 2008• Major Presentation at APA 2008

Develop SOW – Scope of Revision and Timeline

Appoint Members to Tech Cmte

Negotiate Timeline and Revisions to SOW with Tech Cmte

Provide Updates to Sponsoring Associations

Questions?