Psychological Testing Test Administration1

Test AdministrationKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

ObjectivesDiscuss how the relationship between the examiner and the test taker can affect test scoresExplain how an expectancy effect might affect a test scoreExamine the relationship between test performance and reinforcing particular responses

Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

ObjectivesOutline some of the advantages of computer-assisted test administrationList what characteristic of the state of the subject should be considered when a test score is evaluatedKnow what problems you would need to consider in training your observers if you were in-charge of a behavioral observation studyKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.


OUTLINEI. The Examiner and the ExamineeRelationship between examiner and examinee Training of Test Administrators Computer-Assisted Administration Race of the Examiner Expectancy EffectsMode of Administration Language of the Examinee (Test Taker) Effects of Reinforcing Responses Subject VariablesII. Behavioral Assessment and MethodologyReactivityExpectanciesStatistical Control of Rating ErrorsDriftDeception

We have a tendency to think that an observed score really represents the TRUE ability or TRAIT we are trying to measure

Does a score really represents TRUE ABILITY or TRAIT? Explain Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

WHAT concept is implicated in the relationship between a TRUE and OBSERVED score?

What is the focus of RELIABILITY?

What does a RELIABILITY estimate SIGNIFY?

What does it mean if a test has HIGH RELIABILITY estimate?Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Reliability theory is primarily concerned with random sources of errorKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

What are the other potential sources of ERROR?


Testing situationExaminer (tester) characteristicExaminee (test-taker) characteristicKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.


OUTLINEI. The Examiner and the ExamineeRelationship between examiner and examinee Training of Test Administrators Computer-Assisted Administration Race of the Examiner Expectancy EffectsMode of Administration Language of the Examinee (Test Taker) Effects of Reinforcing Responses Subject VariablesII. Behavioral Assessment and MethodologyReactivityExpectanciesStatistical Control of Rating ErrorsDriftDeception

The Examiner and the ExamineeBoth the behavior of the examiner and his or her relationship to the examinee can affect test scores

Enhanced rapport vs. neutral rapport Approving comments vs. Disapproving comments Familiarity vs. Unfamiliarity with proctor Substantial interviewer effects

** Examiners should be aware that their rapport with test takers can influence the resultsKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

WISCEXAMINERs friendly conversation and verbal reinforcement

higher IQ scores for 5th to 9th grader than those 5th to 9th grader who had a NEUTRAL examiner BUT little effect on 1st to 3rd grader (Feldman & Sullivan, 1960)


Saying GOOD or FINE vs. I thought you could do better than that or

Children who took the test under a disapproving examiner received lower scores than did children exposed to a neutral or approving examiner (Witmer, Bernstein, & Dunham, 1971) Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Among 137 young children who took a reading test, reading scores were significantly lower when the proctor was unfamiliar (DeRosa & Patalano, 1991)


In Fuchs and Fuchs (1986) quantitative review of literature (22 different studies which involved 1489 children) the average test performance was roughly 4 IQ points higher when examiner was familiar with the examineeFor those children from low SES, familiarity accounted for 7.6 IQ points

Familiarity can either positively or negatively bias test results


Respondents may give response that they perceive to be expected by the interviewer

People tend to disclose more information in a self-report format forms than they do to an interview (Moun, 1998)

People report more symptoms and health problems in a mailed questionnaire than they do in face-to-face interview (McHorney, 1999) Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Computer administration is at least as reliable as traditional test administration (Bergstrom & Lunz, 1999; Campbell, Rohlman, Storzbach, & Binder, 1999)


In most testing situations, examiners should be aware that their rapport with test takers can influence the results


The Race of the ExaminerRacial Bias

Sattler (2002, 2004) concluded that there is little evidence that the race of the examiner significantly affects intelligence scores

Only few studies (4 out of 29) have referrence to the belief that white examiners impede the performance of African American children (Sattler, 1979a) Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Why Few Studies Show Effects of Examiners Race on IQ tests The procedures for properly administering an IQ test are specificMost standardized test require a strict administration procedure. However, the examiner can still communicate a hostile or a friendly atmosphere, or an inquisitorial or therapeutic roleExaminer effects tend to increase when they are given more discretion about the use of the testOLSAT test self reading vs. audiotaped instructionKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Bias in the way Items are presentedComparison between African American and White Children on Preschool Language Assessment Instrument2 conditions: standard vs. use of context and themesAA children scored higher when items are administered in a thematic modeSignificant increase in performance on more complex and difficult items (Fagundes, Haynes, Haak, & Moran, 1998) Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Otis-Lennon School Ability Test (OLSAT)

It was hypothesized that AA children may score lower on IQ tests due to poorer reading skills4th to 5th grade AA students who heard an audiotaped version scored significantly higher on the test (Warner-Benson, 2001) Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Although standardization is required, VARIATIONS from the standard might reveal some unnoticed skills in disadvantaged test takers

What can be done?

Though the examiners race have minimal effects on IQ scores, should it be taken for granted or disregarded? Why? Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Language of the Test TakerIssues concerning testing individuals with different linguistic backgroundsSome tests are inappropriate for people who have limited knowledge of the language used in a test- What is compromised?In original test vs. translated test, what may be the primary concern?If examinee is bilingual, which version of the test should be used?Any opinion on test interpreters? Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Training of ExaminersDifferent assessment procedures require different levels of training (ex. License for psychometrician) SCID- Structured Clinical Interview for DSM-IVUsed for psychiatric diagnosis, are administered by licensed psychologistsWAIS-R 10 practice administrations to develop competence Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Expectancy EffectsExpectancies shape our judgments in many important ways (Kirsch, 1999)

Lasallians, UP students, Thomasians, Paulinians? Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Expectancy EffectsAlso called Rosenthal Effects (Robert Rosenthal from Harvard University)2 groups of experimenters were led to believe that average responses will fall on success or failure (1 group expects success, the other, failure)Subjects provided data that that confirm the experimenters expectanciesExperimenters influence is not limited to human subjects = maze bright vs maze dull Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

2 aspects of Expectancy Effects in the Use of Standardized Tests (Rosenthal)Observed even when all of the experimenters followed a standardized script; expectancy effect results from subtle non-verbal communication between experimenter and the subject (Rosenthal)Has a small and subtle effect on scores and occurs in some situation but not in othersMay affect in the scoring of tests (told that subject is bright) *Examiners must be aware that interpersonal and cognitive processes can affect objectivity in administering or scoring of tests. Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

A review of many studies suggests that an expectancy effect exists in some but not in all situationsWhat person characteristic may be implicated in expectancy effect?

It is important to pay attention to the potentially biasing effect of expectancy and to eliminate it as possible Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Effects of Reinforcing Responses Examiners should always administer tests under controlled conditionsInconsistent use of feedback can damage the reliability & validity of test scoresMost potent effects of reinforcement arise in attitudinal scales Reinforcement & feedback guide the examinee toward a preferred response* Examiners should exert strict control over the use of feedbackKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Psychological Testing in Everyday Life: Incentive Scoreboard Effects of token and social reinforcementSattler (1988) reviewed 34 studiesEvaluated the effect of incentives: tokens, praise, candy, money, and social reinforcementSubjects in the experiments are normal and handicapped children of various ethnic groups41 % (14 studies): Incentives or feedback did not affect performance38 % (13 studies): Mixed results 21 % (7 studies) : Evidence that reinforcement either improved or hindered performanceKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

One issue raised by research on incentive effects concerns what the results imply for test interpretation

What important characteristic of the test is affected?


Test Manuals Should clearly indicate directions for administrationShould be detailed to be duplicated in all situations in which the test is givenA good manual includes the exact words to be read to the subjectsInclude FAQs and how examiners should answer them*Standardized Administration: Ensures the reliability & validity of the testKaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Advantages of Computer-Assisted Test Administration Excellence of standardizationIndividually tailored sequential administrationPrecision of timing responsesRelease of human testers for other dutiesSubjects are not rushedControl of biasObjective and cost-effectiveAllow for more experimental controlReduced scoring errors*What may be the disadvantages/possible problems? Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Mode of AdministrationTests that are self administered vs test that are administered by an examiner or trained interviewer

People are more likely to show good health during interviews than self-completed questionnaires

Phone interviews yielded higher health scores than filling out questionnaires Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

ExampleTest Administration Types and Self-ReportingColon and rectal cancer screening tests Mail, phone, face-to-faceMode of administration did not make much difference; equally accurate when compared to medical records Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

In educational testing Reading test administered to K to 12 studentsComputer-administered vs paper-and-pencil testsNo significant differences between modes (Wang, Jiao, Young, Brooks, & Olson, 2008)


Mode of asking questions makes a difference (psychiatric disability)

For younger, more distress and disability were reported in questionnaires than in interviews (using the same questionnaire)



OUTLINEI. The Examiner and the ExamineeRelationship between examiner and examinee Training of Test Administrators Computer-Assisted Administration Race of the Examiner Expectancy EffectsMode of Administration Language of the Examinee (Test Taker) Effects of Reinforcing Responses Subject VariablesII. Behavioral Assessment and Methodology ReactivityExpectanciesStatistical Control of Rating ErrorsDriftDeception

Subject VariablesA serious source of errorMotivation and anxiety can greatly affect test scoresTest anxiety difficulty focusing attention on the test items and distracted by other thoughtsI am not doing well, I am running out of time (Sapp, 1999)3 components: Worry, Emotionality, &Lack of self-confidence Illness, health concerns, hormones may also affect perceptual, spatial and motor performances Kaplan, R.M., & Sacucuzzo, D.P. (2011). Psychological Testing. Cengage Learning Asia Pte. Ltd: Singapore.

Activities for Next WeekLab on Test Administration (Manual)QuizDiscussion on Interviewing Techniques (read Chapter 8; pp 182-220)


Test administration methodological issue

*Reinforcement misguides the subject

A variety of studies have demonstrate that random reinforcement destroys the accuracy of performance and decreases the motivatio to respond (Eisenberger & Cameron, 1998)

The ffects of random feedback are severe, causing depression, low motivation for responding, and inability to solve problemslearned helplessness (Abramson, Alloy, & Metalsky, 1995)

Because different test takers make different responses, one cannot ensure that the advantages resulting from reinforcement will be the same for all people as a result, most test manuals and interviewer guides insist that no feedback given*

Psychological Testing Test Administration1

Documents

Transcript of Psychological Testing Test Administration1