PTEA Preliminary Estimates of Concordance 4Aug10 v2

8/6/2019 PTEA Preliminary Estimates of Concordance 4Aug10 v2

http://slidepdf.com/reader/full/ptea-preliminary-estimates-of-concordance-4aug10-v2 1/7www.pearsonpte.com

Preliminary Estimates of Concordance

between Pearson Test of English Academicand Other Measures of English

Language Competencies

Introduction

This document outlines the research carried out to produce preliminary concordance between Pearson Test o English Academic(PTE Academic) and other measures o English language, including the Common European Framework (CEF or CEFR, Council o

Europe, 2001 ), TOEFL® iBT and IELTS™.

Tests comparisons using feld test data

PTE Academic has been eld tested using over 10,400 test takers. Field testing took place in 2007 and 2008. Test takers were

representative o the global population o students seeking admission to universities and other tertiary education institutions

where English is the language o instruction. Test takers were born in158 dierent countries and spoke 126 dierent languages.

During the eld tests several sets o secondary data were collected. Among these were ratings or all test takers on descriptive

scales published by the Council o Europe ( 2001 ). In addition, a number o test takers reported their scores on other tests o

English, including TOEIC, TOEFL PBT, TOEFL CBT, TOEFL iBT and IELTS.

A limited number o the sel-reported data were invalid as the reported scores were outside the possible score range or the

particular test. A small number o the test takers also submitted copies o their ocial score reports on the tests or which they ha

provided sel-reported data. Table1 ( p. 2) shows the ollowing or each test: the numbers o sel-reported data, how many o thes

were valid, the mean sel-reported scores, the number o ocial score reports sent in, the mean ocial scores and the correlation

with the PTE Academic eld test scores. All correlations are signicant at p<.011.

From Table 1 it can be concluded that the sel-reported scores are in general, quite accurate. Indeed, the correlation between the

sel-reported results and the ocial score reports was .82 or TOEFL iBT and .89 or IELTS. This nding is in agreement with earlier

research on sel-reported data. For example, Cassady ( 2001 ) ound students’ sel-reported Grade Point Average (GPA) scores to be

‘remarkably similar’ to ocial records. The data are also consistent. According to ETS ( 2005, p. 7 ) the score range 75–95 on TOEFL

iBT is comparable to the score range 213–240 on TOEFL CBT and to the score range 550–587 on TOEFL PBT. The mean sel-reportescores in Table 1 or these three tests are thereore comparable.

1 Signicant at p<.01 means there is less than 1% chance to observe this correlation i the measures are not related.

© Copyright 2010. All rights reserved; no part of this publication may be reproduced without the prior written permission of Edexcel Limited (a Pearson company).

July 2010 Version 2.



Table 1: PTE Academic eld tests: test takers on other tests o English

Sel-reported data Ofcial score report

n Total

328

96

110

144

2436

n Valid

327

92

107

140

2432

Mean

831.5

572.3

240.5

92.9

6.49

Correlation

0.76

0.64

0.46

0.75

0.76

Test

TOEIC

TOEFL PBT

TOEFL CBT

TOEFL iBT

IELTS

n

no data

no data

no data

19

169

Mean

-

-

-

92.1

6.61

Correlation

-

-

-

0.95

0.73

In addition, according to ETS ( 2001, p. 3 ) a score range o 800–850 on TOEIC corresponds to a score range o 569–588 on TOEFL

PBT, which makes the sel-reported TOEIC mean score o the test takers on the PTE Academic eld test also all in line with data

published by ETS.Based on the data presented in Table 1, concordance between PTE Academic and other tests o English can be estimated,

taking into account a less than optimal eort o test takers during eld testing where test results have no direct relevance to the

test takers.

Relation to the Common European Framework

The preliminary relation o the PTE Academic score scale with the descriptive scale o the Common European Framework or

Languages (CEF) is based on both an item-centered and a test taker-centered method. For the item-centered method the

CEF level o all items was estimated by item writers, reviewed and, i necessary, adapted in the item-reviewing process. For

the test taker-centered method, three extended responses (one written and two spoken) per test taker were each rated by two

independent, trained raters. (On disagreement between the two independent raters, a third rating was gathered and the twoclosest ratings were retained). A dataset o over 26,000 ratings (by test takers sel-reporting, by items, by raters) on up to 100

dierent items was analyzed using the computer program FACETS (Linacre, 1988; 2005 ). Estimates o the lower boundaries o the

CEF levels, based on the item-centered method, correlated at.996 with those based on the test taker-centered method.

Validity check using BETA testing data

In addition to the initial eld testing o 10,400 students during 2007–08, a urther 364 test takers participated in the 2009 BETA

testing o PTE Academic. The concordance between the score scale o PTE Academic and the score scales o TOEFL iBT and IELTS,

each estimated rom the eld test data, were used as predictors o TOEFL iBT and IELTS scores o test takers participating in BETA

testing. Test takers provided sel-reported scores and a smaller, partially overlapping number o test takers sent in

copies o their ocial score reports.



Table 2 shows the mean scores as sel-reported and rom the ocial score reports, the mean scores or the same test takers as

predicted rom their PTE Academic score and the correlations between the reported scores and the predictions rom PTE Academic

All correlations are signicant at p<.011.

It can be concluded that this preliminary concordance produces airly accurate and coherent predictions.

Table 2: PTE Academic BETA: test takers on other tests o English

Sel-reported data Ofcial score report

n

42

57

Mean

98.9

6.80

Predicted

97.3

6.75

Correlation

0.75

0.73

Test

TOEFL iBT

IELTS

n

13

15

Mean

92.2

6.60

Correlation

0.77

0.83

Predicted

98.2

6.51

Concordance o PTE Academic with other measures o English

Based on the research described, Pearson has produced preliminary concordance tables. The rst table (pp. 4–5 ) shows the

relationship between scores on TOEFL iBT and PTE Academic. The second table (pp. 6–7 ) shows the relationship between scores o

IELTS and PTE Academic. Both tables also show Pearson’s current best estimate o the relationship between these scores and the

CEF. Pearson will conduct more research in collaboration with independent scholars to increase the evidence or the validity and

accuracy o these concordance tables.

Caution while interpreting concordance tables

It must be noted that any individual level prediction o a score on a particular test based on the score observed on another test

will contain measurement error, caused by error in each o the tests in the comparison and in the estimate o the concordance.

Furthermore, tests in the comparison do not measure exactly the same construct.

For more inormation and to contact us, visit our website at www.pearsonpte.com

Reerences

Cassady, Jerrell C. ( 2001 ) Sel-Reported GPA and SAT Scores. ERIC Digest. ERIC Identier: ED458216.

Council o Europe ( 2001 ) Common European Framework o Reerence or Languages: Learning, Teaching, Assessment.

Cambridge: CUP.

ETS ( 2001 ) TOEFL Institutional Testing Program (ITP) and TOEIC Institutional Program (IP): Two On-Site Testing Tools rom ETS

at a Glance. Handout Berlin Conerence 2001. Princeton: Educational Testing Service.

ETS ( 2005 ) TOEFL ® Internet-based test: Score comparison tables. Princeton: Educational Testing Service.

Linacre, J.M ( 1988; 2005 ) A Computer Program or the Analysis o Multi-Faceted Data. Chicago, IL: Mesa Press.



Preliminary estimates o concordance between PTE Academic, TOEFL iBT and the descriptive scaleo the Common European Framework o Reerence or Languages

TOEFL

iBTScore

120

119

118

117

115 – 116

114

113

112

110 – 111

109

107 – 108106105

103 – 104102

10199 – 100

98

9795 – 96

9493

91 – 929089

87 – 88

PTE

AcademicScore

90

89

88

87

86

85

84

83

82

81

80

79

78

77

76

7574

73

7271

70

6968

6766

65

6463

62

6160

59

Common

EuropeanFramework

C2

C1

B2

Level Descriptor

© Council o Europe

Can understand with ease virtuallyeverything heard or read. Cansummarize inormation romdierent spoken and written sources,reconstructing arguments andaccounts in a coherent presentation.Can express him/hersel spontaneously,very fuently and precisely,dierentiating ner shades o meaningeven in more complex situations.

Can understand a wide range o demanding, longer texts, and recognizeimplicit meaning. Can express him/hersel fuently and spontaneouslywithout much obvious searchingor expressions. Can use languagefexibly and eectively or social,academic and proessional purposes.Can produce clear, well-structured,detailed text on complex subjects,showing controlled use o organizational patterns, connectors

and cohesive devices.

Can understand the main ideas o complex text on both concrete andabstract topics, including technicaldiscussions in his/her eld o specialization. Can interact with adegree o fuency and spontaneity thatmakes regular interaction with nativespeakers quite possible withoutstrain or either party. Can produceclear, detailed text on a wide range o subjects and explain a viewpoint ona topical issue giving the advantages

and disadvantages o various options.

What does this mean or a score user?

C2 is a highly procient level and astudent at this level would be extremelycomortable engaging in academicactivities at all levels.

C1 is a level at which a studentcan comortably participate in allpostgraduate activities, includingteaching. It is not required or studentsentering university at undergraduatelevel. Most international students whoenter university at a B2 level wouldacquire a level close to, or at, C1 aterliving in the country or several yearsand actively participating in alllanguage activities encounteredat university.

B2 was designed as the level requiredto participate independently in higherlevel language interaction. It is typicallythe level required to be able to ollowacademic level instruction and toparticipate in academic education,including both coursework andstudent lie.

© The copyright o the descriptive scales reproduced in this document belongs to the Council o Europe.



TOEFLiBT

Score

8685

83 – 84

82

81

79 – 80

78

76 – 77

74 – 75

72 – 73

70 – 71

67 – 69

65 – 66

63 – 64

60 – 62

57 – 59

54 – 56

52 – 53

48 – 51

45 – 47

40 – 44

PTE

AcademicScore

5857

56

55

54

53

52

51

50

49

48

47

46

45

44

43

42

41

40

39

38

35

34

3332

31

30

29

10

CommonEuropean

Framework

B1

A2

A1 orbelow

Level Descriptor© Council o Europe

Can understand the main points o clear standard input on amiliarmatters regularly encountered inwork, school, leisure, etc. Can dealwith most situations likely to arisewhilst in an area where the languageis spoken. Can produce simpleconnected text on topics, which areamiliar, or o personal interest. Candescribe experiences and events,dreams, hopes and ambitions andbriefy give reasons and explanationsor opinions and plans.

Can understand sentences andrequently used expressions relatedto areas o most immediate relevance(e.g. very basic personal and amilyinormation, shopping, local geography,employment). Can communicate insimple and routine tasks requiringa simple and direct exchange o

inormation on amiliar and routinematters. Can describe in simple termsaspects o his/her background,immediate environment and mattersin areas o immediate need.

Can understand and use amiliareveryday expressions and very basicphrases aimed at the satisaction o needs o a concrete type. Canintroduce him/hersel and othersand can ask and answer questionsabout personal details such as where

he/she lives, people he/she knowsand things he/she has. Can interactin a simple way provided the otherperson talks slowly and clearly and isprepared to help.


B1 is insucient or ull academic levelparticipation in language activities. Astudent at this level could ‘get by’ ineveryday situations independently.To be successul in communication inuniversity settings, additional Englishlanguage courses are required.

A2 is an insucient level or academiclevel participation.


N o

d a t a

a v a i l a

b l e



IELTS

Score

9.0

8.5

8.0

7.5

7.0

6.5

PTE

AcademicScore

90

89

88

87

86

85

84

83

82

81

80

79

78

77

76

7574737271706968676665

646362616059

Common

EuropeanFramework

C2

C1

B2

Level Descriptor

© Council o Europe

Can understand with ease virtuallyeverything heard or read. Cansummarize inormation romdierent spoken and written sources,reconstructing arguments andaccounts in a coherent presentation.Can express him/hersel spontaneously,very fuently and precisely, dierentiatingner shades o meaning even in morecomplex situations.

Can understand a wide range o demanding, longer texts, andrecognize implicit meaning. Canexpress him/hersel fuently andspontaneously without much obvioussearching or expressions. Can uselanguage fexibly and eectively orsocial, academic and proessionalpurposes. Can produce clear,well-structured, detailed text on complexsubjects, showing controlled use o organizational patterns, connectorsand cohesive devices.

Can understand the main ideas o complex text on both concrete andabstract topics, including technicaldiscussions in his/her eld o specialization. Can interact with adegree o fuency and spontaneity thatmakes regular interaction with nativespeakers quite possible withoutstrain or either party. Can produceclear, detailed text on a wide range o subjects and explain a viewpoint on

a topical issue giving the advantagesand disadvantages o various options.


C2 is a highly procient level and astudent at this level would be extremelycomortable engaging in academicactivities at all levels.

C1 is a level at which a studentcan comortably participate in allpostgraduate activities, includingteaching. It is not required or studentsentering university at undergraduatelevel. Most international students whoenter university at a B2 level wouldacquire a level close to, or at, C1 aterliving in the country or several yearsand actively participating in all languageactivities encountered at university.

B2 was designed as the level requiredto participate independently in higherlevel language interaction. It is typicallythe level required to be able to ollowacademic level instruction and toparticipate in academic education,including both coursework andstudent lie.

Preliminary estimates o concordance between PTE Academic, IELTS and the descriptive scale o theCommon European Framework o Reerence or Languages



IELTSScore

6.5

6.0

5.5

5.0

4.5

PTE

AcademicScore

5857

56

55

54

53

52

51

50

49

48

47

4645

44

43

42

41

40

39

38

37

36

3534

33

32

31

30

29

10

CommonEuropean

Framework

B1

A2

A1 or

below

Level Descriptor© Council o Europe

Can understand the main points o clear standard input on amiliar mattersregularly encountered in work, school,leisure, etc. Can deal with mostsituations likely to arise whilst in anarea where the language is spoken.Can produce simple connected texton topics, which are amiliar, oro personal interest. Can describeexperiences and events, dreams,hopes and ambitions and briefy givereasons and explanations or opinionsand plans.

Can understand sentences andrequently used expressions relatedto areas o most immediate relevance(e.g. very basic personal and amilyinormation, shopping, local geography,employment). Can communicate insimple and routine tasks requiringa simple and direct exchange o

inormation on amiliar and routinematters. Can describe in simpleterms aspects o his/her background,immediate environment and mattersin areas o immediate need.

Can understand and use amiliareveryday expressions and very basicphrases aimed at the satisaction o needs o a concrete type. Can introducehim/hersel and others and can askand answer questions about personaldetails such as where he/she lives,people he/she knows and things he/she has. Can interact in a simple wayprovided the other person talks slowlyand clearly and is prepared to help.


B1 is insucient or ull academic levelparticipation in language activities. Astudent at this level could ‘get by’ ineveryday situations independently.To be successul in communication inuniversity settings, additional Englishlanguage courses are required.

A2 is an insucient level or academiclevel participation


N o

d a t a

a v a i l a

b l e

PTEA Preliminary Estimates of Concordance 4Aug10 v2

Documents

Transcript of PTEA Preliminary Estimates of Concordance 4Aug10 v2