Eng I Fall 2010: Diagnostics & Results

David MinughEnglish Department, SU

May, 2011

Diagnostic Tests: Key aspects

• Intake hurdle or not: a basis for admission?• If students are already accepted, is the test

mandatory or not?• Costs (no “not” alternative here!)• Information to students or a placement test?• Computerized or not?• Reusable or not?

Intake Hurdle: A basis for admissions?

• If we limit intake to those with scores better than X,– We need a new test every time (or a large pool of items)– It needs to be legally acceptable (including student IDs)– What about our intake profile (broader recruitment?)– Teacher candidates will have to be treated separately– We need to have good validity – What about motivation and test-taking skills?– Time: a summer test?!– Retakes for those who were sick, abroad, etc.?– Is it relevant if we can’t afford to turn away students?– Current status: offered to those who actually register

Mandatory or not?• Our paper-based test (up to 2006) was “mandatory,”

– no sanctions for those who didn’t take it– identical from year to year (over a 30-year span)– make-ups were offered on an ad-hoc basis– dropped to save marking and room costs

• Our most recent (voluntary) net-based diagnostic test was taken by 139 of about 300 students– No way of checking which students took it– The process took several weeks

• The new, scheduled diagnostic test (Sept. 3) reached 300 of about 330 students– On their schedule (three 100-student waves), Fri 9-11 am, 11-1, 1-3 pm

Information or Placement Test?

• Any diagnostic test contains potential information for students– however, they may not know what it means– we may not know what it means, either– motivation and commitment are not easily and cheaply

measured• It could be used for group placements

– SU groups are not differentiated: 10 regular + 2 future-teacher groups

– Students choose groups by schedule or teacher or books; 11 different combinations of teachers

• Group placement implies non-recyclable tests

Costs• Test-making:

– One-time development cost for our test at SU– Higher for Q pools, much higher for new tests each semester– Oxford’s test (A1-C2 rating) runs about 20 cr/student– DiaLang (an EU project) is free, but now unsupported

• Room costs: regular exam halls v ones with computers– At SU the cost is the same (!)– 15-20 TKR, c. 3000 cr/hr for a 140-person hall + 5 computer-literate

proctors, 3 waves– IT support costs and level of support?

• Marking: – Free for computerized tests– Student/admin markers for paper-based version– Admin time is at a premium when the semester starts

Computerized or Not?

• Volume: some 300 new students per semester• Advantages:

– no mountain of papers– instant feedback– easy administration of results and student contacts– students get into the VLE system quickly– error analysis (practical or research)

• Disadvantages:– test limited to electronic marking and relatively simplistic questions;

very limited productive component– student unfamiliarity with the system– costs to install and run

Starting Points for SU• 285 new English I students took our net-based diagnostic test on Sept. 3,

2010, at the start of the Fall Semester 2010– A few others from other courses were excluded from this discussion

• The test was strictly proficiency-oriented: Grammar (60), vocabulary (60), collocations (20), phonemes (10), word stress (10)

• The maximum score was 160; actual scores ranged from 39 to 152, a bell curve peaking in the 97-128 range (see next slide)

• Similar results were obtained in the Spring 2011 diagnostic test

• Those who had 80 or less received an individual e-mail about this, but were not called to mandatory counseling

• For the results shown here, all tests have been offered twice, and all papers graded, i.e. a full semester had been completed when the results were calculated (March 2011)

Eng I Diagnostic Test HT10

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-28

129-44

145-60

0

10

20

30

40

50

60

70

N = 295

# of students

Credits Obtained (max 30) v. Diagnostic Results

0 30 60 90 120 1500

5

10

15

20

25

30

Distribution:

Credits Obtained

Credits Achieved (max 30) and Diagnostic Results

0 30 60 90 120 1500

5

10

15

20

25

30

Credits Achieved

Credits Achieved

72 lowest for 30 credits.

Credits Achieved (max 30) and Diagnostic Results

0 30 60 90 120 1500

5

10

15

20

25

30

Credits Achieved

Credits Achieved

72 lowest for 30 credits.

Voc Test 1.5 credits

Note the high-scoringnon-achievers and dropouts!

Percentage Course Dropouts per Diagnostic Results (n = 25)

0-16 17-32

33-48

49-64

65-80

81-96

97-112

113-128

129-144

145-160

0%10%20%30%40%50%60%70%80%

Diagnostic Results

Percent who dropped

The red zone

Course Drop-outs/Finishers per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0

10

20

30

40

50

60

70

80

FinishersDrop-outs

Course Completers/Partials/Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0

10

20

30

40

50

60

70

80

CompletersPartialsDrop-outs

Percentage Course Completers/Partials/ Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0%

20%

40%

60%

80%

100%

CompletersPartialsDrop-outs

Literature (12 cr): Completers/Partials/Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0

10

20

30

40

50

60

70

80

CompletersPartialsDrop-Outs

% Literature (12 cr): Completers/Partials/Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0%

20%

40%

60%

80%

100%

CompletersPartialsDrop-Outs

Linguistic Analysis (6cr): Completers/Partials/Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0

10

20

30

40

50

60

70

80

CompletersPartialsDrop-Outs

% Linguistic Analysis (6 cr): Completers/Partials/ Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0%

20%

40%

60%

80%

100%

CompletersPartialsDrop-Outs

Linguistics Test (6 cr): Completers/Partials/Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0

10

20

30

40

50

60

70

80

CompletersPartialsDrop-Outs

% Linguistics Test (6 cr): Completers/Partials/Drop-outs per Diagnostic Results

0-16 17-32 33-48 49-64 65-80 81-96 97-112

113-128

129-144

145-160

0%

20%

40%

60%

80%

100%

CompletersPartialsDrop-Outs

Eng I Final Grade v Diagnostic Score

60 70 80 90 100 110 120 130 140 150 1600

1

2

3

4

5

Grade (A=5, B=4, C=3, D=2, E=1)

Grade

5%

30%

50%

15%

0%

Tentative Conclusions• Proficiency alone will not guarantee success

• Drop-outs (25 10%): – Half had scored < 80 on the diagnostic test– Almost 30% had scored above 100– Another 18 took 0 credits (40% of them scored < 80)

• Results:– No student scoring under 72 finished within one semester

– The first linguistic test (lectures based on an introductory linguistics textbook (Yule, The Study of Language) is the biggest stumbling block

– Quite a few lack only one or two components

– Very few A grades were granted, and no Es (this may change as further make-up papers and spring retakes come in)

– Both the literary and the linguistic course seem accessible even to quite weak students (60% in the 65-80 range passed the literary and/or linguistic course)

– The diagnostic test is not a strong predictor for most students, but is relatively reliable at the lower and upper ends of the scale

– The tipping point (50% chance of succeeding within a semester) appears to be around 100 on the diagnostic test