EARLI 11th Biennial conference 2005 - Development of edumetrical indices for the analysis of the...

EARLI 11th Biennial conference 2005 - Development of edumetrical indices for the analysis of the spectral quality of higher education standardized tests – Jean-Luc Gilles 1

EARLI 11th Biennial conference 2005

Development of edumetrical indices for the analysis of the spectral quality of higher

education standardized tests

Jean-Luc Gilles

http://www.smart.ulg.ac.be/earli-2005/spectral-analysis.ppt

Système Méthodologique d’Aide à la Réalisation de Tests University of Liège - Belgium


Example of Multiple Choice Question:(Hansen & al, 2000)

2,8

cm

3,9 cm

0,6

cm

1,5 cm

This ultrasound of the left ovary of a 6 years old cow, has been realized one week after the oestrus.Diagnostic :

1. a follicle of de Graaf2. a follicular cyst3. a yellow body4. a cavitary yellow body5. an artifact6. none7. all

With an optical mark reading device it is possible to read automatically answers and confidence degrees of hundreds of students:

Introduction:Introduction:

If you consider you have a probability of correct answer of

0 %

20 %

40 %

60 %

80 %

100 %

0

1

2

3

4

5

Check

Example of confidence degrees scale (Leclercq & Gilles, 2000): The student is invited to accom-pany his choice by the percen-tage of chances that his answer will be correct.


Without confidence degrees, the selected solution of a MCQ is either correct or incorrect.

incorrect correctThe answer is:

Knowledge Ignorance Knowledge

dangerous partial perfectmistaken

With the confidence degrees technique it is possible to deliver more subtle feedback on each person's spectral performances!

(Jans & Leclercq, 1999)

Introduction:Introduction:

100% 100%0%60% 40% 40% 60%

5 4 3 2 1 0 543210Confidence degrees

Confidence degrees

With the confidence degrees technique it is possible to bypass the "binary" character of students assessment performance and to consider a more spectral conception of a person’s performances

The worstsituation

Assuredperformance


Confidence degrees of Confidence degrees of the students who the students who

answer incorrectlyanswer incorrectly

Confidence Confidence degrees of the degrees of the students who students who

answer correctlyanswer correctly

Starting intuition for the new indices used for highlighting problematic Starting intuition for the new indices used for highlighting problematic MCQs and, within those, the proposals which contain anomalies:MCQs and, within those, the proposals which contain anomalies:

2,8

cm

3,9 cm

0,6

cm

1,5 cm



2,8

cm

3,9 cm

0,6

cm

1,5 cm

2,8

cm

3,9 cm

0,6

cm

1,5 cm

2,8

cm

3,9 cm

0,6

cm

1,5 cm



Spectral quality

of the question

We have exploited the confidence degrees provided by the students to provide spectral information on the quality of the questions (as opposed to information on the quality of students’ performances).


Our starting intuition is as follows:

logically the students who answer a question correctly (here in green) should provide percentages of certainty higher than the students who answer incorrectly (in red)






0

1

2

3

4

5

0

1

2

3

4

5

0

1

2

3

4

5

Spectral coherenceSpectral coherence

Spectral inconsistencySpectral inconsistency

0

1

2

3

4

5


How can we calculatespectral coherence ?

We propose to distinguish the case of the spectral coherence of a correct answer and the case

of spectral coherence of an incorrect solution…


An example of MCQ ...

Complete:

He ……….. a heart attack while he ……….. for the bus.

1. was having / was waiting 2. had / waited 3. had / was waiting4. was having / waited

Correct answer

Spectral contrasted point biserial correlation for the correct solution:Spectral contrasted point biserial correlation for the correct solution:

pbr CSC

Student 1

Student 2

Student 3

Student 4Student 5

Student 6

Student 8

Student 7

Student 9

Student 10

Student 11

Student 12

Student 14

Student 13

Student 15

Student 16

Student 17

Student 19

Student 18

Student 20

1. w

as h

avin

g / w

as w

aitin

g

S1

2. h

ad /

wai

ted

S23.

had

/ w

as w

aitin

g

S3

4. w

as h

avin

g /

wai

ted

S4OMConfidence Degrees (CD) of the students.

CD


Complete:




pbr CSC

Confidence Degrees (CD) of the students.

CD

3. h

ad /

was

wai

ting

S3

Student 1

Student 2

Student 3

Student 4Student 5

Student 6

Student 8

Student 7

Student 9

Student 10

Student 11

Student 12

Student 14

Student 13

Student 15

Student 16

Student 17

Student 19

Student 18

Student 20

Correct answer



0

1

2

3

4

5

Which are the confidence degrees of the students who chose the correct solution “S3”?


Complete:




pbr CSC


CD

3. h

ad /

was

wai

ting

S3

Student 1

Student 2

Student 3

Student 4Student 5

Student 6

Student 8

Student 7

Student 9

Student 10

Student 11

Student 12

Student 14

Student 13

Student 15

Student 16

Student 17

Student 19

Student 18

Student 20

Correct answer



0

1

2

3

4

5

0

1

2

3

4

5



Which are the confidence degrees of the students who answer incorrectly?



pbr CSC

pqDaDx

SCCC

C

pbr

with

DxC = the average of the confidence Degrees of the students who chose the correct solution

DaC = the average of the confidence Degrees of the other students who did not choose the correct solution

= the standard deviation of the confidence degrees of all the students

p = the proportion of students who chose the correct solution of the question

q = the proportion of students who did not choose the correct solution of the question (q = 1 - p)


Complete:




pbr CSC


CD

3. h

ad /

was

wai

ting

S3

Student 1

Student 2

Student 3

Student 4Student 5

Student 6

Student 8

Student 7

Student 9

Student 10

Student 11

Student 12

Student 14

Student 13

Student 15

Student 16

Student 17

Student 19

Student 18

Student 20

Correct answer

rpb SCC for the correct answer

« S3 » = 0,76

0 1The results obtained for the correct solution “S3” are coherent from a

spectral point of view

-1

Dxc Dac

pqrpb SCC =

By applying the formula:


and for the measurement of

the spectral coherence of an incorrect

solution ?

It is here that we propose

a Contrasted treatment!


Inco

rrec

t so

lutio

n

S1 CD

Student 2

Student 3

Student 4

Student 6

Student 9

Student 12

Student 14

Student 13

Student 19

Student 20

Spectral contrasted point biserial correlation for an incorrect solution:Spectral contrasted point biserial correlation for an incorrect solution:

pbr ISC Contrasted TreatmentContrasted Treatment=

Elimination of the « background noise »

generated by the data of the other incorrect answers

After contrasted treatment, we calculate the point biserial correlation using the columns « S1 » and « CD »:

rpb SCI of « S1 »

= -0.83

0 1-1

Corr

ect so

lutio

n

S3

011

1

1

10

0

01


The turbo analysis?


S3

Student 3

Student 4

Student 5

Student 6

Student 8

Student 7

Student 9

Student 10

Student 11

Student 12

Student 14

Student 13

Student 15

Student 16Student 17

Student 19

Student 18

Student 20

Student 1

Student 2

Complete:



We calculate the index of

Realism (Rs) for each studentREALISM (Rs)

Correct answer

Very good realism!Very bad realism!

CD

Turbo analysis principle and realism calculation for the Spectral Turbo analysis principle and realism calculation for the Spectral

Contrasted point biserial correlation (rContrasted point biserial correlation (r pbpb SCT):SCT):


REALISM (Rs)

Student 3

Student 4

Student 5

Student 6

Student 8

Student 7

Student 9

Student 10

Student 11

Student 12

Student 14

Student 13

Student 15

Student 16Student 17

Student 19

Student 18

Student 20

Student 1

Student 2

CD

S3

Elimination of the data of each student who obtain a realism

lower than 80...

…Calculation of the point biserial Spectral Contrasted correlation

0 1-1

r pb SCT80 of « S3 » = 0,94

Turbo analysis principle and realism calculation for the Spectral Turbo analysis principle and realism calculation for the Spectral

Contrasted point biserial correlation (rContrasted point biserial correlation (r pbpb SCT):SCT):Turbo analysisTurbo analysis = Increasing the reliability of information related to the spectral indices by restricting the data to the students who make

less errors in their self-assessments


Are these new spectral indices

tested in full-size conditions?


Full-size testFull-size test

We test these new Spectral Contrasted point biserial

correlations indices with the data of several thousands of

answers and confidence degrees collected during the ten

standardized tests of the MOHICAN project supported by the

French speaking community of Belgium.

The aim of the MOHICAN project was to gather reliable data

on the knowledge of the students entering university.

(Leclercq & al, 2003)


• Vocabulary (45)

• Syntax and logical structuring of ideas (12)

• Understanding of texts (6)

• Reading of maps, graphics and tables in geography (10)

• Mathematics (22)

• Physics (10)

• Chemistry (8)

• Biology (10)

• General knowledge in arts and culture (25)

• General knowledge in History and Socio-economy (25)

Item bank of the “MOHICAN” project :Item bank of the “MOHICAN” project :


The academic institutions who participated:

• the Catholic University of Louvain (UCL);

• the Free University of Brussels (ULB),

• the University of Liège (ULg);

• the Academic Faculties Notre-Dame de la Paix in Namur;

• the University of Mons-Hainaut (UMH);

• the Catholic Academic Faculties of Mons (FUCAM);

• the Saint-Louis Academic Faculties in Brussels (FUSL);

• the Polytechnics Faculties of Mons (FPM) and the Agronomics

• the Academic Faculties of Gembloux (FUSAG).

Population:Population:


Population:Population:

,


What are the results of

the full-size test of these new spectral

indices?


Full-size test results: Full-size test results: The sizes of the populations at 10 turbo analysis levelsThe sizes of the populations at 10 turbo analysis levels

10 t

urbo

ana

lysi

s le

vels


Full-size test resultsFull-size test results: :

Comparison of the rComparison of the rpb pb , r, rpbpb SC and rSC and rpbpb SCT80 values obtained by a SCT80 values obtained by a

particularly discriminating questionparticularly discriminating question


Full-size test results: Full-size test results:

Comparison of the rComparison of the rpb pb , r, rpbpb SC and rSC and rpbpb SCT80 values obtained by a SCT80 values obtained by a

particularly problematic questionparticularly problematic question


Does spectral analysis

better highlight the problems

than the classical analysis?


CONCLUSIONS:CONCLUSIONS:

Summary of the full-test results:

• The 10 MOHICAN tests = 173 MCQs• 16 MCQs presents abnormal values for the classical and/or the spectral correlations• The experts point 5 MCQs with real problems

Concerning the DETECTION of the 5 MCQs highlighted by the experts:

Classical correlation (r pb) Detects 2 problematic MCQs out of 5.

New spectral correlation (r pb SC) Detects all the 5 MCQs highlighted by the experts.

New spectral correlation Detects 2 problematic MCQs out of 5.

with turbo analysis (r pb SCT80)

Concerning the FALSE ALARMS (detection of problem whereas the experts do not detect a particular anomaly):

Classical correlation (r pb) Activates 10 false alarms.

New spectral correlation (r pb SC) Activates 7 false alarms.

New spectral correlation Activates 1 false alarm.

with turbo analysis (r pb SCT80)


CONCLUSIONS:CONCLUSIONS:

Qualitativeanalysis:

posterior opinions of the experts

Classical analysis of

discrimination:rpb

Spectral analysis:

rpb SC & rpb SCT

From the point of view of

« less false alarms »: rpb SCT80 is more effective

rpb SC is more effective than the classical rpb

22

From the point of view of « detection »: rpb SC is more effective

11

Spectral analysis is « test free » 33


Thank you for your attention!

For more information concerning

the spectral quality analysis

of standardized tests,

do not hesitate to contact me:

[email protected]

http://www.smart.ulg.ac.be/earli-2005/spectral-analysis.ppt

EARLI 11th Biennial conference 2005 - Development of edumetrical indices for the analysis of the...

Documents

Transcript of EARLI 11th Biennial conference 2005 - Development of edumetrical indices for the analysis of the...