Users’ reactions to innovative computerised feedback – the case of DIALANG
Ari HuhtaCentre for Applied Language Studies
University of JyväskyläFinland
What is DIALANG? computerized diagnostic language
assessment system in 14 languages (www.dialang.org)
feedback (& activities) that help learners to diagnose their strengths and weaknesses plan further language learning raise awareness
unusually wide range of feedback for a test; partly unique / innovative, e.g. self-assessment
What is ’feedback’? Why feedback?
In educational contexts FB can be: ”… any communication or procedure to inform a
learner of the accuracy of a response, usually to an instructional question” (Knowledge of the Correct Response)
but also ”… other information such as precision, timeliness, learning guidance, motivational messages, ... critical comparisons, and learning focus”
”… any message or display that a computer presents to the learner after a response” (Mory 2004)
improve performance & learning (directly) but also to motivate, make learning more interesting,
encourage different learning, raise awareness, …
Why study (DIALANG) feedback?
feedback is an integral part of all behaviour, performance and learning social psychology, organizational theory,
education ...
feedback from tests is not much researched (cf. teachers as assessors)
feedback from computerized (language) tests is particularly little researched
DIALANG FEEDBACK
(test result)Level
VSPTfeedback
Se lf-a ssmfe e d b a c k
Immediateitem review
Post-te stite m re vie w+ sub skill
Extendedlevel descriptions
Advice
Infoabout SA
Research topics / questions
This study: What are users’ reactions to different types of
feedback in DIALANG? How do new types of feedback compare with more traditional feedback?
Are users’ reactions associated with their background characteristics (e.g. sex, age, language use, language proficiency)? How might possible associations be explained?
--------------------------------------------------------
related, more in-depth studies users’ reactions to self-assessment and Vocabulary Size Placement Test
Research methods & data
Quantitative & qualitative (cross-sectional)
Questionnaire 553 respondents in two countries mostly non-language majors at universities
(typical users)
Interviews of users and their teachers Written reports from users & teachers
Users’ ’reactions’?
Defined in terms of 1) Reading / using particular
feedback? 2) Interest in particular feedback? 3) Perceived usefulness of particular
feedback? 4) Other reactions (e.g. ease of use)
Likert scale + open-ended questions cf. Technology Acceptance Model
Who were the respondents?
0
20
40
60
80
%
female male
Gender of respondents
Age of respondents
0
10
20
30
40
50
60
70
80
Percent
under 18 18 - 25 26 - 35 36 - 45 46 - 55
Age
Age of respondents
# of languages & tests taken
0
5
10
15
20
25
30
%
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Number of tests
Number of tests taken
0102030405060708090
Percent
1 2 3 4
Number of languages
Number of language taken
Most common test languages
0
10
20
30
40
50
%
English French Swedish German Finnish
Most common test languages
Results: Overall reaction to FB- use & interest & usefulness
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to Level (test result)
0
10
20
30
40
50
60
%
Clearlynegative
on average
Somewhatnegative
on average
Somewhatpositive on
average
Verypositive on
average
Level (interest & usefulness)
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to Level (test result)
the most popular part of feedback the fact that the meaning of A1 etc
was defined was appreciated associations with user background:
correlated with proficiency: higher proficiency – more positive reactions
men were less interested in this than women
0
10
20
30
40
50
%
Clearlynegative on
average
Somewhatnegative on
average
Somewhatposivite on
average
Verypositive on
average
VSPT feedback
Reactions to Vocabulary Size Placement Test feedback / result
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to Vocabular Size Placement Test & its feedback
VSPT and its result was the most controversial part of DIALANG although, overall, the reactions were positive
strong affective reactions, opinions rather divided “Interesting, I had never thought of a way like this to
test language proficiency. Fun to do!” “Absolute nonsense! This can’t be my level!”
background: in general, those who got low VSPT results regarded it very negatively, whereas those who got good results regarded it very positively in particular, women at level C1/C2 liked VSPT
feedback
explanation: VSPT scoring is affected by test-taking strategy and produces very low scores too easily
0
10
20
30
40
50
%
Clearlynegative
onaverage
Somewhatnegative
onaverage
Somewhatpositive on
average
Verypositive on
average
Immediate item review
0
10
20
30
40
50
%
Clearlynegative
onaverage
Somewhatnegative
onaverage
Somewhatpositive on
average
Verypositive on
average
Post-test item review
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to immediate item review
almost 60% preferred immediate review, 30% post-test review, 10% preference
Why like: want to know immediately, remember better if see the result right away, want to learn (during test)
Why don’t like: disturbs test-taking, can be depressing background:
beginners liked it more than advanced learners in particular, C-level (C2) learners considered it less useful
and interesting C-level men used it particularly infrequently
possible explanation: many beginners & intermediate learners want to learn
from tests?? advanced learners make so few mistakes that they may
feel immediate feedback is waste of time??
Reactions to post-test item review
less popular than immediate review but a fair number of users absolutely want to have it
Background: more experienced users (# of languages used
& length of language studies) considered this more interesting & useful
possible explanation (very tentative): more experienced language learners may find
information about subskills interesting??
Reactions to self-assessment feedback (comparison of SA and test result)
0
10
20
30
40
50
%
Clearlynegative
onaverage
Somewhatnegative
onaverage
Somewhatpositive on
average
Verypositive on
average
Self-assessment feedback
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to self-assessment feedback
according to users, SA is important part of (a system like) DIALANG, as ‘popular’ as e.g. item review (joint second)
background: men had more often very negative opinions
compared with women the youngest group (under 18) found it less
interesting than the older groups possible explanation:
younger learners may not (yet) be so interested in the kind of reflection on one’s skills involved in self-assessment??
Reactions to self-assm. information (why SA and test result may not match)
0
10
20
30
40
50
%
Clearlynegative on
average
Somewhatnegative on
average
Somewhatpositive on
average
Verypositive on
average
Information about SA
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to SA information
the least used / interesting / useful part of feedback Background: negative reactions were associated with
higher proficiency, especially C1/C2 women more experience in lg learning (more lgs studied) male respondents (more very negative opinions) young age (under 18)
positive reactions were associated with: lower proficiency, especially A1/A2 men less experience in lg learning (fewer lgs studied)
men were divided in their opinions of SA information possible explanations:
if SA and test agreed, no need to read this beginners have less experience in what is described in SA
information and may find it new & potentially useful; advanced learners may feel they already know all that
Reactions to extended descriptions
0
10
20
30
40
50
%
Clearlynegative
on average
Somewhatnegative
on average
Somewhatpositive on
average
Verypositive on
average
Extended level descriptions
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to extended descriptions
fairly interesting, useful and used but less so than e.g. item review
background: men made more often very negative
evaluations of this feedback than women
Reactions to advice
0
10
20
30
40
50
%
Clearlynegative
onaverage
Somewhatnegative
onaverage
Somewhatpositive on
average
Verypositive on
average
Advice
VSPT feedback
Advice
Extended descriptions
SA information
Self-assessm. FB
Post-test item review
Immediate item review
Level
0,00 0,50 1,00 1,50 2,00 2,50 3,00
Mean
Reactions to advice
quite popular but not among all users – surprise?
Background: high proficiency (C1/C2) & more experience more
negative views of advice men had more very negative views than women A1 / A2 learners were divided: they often chose it as
one of the 3 most useful types of FB but many also chose it as one of the least useful types of FB
possible explanation: C-level learners no longer need advice, whereas A-
level learners do, but some of them may feel the particular advice in DIALANG is not useful??
Implications / questions
implications for DIALANG and/or other such computerized systems
users clearly like to have feedback about their self-assessment younger learners may need to be
motivated first good to have two types of item
review: immediate and post-test, based on choice preferences differ; level of proficiency
Implications / questions
VSPT divides opinions: love – hate scoring needs revising
advice appears useful for beginners in particular challenges”I don’t bother to read the advice because I know I’ll
do things my way anyway””... you don’t remember them any more when you
study”
longitudinal studies how best integrate systems like DIALANG into
teaching (long-term) effectiveness of feedback?
Top Related