15 March 2005
Introducing Diagnostic Assessment at UNISA
Carol Bohlmann
Department Mathematical Sciences
Reading Intervention Results Phase I
Overall reading scores < 60% => unlikely to pass maths exam
High reading scores do not guarantee mathematical success.
Low reading score a barrier to effective mathematical performance.
Results – Phase II
Reading scores improved by 10% (45% to 56%); but still < 60%.
Reading skills and mathematics exam results considered in relation to matric English and matric first language (African language) results, and to mean mathematics assignments.
Strongest correlation was between pretest reading scores and mathematics exam mark; highly significant.
Results – Phase III (2002)
Reading speed and comprehension
test data (n = 78) (voluntary
submission) Mean reading speed 128wpm (lowest
14 wpm - 11% in final exam) Mean comprehension score approx.
70%
Results – Phase III (2003)
Reading speed and comprehension
test data (n = 1 345) (1st assignment) Mean reading speed 115 wpm
(many < 20 wpm) Mean comprehension score approx. 70% Reading skill (total 54): anaphoric reference
(17), logical relations, academic vocabulary (in relation to maths), visual literacy
High correlation between anaphor score and total reading score (0,830)
Moderately high correlation between comprehension and total reading score (t = 0,581; t2 = 0,338)
Total reading score correlated more strongly with exam mark than did other aspects of reading (t = 0,455; t2 = 0,207)
Attrition rate: 27% in 1997 to 64% in 2004 –effect of poor reading skills for drop-outs?
Correlations
A longer intervention programme had little impact (measured in terms of exam results): students did not/ could not use video effectively on their own (feedback).
Students made aware of potential reading problems might be more motivated to take advantage of a facilitated reading intervention programme.
Diagnostic process necessary
Project assignment: 2001, 2002, 2003
Reading/language problems Meta-cognitive problems: Lack of
critical reasoning skills undermines conceptual development
General knowledge problems: Poor general knowldege impedes students’ ability to learn from examples used to illustrate concepts
Assessment internationally and nationally accepted.
A New Academic Policy for Programmes and Qualifications in Higher Education (CHE, 2001) proposed an outcomes-based education model; commitment to learner-centredness. Learner-centredness => smooth interface between
learners and learning activities, not possible without a clear sense of learner competence on admission.
Pre-registration entry-level assessment can facilitate smoother articluation.
Diagnostic assessment accepted (2003)
The official view …
nstitutions ‘will continue to have the right to determine entry requirements as appropriate beyond the statutory minimum. However, ..., selection criteria should be sensitive to the educational backgrounds of potential students ... .’ (Gov. Gazette 1997)
SAUVCA’s role
National initiative to develop Benchmark Tests academic literacynumeracy maths
Piyushi Kotecha (SAUVCA CEO):
“This is a timeous initiative as the sector is ensuring that it will be prepared for the replacement of the Senior Certificate by the FETC in 2008. Specifically, national benchmark tests will gauge learner competencies so that institutions can better support and advise students. It will also enable higher education to ascertain early on the extent to which the FETC is a reliable predictor of academic success.
Piyushi Kotecha (cont.)
This exercise will therefore create a transparent standard that will ensure that learners, teachers, parents and higher education institutions know exactly what is expected when students enter the system. As such it will also allow for greater dialogue between the schooling and higher education sectors.
Purpose of Diagnostic Assessment
Better advice and support; scaffolding for ‘high-risk’ students.
Critical information can address increased access and throughput.
Admissions and placement testing can lead to a reduction in number of high-risk students admitted to various degree programmes.
Benchmark Tests can provide standardised tests without the process dictating how different universities should use the results.
Some difficulties …
Economies of scale favour large enrolments
ODL principles embrace all students Moral obligation to be honest with
potential students and responsible in use of state funding
Cost – benefit considerations
Establishing the testing process
Meetings with stakeholders Assessment not approved as a prerequisite
to study, but as a compulsory (experimental) co-registration requirement: students thus register for mathematics module and simultaneously for diagnostic assessment, even if they register late and are assessed mid year. BUT later assessment allows less time for remedial action; students advised to delay registration until after assessment.
Computerised testing the ideal; not initially feasible.
The process (cont.)
Two module codes created for the two assessment periods: (i) supplementary exam period in Jan and (ii) end of first semester
Procedures managed by Exam Dept Explanatory letter for students, Marketing; Calendar; Access Brochure
Selecting content: some considerations
Internationally accepted standards Assessment tools adapted to suit specific
UNISA requirements Reliability and validity important criteria Need for practical, cost effective measures Various options investigated, e.g. AARP
(UCT); ACCUPLACER and UPE adaptations; UP
UCT & UPE
AARP possibly most appropriate (trial undertaken at UNISA in 1998), but time consuming and not computer-based.
UPE demographically and academically similar to UNISA, research into placement assessment since 1999. ACCUPLACER (USA/ETS) found appropriate.
ACCUPLACER at UPE
Computerised adaptive tests (unique for each student) for algebra, arithmetic and reading (pen-and-paper options available – only one test).
A profile developed across all the scores in the battery (including school performance)
Regression formulae and classification functions used to classify potential students with respect to risk.
Formulae based on research.
ACCUPLACER Reading Comprehension
Established reliability and validity Possible bias: Seven questions possibly
ambiguous or with cultural bias - did not detract from items’ potential to assess construction of meaning from text. Left in original format for first round of testing; can delete or adapt later.
Own standards
Basic Arithmetic Test
‘Home grown’ Testing for potential Assesses understanding rather than recall Items based on misconceptions that are
significant barriers to understanding the content and recognised problem areas
Experiment on benefit of drill-and-practice Reliability, validity to be established
Test format and data capture
Three hours, 35 MCQs (four options) in each category (Total 70)
Mark reading sheets; return papers Assignment section captured marks Computer services processed marks and
determined categories Marks made public
Aspects assessed in ARC
Aspect tested QuestionsCausal relations 2, 4
Contrastive relations 3, 7, 8, 23
Recognition of sequence
5
Interpretation of implied or inferred information
10, 14, 15, 17, 20, 27, 32, 33, 35
Aspects assessed in ARC (cont.)
Aspect tested QuestionsComprehension/interpretation of factual information; detailed/ general
9, 11, 23, 28, 29
Academic vocabulary All except 5, 26
Number sense 23
Recognition of contradiction, inconsistency
12
Substantiation 1, 6
Aspects assessed in BAT
Aspect tested QuestionsSimple arithmetic operations (whole numbers)
19
Simple arithmetic operations (fractions/decimals)
12
Pattern recognition 3
Number sense 5
Conversion of units 3
Academic (maths) /Technical vocab 3 / 4
Aspects assessed in BAT (cont.)
Aspect tested QuestionsComparison 10
Time - how long /Time - when 3/2
‘Translation’ from words to mathematical statements
17
Recognition of insufficient or redundant information
4
Learning from explanation 2
Spatial awareness 3
Insight 14
Grading criteria Three categories considered (exlp. in T/L 101)
Category 53: Students likely to be successful with no additional assistance.
Category 52: Students likely to be successful provided they had support.
Category 51: Students unlikely to be successful without assistance beyond that available at the university.
Criteria for classification based on ACCUPLACER guidelines and empirical results following Phases I, II and III of the reading intervention.
Conversion table in ACCUPLACERCoordinator’s Guide converts raw score out of35 to a score out of 120 - some form of weighting takes place. 0 to 4 out of 35 equivalent to 20 points. Increment between the numbers of correct
answers increases gradually from 1 to 4. ‘Reward’ for obtaining a greater number of
correct answers.
Criteria - Reading Compreh.
Three categories: Weakest: 51 out of 120 ~ 31%Moderate: 80 out of 120 ~ 60%Good: 103 or higher ~ 83%Scores reflect different reading skills, outlined in Coordinator’s Guide. From Phases I and II of the reading project: 60% ~ threshold below which reading skills too weak to support effective study.
ACCUPLACER recommendations
Our score
Giving students the ‘benefit of the doubt’the upper boundary of the lowest categoryin the MDA (students at risk with respect to reading) was pitched lower, at 70 out of 120 (~ 50%).
Comparison of boundaries
High-risk category
Raw score
(0 – 35)
Converted score
(0 – 120)%
Upper boundary
UNISA
22 67 47
23 71 51
Upper boundary
ACCUP
25 79 59
26 84 64
Criteria - Basic Arithmetic Test
Questions weighted: intention to enable students who demonstrated greater insight to score better than those who had memorised procedures without understanding.
Simplest, procedural questions (such as addition or multiplication): W = 1 or 1,5
Questions requiring some insight and interpretation of language: W = 2, 2,5 or 3
Raw score out of 35 ~ final score max 69
Weight distribution of BAT questions
Categ Weight % of
total
No. items
Aspect assessed
Easy 1 10 7 Simp arith
Easy 1,5 13 6 + frac/dec
Mod easy
2 30 10 + no. sense
lang, time
Mod
diff
2,5 22 6 + money,
insight
Diff 3 25 6 + pattern
Cumulative totals
Score up to 53% by correctly answering easy to moderately easy items (W = 1, 1,5 or 2)
Score up to 75% by correctly answering ‘easy’, ‘moderately easy’ and ‘moderately difficult’ items (W up to 2,5)
Score over 75% only if ‘difficult’ items (W = 3) also answered correctly.
Setting the lower limit
10 items (17% of total) computational - no reading skills. Possible for all students to answer these questions correctly.
25 items (83% of total) dependent on reading and other skills.
60% ‘threshold’ => 17% + (60% of 83%) i.e. 67% set as the lower boundary for BAT (~ raw score 46 out of 69).
Students with < 46 at risk wrt numerical and quantitative reasoning.
Setting the upper limit
ARC top category begins at score of 103 (approximately 83%)
BAT equivalent: 57 out of 69 No other empirical evidence – 57 set as
cut-off point for high achievement in BAT
MDA categories
Category Reading (weighted)
BAT (weighted)
51 S < 70 OR S < 46
53 S > 103 AND S > 59
52 All other scores
The assessment process
Procedural issues Problems with co-registration
requirement Several administrative problems
ARC results
Good item-test correlations. Only five questions with correlations of
below 0,5. Students scored reasonably well on
most of the potentially problematic items. Only three of these had low item-test correlations.
BAT results
Weaker item-test correlations than Reading Comprehension score.
Low (< 0,30), moderate and high item-test correlations in all question categories.
Reading may play greater role in categorisation.
Consolidation of results
January:
Category 53 10 3%
Category 52 93 29%
Category 51 223 68%
Total 326
Note: Oct 03/Jan 04 exams: 76% failed
Consolidation (cont.)
June (after exposure to mathematics and English study material):
Category 53 35 4%
Category 52 176 21%
Category 51 623 75%
Total 834
Further analysis of results
Assignments (January group):
Category 51 mean = 39%
Category 52 mean = 48%
Category 53 mean = 65%
Exam results (all students)
Registered: 1 518 Not admitted to the exam: 912Obtained exam admission: 606Wrote exam: 551 Passed: 162 (October: 145; January: 17) Exam mean: 27% MDA students with exam results: 463MDA exam mean: 35%
Exam results by risk category
Category No. of students
Mean exam score
51 332 (72%) 30%
52 106 (23%) 45%
53 25 (5%) 57%
Exams – no ARC
Category No. of students
Mean exam score
51 136 (29%) 26%
52 189 (41%) 32%
53 138 (30%) 48%
No exam admission
Cat. No. of MDA students in
category
n = 1 160
No. of MDA students without
admission
n = 698
% of MDA students in
category without exam
admission
51 846 514 (74%) 514/846 = 61%
52 269 164 (23%) 164/269 = 61%
53 45 20 (3%) 20/45 = 44%
Comparison between students who wrote/did not write MDA
Wrote MDA
(n = 463)
Did not write
MDA(n =101)
Did not write / wrote
Pass
( n = 162)
125 (77%) 37 (23%) 37/125 = 30%
Fail
( n = 401)
337 (84%) 64 (16%) 64/337 = 19%
Implications of assessment
Counselling essential, especially for Category 51and 52 students.All potential support options dependent on staff and resource allocation. Options: The Institute for Continuing Education (ICE): advice
regarding alternative directions of study, or measures to upgrade academic skills before studying. Initially agreed to investigate such options, but no progress to date.
Implications of assessment
National Tutorial Support Coordinator seemed in favour of using information obtained in the assessment process to inform the tutorial programme. No information forthcoming on support options or the necessary data collection procedures.
The Bureau for Student Counselling and Career Development (BSCCD) staff willing to assist where possible, but staff not deployed at all centres.
Implications of assessment
The Povey Centre (instruction in English language proficiency, reading and writing skills) possibly able to provide some reading instruction via the Learning Centres, at no additional charge (other than the basic Learning Centre registration fee required from all students who registered for tutorial classes) (START programme). No clarity yet regarding extent to which this will be rolled out; impact for mathematics to be investigated.
Implications for qualitative research
Psychological implications of ‘going to university’ but needing first to be taught how to count and read?
Cost implications of referral? Gate keeping or gateway?
Pre-registration assessment…
… is critically important. Costs, benefits, advantages and
disadvantages of instituting diagnostic assessment for mathematics at UNISA need to be thoroughly investigated.
Information regarding the implementation process must be well analysed and utilised.
True access depends on providing students with appropriate career guidance.
Ongoing research (quantitative & qualitative) into specific test aspects and components essential.
Top Related