Conceptualising and Testing Generic Skills in an International Assessment

Of Higher Education Learning Outcomes

Doug McCurry

Australian Council for Educational Research

OECD’s Assessment of Higher Education Learning Outcomes (AHELO) Feasibility Study

17 countries

248 higher education institutions

4 900 faculty

23 000 final-year engineering and economics students

The Feasibility Study aimed to

•test what is or could be feasible in international testing of learning in higher education

•determine what instruments would work well in such testing

•provide lessons on how learning outcomes might be effectively assessed in the future

The Feasibility Study looked at outcomes in

•the discipline-specific knowledge and skills of economics and engineering courses

•the GS common to all students (such as critical thinking, analytical reasoning, problem-solving, and written communication)

Why test generic skills?

The attention to GS reflects

• a recognition that competence means more than technical knowledge and skills

• a demand for an increasingly skilled workforce

• dissatisfaction with work readiness of some exit students

• increasing internationalisation and greater international competition

The generic skills debate

The rejecters

Cognitive performance is knowledge dependent, everything is learned in a context, transfer of learning across contexts is uncertain, and competence and expertise are knowledge rich

GS should be tested within domains

The accepters

GS are the fluid capacities that are used to learn new things, and that are developed by learning things

GS assessment should be focussed on the ability to think about something new and unfamiliar rather than focussed on what has been learned in the past

To the accepters the notion of ‘domain-specific generic skills’ is meaningless

Testing GS in domains is testing the domain specific not the generic

What is meant by the term generic skills?

These GS are distinguished from• the specific knowledge and skills needed for a particular

performance• what the expert knows• competencies or an area in which you have to be competent• a class of tasks

GS are the fluid capacities used to learn new things as distinct from what has been learned in the past

The assessment of general abilities is focused on aptitude to learn rather than achievement or competence

What is a generic skills test?

• aims to minimise the testing of prior knowledge• is stimulus based, and uses generally accessible,

non-specialist public domain material as stimulus• aims to maximise testing of the ability to

understand unfamiliar material and undertake novel thinking tasks

• involves a broad and balanced set of tasks• involves a range of tasks, but is consistent and

coherent enough to be summarised in a single score

Some other international generic skills studies

Prepared for life?: how to measure cross-curricular competencies 1997

OECD Program of International Student Assessment (PISA)

The OECD Definition and Selection of Competencies project (DeSeCo)

IEA Composition Study 1988

The Collegiate Learning Assessment

Council for Aid to Education

The CLA is a stimulus-based performance assessment of

• analytic reasoning and evaluation• problem solving• writing effectiveness• writing mechanics

The GS MCQ

The CC GS skills tested by ACER

• are seen as developed rather than innate abilities• do not involve specialized knowledge or skills• involve integrated and holistic thinking• aim at higher-order skills rather than basic skills or abstract

and de-contextualised information processing• involve thinking about the material world and socio-cultural

issues• aim to be generally accessible and test a broad and balanced

range of cognitive tasks

The ACER Tradition of CC TestingThe art of item writing

Psychometric methods but not psychometric constructs

Based on rich stimulus – specific, real-world, socio-cultural

Minimising the important of prior knowledge

Meaningful questions about real problems

Not abstract exercises or meaningless games

Unfamiliar questions and higher-order thinking

The common central core of cognitive abilities is tested by

•minimising the testing of prior knowledge

•maximising the ability to understand unfamiliar material and undertake novel thinking tasks

•basing testing on generally accessible, non-specialist, public domain stimulus material

•using a set of tasks that is broad enough to involve a complex of abilities, but coherent enough to be summarised in a single score

Categories and characteristics for the AHELO MCQ classification

International reactions to proposed material

Too logico-deductive and quantitative

Too much emphasis on concrete and material issues and information processing

Not conceptual enough

Not girl friendly enough

Some problems with language and cultural references

Some socio-political problems

Development of the proposed GS MCQ

Removed a unit of harsh and lenient prisons, but retained in similar set on a political issue

Added passages by Tolstoy and Ibsen about socio-cultural issues

Added part of a speech by Ghandi with historical and political significance

The 55 items used were broad, rich and intelligible from all sorts of perspectives

Adapting material to work in other languages

The risk of changing material that worked in English (in Australia) for easier transition to other languages

51 of 55 items had satisfactory to good difficulty and discrimination

Only 4 items misfitted in general and were removed from the analysis

A small number of items were removed for the analysis in individual countries

In some respects the GS MCQ were more successful than the domain-specific material

Towards an optimal generic skills test for international use

Testing GS across languages and cultures through unfamiliar but generally accessible material was successful

The feasibility study showed that • GS MCQ test did work well across languages

and cultures• CRI tests could work within languages and

cultures

A optimal test of GS for international use would

• have a number of different tasks rather than one theme to give breadth and balance

• integrate CRI and MCQ• be focussed on thinking rather than literacy• have some specific and some broad and open

tasks• have item specific scoring for some SAQ and

global scoring for some broad and open tasks

The AHELO feasibility study warrants an international GS test that is specific in some respects and broad and open in other respects

Such a test can deal with cultural and socio-political issues

A very encouraging and satisfying result