Two taxonomies are better than one: towards a method of … · 2019-09-05 · Applying Taxonomies...

Educationalfutures Greatorex, Ireland and Coleman

Vol.10(1) July 2019 Two taxonomies are better than one __________________________________________________________________________________________

e-journal of the British Education Studies Association 3

ISSN: 1758-2199

Two taxonomies are better than one: towards a method of analysing a variety of domains and types of thinking in an assessment

Jackie Greatorex, Research Division, Cambridge Assessment, UK

Jo Ireland, Research Division, Cambridge Assessment, UK

Victoria Coleman, Research Division, Cambridge Assessment, UK

Corresponding author: Jackie Greatorex

Email: [email protected]

Abstract

It has been established practice to analyse the domains and types of thinking involved

in undertaking educational activities using a taxonomy. Many taxonomies focus mainly

on cognition and are therefore unsuitable for analysing non-cognitive domains, these

are often assessed by means other than traditional written assessments. The aims of

this research were to find a taxonomy suitable for analysing a variety of domains and

types of thinking which might in turn be suitable for use across a variety of assessment

tasks and to ascertain whether assessment experts perceived it to be useable. To find

a suitable taxonomy, several were evaluated against pre-defined suitability criteria.

While no individual taxonomy met the criteria, the taxonomies of Marzano and Kendall

(2008) and Hutchins et al. (2013) were merged to form a combined taxonomy which

was used for analysing domains and types of thinking. Six assessment experts

analysed selected assessments including a written examination, a presentation and a

reflection using these taxonomies to inform the research. Findings included the

taxonomic analysis facilitated comparisons between different assessments, all of the

assessments tested a high level of thinking but the interpersonal domain was only

evident in the presentation and reflection assessments. The experts generally found

the combined taxonomy accessible and suitable for analysing the domains and types

of thinking and it could be used successfully to research the alignment between

assessment(s) and/or classroom activities.

Keywords

taxonomy, demand, domain, assessments




ISSN: 1758-2199

Introduction

Since the seminal work of Bloom, Engelhart, Furst, Hill and Krathwohl (1956), it has

been established practice to use taxonomies to analyse the domains and types of

thinking employed by students undertaking academic activities. Taxonomy refers to a

system for classifying domains and types of thinking. Domains are an area of

knowledge or activity;

• The cognitive domain focuses on thinking, including thinking about thinking

(also referred to as metacognition).

• The affective (intrapersonal) domain relates to interests, attitudes, emotions

and values.

• The psychomotor domain centres on physical movements.

• The interpersonal domain concentrates on relationships between people.

Types of thinking are different cognitive activities such as recalling or analysing while

demand generally refers to the level of challenge involved in undertaking a task. There

are nuanced variations of the definition of demand for different taxonomies. In some

taxonomies one type of thinking is considered to be more cognitively demanding than

another. For example, analysing concepts is more demanding than recalling concepts

used to study the topic (Bloom et al., 1956).

There are many reasons to analyse the domains and types of thinking involved in

curriculum and assessment. For example designing assessment strategies by starting

tests or topics with less demanding tasks (such as simple description tasks), before

increasing the demands in the tasks as students become more experienced (such as

evaluative examination questions) and checking that the assessment assesses the

domains and types of thinking required in the curriculum (Ebadi and Shahbazian,

2015; Luebke and Lorié, 2013; Webb, Herman and Webb, 2007). Such practices are

used at all levels of education (primary to post-compulsory) and the work generally

involves experts using a taxonomy to judge which domains and types of thinking

students use when they respond to a task.

Awarding organisations in England and the national regulator of qualifications (Ofqual)

often analyse the cognitive demand of syllabuses such as examination questions and




ISSN: 1758-2199

mark schemes (Pollitt, Ahmed and Crisp, 2007; Shaw and Crisp, 2012). Such analyses

are used to compare the cognitive demands of questions. For example:

• With mark schemes (Shaw and Crisp, 2012).

• From one year with questions from another (Crisp and Novaković, 2009).

• From one assessment provider with questions from another (QCA, 2008).

Such analyses are used in validation work to consider:

• The alignment between question papers and mark schemes (Shaw and Crisp,

2012).

• Whether a variety of demands were placed on the students (Greatorex, Shaw,

Hodson and Ireland, 2013).

Studies tend to focus on the cognitive domain and omit other domains, but the

affective, interpersonal and psychomotor domains are also important as they are all

used in education (Bichi, Hafiz and Abdullahi, 2017; Hiong and Osman, 2013) and/or

employment (Bandaranaike and Willison, 2015; Nair, 2012). For example, the affective

domain is used in architecture curricula (Savic and Kashef, 2013), the cognitive

domain is tested in accounting examinations and used by clinicians (Davidson, 1996;

Laxmisan et al., 2007), the interpersonal domain is employed in many army roles

(Carpenter, Wisecarver, Deagle-III and Mendini, 2005, Hutchins et al., 2013) while the

legal profession use the psychomotor domain in court (Cradduck and Thomas, 2016).

All domains were seen as necessary and included in this research.

The research was done by a team of three researchers working for a national

assessment board and the aims of the project were therefore to:

• find a taxonomy for analysing a variety of domains and types of thinking.

• ascertain whether the taxonomy is suitable for use with a variety of

assessment tasks (not just cognitively orientated written tasks).

• ascertain whether experts perceived the taxonomy to be useable.

Applying Taxonomies

In previous taxonomy research, the participants (those applying the taxonomy) were

often experts with substantial experience in assessing the target field (Chan, Tsui,




ISSN: 1758-2199

Chan and Hong, 2002; Näsström and Henriksson, 2008; Webb et al., 2007). However

many studies gave sparse, if any, description of the experts’ experience in the target

field (for example, Chan et al., 2002; Ebadi and Shahbazian, 2015; Edwards, 2010;

FitzPatrick and Schulz, 2015; Kendall et al., 2008; Palmer and Devitt, 2007). In these

studies, two or three experts tended to use a single taxonomy to categorise items such

as assessment tasks, learning outcomes etc. More unusually, where a larger number

of experts took part (20 was the highest number found in the review of relevant

literature) their experience was given in some detail (Herman, Webb and Zuniga,

2007).

Participant Perceptions

Depending on the situation participants using taxonomy in research may be students,

teachers, researchers or assessors. Whilst there are many taxonomies and a wealth

of taxonomy research, there are only a few studies which investigate participant

perspectives. For example, Dubas and Toledo (2016) researched student perceptions

of the learning objectives that were produced a taxonomy of educational objectives

(Marzano and Kendall, 2007). Nakyam, Kwangsawad and Sriampai (2013) drafted a

curriculum and associated assessments using Marzano and Kendall’s taxonomy and

once the curriculum was implemented the students were surveyed and their views on

the curriculum were found to be positive. Crowe et al. (2008) used Bloom’s taxonomy

in several ways including developing assessments, and tasking students with writing

and classifying test items using the taxonomy. They also gathered student perceptions

on how Bloom’s taxonomy was applied in particular courses and whether it facilitated

learning. However, studies investigating how easy a taxonomy is to use, and whether

it is applicable to the area being studied are rare, yet it is important to know whether a

taxonomy is difficult to apply or appropriate for a particular subject before it is widely

used as this facilitated good educational practice.

Arising from the above literature, the study was designed in three consecutive parts:

Part 1: Selecting a taxonomy for analysing a variety of domains and types of

thinking.

Part 2: Ascertaining whether the selected taxonomy is suitable for use with a

variety of assessment tasks, not just written examinations.




ISSN: 1758-2199

Part 3: Ascertaining the usability of the selected taxonomy from the

participants’ perspective.

The studies were conducted sequentially. A taxonomy was selected in part 1 of the

study, used in part 2 of the study, and finally evaluated by participants in part 3 of the

study. Findings from the earlier studies informed the later studies.

Study 1: Selecting a taxonomy

Aim

The aim of this part of the study was to select a taxonomy to analyse a variety of

domains and types of thinking.

Procedure

The first step was to devise criteria against which we would evaluate taxonomies and

select the most suitable one. We decided our taxonomy should be:

• authoritative and based on current established theory and/or have an

empirical basis to ensure it was robust.

• broad enough to incorporate the cognitive, psychomotor, affective and

interpersonal domains for analysing a variety of domains and non-cognitive

tasks.

• hierarchical or cumulative, with higher levels or categories to distinguish the

more demanding tasks and different types of thinking.

• have been used in assessment in an academic, vocational or applied setting.

(This criterion would ensure that the taxonomy is suitable to use in a variety of

assessment tasks, as per the study aims).

• be applicable to the studied assignments assessed objectives.

Using a taxonomy with no conceptual linkage to the assessment objectives is bad

practice (Johnson and Mehta, 2011) so one researcher searched for and read

literature about taxonomies noting whether the taxonomy met our criteria.

Creating a Combined Taxonomy

It is beyond the scope of this paper to describe all the taxonomies we studied in detail,

but it is important to summarise the selected taxonomies used in Studies 2 and 3. After




ISSN: 1758-2199

investigating a range of taxonomies (Anderson et al., 2001; Atkinson, 2013; Biggs and

Collis, 1982; Carpenter and Wisecarver, 2004; Harrow, 1972; Hauenstein, 1998;

Hutchins et al., 2013; Klein, DeRouin and Salas, 2006; Marzano and Kendall, 2007;

Marzano and Kendall, 2008), it was evident that none met all of the selection criteria

we had set - see above and Table 3 for the criteria. The taxonomy from Marzano and

Kendall (2007, 2008) came nearest and while it omitted the interpersonal domain it

was felt this would prove the most useful. This is summarised in Table 1.

Level Process Description

6 Self-System Thinking

Identifies learning motivation

Identifies emotional responses

Identifies improving ability or understanding

Assess own learning

5 Metacognition Gauge own level of accuracy

Determine own understanding

Monitor own progress toward a goal

Outline learning goals and plan to achieve them

4 Knowledge Utilization

Investigate outcomes

Experiment to find different outcomes

Test theories

Solve Problems and make decisions

3 Analysis

Inform of consequences

Make generalisations

Spot errors

Categorise

Note similarities and differences

2 Comprehension

Draw up information

Design information outputs

Combine information and summarise

Structure information

1 Retrieval Displaying more complex processes

Simple procedural actions

Recollection of simple information

Producing information

Recognising information appropriateness

Identifying information and assessing for accuracy

Table 1: Summary of Marzano and Kendal’s New Taxonomy (2008)

Therefore we chose Marzano and Kendall’s (2007, 2008) taxonomy for use in

combination with Hutchins et al. (2013), taxonomy to provide a comprehensive overall




ISSN: 1758-2199

taxonomy which allowed us to include an up-to-date articulation of the interpersonal

domain.

Marzano and Kendall’s (2007, 2008) taxonomy has two dimensions: (1) knowledge

domains; the things we know or remember, and (2) mental processing how we

understand the things we know or remember. Within the knowledge dimension there

are three domains: information (factual knowledge); mental procedures (how to

mentally carry out a task); and psychomotor procedures (how to physically carry out a

task). While there is no hierarchical relationship amongst these domains, each domain

comprises of further sub-categories, which are hierarchically organised.

Within the mental processing dimension there are three thinking systems delineated

into six levels which mentally process types (domains) of knowledge (Marzano and

Kendall, 2007, 2008). The highest or most complex level is the self-system (Level 6),

which refers to an assembly of interrelated beliefs, attitudes and emotions that control

attention and motivation. These determine the candidate’s level of engagement with a

new task. When a candidate embarks on a new task, the metacognitive and cognitive

systems are activated. The metacognitive system (Level 5) exercises executive

control as it monitors, evaluates and regulates cognition or thinking. Metacognition

involves stipulating goals and planning how to achieve the goals.

Category Abilities needed to….

Interpersonal

communication skills

Express and integrate information in social interaction. This

encompasses listening, speaking, writing, sending/receiving non-verbal

signals in an empathic, attentive, responsive and confident way.

Relationship building skills Develop and keep relationships, to support others, and build strong

beneficial unions as well as manage and resolve conflicts.

Peer leadership skills Coach, advice, motivate and empower group members. Happily interact

with a team, gain trust, dynamically join in problem solving.

Social/behavioural agility

skills

Monitor and understand our own and other’s behaviours and alter self-

presentation to influence the interaction.

Table 2: Summary of Marzano and Kendal’s New Taxonomy (2008)

The remaining four levels process the knowledge required to do the task and form the

third Thinking system: cognitive (Marzano and Kendall, 2007, 2008), knowledge

utilisation (Level 4) entails using knowledge in a particular situation, analysis (Level 3)




ISSN: 1758-2199

involves reasoning which creates new thought and information, comprehension (Level

2) refers to the interpretation of knowledge into a mode appropriate for storing in long

term memory, retrieval (Level 1) is remembering information and drawing it into

consciousness ready for use. All six levels are subdivided into operations, which are

cumulatively organised.

The cognitive levels in Marzano and Kendall’s taxonomy (2007, 2008) are delineated

in terms of the cognitive control of the thought processes needed to complete tasks.

The demand of the tasks is not related to the cognitive demand intrinsic to the task,

but to how intentional the students must be in their thinking of an answer to tasks. In

this context, the term intentional refers to deliberate and conscious thinking rather

than, for example, practised recall which has become automatic

Hutchins et al. (2013) created a taxonomy from literature about training and skills,

including negotiation and conflict resolution. We judged this to be a taxonomy focusing

on the interpersonal domain. The main skill groupings in the taxonomy are:

relationship building skills; peer leadership skills; interpersonal communication skills;

and social/behavioural agility skills. Each main grouping comprises additional

subcategories of skills. There is no hierarchy within the taxonomy (see Table 1).

However, we found no interpersonal taxonomies which were hierarchical.




ISSN: 1758-2199

Suitability Criteria The taxonomy must

An

ders

on e

t al. (

20

01)

Atk

inso

n (

20

13)

Big

gs a

nd C

olli

s (

19

82)

Carp

ente

r an

d W

isecarv

er

(2004)

Harr

ow

(19

72)

Haue

nste

in (

19

98)

Hutc

hin

s e

t a

l. (

2013)

Kle

in e

t a

l. (

2006)

Marz

ano a

nd

Kend

all

(20

07,

2008)

Be established ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Be based on theory ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Include the (meta)cognitive domain

✓ ✓ ✓ ✓ ✓ ✓

Include the psychomotor domain

✓ ✓ ✓ ✓ ✓

Include the affective domain

✓ ✓ ✓

Include the interpersonal domain

✓ ✓ ✓

Be hierarchical or cumulative

✓ ✓ ✓ ✓ ✓ ✓

Be used in assessment of academic achievement

✓ ✓ ✓ ✓

Evidence of the taxonomy being used in assessment of academic achievement

FitzPatrick

and

Schulz

(2015)

Rembach

and Dison

(2016)

Yang

(2013)

Dubas and

Toledo (2016);

Faragher and

Huijser (2014)

Be used in assessment of vocational or applied knowledge and skills

✓ ✓ ✓ ✓

Evidence of the taxonomy being used in assessment of vocational or applied knowledge and skills

Mayer

(2002)

İlgüy, İlgüy,

Fişekçioğland

Oktay (2014)

Chan,

Chan,

Tsui

and

Kwok

(2014)

Pate and Miller

(2011)

Haolader, Ali

and Foysol

(2015)

Edler, Eberman,

Kahanov, Roma

and Mata

(2015)

Table 3: Judging taxonomies against suitability criteria

Note, ✓ denotes that the taxonomy meets the criteria, and a blank cell denotes that it does not meet the criteria.




ISSN: 1758-2199

Turning to the issue of how we combined the taxonomies, cognition

(interpreting/understanding communication signals) and intrapersonal ability

(managing emotions, beliefs) are key mediating variables for the deployment of

interpersonal skills (Klein et al., 2006). As mentioned in the introduction, it is important

to examine domains beyond just the cognitive domain, as these are also used in both

education and employment. Therefore, we theorised that the levels of mental

processing (Marzano and Kendall, 2007; 2008) also apply to the interpersonal domain

(Hutchins et al., 2013). Greatorex and Suto (2016) made the same theorisation. The

taxonomies focus on each individual candidate and can apply to a variety of academic

and vocational subjects (Hutchins et al., 2013; Marzano and Kendall, 2007, 2008).

Within our combined taxonomy there are three dimensions of demand: (1) the number

of domains; (2) the knowledge category; and (3) the level of mental processing. The

demand of a task can be increased by including more domains, using a higher

knowledge category, or using a higher level of mental processing. Similarly, the

demand can be decreased by reducing the number of domains tested, requiring a

lower knowledge category or a lower level of mental processing.

In conclusion, the selected taxonomy was a combination of Marzano and Kendall

(2007, 2008) and Hutchins et al. (2013). We theorise that the levels of mental

processing described by Marzano and Kendall (2007, 2008) also apply to the

interpersonal domain from Hutchins et al. (2013). This is shown in Table 4: Overview

of the combined taxonomy (Hutchins et al., 2013; Marzano and Kendall, 2007, 2008).

Thinking system/level of

mental processing

Domain

Information Mental

procedures

Psychomotor

procedures Interpersonal

Self system

Meta-cognitive system

Knowledge utilisation1

Analysis1

Comprehension1

Retrieval1

Table 4: Overview of the combined taxonomy (Hutchins et al., 2013; Marzano and Kendall, 2007; Marzano and

Kendall, 2008) Note: Together these levels of mental processing form the Cognitive system.




ISSN: 1758-2199

Study 2: Using the selected taxonomy

Aim

The aim of this stage was to ascertain whether the new combined taxonomy was

suitable for use with a variety of assessment tasks and if the data generated from

applying the taxonomy could be used to compare the:

• Domains, categories and operations elicited by each task.

• Demands of various assessments.

Method

Six experts in summative assessment of the target field were recruited as participants

in the study. Here, expert refers to individuals with substantial experience of

summative assessment, particularly in marking and determining grade boundaries.

Each expert categorised the questions from the assessments under the same

conditions using the combined taxonomy from Study 1. The discipline/level was

purposefully omitted to protect the anonymity of participants. This is due to the small

number of experts within this specific area of international education.

The following summative assessments from a secondary level international academic

qualification were used:

• An assignment: for which students study a topic then write a report based on

their studies.

• A reflection: (reflective article) about their team’s solutions to a problem

which the team researched.

• A presentation: about their own research and proposed solutions to the

aforementioned problem.

• A written exam: comprising questions which required short and long

responses.

These summative assessments were used as they varied in style and were not all

timed written tests.




ISSN: 1758-2199

Five Example Exam Questions

Five example questions were devised by the researchers using guidance in Marzano

and Kendall (2007, 2008). The example questions were designed specifically for the

research and the orientation exercise and are shown in Appendix 1.

Procedure

Preliminary work

Johnson and Mehta (2011) explain that applying a taxonomy relies on the participants

being able to relate their subject knowledge to the taxonomy. They therefore advise

that, before applying a taxonomy, researchers map the dimensions in the taxonomy

to the subject. Therefore, the assessment objectives from the above-mentioned

summative assessments were mapped to the selected taxonomies. There were two

steps to this procedure:

a) A researcher systematically went through the verb in each assessment

objective and noted the level of mental processing it corresponded to in

Marzano and Kendall (2007, 2008), by matching it to the example verbs, or

where they were synonyms of the verbs. Marzano and Kendall (2007, 2008)

give examples of verbs from educational objectives connected with each level

of mental processing. For instance, the verb ‘state’ is frequently used in

learning objectives for retrieval (Level 1).

b) A researcher noted the object of each assessment objective and judged which

domain(s) were targeted by the assessment. Marzano and Kendall (2007,

2008) argue that the object of educational objectives signpost the domain(s)

of knowledge on which an assessment concentrates.

The results of the mapping are given in Table 4, which shows the assessment

objectives mapped to the combined taxonomy showing the combined taxonomy was

suitable for analysing assessment tasks.

Orientating experts to the taxonomy

One researcher briefed the experts on the combined taxonomy to engender a shared

understanding of the taxonomy categories. They then applied the combined taxonomy

categories to the five example questions with facilitation by the researcher. Here the




ISSN: 1758-2199

aim was not for experts to agree on which questions fitted into a taxonomy category

but for orientation.

Applying the taxonomy

The experts were told they needed the assessment tasks (question papers or

equivalent), the written document summarising the combined taxonomies and a

document for recording responses. It was also explained that the exercise was about

the types of knowledge and thinking elicited by the assessment tasks. The researchers

asked the experts to work individually and to re-familiarise themselves with the

summary of the combined taxonomies. The researchers advised that some

assessment tasks may use one domain and others may use more. They were then

asked to do the following for each assessment task in turn starting with the first

question of the written examination:

• Read the assessment task.

• Decide which domain(s) the assessment task was likely to elicit.

• For each domain, decide which thinking operation(s) the assessment task

was likely to elicit.

• Record the decisions.

In summary, the experts individually categorised each assessment task using the

combined taxonomy.

Analysis

To categorise assessments with operations and domains we calculated the number of

experts allocating an assessment task (examination question) to an operation and

domain category. Several studies use any agreements between two experts to

indicate where an item is classified within the taxonomy. Examples include Edwards

(2010); Kendall et al. (2008); Näsström and Henriksson (2008). Other studies, with

between six and twenty coders, take between 56% to 67% of coders assigning an item

to a particular taxonomy category, to indicate where an item is classified within the

taxonomy (Herman et al., 2007; Webb et al., 2007). In this study when four of the six

experts said that a question elicited a particular domain or operation the question was

then added to the respective domain or operation.




ISSN: 1758-2199

This allowed an assessment of whether experts thought that assessment tasks elicited

a variety of domains, categories, levels of processing and operations. The orientation

task was designed to develop a shared understanding of the taxonomy but not

necessarily a shared interpretation of the assessment tasks.

Results and Discussion

The assessments were compared on the basis of the domains selected by the experts

and the results are shown in Table 5. Both the written examination (questions One to

Four) and the assignment evoked two domains: information and mental procedures.

The presentation and reflection were the most demanding as they educed all

domains. In this situation, one assessment being more demanding than another is

satisfactory as there is no requirement for the assessments to be of equal demand.

Domain Category

Qu

esti

on

1

Qu

esti

on

2

Qu

esti

on

3

Qu

esti

on

4

Assig

nm

en

t

Pre

sen

tati

on

Refl

ecti

on

Information

Principles ✓ ✓

Generalizations ✓

Time sequences

Facts ✓ ✓ ✓ ✓

Vocabulary terms ✓ ✓

Mental

procedures

Macro-procedure ✓ ✓ ✓ ✓ ✓

Tactics

Algorithm

Single rule

Psychomotor

procedures

Complex combination rules ✓

Foundational procedures

Simple combination rules

Interpersonal

Communication skills ✓ ✓

Peer leadership skills

Relationship building skills ✓

Social agility skills ✓

Table 5: Questions that elicited each category




ISSN: 1758-2199

The assessments were then compared in terms of the categories elicited within each

domain. The assignment and presentation prompted a higher information category

than the written examination with the assignment and presentation equally

demanding. Each assessment stimulated the highest category of mental procedures.

The presentation achieved the highest psychomotor procedures category making it

the most demanding assessment in the psychomotor procedures domain. The

combination of assessment tasks drew from a range of domains and categories. The

set of questions elicited each domain and nine of the 16 categories, including both

high and low categories (see Table 5). This suggests a good assessment regime as

the questions required a breadth of knowledge and there were less demanding tasks

where lower ability candidates were more likely to achieve, and more demanding tasks

to stretch higher ability candidates.

There was a relationship between question type and the domain(s) assessed, as might

be expected. Three of the questions (1, 2 and 4) from the written examination each

elicited one domain, either the information or mental procedures domain. The other

written assessment tasks the assignment and the reflection each stimulated

candidates to use two domains. The presentation prompted examinees to use all four

domains. It seems that a greater variety of domains were elicited by questions and

assessments requiring longer constructed responses using a medium other than a

written examination. There was a relationship between the type of question

(openness of the constructed response) and whether higher or lower categories were

elicited. Questions 1 and 2 (those requiring the shortest responses) elicited lower

categories within the information domain. Questions 3 and 4, the assignment,

presentation, and reflection each prompted candidates to use the highest category

in a domain. Generally, open constructed response questions educe the higher

categories within the domain.




ISSN: 1758-2199

Table 6: Questions which elicited each operation

The assessments were compared in terms of the operations elicited by the

assessments. The written examination questions evoked lower operations than the

assignment (see Table 6). The presentation and reflection educed higher

operations than the assignment. Therefore, the presentation and reflection were of

Level Operation

Qu

esti

on

1

Qu

esti

on

2

Qu

esti

on

3

Qu

esti

on

4

Assig

nm

en

t

Pre

sen

tati

on

Refl

ecti

on

Self system

Examining overall motivation ✓

Examining emotional response ✓ ✓

Examining efficacy ✓

Examining importance ✓ ✓ ✓

Meta-cognitive

system

Monitoring accuracy ✓ ✓ ✓

Monitoring clarity ✓ ✓

Process monitoring ✓ ✓ ✓

Specifying goals ✓ ✓

Knowledge

utilisation

Investigating ✓ ✓ ✓

Experimenting

Problem solving ✓

Decision making ✓ ✓ ✓ ✓

Analysis

Specifying ✓ ✓

Generalising ✓ ✓ ✓ ✓

Analysing errors ✓ ✓ ✓

Classifying ✓ ✓ ✓

Matching ✓ ✓ ✓

Comprehension Integrating ✓

Symbolizing

Retrieval

Recognition ✓ ✓

Recall ✓ ✓

Execute ✓ ✓ ✓ ✓




ISSN: 1758-2199

the greatest demand. In this situation, one assessment being more demanding than

another is suitable as there was no requirement for the assessments to be of equal

demand. It is striking that the longer constructed response questions, particularly

those drawing on group work, elicited higher operations (see Table 6). In other words,

the longer constructed questions and the questions drawing on group work placed

higher demands on candidates.

In summary, the data generated by the taxonomic analysis allowed us to compare the

domains, categories and operations elicited by each task, and compare the demand

of various assessments. Therefore, the combined taxonomy was suitable for use with

a variety of assessment tasks.

Study 3: Experts evaluate the utility of the taxonomy

Aim

To ascertain whether experts perceived the taxonomy to be useable.

Method

The same experts from Stage 2 subsequently completed an online questionnaire

including three closed response questions about the accessibility, appropriateness

and ease of use of the taxonomies. There was also an open-ended question that

asked for a list of knowledge and skills not covered by the Thinking systems.

To analyse the closed response questions the frequency of responses were tabulated

for positive or negative responses. For the open questions, the responses were

summarised or quoted, as agreed with the experts and themes drawn out.

Results

The accessibility, appropriateness and ease of use of the taxonomies were evaluated

using three closed response questions. All or most of the participants found each

aspect of the taxonomies accessible (see summary in Appendix 2). All the participants

felt that most aspects of the taxonomies were appropriate for the target examination

questions (see summary in Appendix 2). The only aspect which they felt was not

appropriate was the psychomotor procedures domain. Generally the participants

found it neither easy nor difficult to analyse the domains and thinking systems involved

in the tasks (see summary in Appendix 2). Overall, the taxonomy was found by the




ISSN: 1758-2199

experts to be useable in our study and therefore may be considered for use with similar

assessments.

Experts were asked to list the knowledge and skills from the assessment which were

not covered by the domains or thinking systems. Responses indicated that three areas

of knowledge/skills were omitted. Firstly “Identifying terms/phrases,” (Expert 2) from

questions 1 to 4 was not evident but it could be argued it is covered by recognition.

Secondly “Much of the information domain appears to be focused upon utilising

knowledge and not gleaning knowledge,” (Expert 4). This comment is accurate, but

‘gleaning knowledge’ is arguably captured in the operation experimenting and

investigating in the cognitive system. Thirdly, the field assessed by the examination

questions “assesses ability to utilise a system above that of self,” one might term this

‘society system.’ This supports the notion that knowledge and skills from the

qualification were largely incorporated in the domains or thinking systems.

Overall discussion and conclusions

The research had two key purposes. These were to ascertain:

• Whether the taxonomy combining Hutchins et al. (2013) and Marzano and

Kendall (2007, 2008) could be used to analyse domains, types of thinking and

demand.

• Whether assessment experts perceived it to be useable.

The main findings were that the combined taxonomy:

• Could be used to analyse domains, types of thinking and demands.

• The experts’ found the combined taxonomy accessible and appropriate for a

variety of assessments and subjects.

There are several limitations to the study. Firstly, the research focused on only one

subject, although the experts felt that the taxonomy could be used in other subjects.

Secondly, the participants may not be a representative sample of the population of

assessment experts in a variety of subjects, and therefore it is unclear how far their

participant perceptions can be generalised. Conversely, there is no clear reason why

they would be significantly atypical of assessment experts.




ISSN: 1758-2199

Regardless of these limitations, our findings prove useful. The combined taxonomy

has some advantages over CRAS (Complexity, Resources, Abstractness, Strategy),

which is the established and widely used tool for analysing demands in school

examinations. CRAS focuses on cognitive demands and, consequently, has limited

usefulness with examinations which claim to prepare students for employment, an

industrial sector, or a particular occupation. This is because many careers draw upon

non-cognitive as well as cognitive domains (Bandaranaike and Willison, 2015;

Carpenter. et al., 2005; Cradduck and Thomas, 2016; Davidson, 1996; Hutchins et al.,

2013; Laxmisan et al., 2007; Nair, 2012; Savic and Kashef, 2013). In contrast, the

domains in the combined taxonomy broadly align with those in the literature. Also

CRAS may not align with the non-cognitive aspects of assessments, which are key

features of applied and vocational qualifications. The combined taxonomy may be

more relevant to such qualifications.

The domains, types of thinking and demand invoked by a variety of item types, such

as short/long constructed response, and modes of assessment, such as written

examination and presentations, were explored and compared using the procedures

and the combined taxonomy. In the future this technique could be applied to

comparability studies, which include a variety of question types. Such comparability

studies may compare examination questions from different years or assessment

providers. The combined taxonomy, together with the research procedure, effectively

identified if a variety of demands from a range of domains were placed on students as

well as highlighting whether there were questions of relatively low demand on which

less able candidates might achieve. It also identified relatively high demand questions

which will challenge able candidates. Thus, this approach could be applied to

validation studies, particularly when the assessment includes non-cognitive domains.

The research was unusual in gathering participants’ views on whether the combined

taxonomy facilitated the analysis of domains, types of thinking and demand.

Participants understanding of the concepts drew from their expertise and the

orientation task. Such work feeds into key assessment work on validity and

comparability, as already mentioned. For the combined taxonomy to aid analysing

domains, types of thinking and demand it must be readily applicable and appropriate

for the subject. The assessment experts’ views were largely positive on all of these




ISSN: 1758-2199

issues. This suggests that the combined taxonomy is a useful tool for analysing

domains, types of thinking and demands.

We followed established procedures for experts applying taxonomies. Examples

include experts in the examination questions participating in the research and

familiarising participants with the taxonomy used. The present research allowed

experts to record as many categories and operations as they wished for each item, as

the item might elicit more than one domain and could be answered at various levels

of thinking. For instance, it is likely that doing a presentation uses interpersonal skills

and information. This approach aligns with prior research (Johnson and Fuller, 2006;

Näsström and Henriksson, 2008). In other examples, experts were asked for one

taxonomy category per item (Ebadi and Shahbazian, 2015). Arguably, future research

could ask for participants to identify each domain category and the highest operation

elicited in association with that domain category. It could then be assumed that the

question might also elicit all lower categories and operations.

Greatorex and Suto (2016) theorised that the thinking systems and operations

(Marzano and Kendall, 2006, 2007, 2008) may be used to process interpersonal

knowledge. The present research provides evidence to support that theory. Finally,

the findings are important for qualification development. Taxonomies are often used

as guidance for writing educational objectives, items and mark schemes as well as

aligning these during the development process (Marzano and Kendall, 2007, 2008).

As the participants felt the combined taxonomy may be appropriate for many

qualifications and subjects, it follows that it may be useful in guiding syllabus and

item/mark scheme writing in many areas. In this regard our findings accord with those

of Greatorex and Suto (2016).

In conclusion, we found that the taxonomies of Hutchins et al. (2013) and Marzano

and Kendall (2007, 2008) can be combined. Moreover, the combined taxonomy was

successfully used to compare the domains, categories and operations elicited by a

variety of tasks and assessments, as well as compare the demand of such tasks and

assessments. Furthermore, this taxonomy is accessible and appropriate for a variety

of assessments and subjects.




ISSN: 1758-2199

References

Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E.,

Pintrich, P. R., Raths, J. and Wittrock, M. C. (2001) A taxonomy for learning, teaching

and assessing: a revision of Bloom’s taxonomy of educational objectives. New York:

Longman.

Atkinson, S. P. (2013) Taxonomy circles: Visualizing the possibilities of intended

learning outcomes. Learning and Teaching Working Paper. Available at:

https://spatkinson.files.wordpress.com/2015/01/taxonomy-circles-atkinson-aug13.pdf

Bandaranaike, S. and Willison, J. (2015) Building capacity for work-readiness:

Bridging the cognitive and affective domains. Asia-Pacific Journal of Cooperative

Education, Special Issue, Vol. 16(3), pp. 223-233.

Bichi, A. A., Hafiz, H. and Abdullahi, S. (2017) Evaluating secondary school students’

science achievement: Implication for curriculum implementation. International Journal

for Social Studies, Vol. 3(1), pp.113-121.

Biggs, J. B. and Collis, K. F. (1982) Evaluating the quality of learning – the SOLO

taxonomy. 6th ed. New York: Academic Press.

Bloom, B. S., Engelhart, M. D., Furst, E. D., Hill, W. H. and Krathwohl, D. R. (1956)

Taxonomy of educational objectives: the classification of educational goals. Handbook

1, Cognitive domain. London: Longmans.

Carpenter, T. D. and Wisecarver, M. M. (2004) Identifying and validating a model of

interpersonal performance dimensions. Alexandria: Army Research Institute for the

Behavioral and Social Sciences.

Carpenter, T.D., Wisecarver, M. M., Deagle III, E. A. and Mendini, K. G. (2005)

Special Forces interpersonal performance assessment system. FEDERAL

MANAGEMENT PARTNERS INC ALEXANDRIA VA.

Chan, C. CH., Chan, K., Tsui, M.-S. and Kwok, A. (2014) Towards a competence-

based evaluation framework: The personal growth of volunteers in a child injury

prevention program Journal of Social Work, Vol.14(5). doi:

10.1177/1468017313479080.




ISSN: 1758-2199

Chan, C. C, Tsui, M. S., Chan, M. Y. C. and Hong, J. H. (2002) Applying the

Structure of the Observed Learning Outcomes (SOLO) taxonomy on student’s

learning outcomes: an empirical study. Assessment & Evaluation in Higher

Education, Vol. 27(6), pp. 511- 529. doi: 10.1080/0260293022000020282.

Cradduck, L. Thomas, M. (2016) From the waist up: developing psychomotor skills

for the court room. International Journal of the Legal Profession, pp. 1-21. doi:

10.1080/09695958.2016.1226844.

Crisp, V. and Novaković, N. (2009) Is this year's exam as demanding as last year's?

Using a pilot method to evaluate the consistency of examination demands over time.

Evaluation and Research in Education, Vol.22(1), pp. 3-15. doi:

10.1080/09500790902855776.

Crowe, A., Dirks, C. and Wenderoth, M. P. (2008) Biology in Bloom: Implementing

Bloom’s taxonomy to enhance student learning in biology. CBE—Life Sciences

Education, Vol. 7, pp. 368-381. doi: 10.1187/cbe.08-05-0024.

Davidson, R. (1996) Cognitive complexity and performance in professional accounting

examinations. Accounting Education, Vol.5(3), pp. 219-231.

Dubas, J. M. and Toledo, S. A. (2016) Taking higher order thinking seriously: Using

Marzano’s taxonomy in the economics classroom. International Review of Economics

Education, Vol. 21, pp. 12-20. doi: 10.1016/j.iree.2015.10.005.

Ebadi, S. and Shahbazian, F. (2015) Exploring the cognitive level of final exams in

Iranian high schools: Focusing on Bloom’s taxonomy. Journal of Applied Linguistics

and Language Research, Vol. 2(4), pp. 1-11.

Edler, J., Eberman, L., Kahanov, L., Roman, C. and Mata, H. (2015) Athletic trainers’

knowledge regarding airway adjuncts. Athletic Training Education Journal, Vol.10(2),

pp. 164-169. doi: 10.4085/1002164.

Edwards, N. (2010) An analysis of the alignment of the grade 12 physical sciences

examination and the core curriculum in South Africa. South African Journal of

Education, Vol.30, pp. 571-590.




ISSN: 1758-2199

Faragher, L. and Huijser, H. (2014) Exploring evidence of higher order thinking skills

in the writing of first year undergraduates. The International Journal of the First Year

in Higher Education, Vol. 5(2), pp. 33-44. doi: 10.5204/intjfyhe.v5i2.230.

FitzPatrick, B. and Schulz, H. (2015) Do curriculum outcomes and assessment

activities in Science encourage higher order thinking? Canadian Journal of Science,

Mathematics and Technology Education, Vol.15(2), pp. 136-154. doi:

10.1080/14926156.2015.1014074.

Greatorex, J., Shaw, S., Hodson, P. and Ireland, J. (2013) Using scales of cognitive

demand in a validation study of Cambridge International A and AS Level Economics

Research Matters: A Cambridge Assessment Publication (15), pp.29-37.

Greatorex, J. and Suto, W. M. I. (2016) Extending educational taxonomies from

general to applied education: can they be used to write and review assessment

criteria? (Roundtable). Paper presented at the European Association for Research on

Learning and Instruction, Ludwig-Maximilians-Universität Munich, Germany.

Haolader, F. A., Ali, M. R. and Foysol, K. M. (2015) The taxonomy for learning,

teaching and assessing: Current practices at polytechnics in Bangladesh and its

effects in developing students' competences. International Journal for Research in

Vocational Education and Training, Vol.2(2), pp. 99-118. doi:

10.13152/IJRVET.2.2.2.

Harrow, A. J. (1972) A taxonomy for the Psychomotor domain: A guide for developing

behavioural objectives. New York: David McKay.

Hauenstein, A. D. (1998) A conceptual framework for educational objectives: A holistic

approach to traditional taxonomies. Lanham, MD: University Press of America.

Herman, J. L., Webb, N. M. and Zuniga, S. A. (2007) Measurement issues in the

alignment of standards and assessments. Applied Measurement in Education &

Training, Vol. 20(1), pp. 101-126. doi: 10.1080/08957340709336732.

Hiong, L. C. and Osman, K. (2013) A conceptual framework for the integration of 21st

Century Skills in biology education. Research Journal of Applied Sciences,

Engineering and Technology, Vol.6(16), pp. 2976-2983.




ISSN: 1758-2199

Hutchins, S., McDermott, P., Carolan, T., Gronowski, M., Fisher, A. and DeMay, M.

(2013) Interpersonal skills summary report Research Note 2013-03. Fort Belvoir: Alion

Science and Technology, Inc.

İlgüy, M., İlgüy, D., Fişekçioğlu, E. and Oktay, I. (2014) Comparison of case-based

and lecture-based learning in dental education using the SOLO taxonomy. Journal of

Dental Education, Vol.78(11), pp. 1521-1527.

Johnson, C. G. and Fuller, U. (2006) Is Bloom’s taxonomy appropriate for computer

science? Paper presented at the 6th Baltic Sea conference on computing education

research: Koli Calling 2006.

Johnson, M. and Mehta, S. (2011) Evaluating the CRAS framework: Development and

recommendations. Research Matters: A Cambridge Assessment Publication (12),

pp.27-33.

Kendall, J. S., Ryan, S., Weeks, S., Alpert, A., Schwols, A. and Moore, L. (2008)

Thinking and learning skills: What do we expect of students. Denver: Mid-continent

research for education and learning.

Klein, C., DeRouin, R. E. and Salas, E. (2006) Uncovering workplace interpersonal

skills: A review, framework, and research agenda. In: G. P. Hodgkinson and J. K. Ford

(Eds.), International Review of Industrial and Organizational Psychology 2006, Vol.

21. West Sussex: John Wiley & Sons.

Laxmisan, A., Hakimzadaa, F., Sayan, O. R., Green, R. A., Zhang, J. and Patel, V. L.

(2007) The multitasking clinician: Decision-making and cognitive demand during and

after team handoffs in emergency care. International Journal of Medical Informatics,

Vol. 76, pp. 801-811.

Luebke, S. and Lorié, J. (2013) Use of Bloom’s taxonomy in developing reading

comprehension specifications. Journal of Applied Testing Technology, Vol. 14.

Marzano, R. J. and Kendall, J. S. (2007) The new taxonomy of educational objectives.

2nd ed. USA: Corwin Press Inc.

Marzano, R. J. and Kendall, J. S. (2008) Designing and assessing educational

objectives: Applying the new taxonomy. United States of America: Corwin Press Inc.




ISSN: 1758-2199

Mayer, R. (2002) A taxonomy for computer-based assessment of problem solving.

Computers in Human Behavior, Vol.18(18), pp. 9. doi: 10.1016/S0747-5632(02)00020-1.

Nair, P. J. (2012) Is talent management accentuated by competency mapping? With

special reference to educational sector. International Journal of Social Science &

Interdisciplinary Research, Vol.1(11), pp. 132-147.

Nakyam, J., Kwangsawad, T. and Sriampai, P. (2013) The development of foreign

language substance group curriculum based on Marzano’s taxonomy. Educational

Research and Reviews, Vol. 8(14), pp. 1109-1116.

Näsström, G. and Henriksson, W. (2008) Alignment of standards and assessment: A

theoretical and empirical study of methods for alignment. Electronic Journal of

Research in Educational Psychology, Vol.6 (3), pp. 667-690.

Palmer, E. J. and Devitt, P. G. (2007) Assessment of higher order cognitive skills in

undergraduate education: modified essay or multiple choice questions? BMC medical

education, Vol. 7(49), pp. 1-7. doi: 10.1186/1472-6920-7-4.

Pate, M. L. and Miller, G. (2011) Effects of think–aloud pair problem solving on

secondary–level students’ performance in career and technical education courses.

Journal of Agricultural Education, Vol.52(1), pp. 120-131. doi: 10.5032/jae.2011.01120

Pollitt, A., Ahmed, A. and Crisp, V. (2007) The demands of examination syllabuses

and question papers. In: P. Newton, J. Baird, H. Goldstein, and H. Patrick (Eds.),

Techniques for monitoring the comparaility of examination standards (pp. 166-206).

London: Qualifications and Curriculum Authority.

QCA. (2008) Inter-subject comparability studies. Study 1a: GCSE, AS and A Level

geography and history: QCA.

Rembach, L. and Dison, L. (2016) Transforming taxonomies into rubrics: Using SOLO

in social science and inclusive education. Perspectives in Education, Vol.34(1), pp.

68-83. doi: 10.18820/2519593X/pie.v34i1.6.

Savic, M. and Kashef, M. (2013) Learning outcomes in affective domain within

contemporary architectural curricula. International Journal of Technology and Design

Education, Vol.23(4), pp. 987-1004. doi: 10.1007/s10798-013-9238-8.




ISSN: 1758-2199

Shaw, S. D. and Crisp, V. (2012) An approach to validation: Developing and applying

an approach for the validation of general qualifications. Research Matters: A

Cambridge Assessment Publication, Special Issue 3.

Toledo, S. and Dubas, J. M. (2016) Encouraging higher-order thinking in general

chemistry by scaffolding student learning using Marzano’s taxonomy. Journal of

Chemical Education, Vol.93, pp. 64−69. doi: 10.1021/acs.jchemed.5b00184.

Webb, N., M. Herman, J., L. and Webb, N., L. (2007) Alignment of mathematics state-

level standards and assessments: The role of reviewer agreement. Educational

Measurement: Issues and Practice, Summer, Vol.17 (2).

Yang, D. (2013) Comparing assessments within junior geography textbooks used in

mainland China. Journal of Geography, Vol.112(2), pp. 58-67. doi:

10.1080/00221341.2011.648211.




ISSN: 1758-2199

Appendices

Appendix 1: The Five Example Questions for the Exam Assessment

1. Complete the following sentence.

‘Global warming is…..’

a. the world getting hotter

b. the world getting colder

c. ice caps melting.

2. Here is a diagram in Draw (an Open Office computer programme). Save the file.

(Diagram Given)

3. Generate and test hypotheses about which is the best mode(s) of communication

(written, oral, face to face etc) for training overseas teachers to use a coursework mark

scheme.

4. Your task is to park and secure two cherry pickers on a lorry for transportation.

Set a goal for your performance. As you perform the task, identify and describe how

effectively you think you are performing.

5. Here are the price lists for running a conference at each of three Cambridge Colleges.

i. Select the best option.

ii. Explain how the information helped you select the best option.

Appendix 2: Summary

Responses to the question: ‘Which of the following were accessible? The description of…’

Number of positive responses

The Information domain 6

The Mental procedures domain 5

The Psychomotor procedures domain 5

The Interpersonal domain 6

The Thinking systems 5




ISSN: 1758-2199

‘Which of the following were appropriate for analysing the knowledge and thinking in [the

examination questions]1?’

Number of positive responses

The Information domain 6

The Mental procedures domain 6

The Psychomotor procedures domain 2

The Interpersonal domain 6

The Thinking systems 6

‘How easy was it to analyse the domains and thinking systems involved in the

assessment tasks?’

Easy Neutral Difficult

Written examination 1 3 1

Assignment 0 5 0

Presentation and Reflection 1 4 0

1 The original question gave the subject and qualification, which were anonymised for this paper.

Two taxonomies are better than one: towards a method of … · 2019-09-05 · Applying Taxonomies...

Documents

Transcript of Two taxonomies are better than one: towards a method of … · 2019-09-05 · Applying Taxonomies...