Drivers and Barriers for the adoption of Computer Assisted Assessment Geoff Chapman Thomson...

Drivers and Barriers for the adoption of Computer Assisted Assessment

Geoff Chapman

Thomson Prometric

July 5th, 2005

Drivers and Barriers to CAA – Thomson Prometric

Rationale

QCA Blueprint for e-Assessment

Vision and Timeline

Systematic and independent Awarding Body survey

Understanding of needs and concerns


Research Context

Charts show research findings on a question-by-question basis

Findings broken by sub-group with significant differences– Currently delivering versus not currently delivering e-Assessment– Preferred testing location– QCA vision - Achievable versus Not Achievable– Reasons for not developing/piloting e-Assessment– Willingness to share test delivery locations and test design

with third parties

Few significant differences found amongst these sub-groups

Majority are those who are and aren’t currently delivering e-Assessment


Statistical Confidence

87 Awarding Body respondents

Findings give 95% confidence level with an approximate 5% confidence interval

26% of respondents currently deliver e-AssessmentOf these

% of programme 0-30% 31-60% 61-90% Over 90%

% of respondents 61% 22% 9% 4%

Average 29% of current programme being delivered


e-Assessment Delivery

Of the 74% of Awarding Bodies who do not currently deliver e-Assessment

– 46% are in the process of developing or piloting

Reasons given by the 54% not in the process of developing/piloting e-Assessment are provided on the right

– Perceived not suitable for nearly a third

9%

9%

9%

11%

17%

20%

20%

Small/new AB

Not suitable - deliver NVQ's

Not suitable - hold practicalassessments

Investigating with tech/swproviders

Resource/cost issues

Still at very early stages

Potential for the future

Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)


Robustness

84% believe e-Assessment to be equally as or more robust than paper-based

– There are few significant differences in responses given by those currently delivering and those not

– Those who believe that over 60% of their programme is suitable for e-Assessment are more likely to see it as more robust than a paper-based system

• 6%

39%

56%

6%

13%

13%

69%

6%

8%

19%

67%

6%

17%

22%

57%

6%

10%

20%

64%

Less robust

Don't know

More robust

Equally as robust

Overall

Currently running

Not currently running

Up to 60% of programmebelieved suitable

Over 60% of programmebelieved suitable

Q. Do you think that e-Assessment is a more robust, less robust or equally as robust method of assessment as paper-based assessment, on the whole? (Prompted)



Leading Awarding Bodies

Majority unable to strongly associate leadership with any UK Awarding Bodies

City & Guilds associated most

Minimal mention of SQA, OCR & AQA

Other ‘Awarding Bodies’ mentioned less frequently were

– Open University– American Certified Public Accountants – Teacher Training Agency

24%

24%

25%

13%

13%

53%

8%

71%

14%

3%

12%

9%

6%

8%

5%

SQA

AQA

OCR

Edexcel

City & Guilds

Other

No mention/don'tknow

1st mention

2nd mention

3rd mention

Q. Which Awarding Bodies do you associate with leadership in the field of e-Assessment? (Spontaneous)



Market insight: e-Assessment Examples

Overall, the majority of respondents who have seen existing programmes listed, thought that the City & Guilds ‘Gola’ system was the best demonstration

– The Theory Test delivered by the Driving Standards Agency received highest recognition for having the most encouraging outcome

21%

25%

29%

37%

58%

60%

10%

12%

12%

15%

14%

21%

ConstructionIndustry Health &

Safety Test

Institute of theMotor Industry

EDI system forLondon Chamber

of Commerce

City & Guilds’‘Gola’ system

Edexcel’s KeySkills Assessment

Driving StandardsAgency Theory

Test

Seen

Read/heard about


Q. Which of the following e-Assessment prototypes, trials and existing programmes have you either seen in action, read or heard about? (Prompted)


E-Assessment Suitability

Adult basic skills testing perceived to be suited to e-Assessment by the largest proportion of respondents

Other differences shown not significant

3%

10%

24%

40%

16%

6%

12%

33%

28%

13%

6%

16%

30%

24%

15%

3%

15%

39%

23%

10%

Not at allsuited

Not very wellsuited*

Neutral*

Quite wellsuited*

Extremely wellsuited

Vocational

Professional

Schools

Adult basic skills *Definitions derived from 5 point numeric scale


Q. On a scale of 1-5 where 5 is extremely well suited and 1 is not at all suited, how well suited do you feel e-Assessment is to each of the following types of education? (Prompted)


E-Assessment: Suitable Circumstances

0%

10%

20%

30%

40%

50%

Overall Currently delivering Currently not delivering

Q. In your opinion, in what types of situations or circumstances does e-Assessment work particularly well? (Spontaneous)



E-Assessment: Unsuitable Circumstances

0%

10%

20%

30%

40%

Data Currently delivering Currently not delivering

Q. In your opinion, what types of situations or circumstances is e-Assessment less well-suited to? (Spontaneous)



Advantages

0%

10%

20%

30%

40%

50%

60%

70%

80%


Q. What are the three main advantages of using e-Assessment, in your opinion, overall? (Spontaneous)



Disadvantages

0%

10%

20%

30%

40%

50%


Q. What are the three main disadvantages of using e-Assessment, in your opinion, overall? (Spontaneous)



E-Assessment: Risks associated

0%

10%

20%

30%

40% Overall Currently delivering Currently not delivering

Q. What risks do you associate with the introduction of e-Assessment, if any? (Spontaneous)



Costs

Some comments for reasoning

Cost effective as long as scaleable – i.e. larger AB’s who deliver more tests could justify high start-up costs, whereas smaller AB’s who run fewer exams may not be able to

Too early for some respondents to determine long-term savings or otherwise (may be easier to realise once up and running for a while)

16%

8%

11%

66%

0%

17%

13%

61%

14%

10%

12%

64%

Don't know

Less

Approx. equalto

More

Overall

Currently delivering

Not currently delivering

Q. Three years after implementation, will e-Assessment be more, less or equally cost-effective as paper and pencil tests?



Driving Uptake

0%

10%

20%

30%

40% Overall Currently delivering Currently not delivering

Q. Within the vocational/professional assessment arena, what do you believe to be the top three factors driving the uptake of e-Assessment? (Spontaneous)



Further Driving Uptake

0%

10%

20%Data Currently delivering Currently not delivering

Q. In your opinion, what additional factors could further drive take-up rates of e-Assessment? (Spontaneous)



Culture for e-Assessment

0% 5% 10% 15% 20% 25% 30%

1

2

3

4

5

6

7

8

9

10

Cu

ltu

re f

or

e-A

ssessm

en

t

%

Not running e-Assessment

Already running e-Assessment

Q. On a scale of 1-10 where 10 is completely and 1 is not at all, how ready is the culture within your Awarding Body for e-Assessment?



E-Assessment: Importance Factors

44.5

55.5

66.5

77.5

88.5

99.5

Overall Importance Already running e-AssessmentNot currently running e-Assessment

Q. On a scale of 1-10 where 10 is extremely important and 1 is not important at all, how important are each of the following factors to you in relation to e-Assessment? (Prompted)



Location Preference

Further spontaneous responses include:It depends

– Flexibility/suitability for assessment or candidate (17%)

– Candidate preference (5%)

Place of learning– Convenience/candidate familiarity (10%)– Practicalities/convenience for AB (10%)

Dedicated test centre– Quality controlled conditions (8%)– The only option/tradition/policy (7%)

Place of work– Policy/tradition (5%)– Office facilities available (3%)

13%

19%

31%

38%

17%

39%

17%

22%

14%

24%

28%

33%

Place of work

Dedicated testcentre

Place of learning

It depends

Overall

Currently running e-Assessment

Not currently runninge-Assessment

Q. Where does your organisation prefer to test? (Prompted)



Technology Providers – Assessment Location

0% 10% 20% 30%

1

2

3

4

5

6

7

8

9

10

Sh

are

d D

eli

very

Lo

cati

on

s

%



Q. On a scale of 1-10, how willing would you be to work collaboratively with third party technology solutions providers on shared e-Assessment locations? (Prompted)



Technology Providers – Test Design and Development

0% 10% 20% 30%

1

2

3

4

5

6

7

8

9

10

Willin

gn

es

s t

o C

olla

bo

rate

%



Q. On a scale of 1-10 where 10 is completely and 1 is not at all, how willing would you be to work collaboratively with third party technology solutions providers on test design and question development? (Prompted)



Technology Provider Competencies

• Cost effectiveness was highest 1st mention

• Overall adaptability, IT skill/support and affordability are top three attributes required in a technology provider

• Understanding of the market, reliability and flexibility feature as lower mentions

10%

12%

15%

16%

12%

17%

9%

6%

7%

9%

8%

14%

6%

10%

1%

2%

3%

6%

7%

13%

20%

Reliability

Understanding ofmarket/industry

Reputation/trackrecord/credibility

Experience/expertise

IT skills/support/customerservice

Adaptability/ability tolisten/tailor solutions

Costeffectiveness/affordability

1st mention

2nd mention

3rd mention


Q. What are the top three competencies, attributes or qualities would you seek in a third party technology provider? (Spontaneous)


QCA & FAB

Yes 22% (Currently delivering: 39%, Not currently delivering: 16%)

No 76% (Currently delivering: 57%, Not currently delivering: 83%)



Don’t know 25% (Currently delivering: 39%, Not currently delivering: 20%)



Don’t know 15% (Currently delivering: 22%, Not currently delivering: 13%)

Q. Do you believe that the QCA’s vision of full e-Assessment adoption by 2009 is achievable in the vocational/professional market?

Q. Do you perceive QCA to have the competencies to assess the technical capabilities and securities of e-Assessment used by Awarding Bodies?

Q. Do you believe that the FAB should adopt a significant and active role in driving e-Assessment knowledge?



Future of e-Assessment

2%

7%

15%

31%

39%

0%

100%

60 - 99%

31 - 59%

1 - 30%

6%

9%

11%

25%

42%

0%

100%

61 - 99%

31 - 59%

1 - 30%

Q. Approximately what percentage of your assessment programme do you realistically expect to be e-Assessment by 2009? (Spontaneous)

Q. What percentage of your programme do you believe could potentially be assessed by sophisticated multiple choice questions? (Spontaneous)



Further Feedback

Verbatim:

“There is a need for a clear definition of e-assessment. People tend to think it is very simplistic which is clearly not the case”

“It would be useful to receive information on what is going on with the developments and providers of e-assessment”

“E-Assessment is not a solution for everything but it is a very good solution for some things, for us this is only a small portion of we need”

“We are very willing to have discussions as to what an outside body could recommend for e-assessment for our subject”

“I have a worry that Government cuts in funding for qualifications will cause a conflict between cost and benefit”

“The QCA do not have a high enough profile to add to this campaign. A worry is that this is the way education is going to be taken over by businesses”

3%

5%

5%

5%

6%

8%

Reservation about ITcapabilities/range

Willing to look into/developfurther

Still working todevelop/investigate e-

Assessment

Organisation is positivetowards e-Assessment

e-Assessment is only partof the solution

Not enough informationavailable/publicised/shared

Thank You

[email protected]

July 5th, 2005

Drivers and Barriers for the adoption of Computer Assisted Assessment Geoff Chapman Thomson...

Documents

Transcript of Drivers and Barriers for the adoption of Computer Assisted Assessment Geoff Chapman Thomson...