Rapid Reaction and Response Project

Rapid Reaction and Response - R³

The in-class use of mobile technologies to support formative assessment and feedback

Dr. Ann OomsSenior LecturerFaculty of Health and Social Care SciencesKingston University and St George’s University of London

National Student Satisfaction Survey

Question

2007NSSUK

2008NSSUK

2007NSSKU

2008NSS KU

1 - 4 The teaching on my course

82 83 79 82

5 – 9 Assessment and feedback

62 64 59 64

10 – 12 Academic support 71 73 67 72

13 – 15 Organisation & management

71 73 68 73

16 – 18 Learning resources 80 81 73 77

19 - 21 Personal development 77 78 75 77

22 Overall satisfaction 81 82 78 81

RationaleIn-class use of mobile technologies

Students identify misunderstandings and weaknesses

focus their learning activities and efforts

diminish misunderstandings

Educators identify students’ misconceptions, challenges

information regarding teaching practices

adapt teaching practices

Immediate feedback

Technologies to support rapid feedback

Interactive Blackboard Tools Blog Wiki

Electronic Voting Systems Tablet PC’s Interactive tablets Mobile phones (EduTXT Texting

Service)

Electronic Voting Systems

Tablet PC

Interactive Tablets

EduTXT Texting Service

Rapid Reaction and Response Project

•HEA Funded Research Project : £42,600•Aim: To examine the effective use of in-

class mobile technologies to enhance formative assessment and feedback

Research Questions

•Technology▫Under which conditions can each of the

technologies be effectively and efficiently used for formative assessment and feedback?

Research Questions

•Learning▫ What is the impact of the in-class use of mobile

technologies for formative assessment and feedback on: Students’ attitudes toward the module? On students’ conceptual understanding? On students’ test results?

Research Questions

• Teaching▫ What is the impact of the R3 project on:

Teaching practices (and will they be sustained?)

Assessment practices (and will they be sustained?)

What indicators are there of institutional commitment?

Research Methodology

•Mixed methods methodology▫Student data

Questionnaires Attendance records

▫Participant data Questionnaires Interviews Reflective journals

Lecturers

Faculty Lecturers

FADA 2

FASS 1

Business and Law 2

CISM 2

Engineering 2

HSCS 2

Science 2

13

Support Provided to Lecturers

•Workshop on assessment and item-writing (Bob Rotherham)

•Workshops on use of the technologies•Support from two mentors •Peer support meetings (with mentors)•Support from e-assessment developer•Group meetings (reflection and additional

workshops)

Data from Lecturers

Use of Technology

Technology Number of lecturers

Electronic Voting System 9

Tablet PC 5

BlackBoard Content System 2

SchoolPad Interactive Pads 1

Text Messaging 1

i-Pods 2

Effectiveness of technologies

Rank order of effectiveness of technologies per interaction and feedback pattern

Technology L-S S-L S-S

Electronic Voting System

1 1 3

Tablet PC 3 1 1

BlackBoard Content System

2 2 2

SchoolPad Interactive Pads

Text Messaging 2

How satisfied are your students in terms of formative assessment activities you provided?

Dissatisfied Somewhat dissatisfied

Somewhat satisfied

Satisfied

Pre 3 15 3

Post 0 10 9

How satisfied are your students in terms of feedback they received?


Somewhat satisfied

Satisfied

Pre 1 11 6 3

Post 6 13

How satisfied are you in terms of your ability to provide feedback to your students?


Somewhat satisfied

Satisfied

Pre 9 11 2

Post 1 12 6

Impact of project

No impact

Minor impact

Impact Major impact

Mean

Students’ enjoyment of classroom sessions

0 6 12 2 3.67

Students’ attendance 7 5 6 1 2.22

Students’ engagement 1 2 14 3 4.00

Students’ conceptual understanding

3 5 12 0 3.00

Students’ test results 6 9 4 0 1.78

Impact of project

No impact

Minor impact

Impact Major impact

Mean

Overall teaching experience

0 3 15 2 4.00

Information you gained about your students’ misunderstandings

0 2 12 5 4.22

Information you gained about your teaching practices

0 1 16 2 4.22

Informed your reflection on your teaching practices

1 1 15 1 4.00

Positive impact of the use of mobile technologies on the interaction and feedback patterns

No impact

Minor impact

Impact Major impact

Mean

M1 L – S 1 1 7 1 1.80

S – L 9 1 2.10

S – S 1 3 6 1.50

M2 L – S 1 2 3 4 2.00

S – L 7 3 2.30

S – S 2 6 1 1.89

To what extent do you feel the use of mobile technologies enhanced your teaching

Learning curve• “I could have used it more. I think maybe in the

future when I am more confident with it, probably.”

• “I did the right thing by not trying to do too much, and gauging what’s comfortable for me and the students.”

• “I only used it at the beginning and end of the lecture. I did not use it continuously throughout and that probably might be my next aim.”

• “I need to think deeper away from the technology and what I am doing.”

Benefits•Students were more engaged

▫“I was able to engage students who would not normally participate”

▫“Students responded quicker than they normally do when I ask them a question”

▫“There was a real ‘buzz’ in the room, with students engaging with each other.”

▫“I liked the way they worked as teams and they did too!”

▫“Students were generally excited at using the pads.”

Benefits• Gained insights on students’ understanding

▫“It made the students more aware of what they didn’t know, rather than what they did know.”

▫“I think the real time feedback and the anonymity help to highlight some of the weaknesses or the strengths.”

▫“When I see a third of them don’t actually answer the question correctly, then I think either ‘Well they are not good enough students’ or ‘I need to explain it in a different way’, or a mixture of both.”

▫“Reassured me that my verbal explanations were being understood.”

Benefits•Lecturers gained insights on their teaching

▫“It made me more aware to interact with the students in the class”

▫“I am understanding more about what the students understand so I can act on that”

▫“I was disappointed by their responses, they were lower that I had anticipated”

▫“The technology allowed me to identify areas of concept development that needed to emphasised and to improve my own input for future lectures.”

Challenges

• Access to the technologies ▫“Collecting and returning the equipment from

other campuses is a bit time consuming.”• Unreliability of the technologies

▫“Students did get impatient with the failure of the wireless network.”

• Time it takes to set it up in class▫“Allow time to test the technology before the

students arrive.”▫“Allow 10 minutes to set up.”▫“Have the technology pre-installed in some

classrooms to avoid long set up time.”

Challenges

•Time it takes in class ▫“I ended up about a lecture behind so that

is about 10%.”▫“Allow 5 minutes minimum per voting

slide.”•Time to redesign lecture

▫“To use it effectively you need to integrate it with your lecture, and not just as an add-on and in order to do that I need time to redesign the lecture.”

Data from Students

Impact of the in-classroom use of technologies

n Disagree

Somewhat disagree

Somewhat agree

Agree

Made it more enjoyable to attend the classroom sessions

455 3.7 7.0 40.7 48.6

Made the classroom sessions more interactive

454 3.1 6.4 32.6 57.9

Had a positive impact on my motivation to study for this module

452 6.0 19.2 43.1 31.6

Impact of the in classroom use of technologies

n Disagree

Somewhat

disagree

Somewhat agree

Agree

Was useful for feedback on my understanding

452 3.8 17.1 40.9 38.2

Provided information about my misunderstanding of certain components

455 7.0 20.0 44.2 28.8

Had a positive impact on my understanding of the material taught

451 3.8 14.9 47.7 33.7

Assisted me in focusing my study efforts

450 7.3 23.1 38.4 31.1

Descriptive Statistics

Question Mean

More interactive 2.45

More enjoyable 2.34

Feedback on understanding 2.14

Impact on motivation 2.00

Impact on understanding 2.11

Info about misunderstandings 1.95

Focusing study efforts 1.93

Overall, the use of the new technologies was a positive experience for me as a student

n = 456 Percent

Disagree 3.5

Somewhat disagree 9.2

Somewhat agree 36.8

Agree 50.4

I would advise my lecturer to keep using new technologies in his/her modules

n = 456 Percent

Disagree 2.6


Somewhat agree 31.8

Agree 59.9

I would like other lecturers to used new technologies in their modules

n = 452 Percent

Disagree 2.9


Somewhat agree 33.0

Agree 55.5

Open-ended Questions: Positive Responses• “The clickers made things more “fun” and made us

involved. Keep them!”• “They were fun to draw on.”• “Innovative and enjoyable to use.”• “For less motivated students it gave them an opportunity

to get involved instead of just sitting there.”• “Areas of weakness and strength are highlighted.”• “It makes the lecture very engaging and interesting.”• “It would be nicer if they did it in every module.” • “Using the tablets for group work ensured that you had to

do the work because there was a chance that it was going to be shown on screen for everyone to see. That was good.”

Open-ended Questions: Suggestions for improving

“Use them more regularly”“Have the technology pre-installed”“More clickers so that everyone can have

one”“Use in seminars”“Useful for revision lectures”“Make sure they all have batteries”“More questions and more time explaining

the answers”“Increase the reliability of the technologies”

Sustainability - Dissemination

•Seven lecturers became mentors•One mentor became master-mentor•Conference presentations•Publications: in progress

▫Two-day writing retreat in April 2009 7 participants 1 mentor Management team Dr Martin Oliver (Institute of Education) Dr Jane Seale (University of Southampton)

Contact Information

•Ann Ooms: [email protected]

•R3 Web site: ▫http://staffnet.kingston.ac.uk/

~ku36708/RRR

Rapid Reaction and Response Project

Education

Transcript of Rapid Reaction and Response Project