Bb Teaching & Learning Presentation

25
Achieving High Student Evaluation of Teaching Response Rates

description

 

Transcript of Bb Teaching & Learning Presentation

Page 1: Bb Teaching & Learning Presentation

Achieving High Student Evaluation of Teaching Response Rates

Page 2: Bb Teaching & Learning Presentation

2

OverviewRationale

Pilot Project

Process

Things to Consider

Page 3: Bb Teaching & Learning Presentation

3

Who we areThe Office of Quality, Teaching, and Learning is a service unit for academics and students. We are made up of three services – Quality Assurance (QA), Teaching and Learning Services (TLS), and Student Learning Support (SLS).

Services we provide:

• Teaching approaches, strategies and ideas

• Professional development as a teaching academic

• Student learning support

• Assessment, assignments, and marking

• Applying for Teaching and Learning Awards and Grants

• Student evaluation of teaching (TEVAL) processes and applying student feedback

Page 4: Bb Teaching & Learning Presentation

4

Fast factsBond University received the most five-star ratings out of any university in Australia in the independent 2012 Good Universities Guide

Bond had an average student enrolment of 4,480 students in 2011, 1,449 of which were international students representing 85 nationalities.

There are four Faculties and one Institute:• Faculty of Business: School of Business; School of Information Technology; School of Hotel, Resort & Tourism

Management

• Faculty of Health Sciences & Medicine

• Faculty of Humanities & Social Sciences

• Faculty of Law

• Institute of Sustainable Development & Architecture

Page 5: Bb Teaching & Learning Presentation

5

Background• Traditionally decentralised paper-based system

• New electronic format inspired by former executive BUSA members

• Mandatory evaluations voted in by University Teaching and Learning Committee and Academic Senate

• Pilot project Semester 3, 2009 - Success

Page 6: Bb Teaching & Learning Presentation

6

ResearchSource Response Rate (%)

Paper-Based SETsResponse Rate (%)

Online SETs

Baruch (1999) - 40Ballantyne (2005) 55 47Brigham Young* 71 50Cook et al. (2000) 56 -Cornell University* 78 50Dommeyer et al. (2004) 75 43Griffith University (2005) 57 20Krieg and Hartsoch (2010) 74.2 56.8Nair et al. (2005) 56 31Ogier (2005) 65 30Sweep (2006) 56 23University of North Carolina* 92 59Watt et al. (2002) 33 33Average 64.7 40.7

Table 2: Published response rates

Table adapted from Nulty, D. (2008). ‘The adequacy of response rates to online and paper surveys: What can be done?’ Assessment & Evaluation in Higher Education, 33(3), 303. *Results cited in Hartshorne, R., Lambert, R., Algozzine, B., McAlpin, V., Algozzine, M., Norris, L. & Pyke, J.G. (2011). A comparison of web-based and paper-based course evaluations at UNC Charlotte: A report prepared by The Center for Educational Measurement and Evaluation and The Center for Teaching and Learning. Retrieved from University of North Carolina Charlotte website: http://studentcourseevals.uncc.edu/sites/all/files/Website%20Report%20-%20Course%20Evaluation%20Study.pdf

Page 7: Bb Teaching & Learning Presentation

7

Instrument - EvaluationKit

1. Blackboard building block – single sign-on with LMS and access to Blackboard groups/class data

2. Highest security rating

3. Extensive troubleshooting & customer support

http://www.evaluationkit.com

Page 8: Bb Teaching & Learning Presentation

8

Key Functional Requirements/Consultations

• Mandatory evaluations - LMS pop-up module to encourage participation

• Skip/Opt-out option

• Groups tool – maintains accuracy of class allocations

• Administrator level automated report builder

• Content Analysis Software integration

Page 9: Bb Teaching & Learning Presentation

9

Process - Overview

• eTEVALs administered at the end of each semester

• Timeframe: Week 10 - 14

• Two separate evaluations: Subject and Educator evaluations (5 point Likert)

• Subject Evaluation: 5 questions, 1 open-text comment box; Educator Evaluation: 10 questions, 2 open-text comment boxes

• Instructor/Administrator reports available after student results are released

Page 10: Bb Teaching & Learning Presentation

10

The Process• Access via email

Page 11: Bb Teaching & Learning Presentation

11

The Process• Access via iLearn (LMS)

Page 12: Bb Teaching & Learning Presentation

12

The Process• eTEVAL Dashboard

Page 13: Bb Teaching & Learning Presentation

13

The Process• Skip/Opt out option

Page 14: Bb Teaching & Learning Presentation

14

ResultsImposed Week 12 sanction increased response rate

Table 1: Week 12 Response Rate Results - (30/03/2012)

Response Rates (%)Level Subject evaluation Educator evaluation

Overall Bond Educator Evaluation 50.66% (+12.91%) Bond Subject Evaluation 55.72% (+13.06%) Faculty of Business

Business 58.98% (+13.82%) 55.54% (+14.03%) IT 55.95% (+10.41%) 51.38% (+11.32%) HRTM 56.69% (+14.01%) 51.59% (+13.37%)Institute of Sustainable Development and Architecture School of Sustainable Development 55.30% (+17.25%) 49.77% (+18.94%) School of Architecture 46.10% (+10.39%) 43.51% (+8.45%)Faculty of Humanities & Social Sciences 53.27% (+11.98%) 46.13% (+11.72%)Faculty of Health Sciences & Medicine School of Health Sciences 58.13% (+7.72%) 54.31% (+10.96%) School of Medicine 60.49% (+3.70%) 53.81% (+3.01%)Faculty of Law 54.27% (+15.74%) 50.80% (+16.96%)Bond College/BUELI 71.38% (+21.69%) 69.47% (+28.84%)

Opt-Out Option ResponsesOpt-Out 400 (+312) 970 (+630)

Page 15: Bb Teaching & Learning Presentation

15

ResultsResponse rates of nearly 90% on the Likert scale items of online student evaluation of teaching in the first semester of whole-of-university implementation.

Table 2: Final Response Rate Results - (23/04/2012)

Response Rates (%)Level Subject evaluation Educator evaluation

Overall Bond Educator Evaluation 86.04% (+0.03%) Bond Subject Evaluation 89.22% (+0.23%) Faculty of Business

Business 91.54% (+0.08%) 90.48% (+0.10%) IT 86.61% (+0.45%) 83.03% (+0.16%) HRTM 88.54% (+0.00%) 87.90% (+0.00%)Institute of Sustainable Development and Architecture School of Sustainable Development 90.52% (+0.14%) 89.72% (+0.69%) School of Architecture 82.47% (+0.65%) 82.47% (+0.00%)Faculty of Humanities & Social Sciences 87.63% (+0.31%) 85.19% (+0.60%)Faculty of Health Sciences & Medicine School of Health Sciences 88.95% (+0.00%) 88.24% (+0.00%) School of Medicine 72.84% (+1.85%) 61.52% (+0.47%)Faculty of Law 90.94% (+0.24%) 89.51% (+0.37%)Bond College/BUELI 88.05% (+0.94%) 86.95% (+0.84%)

Opt-Out Option ResponsesOpt-Out 1097 (+6) 2514 (+13)

Page 16: Bb Teaching & Learning Presentation

16

Trend of responses during the evaluation period

Results

Notable spike at Week 12 – Imposed sanction for non-respondents

Page 17: Bb Teaching & Learning Presentation

17

Results

Continued success – Semester 2, 2012

• Increased awareness and support from students and faculty

• Contributing to a culture of evaluation

Subject Evaluation Educator Evaluation

Actual Response Rate 82.57% (+2%) 82.84% (+8%)

Opt Out Response Rate 1,478 (+13%) 2,516 (+13%)

Aggregated Response Rate 96.31% (+7%) 96.16% (+10%)

Page 18: Bb Teaching & Learning Presentation

18

Campaigns

http://www.bond.edu.au/etevals

Page 19: Bb Teaching & Learning Presentation

19

Subject Evolution Reports (SERs)

• Closing the loop on student feedback

• Inspired by former executive members of BUSA

• TEQSA - Provider Standards: - Standard 5.6 - Standard 4.2

• Bond University initiative

Page 20: Bb Teaching & Learning Presentation

20

The SER ProcesseSETs close

(Semester a)

Quantitative and Qualitative data

analysed

Administrator eSET report access(Semester b)

Administrator review and action SER items

(if applicable)

Faculty Associate Dean Teaching &

Learning review and approve

Quality Assurance and approved for

publication(Week 2 Semester c)

Subject Coordinators may decide that the comments cannot be actioned (“didn’t like this subject”);

there may be no negative comments to action; Positive comments cannot be actioned (“loved this

subject”)

Subject Coordinator selects the “No action Taken”

Button and "Submit" button and SER remains unchanged

Page 21: Bb Teaching & Learning Presentation

21

Subject Evolution Reports (SERs)

Page 22: Bb Teaching & Learning Presentation

22

Subject Evolution Reports (SERs)

Page 23: Bb Teaching & Learning Presentation

23

Things to consider

eTEVALs

Communication

AdministrationCollaboration

Publication

Page 24: Bb Teaching & Learning Presentation

24

Research

Kinash, S., Naidu, V. & Wood, K. (2012). Electronic teaching evaluation: Student perceptions and teacher responses. Education Technology Solutions, 48, 60-62.

Knight, D., Naidu, V. & Kinash, S. (2012, Oct). Achieving high student evaluation of teaching response rates through a culture of academic-student collaboration. Paper to be presented at the Australasian Higher Education Evaluation Forum 2012, Rockhampton, Queensland, Australia.

Kinash, S., Knight, D. & Hives, L. (2011). Student perspective on electronic evaluation of teaching. Studies in Learning, Evaluation, Innovation and Development, 8(1), 86-97.

http://works.bepress.com/shelley_kinash

Page 25: Bb Teaching & Learning Presentation

25

Questions