1 June 10-15, 2012 Growing Community; Growing Possibilities Switching to on-line evaluations for...

1

June 10-15, 2012

Growing Community; Growing Possibilities

Switching to on-line Switching to on-line evaluations for courses evaluations for courses at UC Berkeleyat UC Berkeley

Daphne Ogle, Lead Design, UC BerkeleyAaron Zeckoski, Lead Developer,

Unicon

22012 Jasig Sakai Conference

SpeakersSpeakers Daphne Ogle Berkeley Design Lead

OAE since 2010, Fluid Project, Sakai since 2004

Aaron Zeckoski Unicon Development Lead

Evaluation System original lead (2006), Evals software since 2001, Sakai since 2004


Unicon ProfileUnicon Profile


Unicon technology & domain Unicon technology & domain expertiseexpertise


Unicon service capabilities / offeringsUnicon service capabilities / offerings


Intro to UC Berkeley on-line course evaluation project

Demo: Updates to CLE Evaluation System for Spring pilot

Future work What we learned Questions & Answers

Today’s topicsToday’s topics


Introduction to online Introduction to online evaluations project at UC evaluations project at UC

BerkeleyBerkeley

Overall project initiative


Online evaluations for coursesOnline evaluations for courses New shared questions

◦ campus-wide◦ instructional format-based (14 formats)◦ piloted first

On-line◦ pilot CLE Evaluation System◦ development to meet Berkeley needs & overall

system improvement◦ Berkeley & Unicon collaboration


Goals of initiativeGoals of initiative Rapid access to useful information to

help faculty improve their courses and their teaching effectiveness

Improvements to the quality & integrity of data that the campus uses to understand and recognize teaching contributions

21st century tools for staff to perform their jobs efficiently and free up time for direct service to students and faculty

High quality information for students about courses and instructors at Berkeley

Cynthia Schrager, Assistant Vice Provost

10


Project time lineProject time line

Summer 2010

Spring 2011

Summer 2011

Fall 2011

Spring 2012

Summer 2012

Fall 2012

Research new questions & instructional formats

Evaluation systems comparative analysis

Design, analysis & development CLE eval sys

(in UCB branch)

Campus buy-in

Fall 2010

Pilot new questions & instructional formats

Pilot CLE eval sys

Assess pilot

Design & development (intend to move to trunk)

Larger 2nd pilot

11


Spring Pilot

Introduction to online Introduction to online evaluations project at UC evaluations project at UC

BerkeleyBerkeley

12


Goals of spring pilotGoals of spring pilot Establish baseline response rate Assess CLE evaluation system

◦ technical longterm sustainability◦ user experience

Build knowledge of CLE evaluation system Build knowledge of infrastructure needs

13


Spring pilot success criteriaSpring pilot success criteria Establish baseline response rate Collect data on function & experience User experience meets or exceeds previous Transparency across campus Elicit higher quality response Gain understanding to inform Fall design &

development effort

14


Spring pilot makeupSpring pilot makeup 8 departments 61 courses (primary & secondary) 2300+ enrollments

15


Spring pilotSpring pilotDemo

16


Spring pilot developmentSpring pilot development Significant time understanding system Overall model & process to follow

◦ “uber” template◦ self-service

Focus on student & instructor experience System areas of focus

◦ Dashboard◦ Evaluation presentation◦ Administration page◦ Notifications

17


Dashboard - studentDashboard - student

18


Dashboard - administratorDashboard - administrator

19


EvaluationEvaluation

20


AdministrationAdministration

21


NotificationsNotifications

22


Bulk import / export configurationsBulk import / export configurations

23


Live DemoLive Demo

http://berkeley-eval.unicon.net/portal

http://berkeley-eval.unicon.net/portal

24


Lessons learnedLessons learnedFood for thought

25


Lessons learnedLessons learned Configurability + flexibility = Complexity

◦ documentation not always up to date◦ allow significant time to “play around”

Unfinished functionality◦ email notifications◦ options in evaluation creation

Global tool◦ acts different than other tools◦ where will it be exposed?

26


Lessons learned - cont’dLessons learned - cont’d Allow a lot of time for QA

◦ see 1st bullet: lots of variables◦ QA’ing up to the last minute

Clean data required◦ setup if not using group provider◦ site roles NOT always evaluation roles

Communicate!◦ be transparent◦ remind users it’s a pilot

27


Lessons learned - cont’dLessons learned - cont’d Learn from other implementors

◦ document models, workflows & implications◦ community project?

Culture & legal requirements vary across campus◦ viewing respondents◦ roles, permissions, viewing each others

evaluations

28


Future plansFuture plansEvaluation system at Berkeley

29


What’s next What’s next (1 of 2)(1 of 2)?? Feedback sessions with users on Spring pilot Needs analysis around results Move our work back into trunk Configuration option to hide respondents page UX improvements

◦ results page◦ multiple instructors (Berkeley-centric)◦ add respondent link to dashboard◦ make display options clear in context◦ configuration page (information architecture)

30


What else is next?What else is next? Investigation

◦ support for instructional formats (Berkeley-centric)◦ give department admins access to results◦ Evalsys tool added to My Workspace automatically◦ group provider & SIS integration◦ partial OAE integration

31


ResultsResults(early idea)(early idea)

32


Results(early idea)Results(early idea)

33


Sneak peak: dashboardSneak peak: dashboard

34


Sneak peak: scale creationSneak peak: scale creation

35


Sneak peak: scale creationSneak peak: scale creation

36


ResourcesResources Berkeley Confluence

◦ https://confluence.media.berkeley.edu/confluence/display/EVAL/Home

Sakai Confluence◦ https://confluence.sakaiproject.org/display/EVALSYS/Home

Sakai Jira◦ https://jira.sakaiproject.org/browse/EVALSYS

https://confluence.media.berkeley.edu/confluence/display/EVAL/Home

https://confluence.sakaiproject.org/display/EVALSYS/Home

37


Thank you!Thank you! Questions, comments, discussion...

1 June 10-15, 2012 Growing Community; Growing Possibilities Switching to on-line evaluations for...

Documents

Transcript of 1 June 10-15, 2012 Growing Community; Growing Possibilities Switching to on-line evaluations for...