Post on 18-Jan-2016
description
Achieving Learning Objectives Online: Not All Platforms are Equal!
Nick FeamsterGeorgia Tech
A Tale of Two MOOCs
• A free Coursera MOOC on Software Defined Networking to over 50,000 enrolled students– Scalable delivery, assignments, and forums
• A for-credit Udacity “MOOC” on Computer Networking for Georgia Tech’s Online MS program, to 200+ enrolled students– Not so scalable…
Subtle differences in platform design have significant effects on a course’s ability to scale.
MOOC #1: Software Defined Networking
• Extremely hot, new topic– No existing “real world” courses on the topic
(yet)– A chance to develop an archetype course
(and material) that others might use in their own courses
– I wanted to be copied. This seemed like a good way.
• Boutique topic means that there were no “set in stone” ways of teaching the course– Easier to think outside the box, since there
was no existing material anywhere
http://blog.sflow.com/2012/05/software-defined-networking.html
Two Offerings So Far:• Summer 2013
(6 weeks)• Summer 2014
(8 weeks)
Who Takes The Course?
• 50,000+ people registered each year– 10,000 people active
each week– 1,000-3,000 watching
lectures(3,500+ in 2014)
• 500+ turn in programming assignments each year
Other Demographics (Summer 2014)
• 55,000+ Enrolled• 197 Countries– 31% from the United States– 15% from India
• Sex and Age– 86% Male– 74% 20-39 years old
• Education and Employment– 58% Employed full-time– 26% Full-time students– 38% have Undergrad degree, 31% have Masters
What the Instructor Sees
How Was the Course Developed?• Course lesson plan, with learning objectives• Top down design: – Figure out modules, continue subdividing until you
get 10-minute “lectures”• Production– Lecture Filming and Production– Assignments– Syllabus
• Operation– Forums
Filming Take 1: Studio
• Started in a studio at Georgia Tech• This quickly proved intractable– Studio is distracting. Lots of people watching
makes recording nerve-wracking.– Studio time is limited, contentious, and expensive.– Post-production quality was not to my liking.– Faculty members need scheduling autonomy.
• A lot different from a TV interview!
Filming Take 2: Camtasia
• Studio people will claim that quality suffers.• I believe this is bogus, for several reasons.– People can take their time recording, take breaks, record
when they are “in the zone”.– Screen captures from a laptop are easy.– Recording quality from a good laptop is quite fine.– Nobody has ever complained about the quality of the
recordings. Content is what matters most.
• Takes a little getting used to at first (looking at the camera, etc.)
• You have to do your own editing.
Producing Lectures in Camtasia
• General rule: 10-15 minutes of lecture == 2 hours of recording and editing– This does not include time to prep slides, etc.
• This is way different than a usual lecture!– Biggest lesson: Silence is quite easy to edit!– Stumbling is OK: Just say the same thing again– You can also “practice” while the video is rolling
• Need to find a quiet, well-lit location• Takes patience, but it starts to be quite fun
General Lecture Production Process• Determine learning objectives• Research the topic • Write slides– General format: Summary/outline, 7-10 content slides,
wrap-up
• Plan demonstrations– Plan on-screen landscape
• Record (one sitting for consistency…big challenge!)• Edit and export• Upload• Note: You can prepare lectures out-of-order!
New Modes of Lecture Production• On-screen demonstrations
– Videos lend themselves very nicely to on-screen demonstrations
– User can see things in “life size”– User can pause, rewind, etc. Very helpful
tutorials for working through assignments• Interviews with luminaries in the field
– Google “On Air” Hangouts– Can be done anywhere, and people are
willing!– Process: Invite, script questions, send,
revise. Stay roughly on script– Many people have since copied this idea
(Nick McKeown, Alex Orso, Mike Hicks, etc.)
Lessons from Lecture Production• Lots of money is not needed. Students want
content, clearly delivered. Camtasia works.• A 45–60-minute lecture is about 15 minutes of
well-polished video. Content distills well.• Lecturer is always “ahead” of the students.
This creates interesting time-shifting effects, with advantages and disadvantages.
• MOOCs are asynchronous and autonomous for the student. This is a big reason for their success. Production should be the same.
How Was the Course Developed?• Course lesson plan, with learning objectives• Top down design: – Figure out modules, continue subdividing until you
get 10-minute “lectures”• Production– Lecture Filming and Production– Assignments– Syllabus
• Operation– Forums
Developing Scalable Quizzes: Stick to the Basics
• Problem: The platform is not very good at handling anything that’s not multiple choice or simple numerical answer.– Short answers become massive regular expressions– …will likely improve as the software gets better
• Stick to multiple choice if possible• Main differences from a normal MC quiz– Helps to write up explanations for each option– Need to figure out if students can re-take the quiz, and if
so, how many times
Developing Assignments is Hard• Wanted to teach concepts and programming– Not a “heavy-duty” programming course, but
some concepts are best explained and learned through short code exercises.
• Code submission possible, but grading must be done on regular expression evaluation of output
• Problem: How to guarantee uniform programming environments?
Uniform Environment: Virtual Machine
• Asked students to:– Install VirtualBox (or equivalent) virtual machine emulator– Download 64-bit VM image that had everything pre-
packaged (no asking students to install software themselves). Kind of like a lab at home.
• This has generally worked. A few problems– VM is huge (~1 GB), so have to get it right the first time– Not everyone has a 64-bit machine– Certain performance, even when run in an emulator, varies
depending on underlying hardware
How Was the Course Developed?• Course lesson plan, with learning objectives• Top-down design: – Figure out modules, continue subdividing until you
get 10-minute “lectures”• Production– Lecture Filming and Production– Assignments– Syllabus
• Operation– Forums
Communicating with 10,000 Students Can Scale!
• I feared absolute disaster– Small glitches in
assignments in a classroom of 50 students create mayhem.
– I feared that I would be spending my life answering minutia in online forums.
• Thankfully, this did not happen!• Lesson: Giving an assignment
to 10,000 students in a MOOC is easier than giving it to 50 in a classroom!
Why Assignments Can Scale• First of all, you do your best to work
out the bugs, mostly out of fear – We attempted everything that
we assigned, sometimes on multiple platforms
• More importantly: Self-selection!– Everyone doing the assignments
wants to be there– This makes a huge difference– People not only fix issues
themselves, they rewrite documentation
• Caveat: Course difficulties vs. platform difficulties are currently hard to tease apart. Some of this is “user error”.
A student’s blog post describing part of the assignment setup in detail…
Some Feedback…
A Tale of Two MOOCs
• A free Coursera MOOC on Software Defined Networking to over 50,000 enrolled students
• A for-credit Udacity “MOOC” on Computer Networking for Georgia Tech’s Online MS program, to 200+ enrolled students
“MOOC” #2: Online MS at GT
• Online version of CS 6250: Computer Networking
• Two offerings thus far (Spring and Summer 2014)– About 120+ students in each offering– Paying tuition for a “degree”– Not specialists in networking
• Technical difference: Platform
How Was the Course Developed?• Top-down planning, with “scripts”• Several full-time staff members– Course developer from Udacity– Video editor from Udacity
• Part-time from others– Three paid TAs/Graders– Many other people handling logistics
• Course recorded in studio, ahead of time, with custom equipment– Recording done in “marathon stints”– Not possible to integrate interviews
• No auto-grading, little visibility of content, etc.• Initially, all material released at once (huge mistake)
Comparing the ApproachesCoursera (SDN MOOC)
Udacity (OMS “MOOC”)
Video Delivery Camtasia-recorded videos, talking head and slides + on-screen demos
Tablet writing + moving hand
Grading Simple MC quizzes, auto-graded programming assignments.
No auto-grading! TAs graded assignments “manually”.
Forums Initially “home brew”. Moved to Piazza.
Piazza, but the students expect more direct instructor engagement.
Pacing One week at a time, synchronous (though people fall behind)
All-at-once release.
Demographics Professionals and students taking the course for free, for enrichment.
Paying students who are not necessarily interested in the subject. Entitled.
Cost 300+ hours of my time, 40+ hours of TA time (all “Free”)
Estimated: $150k (staff salaries, studio time, etc.)
Leave the Pedagogy to the Instructors
• Substance trumps style.
• A MOOC platform should allow instructors to deliver the content in whatever format they see fit.
• Mandating a particular “style” emphasizes style over substance.
GT OMS Degree “MOOCs” Are Neither Massive nor Open
• One of the important features of MOOCs that allows scaling is community support.
• Things that dilute this support:– Students who pay and expect staff contact– Students who are not interested in the content,
but rather trying to get a degree– “Go at your own pace” style (students are on their
own, course staff can’t keep up)• Other platform-specific factors prevent scaling– Instructor has less autonomy in creation and
management of content– Support for auto-grading is critical!
Lesson #1: MOOCs Change the Game for Instruction
• Everyone is watching you– In seven years, never got feedback (or
acknowledgment) about on-campus course– Now, everyone’s eyes are on you– Huge pressure to do a good job
• MOOCs are a “forcing function” for introducing new media into the classroom– I wouldn’t have thought to do “on air” interviews for a
classroom; people would likely decline them anyhow– I will likely re-use some MOOC content in the “real”
classroom
Lesson #2: Time Investment Pays Off• Typical Course Creation (45-minute lecture)
– 1-2 hours to create slides– 1 hour to deliver the lecture– Number of students: 50-100– Total: 3 hours per lecture– … but then you have to do it again! (can’t save an awesome
lecture, students can’t rewatch, etc.)• MOOC Course Creation (10-minute video)
– 1-2 hours to create slides– 2-3 hours to produce the lecture– Total: 3-5 hours per lecture– Number of students: 500-100,000– … but, you have the archive. Can keep improving it, use it to
enhance in-classroom discussion, etc.
Lesson #3: Certain Aspects of MOOCs Scale Surprisingly Well
• Lectures are watchable by thousands– The scale can be a motivator for guest lectures!
• Programming assignments can be graded automatically, if designed well
• Self-motivated, self-selected students fix glitches, help each other out
Conclusions• Scalable MOOCs require– Community engagement– Low barriers to creating content– Support for a diversity of delivery modes to match
the material• Not all platforms are equal– A platform should give the instructor autonomy
and visibility.• For-credit “MOOCs” face far different scaling
challenges than true MOOCs.– At the moment: neither massive, nor open