Evaluation Report on Duke University’s Coursera ... Report on Duke University’s Coursera...

EvaluationReportonDukeUniversity’sCourseraSpecializationsonBusinessAnalyticsandJavaProgrammingKimManturuk,ProgramEvaluator,CenterforInstructionalTechnology

of 15

ExecutiveSummaryIn2015,DukeUniversityrapidlydevelopedandlaunchedSpecializationsinBusinessAnalyticsandJavaProgramming.WhiletheshorttimelineforcreatingtheseSpecializationscreatedseveralchallenges,thecourseshavebeenverysuccessfulintermsofattractingenrollments.Overall,theSpecializationshavebeensuccessfulatthefollowing:

• Providingcareer-oriented,high-demandskillstrainingforpeopleworkingin,orseekingtoenter,professionalfields

• GeneratingrevenueforDukeandforthecourseinstructors• RaisingDuke’svisibilityandreputationasoneoftheleadingeliteAmerican

universitiesofferingopen,onlineeducation• Providingthecatalystforseveralnewmethodstoassesscoursequalitythat

willbeusedinfutureprojectsincludingjust-in-timemodulefeedbacksurveysandin-housebetatesting.

ThereweresomeareasinwhichtheSpecializationswerenotassuccessfulasenvisionedasfollows:

• TheJavaProgrammingSpecializationfellshortofthestatedgoalofattractingwomenandpeopleofcolortothecoursesandthefieldofcomputerscience;additionaleffortsareunderwaytoaccomplishthisgoal

• TheBusinessAnalyticsSpecializationdidnotinitiallyincludesufficientqualityassuranceprocesssorevisionshadtobeundertakenafterthecourseslaunchedtocorrecterrors

• CourserastaffandpotentialindustrypartnersdidnotprovidethetechnicalsupportexpectedbyCITandthefacultymembers;revenueexpectationswerealsonotcommunicatedeffectively

Lessonslearnedfromthesetwoprojectsincludethebenefitsofhavingmoreflexibletimelinesandtheneedtosetrealisticexpectationsatthestartofaproject.Infuturesimilarprojects,effectivequalityassuranceandcommunicationprocesseswillbecriticaltosuccess.Overall,werecommendthatanysimilarprojectundertakeninthefutureexplicitlyspecify,ineithertheprojectagreementortheworkplan,howthefollowingtaskswillbecarriedout:

• ObtaindetailsinwritingofanysupportorresourcesthatwillbeprovidedbycompaniesorpartnersoutsideofDukeUniversity

• Forprojectsinvolvingmultiplefacultymembers,planinadvanceforhowrevenuewillbeadjustediftimecommitmentschangewithinthegroup

• EstablisheffectivequalityassuranceprocessbeforelaunchingcoursesanddesignateaQAleadperson

of 15

ProjectOverviewStartinginAugust2015,OnlineDukeandtheDukeCenterforInstructionalTechnologycollaboratedwithsixinstructorstodeveloptwoCourseraSpecializations.Fourinstructorsworkedtogethertocreatea5-course(4+capstone)SpecializationonJavaProgramming;twoinstructorscreatedaSpecializationonBusinessAnalyticsinthesameformat.Thesetwoprojectswereuniqueformanyreasonsincluding:

• Thecourseteamshadjust4weekstocreateeachofthe5courses,amuchfastertimelineforcoursecreationthanonanypriorproject

• TheprojectleadershadtocoordinatemuchlargercourseteamsthanDukehasusedinpriorprojects

• Thetechnologiesrequiredtocreatethecourseactivitiesandassessmentswerenewtotheprojectteamsonewskillshadtobedevelopedveryrapidly

• Just-in-timecoursefeedbackwasprovidedtocourseteamsthroughtheuseofveryshortfeedbacksurveysembeddedineachcoursemodule

• Courseteamswereawarethatthesecourseswouldlikelybeveryvisibleandgeneratesignificantrevenuefortheuniversity

ThecapstonecoursefortheJavaProgrammingSpecializationwascompletedinmid-January2016andtheBusinessAnalyticscourseteamfinishedproductionontheircapstonecourseinMarch2016.Sincethattime,wehavecollectedsurveydatafromlearnersinbothSpecializationsandtrackedlearneractivitypatterns.WealsoconductedinterviewswithmostofthecourseteammembersonbothSpecializationstounderstandthestaffandinstructorexperience.Thisreportsummarizesthatinformation.NotethatwehavealsouseddatafromseveralsourcestohelpbothcourseteamsidentifyelementsintheirSpecializationsthatwouldbenefitfromrevisions.Becausethoserevisionsarein-progress(JavaProgramming)orhavejustrecentlybeencompleted(BusinessAnalytics),anevaluationoftherevisionprocessandchangeswillnotbecoveredinthisreportbutwillbeforthcoming.Anupdateonrevisionsisprovidedattheendofthisreport

IndustryPartnershipsAfifthuniqueaspecttotheseSpecializationsthatonlypartiallymaterializedwasthattheyinvolvedseveralpartnershipsthatbothcontributedvaluableresourcestotheprojectyetdidnotalwaysdeliverasexpected.TheBusinessAnalyticsteampartneredwithAirbnbtoprovidelearnerswithdataforassignments.ThesameteamalsopartneredwithTerraDataandTableautoprovidestudentssoftwareaccess.Thesepartnershipsdidprovidelearnerswithvaluableresources,butCITprojectteamshadtodomoreworkthanexpectedtograntstudentsaccesstotheseresources.Infutureprojects,teamleadsshouldstartthisprocessassoonaspossibleandexplorewhetherDuke’sOITofficecanprovidesupport.

of 15

TheJavaProgrammingSpecializationwasinitiallygoingtobeofferedinpartnershipwithGoogle,butachangeinprioritiesduetointernalpersonnelchangesledGoogletosignificantlyreducetheroleitwouldplayinthedevelopmentandmarketingofthecourse.TheydecidednottoaffiliatetheGooglenamewiththecourseandlimitedtheirinvolvementtoprovidingsomeshortvideointerviewswithGoogleengineers.AllthefacultymemberswhoparticipatedinthisprojectwereveryfrustratedthatGooglelargelywithdrewfromtheproject;theplannedpartnershipwithGooglewasasignificantfeaturethatdrewthefacultymemberstoparticipateintheprojectbecausetheyanticipatedbeingabletoaccessexpertiseandresourcesfromGoogle.ThefacultymembersalsoanticipatedthatthepartnershipwithGooglewouldincreaseenrollmentsandthusimpactrevenuefromtheSpecialization.HenceGoogle’swithdrawfromtheprojectwasdisappointingforseveralreasons.Whilefuturesimilaroccurrencesarelargelyunavoidable,itisimportanttonotethenegativeimpactitwouldhaveonanyfutureprojectteamifanimportantindustrypartnershipweretofallthrough.

LearnerDemographicsLearnersenrolledintheSpecializationstypicallyfittheprofileoflearnersinothercourses.Table1belowshowssomedemographicmeasuresdividedbycourse.Mostlearnersalreadyhadacollegedegreeandwereemployed.BothSpecializationsenrolledfewerwomenthanmen,andfewerwomenthantheCourseraaverageof39%.TheJavaProgrammingcourseinstructorshadagoalofattractingwomanandpeopleofcolortothecourse,soadditionalmarketingeffortstoattractthosegroupsarerecommended.Table1:SelectedDemographicMeasures

BusinessAnalytics

Course1 Course2 Course3 Course4

Course5(capstone)

NorthAmerica 43% 30% 44% 42% 44%Women 30% 28% 28% 29% 25%Collegedegree 86% 84% 91% 88% 92%Employed 76% 77% 81% 74% 71%JavaProgramming


Course5(capstone)

NorthAmerica 36% 33% 32% 33% 38%Women 29% 20% 20% 17% 24%Collegedegree 63% 75% 78% 77% 88%Employed 70% 70% 70% 73% 79%

of 15

LearnerActivityDataBothSpecializationswere,asanticipated,highlyvisiblelikelyduetothefactthattheyfocusonteachinghigh-demand,career-orientedskillsthataretypicallynotpartofacollegeeducation.CoursetwointheBusinessAnalyticsSpecializationhas,asofthiswriting,attractedalmostahalfmillioncoursevisitorsandenrolledover100,000people.Thenumberofcoursevisitorsandenrollmentsforeachcourseareshowninthechartsbelow.Chart1:VisitorsandActiveLearners–BusinessAnalytics

Chart2:VisitorsandActiveLearners–JavaProgramming

98,660

426,135

98,813 103,672

19,49244,624

126,266

33,592 26,653458

Course1 Course2 Course3 Course4 Course5(capstone)

VisitorsActivelearners

147,178

106,398

66,528 68,786

16,147

52,202

30,28314,722 12,414

366


VisitorsActivelearners

of 15

Charts3and4belowshowthenumberofcoursepaymentsandapprovedfinancialaidapplications1foreachcourse.Alllearnershavetheoptiontoapplyforfinancialaidbywriting3briefessaysdescribingtheirfinancialneed,whytheywanttoenrollinacourse,andhowtheywillmaintainacademicintegrity.LearnerscanapplyforfinancialaidfromeithertheindividualcoursepagesorfromtheSpecializationhomepage,buttheymustapplyindividuallyforeachcourseintheSpecialization;itisnotpossibletocompleteoneapplicationandreceivefinancialaidforallfiveclasses.Asshownbelow,coursepaymentsconsistentlyoutpacefinancialaidapplicationsintheBusinessAnalyticscourses.Notethatthenumbersinthechartsbelowdonotrepresentindividuallearners;theyarecoursepayments,originatingfrombothindividualcoursepagesandfromtheSpecializationpage.Chart3:PaymentsandFinancialAid–BusinessAnalytics

IntheJavaProgrammingchart(below),thenumberstotherightofthelinearepaymentsandthenumberstotheleftarefinancialaidapplication.IncontrastwithBusinessAnalytics,thefirstcourseintheJavaProgrammingSpecializationhasreceivedmorefinancialaidenrollmentsthanpaidenrollments.Subsequentcoursesintheserieshavemorepaidenrollments.

1Courseraallowslearnerstoapplyforfinancialaidbyansweringafewquestions.Ifapproved,theyreceiveafeewaiverforonecourse.ThisprocessisentirelyinternaltoCourseraandisnotassociatedinanywaywithfederalfinancialaidoranyschool-basedaidprograms.

9,544

7,072

5,3814,823

3,262

5,071

3,824

1,876 1,681

216


PaymentsFinancialaid

of 15

Chart4:PaymentsandFinancialAid–JavaProgramming

Thenexttwochartsshowtheconversionpercentwithineachclassfromcoursevisitorstoactivelearners,andfromactivelearnerstopaidenrollments.Notethatweusetheterm“conversion”loosely;weknowthatnotallpaidlearnersstartasactivelearnersandcompleteanactivitypriortopurchasinganenrollment.However,displayingthedatainthismannerishelpfulforillustratingwhatportionofactivelearnersarepaidlearners.Asillustratedpreviously,thesecondcourseintheBusinessAnalyticsserieshasattractedthehighestnumberofcoursevisitorsandenrollmentsofanycourseineitherSpecialization.However,ithasthelowestactive-to-paidconversionrateat6%.Thisindicatesthat94%ofpeoplewhoareactiveinthecoursehavenotpaid.Sinceweknowthatthenumberoffinancialaidlearnersisroughlyhalfthenumberofpaidlearners,wecaninferthataround91%ofactivelearnersareauditingthecourse.

5,718

2,326

1,711 1,555 1,432

5,940

1,884877

632183


PaymentsFinancialaid

of 15

Chart5:ConversionPercents–BusinessAnalytics

AcrossallcoursesintheJavaProgrammingSpecialization,theactive-to-paidconversionrateisbetween8%and13%.Thereisadeclineinthepercentofcoursevisitorswhobecomeactiveinacoursewitheachsubsequentcourseacrosstheseries.Thiscouldbetheresultofthecoursesbuildingoneachothersoitwouldbedifficultforsomeonetovisitacourseinthemiddleoftheseriesandbecomeactiveiftheyhadnotalreadydonethepriorcourses.ThisislesstrueintheBusinessAnalyticsseries.Chart6:ConversionPercents–JavaProgramming

18%

16%

6%

21%

26%

34%

30%

45%

Course4

Course3

Course2

Course1

VisitorconversiontoactivelearnerActivelearnerconversiontopaid

13%

12%

8%

11%

18%

22%

28%

35%

Course4

Course3

Course2

Course1

Visitorconversiontoactivelearner

Activelearnerconversiontopaid

of 15

LearnerFeedbackSurveysAlllearnerswhoenrollinanyofthecoursesineitherSpecializationareemailedasurveyatthebeginningofthecourse.Alllearnerswhosuccessfullycompleteacourseareemailedafollow-upsurveysolicitingtheirfeedback.Thereisanexceptiontothis;pre-coursesurveyswerenotdoneintheJavaProgrammingcoursesbecauseGoogleresearchersweredistributingtheirownsurveysatthebeginningofeachcourseinthatseries.Wereceivedapprovaltoresumesurveysinthosecourseslastmonthsothosedatawillbeavailableinfollow-upstothisreport.Inadditiontothelearnerfeedbacksurveys,wealsocollecteddataonthelearnerexperiencefromtheOnlineCourseBuilderswhowereactiveineachofthecoursesandthushadinsightintowhatlearnersfounddifficultorchallenging.Thereareseverallimitationstothelearnerfeedbacksurveysthatshouldberecognized.First,theyarenotgeneralizabletoalllearners.Onlypeoplewhosuccessfullycompleteacourseareemailedthepost-coursesurveysotheresultsshouldbegeneralizedonlytothatsub-groupofenrolledlearners.Second,evenwithinthegroupofcoursecompleters,theresponseratesarelowandpeopleself-selectwhetherornottotakethesurvey.Itisnotknownhowselectionbiasimpactsthedata,butitlikelydoes.Inspiteofthoselimitations,thedataareveryusefulforindicatingtrendsandpatterns.Whenthedatafromonecourselooksdifferentfromtherest,itindicatesthereislikelyasignificantdifferenceinthatcourse.AnexampleofthisisshowninTable2below.Thistableshowsthemeanoverallcourseratingscoresgiventoeachcourse.Learnerswereasked,oncetheyfinishedtheentirecourse,toassignascorefrom1(verypoor)to7(excellent).AllthecoursesinthetwoSpecializationshaveaveragescoresbetween5.7and6.62withtwoexceptions:BusinessAnalyticscoursetwohasalowerscore(5),andJavaProgrammingcoursefivehasahigherscore(6.6).Basedonthis,weworkedwiththecoursedevelopmentteamandtheinstructorsofthesecondBusinessAnalyticscoursetoidentifywhytheratingwaslowerandtargetrevisionstoaddressproblems.Thoserevisionsarecurrentlybeingcompleted.Table2:OverallCourseRatings(Max.7)


Course5(capstone)

BusinessAnalytics 5.9 5 6.1 5.8 5.7JavaProgramming 5.7 6.0 6.2 6.1 6.6WeusedtheoverallcourseratingsasastartingpointtoidentifywhylearnersgavetheJavaProgrammingcapstonecoursehigher-than-averageratings.Severalstudentscommentedthattheyfoundthecapstoneprojecttobeagoodlearningexperiencebecauseitusedmanydifferentskillstheyhadlearnedthroughouttheseries.Theyalsopraisedthe“real-world”natureoftheprojectandthefactthattheskillstheyusedwerethingstheywouldputtofutureuse.Forexample,astudentnamedCathywrotethefollowingcommentaboutherexperienceafterthecapstone:

of 15

Thiswholespecializationwasfantastic!IwasalinguisticsmajorasanundergradandtriedtolearnJava,butIreallystruggledwithit.ThisfallIamenteringgradschoolforamastersprograminHumanLanguageTechnology.Iamaimingtowardacareerincomputationallinguisticsandnaturallanguageprocessingbutlearningtoprogramhasbeenahugehurdle.NowthatIhavecompletedtheSpecializationIhavesomuchmoreconfidenceinmyprogrammingskillsandIfeelreadytoapplythemingradschool.

Otherstudentscommentedthat,“Thiswasanawesomeproject.ItwasgreatgettingtoimplementeverythingIlearnedoverthecertificateprogram”,and“Thiscapstoneprojectisanicewaytoreinforcethelearningofthematerialinthecourse”.Asanalternativemeasure,wealsocomparedthecoursescoresassignedbylearnersthroughtheCourseraplatform.Learnerscanclickonstarstoassignacoursearatingbetween1and5stars(notethatthescaleusedbyCourseraisa5-pointscalewhileDuke’slearnersurveysreportedinTable2usea7-pointscale).Duke’soverallcourseaverageis4.52andCoursera’soverallaverageis4.63.TheBusinessAnalyticsSpecializationcoursesreceivedratingsof4.5,4.2,4.6,4.7,and4.7.Thisisconsistentwiththepatternseeninthelearnerfeedbacksurveys.TheJavaProgrammingscoresare4.4,4.5,4.7,4.6,and4.7,alsokeepingwiththepatternseeninthelearnerfeedbackandcomparabletoothercoursesofferedontheplatform.Thesehighcourseratesarealsoreflectedinthestudentcommentsprovidedinthefeedbacksurveys.Forexample,alearnerinthefirstcourseintheJavaProgramingSpecializationwrote:

Iwouldliketoexpressmygratitudetothecourseinstructors,Susan,Owen,AndrewandRobertfortheiroutstandingaffordstoteachusprogramming.I'velearntmuchmoreaboutprogrammingandmoreimportantlynowIfeelmyselfconfidenttosolvingprogrammingproblem.TheinstructorsprovideahighqualityofeducationofcomputerscienceinveryusefulmannerandmademefeelI'mreallydoingprogramming.

Similarly,astudentinthethirdBusinessAnalyticscourseofferedpraiseforthewayallthecourseelementsworkedtogether:

ThankstoProf.Jana.Thewholelearningexperienceisexcellent.Thevideolectures,practicequizandgradedquizallworkedtogethertodeliveranexcellentcourse.Itisbyfarthebestinitsclass-beingtechniqueandtooloriented.Prof.Jana'slectureontheneuroscienceofvisualperceptionandattentionisthebestfoundationIhadonvisualizationandcommunication.

of 15

AsecondsetofmeasuresweusedtoevaluatethelearnerexperienceintheSpecializationsarequestionsaskingpeopletoself-assesstheirownlearningandtheextenttowhichtheyfeltthecoursewasengaging.Charts7and8belowindicatethepercentoflearnersineachcoursewhosaidtheyagreedwiththefollowingtwostatements:

• FortheamountoftimeIinvestedinthiscourse,I’mhappywithwhatIlearned

• ThecoursematerialswerepresentedinanengagingmannerThedatafromtheBusinessAnalyticsSpecializationconfirmsthepatternindicatedbytheoverallcourseratingscores.Farfewerlearnersinthesecondcoursearehappywithwhattheylearnedorfindthecoursematerialsengaging.IntheJavaProgrammingSpecialization,thepercentoflearnersindicatingeachoutcomeissimilaracrossallcourses.Overall,thelearnersareextremelysatisfiedwiththecourses.Chart7:LearnerSurveys–BusinessAnalytics

93%

60%

95% 90%100%

91%

52%

97%89% 86%


HappywithwhatIlearnedCoursewasengaging

of 15

Chart8:LearnerSurveys–JavaProgramming

StaffandFacultyExperienceWeconductedinterviewswiththeprojectleads,coursesupportstaff,andinstructorstounderstandthestaffandfacultyexperience.Wealsowantedtoidentifyprocessesorexperiencesthatcouldbechangedtoimprovethecoursedevelopmentprocessinthefuture.Duringthecourseofthoseinterviews,severalthemesemergedthatwererepeatedinthemajorityofconversations.Thefollowingsectiondescribesthosethemes.

SettingExpectationsEarlyItisimportanttosetexpectationsforthecoursedevelopmentprocessandtomakeallexpectationsexplicit,notimplicit.Everypersonwespoketocitedatleastonetime,andsometimesmanytimes,duringdevelopmentwhenpeopleonacourseteamhaddifferentexpectationsforhowsomethingwouldbedoneanditcausedprojectdelays,frustration,andpeoplerepeatingwork.Specifictopicsthatshouldbeaddressedup-frontinclude:

• Specificdivisionoftaskswithintheprojectteam,includinginstructors• Howtohandleerrorsandmistakes–whotocontactandhowtheyrespond• Realisticenrollmentnumbersandroyaltyamounts• Expectationsforfrequencyandformatofcommunication• Aprocessforrevisitingandrevisingtaskassignmentsasneeded

Severalpeoplesuggestedhavingalongerkick-offmeetingtoreviewexpectations.Similarly,onepersonsuggestedhavingavideokick-offmeetingfirstandthenwaitingaweekandhavingafullteamkick-off.Thisletstheinstructorsseethe

88% 92% 95% 93% 94%87% 89% 93% 93% 94%


HappywithwhatIlearnedCoursewasengaging

of 15

typesofvideoavailableandcometotheteamkick-offreadytomakedecisions.Anotherpersonsuggestedhavingawork-orientedkick-offmeetingandaseparatekick-offsocialeventtobuildteamcohesion.

NavigatingShortTimelinesAsdiscussedatthebeginningofthisreport,oneofthebiggestchallengeswithboththeseSpecializationswasthattheywerecreatedonaveryshorttimeline–about4weeksforeachcourse.Almosteveryoneweinterviewedmentionedthisasachallenge,andseveralinstructorssaidtheywouldnotdoaprojectlikethisagainunlessthetimelinewasmuchlonger.Weidentifiedseveralreasonswhythetimelinewasproblematicandchangesthatcouldhelpalleviatetheburdencreatedbythetimeline.First,thetimelinewasproblematicbecausecoursebuildingisnotlinear.Theprocessbecomesfasteraspeoplegainexperience;theJavainstructors,forexample,feltthatbuildingcoursefourwentmuchfasterthanbuildingcourseone.Thesettimeline,however,allowed4weeksforeachcourse.Second,therearesometasksduringcoursedevelopmentthatrequireasignificantamountofinactivetime,suchaswaitingforatechnologytooltobebuiltorapproved.Alineartimelineplacestheseeventssequentiallyinsteadofsimultaneously.Finally,otherdemandsonfacultymembers’timearenotconstant.Severalinstructorssaidthattheyhadtimesduringtheprojectwhentheycouldhavedonemorethantheyhadscheduled,andtimeswhentheywereunabletodoanyworkontheprojectatall.Theobviousrecommendationistoallowforalongertimelineonfutureprojects.Fourweeksisnotenoughtimetobuildacourseundermostcircumstances.Beyondthat,someotherpossiblechangesare:

• Identifyaback-uppersonforeachmajortasksothatifthingsfallbehind,thereisapersontohelp

• Beginallinactivetasks(seekingapprovals,gainingcopyrightclearance,workingwithvendors)atthestartoftheproject

• Overlapcoursedevelopmentwhenpossible,suchashavinginstructorsplanallthecoursesinaseriessotheyhavetheoptiontomakelargeamountsofvideocontentformultiplecourseswhentheyfindtimeavailable

QualityAssuranceThetwoSpecializationteamstookdifferentapproachestoqualityassurance.TheJavaProgrammingteamaskedanOnlineCourseBuilderwithastrongtechnicalbackgroundandcodingexperiencetotakeallofthecoursesasalearner,assumingthatifsheexperienceddifficultylearninganyofthematerial,itwouldposeproblemsformostpeople.Thisapproachhelpedthemidentifyseveralplaceswheretheyneededtomakechangestobothvideosandassessments.Therewaswidespreadagreementthat,whiletimeconsuming,thisapproachtoqualityassurancewasveryeffectiveandallowedthemtomakechangesbeforeacourseopenedorshortlythereafter.

of 15

TheBusinessAnalyticsteamreliedmoreheavilyonthemodulesurveysthatbeganafteracourselaunchedforfeedback.Theyalsolookedatthingslearnerswereflagginginthecourseandfixederrorswhentheywerepresent.Thiswasaneffectiveapproachforcorrectingerrors,butdidnotallowthemtheoptiontofixthembeforethecourseopened.Itwasalsonotalwaysclearwhowasresponsibleforlookingatfeedbackandflagsandmakingchanges;sometimestheprojectstafffeltthattheinstructorsneededtomakechangessincetheywerethecontentexpertsbutsometimestheerrorswereissuesbestfixedbypeoplemorefamiliarwiththeplatform.Bothprojectteamsagreedthatbetatestingofcourseswasveryimportant.Giventheshorttimeline,betatestingdidnotalwayshappenbeforeacourseopened.Whenitdidnothappen,therewerenoticeablymorelearner-reportederrors.

SupportStaffAfull-timededicatedTeachingAssistant(TA)whowashiredbythefacultymembersassistedintheJavaProgrammingcoursesbyansweringlearnerquestionsandparticipatinginthediscussionforums.About5-10hoursaweekofhertimeisspentrespondingtolearnerquestions.Shealsoassistswithdevelopingandimprovingcontent,creatingassessments,anddesigningA/Bteststoexperimentwithnewmaterials.Shealsohelpedthecoursementors2byansweringtheirquestionsandprovidingguidanceonhowtheyshouldrespondtovariousissues;shetypicallyrespondedthesamedayaquestionwasposted.ItislikelythatthereasonpeoplevolunteeredtobementorsisbecausetheTAwasactiveinthecoursefromthebeginningandlearnerssawthediscussionforumsasawaytogetinvolved.ItisclearthathavingadedicatedTAwhoisacontentexperthasgreatlyenhancedthelearnerexperienceintheJavaProgrammingcoursesandbeeninstrumentalinmakingthediscussionforumsatoolforactiveandengagingdiscussions.Incontrast,theBusinessAnalyticsSpecializationsdidnothaveadedicatedTAandtheinstructorswerenotactiveinthediscussionforums.Thisresultedinlearnerpostswithquestionsnotreceivingquickresponseswhenthequestionswerecontent-related(CITstaffmostlyrespondtonon-contentquestionssuchashowtoaccessaresource).Overtime,somelearnershavevolunteeredtobementors,butthey have not received a lot of direction on what they should do. One volunteer mentor posted in the dedicated mentor forum:

I'mhavingtroublekeepingupalreadywiththerequestsforhelpintheforums,especiallysinceI'vebeenawayfromthismaterialfornearlyayearandwillneedtimetore-masterit.Arethereothermentorsforthiscoursewhocanhelp?

2Courseracoursementorsarelearnerswhosuccessfullycompletedthecourseandthenvolunteeredtomentorothersthroughdiscussionforumparticipation.

of 15

Ittook17daysforsomeonetorespondtothisquestion,andtheeventualresponsecamefromanothermentor.SeveraloftheweeklydiscussionforumsintheBusinessAnalyticscoursesdonothaveanyactivediscussionsfor1-2monthsatatime.ThiswasespeciallyproblematicintheMySQLclassbecauselearnersfoundittobeatechnicallydifficultcourseandthereforehadalotofquestions.Thecourseteamrespondedbyhiring2part-timeTAstospend7-10hoursaweekrespondingtoquestionsintheforums.Thecourseteamreportsthatthishasmakeasignificantpositiveimpactonlearnersatisfactionandreducedthenumberofquestionsbeingaskedbecauselearnerscansearchforexistingthreadsthatanswertheirquestion.Wheneverpossible,atleastapart-timeTAwithcontentexpertiseshouldbehiredforfuturecourses.

RevisionsBasedonapreponderanceofevidence,theBusinessAnalyticsteamtargetedcoursetwoforrevisions.Weanalyzedalltheavailabledataandprovidedthecourseprojectteamwithareportrecommendingchangesandidentifyingtargetsbywhichtomeasuresuccess.Thecoursewasre-launchedwithrevisionsasofSeptember1.Atthistime,wehaveonlycollected17post-coursesurveyssoitistooearlytoassesswhethertherevisionshavebeensuccessful.However,basedonreadingstudentcommentsinthemodulesurveys,itappearsthattherevisionshaverectifiedmostifnotalltheproblems.

Evaluation Report on Duke University’s Coursera ... Report on Duke University’s Coursera...

Documents

Transcript of Evaluation Report on Duke University’s Coursera ... Report on Duke University’s Coursera...