dB-SERC Mentor-Mentee Evidence-Based Teaching...
15
dB-SERC Mentor-Mentee Evidence-Based Teaching Award Adam J. Lee William C. Garrison III
Transcript of dB-SERC Mentor-Mentee Evidence-Based Teaching...
dB-SERCMentor-MenteeEvidence-BasedTeachingAward
AdamJ.LeeWilliamC.GarrisonIII
Designingandbuildingsecuresystemsishard!
2
“TheblackartofprogrammingSatan’scomputer”[1]
Longstandingdesigns andimplementations havebeenproveninsecure:
Needham-SchroederMan-in-the-middlediscovered
after17yearsinuse
OpenSSLHeartbleed vulnerabilitydiscovered
2yearsafterintroduced
Formalverificationisverydifficult,evenforexperiencedsoftwareengineers!
CS1653teachessecurityengineeringwithafocusonasemester-longgroupproject
3
CS1653:AppliedCryptographyandNetworkSecurity
Lecturespresentalgorithmsandprotocols,studentsapplytheseinaninterleavedsemesterproject
Inthisproject,studentsmust:• Workingroupsforthefullsemester• Proposetheirownsolutionstoadversarialtasks• Develop,maintain,andextendanon-trivialcodebase(~5klines)
Requiresbothdesign andcoding!
AsummaryoftheCS1653semesterproject
4
Studentsdevelopasecuredistributedfile-sharingsystem
Fivephases,eachconsideringadditionalsecuritythreats
Studentsmeetwithinstructortoproposesolutions,demowithTAaftersubmission
!"#$%&'
()*+,'-$).$)' /#"$'-$).$)'0'
/#"$'-$).$)'!'
!"
!"
!"
12%23$'/#"$4'
5+&6$%782&$'2%9'
12%23$'()*+,4'
Eventhebeststudentsrunintoproblemswiththisproject…
5
Themostcommonproblems:• Unevendistributionofwork• Lackofcommunicationamonggroupmembers• Procrastination,submittinglast-minute• Jugglingdesignandcode• Rushingthroughcode• Combiningcodewrittenbymultiplegroupmembers• Designandcodenotmatching,evolvingout-of-sync
Canusingaversioncontrolsystemhelpmitigatetheseissues?
Whydevelopcodeusingaversioncontrolsystem?
6
InaVCS,anychangetoacodebaseiscalledacommit
TheVCSmaintainsahistory of
previouscommitswithdescriptions
u1
u2u1
u2
mrgAcommitisrelative,toeasethemerging
ofworkfrommultipleusers
Commitlogsaretimeseriesdescribingdevelopmentatafinegranularity,andhavebeenusedforavarietyofexperiments:
• AdoptionofnewAPIsdoesnotkeeppacewiththeirdevelopment[2]• Programminglanguagedesignhasamodesteffectoncodequality[3]• Genderandtenurediversityarepositivepredictorsofproductivity[4]• Functionswithassertshavesignificantlyfewerdefects[5]• AskingquestionsonQ&Asitescatalyzesdevelopment(andviceversa)[6]
HowcanusingaVCSimprovetheCS1653projectforourstudents?
7
Stayorganized:studentsreviewtheirchangeswhencommitting
Commitlogsimprovecommunication:seewhatyourgroupmates havecompleted
Muchsimplermerging whenworkingsimultaneously:nomoreemailingcodeandmanuallycombining!
Continuoussubmission:workuntilthedeadline,committingasyougo
Whataboutusinganalytics?
VCSanalyticstoimprovethecourseproject
8
High-levelgoal:improvegroupperformance…how?
Duringthesemester• Useanalyticstodetectproblemsingroups• Allowtheinstructortointerveneasneeded
Betweensemesters• Useanalyticstodiscoverwhatmakessomegroupsmoresuccessful• Adjustcoursetoencouragebehaviorseen instronggroups
WecollecteddatafromSpring2015offeringofthecourse,andappliedthelessonslearnedtoSpring2016
• 2154:33students,14groups,4projectphases• 2164:33students,12groups,5projectphases
Whatanalyticscorrelatedwithgroupperformance?
9
Distributionoflabor
Per-weekworkcompleted
Others:Goodcommitmessages,workingondocumentationearly
ForSpring2016,ourchangeswereprimarilyinterventionary
10
Duringofficehoursmeetings,checkedlogsforindicators:• Earlyworkondocumentation• Balanceofcommitspermember• Descriptivecommitmessages
Whenconfrontedwithconcerns,studentshadarangeofresponses:
• Expressedregret,admittedtheyneededtoimprove• Defendedtheirgroupmates• Explainedspecialcircumstances
• “Johncouldn't commit,sochangeswentthroughme”• “Wemetatmyplaceandpair-programmed”
Inbetweenphases,offeredhelpmanaginggroupwork,etc.
Overall,studentsseemedto“bounceback”moresuccessfully
11
40
50
60
70
80
90
100
30 40 50 60 70 80
Gradeonnextprojectphaseafteragradeof80or below
Term2154 Term21640
10
20
30
40
50
60
70
80
90
100
MEAN MEDIAN
Gradeonnextprojectphaseafteragradeof80orbelow
Term2154 Term2164
Overall,studentsseemedto“bounceback”moresuccessfully
12
0
10
20
30
MEAN MEDIAN
Gradeincreaseonnextprojectphaseafteragradeof80orbelow
Term2154 Term2164
-10
0
10
20
30
40
50
30 40 50 60 70 80
Gradeincreaseonnextprojectphaseafteragradeof80or below
Term2154 Term2164
Projectgradesoverthecourseoftheterm
13
35
45
55
65
75
85
95
105
P1 P2 P3 P4 P5
2154GradesbyPhase
35
45
55
65
75
85
95
105
P1 P2 P3 P4 P5
2164GradesbyPhase
Futureimprovementstobemadeusingthesetechniques
14
Phase1seemedtobeharderduetoversioncontrol• HoldoffuntilPhase2?• ShorterassignmenttogetusedtoVC?• Grademoreleniently?
Phase3isstillthehardestoverall• Closertrackingofrepositories,evenoutsideofmeetings?• Givemoreguidance,leavelaterphasesmoreopen-ended?• Shorten,movesomematerialtoPhase4?
Outliergroupsneverrecover• Offermorepointedguidance?• Detectthistypeofgroup,breakupearly?
Questions?
15
References:1. RossJ.Anderson andRogerM.Needham,“Programming Satan’sComputer,” In
ComputerScienceToday:RecentTrendsandDevelopments,1995.2. TylerMcDonnell, Baishakhi Ray,Miryung Kim:AnEmpiricalStudyofAPIStability
andAdoption intheAndroid Ecosystem.ICSM2013:70-793. Baishakhi Ray,DarylPosnett,VladimirFilkov,Premkumar T.Devanbu:Alarge
scalestudyofprogramming languagesandcodequalityingithub.SIGSOFTFSE2014:155-165
4. Bogdan Vasilescu,DarylPosnett,Baishakhi Ray,MarkG.J.vandenBrand,AlexanderSerebrenik, Premkumar T.Devanbu,VladimirFilkov:GenderandTenureDiversityinGitHub Teams.CHI2015:3789-3798
5. Casalnuovo Casey,Devanbu Prem,OliveiraAbilio,Filkov Vladimir,andBaishakhiRay:AssertUseinGitHub Projects.ICSE2015
6. Bogdan Vasilescu,VladimirFilkov,AlexanderSerebrenik:StackOverflow andGitHub:AssociationsbetweenSoftwareDevelopment andCrowdsourcedKnowledge.SocialCom 2013:188-195
Thank you!
