2016 Millennial Perspectives on K-12 Education and School Choice - Summary
Student Perspectives of TBL in a CS Course: Summary of ... · Student Perspectives of Team-Based...
Transcript of Student Perspectives of TBL in a CS Course: Summary of ... · Student Perspectives of Team-Based...
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Student Perspectives of TBL in a CS Course: Summary of Qualitative Findings
Michael S. Kirkpatrick SIGCSE Technical Symposium
March 2017
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
TBL implementation
Team-based Learning (TBL)• Flipped pedagogy developed by Larry Michaelsen • Course divided into 5-7 modules • Founded on constructivism and group process dynamics
Preparation (Pre-Class)
Readiness Assurance Diagnostic - Feedback Application of Course Concepts
45-75 minutes class time 1-4 hours class time
1 2 3 4 5 6Individual
Study
Individual TestTeam Test
Written AppealsInstructor Feedback Application
Oriented Activities
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Pre-module preparationModule 1 Reading Guide
Required readings:• Watch the video lectures linked on Canvas on C programming basics• Practical C Programming: sections 4.8, 5.2, 9.2, 13.1, 13.2, 13.3, 17.1, 17.2
• Skim additional sections for reference as needed
Objectives and questions:• Summarize the structure of a typical C program
• What types of #include statements are most similar to Java API imports?• What types of #include statements are most similar to Java imports of classes in
the same package?• What is the purpose of a function prototype?• Every executable program must have what function defined?• If a function is defined in one source code file but used in another, how do you
expose the function so the compiler can check for the correct parameter types?• If assume that counter is a global variable of type int. How would you declare
this variable in a header file to make it visible to multiple source files?• Compile and execute a C program on the command line
• What would you type on the command line to compile hello_world.c into an object file?
• What would you type on the command line to link hello.o and goodbye.o into an executable called greetings?
• What are the 3 components of a Makefile rule?• Assume that a Makefile has a rule with the target name clean. What would you
type on the command line to execute that rule?• Demonstrate basic C programming techniques, including stdio and string processing
• If x is a uint32_t (unsigned 32-bit integer), how would you print its value in C?• If x is a string in C, how would you print its value?• How many bytes are required to store the string ”my name” in C?• Assume you have an array int arr[5]. What would happen if you did arr[10] = 7?
• What would be the C equivalent of the following Java class definition?! class Junk {
! ! int x;! ! char y[4];! }
• Summarize the fundamental segments of the C memory model• Global variables are stored in what segment of memory?• Local variables are stored in what segment of memory?
JMU CS 350: Computer Organization • Spring 2015
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Readiness Assurance Process (RAP)
CS 350 – Spring 2015 Name:
By submitting this paper, I affirm my compliance with the JMU Honor Code.
Module 1 iRAT
1. Which of the following lines of C code is most like these lines of Java?
import java.io.*;import System.out.*;
(A) #include "stdio.h" (C) #include <stdio.h>(B) #include <stdio.c> (D) #include "stdio.c"
2. Which of the following command lines will compile and link rudolph.c intoan executable called reindeer.exe?
(A) gcc -c rudolph.c -o reindeer.exe(B) gcc rudolph.c -o reindeer.exe -O0(C) gcc -g reindeer.exe rudolph.c(D) gcc -c -o reindeer.exe rudolph.c
3. Which phase of the compilation process creates a single executable file bymerging one or more object files?
(A) assembling (C) linking(B) loading (D) preprocessing
4. Which of the following statements about Makefiles is FALSE?
(A) All dependencies for a target must be satisfied before that target’srecipe will be run.(B) The recipe is the sequence of system commands that will be executedwhen make runs a target.(C) If no target is indicated, make will execute the all target by default.(D) The clean target has no inherent meaning, but is commonly used fordeleting object files.
5. Which of the following would be stored on the stack?
(A) A dynamically allocated string that is a duplicate of the string declaredas char str[] = "hi";(B) A function’s local variable node that is a pointer to a part of a datastructure(C) A global variable references that keeps track of how many times afunction gets called(D) The character ’h’ in the local string char *str = "hello world";
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Application activities
C code Assembly language Machine language
x = ~(x >> 2);
y = x + 10;
printf (“%c\n”,y);
x += (char)y;
Two-pass assemblyThe following program has gone through the first pass of a two-pass assembler. Complete the symbol table that is used as input to the second pass. Below that table is the output from the first pass. Write the final bytes of the executable on the line above the first pass bytes.0000! C10000! main:! LDA x,d0003! 710000! ! ADDA y,d0006! E10000! L1:! STA x,d0009! 390000! ! DECO x,d000C! 00! ! STOP000D! 0005! x:! .WORD 5000F! 0002! y:! .WORD 2
C1 00 00 71 00 00 E1 00 00 39 00 00 00 00 05 00 02
Memory layout - C vs. Pep/8Using the memory layout of the C memory model to the right, fill in the missing addresses for the following assembly language program. Are these values realistic for a normal C program? What difficulty would arise, particularly in regard to pointers?
JMU CS 350: Computer Organization • Spring 2015
1239 461238 451237 441236 431235 421234 41
FFFFKernel
FBCFKernel
FBCE
--
--
Address Depend Symbol Value
0000 LDA x,d0003 ANDA y,d0006 STBYTEA z,d0009 CHARO z,d000C STOP000D z: .BYTE 0000E x: .WORD 0xA94F0010 y: .WORD 0xFFFD
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Major assessments
Module 1 Module 2 Module 3 Module 4 Module 5 Module 6
Team Exam
Individual Exam
Team Exam
Final Exam
Project 1 Project 2 Project 3 Project 4
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Prior results (ITiCSE 2015)
4050
6070
80CS 350 Grade
CS
450
Fin
al E
xam
Sco
re
D C B A
Fall 2013 C experienceFall 2013 No CFall 2012 C experienceFall 2012 No C
Figure 5: Visual representation of the linear regres-sion model of final exam score
the primary goal was to find a way to bring active learninginto our systems courses.Finally, we also need to address a commonly cited concern
about switching to active learning techniques: its impact oncoverage. That is, instructors often worry that adopting anew pedagogy will interfere with their ability to cover all re-quired material. In this instance, we do not feel that such aconcern is warranted. After doing an informal retrospectivemapping to the ACM 2013 curriculum recommendations,we found no differences between coverage of Core Tier-1and Core Tier-2 topics. Any differences between coverageof Elective topics was negligible. Consequently, we find thatadopting these techniques will not have any measurable im-pact on coverage.
5. CONCLUSION AND FUTURE WORKIn Fall 2013, we introduced two new techniques into our
OS course in an attempt to increase the amount of activelearning that happened in the classroom. These two tech-niques, the RAP and the RSQC2, require a very modestamount of effort on the part of the instructor, but initial ev-idence indicates that they can be powerful first steps towardimproving student learning. Our statistical analysis, whichcompares the Fall 2012 and Fall 2013 offerings of OS, showsthat there were statistically significant improvements in a va-riety of measurements, even when demographic differenceswere considered. Thus, for any instructor contemplating aswitch to an active learning pedagogy, we recommend thesetwo techniques as a low-cost transitional step.
6. REFERENCES[1] T. A. Angelo and K. P. Cross. Classroom Assessment
Techniques: A Handbook for Faculty. Jossey-Bass, SanFrancisco, CA, 1993.
[2] J. Campbell, D. Horton, M. Craig, and P. Gries.Evaluating an inverted cs1. In Proceedings of the 45thACM Technical Symposium on Computer Science
Education, SIGCSE ’14, pages 307–312, New York,NY, USA, 2014. ACM.
[3] R. M. Felder and R. Brent. Active learning: Anintroduction. ASQ Higher Education Brief, 2(4),August 2009.
[4] S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith,N. Okoroafor, H. Jordt, and M. P. Wenderoth. Activelearning increases student performance in science,engineering, and mathematics. Proceedings of theNational Academy of Sciences, 2014.
[5] R. R. Hake. Interactive-engagement vs. traditionalmethods: A six-thousand-student survey of mechanicstest data for introductory physics courses. AmericanJournal of Physics, 66(1):64–74, 1998.
[6] S. Horwitz, S. H. Rodger, M. Biggers, D. Binkley,C. K. Frantz, D. Gundermann, S. Hambrusch,S. Huss-Lederman, E. Munson, B. Ryder, andM. Sweat. Using peer-led team learning to increaseparticipation and success of under-represented groupsin introductory computer science. In Proceedings ofthe 40th ACM Technical Symposium on ComputerScience Education, SIGCSE ’09, pages 163–167, NewYork, NY, USA, 2009. ACM.
[7] H. H. Hu and T. D. Shepherd. Using pogil to helpstudents learn to program. Trans. Comput. Educ.,13(3):13:1–13:23, Aug. 2013.
[8] P. Lasserre and C. Szostak. Effects of team-basedlearning on a cs1 course. In Proceedings of the 16thAnnual Joint Conference on Innovation andTechnology in Computer Science Education, ITiCSE’11, pages 133–137, New York, NY, USA, 2011. ACM.
[9] C. B. Lee, S. Garcia, and L. Porter. Can peerinstruction be effective in upper-division computerscience courses? Trans. Comput. Educ.,13(3):12:1–12:22, Aug. 2013.
[10] M. Mani and Q. Mazumder. Incorporatingmetacognition into learning. In Proceeding of the 44thACM Technical Symposium on Computer ScienceEducation, SIGCSE ’13, pages 53–58, New York, NY,USA, 2013. ACM.
[11] L. K. Michaelsen, A. B. Knight, and L. D. Fink.Team-Based Learning: A Transformative Use of SmallGroups in College Teaching. Stylus Publishing,Sterling, VA, 2002.
[12] B. Pfaff, A. Romano, and G. Back. The pintosinstructional operating system kernel. In Proceedingsof the 40th ACM Technical Symposium on ComputerScience Education, SIGCSE ’09, pages 453–457, NewYork, NY, USA, 2009. ACM.
[13] L. Porter, C. Bailey Lee, and B. Simon. Halving failrates using peer instruction: A study of four computerscience courses. In Proceeding of the 44th ACMTechnical Symposium on Computer Science Education,SIGCSE ’13, pages 177–182, New York, 2013. ACM.
[14] M. Prince. Does active learning work? a review of theresearch. Journal of Engineering Education, 93(3),July 2004.
[15] M. K. Smith, W. B. Wood, W. K. Adams,C. Wieman, J. K. Knight, N. Guild, and T. T. Su.Why peer discussion improves student performance onin-class concept questions. Science, 323, January 2009.
0%
20%
40%
60%
None Limited Small Large
Offering
Fall 2012
Fall 2013
Figure 3: Self-reported C programming experience
performed significantly better in CS 450 than those in thecontrol group. Figure 4 shows the grade distributions forthe two offerings. The WDF rate improved from 13/49 inFall 2012 to 3/52 in Fall 2013; a one-sided χ2 test yieldedp = 0.004893, indicating significance. Similarly, if we lookat the students who did well in CS 450 (those earning Asand Bs, and only considering the subset of students whoresponded), we see that the ratio of As and Bs improvedfrom 17/44 in Fall 2012 to 35/43 in Fall 2013, which yieldsp = 0.0000596 in a one-sided χ2 test. Thus, the interventiongroup showed a significant improvement in course grades.In addition to overall course grades, we also observed a
significant improvement on programming projects. For in-stance, both offerings were required to complete (workingin groups) portions of the Pintos OS projects [12]. In thesecourses, the userprog project was completed at the end ofthe semester. The average number of test cases (only asubset were required) that the teams passed increased from28/50 to 40/50. A t test yields p = 0.02391, thus indicat-ing significance. As these were team projects, we attemptedto perform a linear regression that modeled the number oftest cases passed as a function of the team’s average C pro-gramming experience and the intervention. The only factorfound to be significant was the intervention (p = 0.0678),but the adjusted R2 value was 0.1535 (overall p = 0.09181for the model), indicating the model produced by the regres-sion is rather weak. However, what is interesting about thisregression is that teams’ average C programming experienceis not a significant predictor of test case success, regardlessof whether or not the intervention was present. This findingis counterintuitive, as the Pintos projects are all written inC, but it is welcome: It suggests that teams with less C ex-perience in both groups were able to catch up with those withmore experience by the end of the semester.
3.3 Impact on final exam performanceAs noted above, our intervention group (Fall 2013) demon-
strated significant improvements in the WDF rate, overall
0%
10%
20%
30%
40%
50%
WDF C B A
Offering
Fall 2012
Fall 2013
Figure 4: Fall 2012 vs. Fall 2013 CS 450 grade dis-tribution
course grade, and programming projects. We also observedthat students performed better on the exams, as well. Toanalyze exam performance in more detail, we performed amultiple linear regression analysis aimed at describing finalexam scores as a function of demographics (C programmingexperience, community college equivalency of the prerequi-site, and prerequisite grade) and our intervention. Linearrelationships between final exam scores and each of the de-mographic and intervention variables were reasonable, andresidual diagnostics for the final model (described below)revealed no issues with model form or assumptions. Wefound the community college equivalency factor to be in-significant with the other factors present, and removed itbased on ANOVA results (p = 0.4958).
Recall that the C programming factor was self-reportedLikert-scale data. While we initially considered all four Lik-ert responses as factors in the regression, we found the in-crease to the third level, “Written several small programs,”relative to having “No experience,” to be the only significantfactor (p = 0.0551). Consequently, we found it appropri-ate to group the responses into two categories: those withlimited or no experience, versus those with at least someexperience with C programming.
Table 1 shows our model, which is significant with an over-all p = 1.153 ∗ 10−12 and R2 = 0.5114, indicating that thismodel captures the behavior well. According to this model,the y-intercept for students in the intervention group was 24points higher on the final exam when all other factors areheld constant. Furthermore, there was a significant inter-action between the prerequisite grade and the intervention,which changes the slope of the line for that class. The resultis that the weakest students (those who scored a D in theprerequisite) in the intervention group benefited the most.At the same time, though, the strongest students in the in-tervention group also benefited but to a lesser extent.
Figure 5 illustrates the model in graphical form. In thisfigure, the dotted lines indicate the model for the studentswho had limited or no C programming experience, whereas
Kirkpatrick, M.S. and Prins, S. “Using the Readiness Assurance Process and Metacognition in an Operating Systems Course.” 20th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), July 2015, http://dx.doi.org/10.1145/2729094.2742594.
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Do students perceive value in using TBL in CS classes?
Does this perception change?
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Setting
• Master’s university, very high undergraduate • More selective, full-time, lower transfer-in • CS department is primarily undergraduate
• 300-level Computer Organization • 30 sophomores, 47 juniors, 34 seniors (111 total) • 28+27 using TBL (one instructor) • 27+29 using traditional lecture (different instructor)
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Mixed-methods design
Quantitative • Value of Teams Survey
(VTS) • Working with Peers • Value of Group Work
Qualitative • Semi-structured focus
groups in 12th week • 8+6 participants (each team
represented) • Anonymized transcripts • Open & axial coding • No inter-rater reliability
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Transcript coding results
10% 15%
15%
20%20%
20%
Value of in-class collaboration Team formationFactors influencing success RAP difficultiesTeam tests Other
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Value of in-class collaboration
Greater learning, but difficult
Correct answers emerged from
discussion
Mixed feelings on making projects
collaborative
“[I]t’s easier to ask your peers that are in your immediate group if you don’t understand something.”
"Even if everyone didn’t know the answer exactly, like when we talk about it we somehow come up with an answer.”
“I mean as programmers we’re gonna be working on teams and developing software programs.”
“I would say it was harder to do things out of class, but in class was a lot more valuable time.”
“I made a joke along the lines of as a group we work really well together, but as an individual entity we all suck.”
“I think there are others that might use that as a way to cruise.”
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Team formation is problematic
Split opinions on balanced vs. unbalanced teams
Survey-based team assignment can be manipulated, but no better
choice
“I think maybe if that was improved then the teams could be more balanced... I feel like there could be better ways to balance the groups and that would really enhance the experience.”
“[I]t all depends how honest you are on the survey too. Cuz if you say that, ‘Okay, I’m not proficient in this,’ and then you actually are, and then you’re put in the group with a bunch of a people who are really good at it.”
If “he puts you in the group [with] people who don’t know, I feel like then you guys will work harder… You guys will teach each other, like, ‘Hey, guys, we don’t have anyone that really knows the stuff in the group, so we’re gonna work hard to get it.’”
“That’s not to say that I’d like to choose the groups on my own because I feel like that could be unfair to the people who don’t know anybody.”
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Factors that influence success
Preferred effort and formative evaluation
over ability
Team dynamics create risks
Team structure mostly encourages
accountability
“I’d rather have you try and get the answer wrong than just to sit there and not do anything at all.”
“[I]f you have [mixed ability groups], they’re supposed to try and teach them, but they’re introverted or don’t care, they’re not gonna teach anything to us.”
“If you feel that they’re gonna rely on you, then you might be more inclined to make sure that you read and make sure you know.”
“[A:] “I feel like [peer evaluations] should be maybe twice during the semester.” [B:] “I know. Cuz at the end of the semester what good is it gonna do [chuckles]?”
“They would go into that stuff so fast, and I was literally so lost. I had so much anxiety the first couple of weeks because I didn’t know what was going on. They didn’t wanna take the time to explain it to me.”
“[I]t’s kind of a self-correcting system. I mean eventually they might kind of reap the rewards for a while, but you’re gonna face what’s happening in the class.”
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
RAP frustrations
Reading assignments are insufficient
preparation
RAP reveals misunderstandings
Sometimes RAP is insufficient feedback
“He gives you a reading, but it’s not...this is not like a history class where you just read it and get all the materials there. It’s kinda hard.”
“I feel a certain point of this is to show you how wrong you are in thinking whatever you thought you knew from studying…”
“They’re not gonna–not everyone’s gonna explain every single answer that they put, and then they’re like, ‘Okay, we got that right,’ but you still have no idea why that was the right answer.”
“We basically had to teach ourselves and then we got graded on that. Then we wouldn’t learn...until we did the activity, which was after the quiz, so not very effective.”
“[T]he iRAT would show me what I think I know versus what I don’t know, which is usually 70 percent or whatever. Then the tRAT sort of allows us to collectively come together and reason out why.”
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Team exams
New process skills and new test-writing approach required
“[I]f we start a problem and then I say what I think the answer is and then my group, they disagree, then we have to talk about it.”
“If we split up the team exam into different sections so certain people do certain parts, that defeats the point of it.”
“[W]e had that idea in the back of our mind, like we practiced that the day before and we had a strategy. That was something we vocally talked about the day before our second exam cuz we didn’t do quite as well on our first exam. Talking about pacing ourselves.”
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Recommendations• Be transparent about the process of assigning students to teams. Use
objective metrics to the extent possible. • Provide explicit instruction for TBL process skills, including how to prepare
for team exams and the iRAT. • Supplement reading assignments with scaffolding materials to facilitate
achieving desired learning objectives for the RAP. • Consider replacing or augmenting the iRAT with a pre-class online quiz. • Ask for additional points of confusion after addressing commonly missed
tRAT questions. • Make team exams shorter than individual exams to allow for discussion and
to be mindful of the impact of absences. • Provide time estimates for team exam problems.
Student Perspectives of Team-Based Learning in a CS Course: Summary of Qualitative Findings SIGCSE 2017 • Kirkpatrick
Thank you