Preliminary findings on quantifying how
engineering students use webcasts of class
lectures
Cheryl Schramm
Alan Steele
Introduction
• Goal: Provide systematic evidence to help
assess the use of video-casts as a teaching
tool in order to strengthen and improve
future academic experience of students
• Method: Formative Hybrid [Program]
Evaluation of providing webcast lectures in
engineering courses.
• Implementation Issues: Preliminary run on
ECOR 1606 Summer 2011
• Impact Issues: Ongoing work
Presentation Overview
1. The Context of the Engineering Webcast
Program
2. Literature Background
3. Program Evaluation: Webcasting ECOR 1606
4. Preliminary Data Analysis – Early Results
5. Research Questions
Organizational Context at Carleton
Faculty of Engineering & Design
Faculty of Science
Faculty of Arts & Social Science
Faculty of Public Affairs
Department of Mechanical & Aerospace Eng
Department of Electronics Alan Steele
Department of Civil Eng
Department of Systems & Computer Eng Cheryl Schramm
Interactive Media Services (IMS)
Education Development Centre (EDC) CUOL
Provost & Vice President Academic
Carleton Centre for Research on Engineering Education (CCEE)
Students of ECOR 1606
Associate VP (Teaching & Learning)
CUOL – Carleton University on Line
“…initiative to provide alternative and flexible access to many Carleton University
credit courses. Now in its 33rd year of operation, more than 80 000 students from
the Ottawa area, and around the world have used CUOL distributed courses to learn
and complete a University degree program”
“CUOL courses are recorded on-campus during a regular class, and the lectures are
then made available to local and distance students … CUOL classes are as close
as possible to being in the classroom – you can listen to classroom discussions,
hear questions, and see the lectures just as they are delivered to fellow students.”
http://www2.carleton.ca/cuol/
My Virtual Classroom
Using multiple teaching technologies
• Lecture Materials Powerpoint Slides
• Laptop Demonstrations (Code Development
and Debugging)
• Chalk board problem solving
• Overhead projectors (Card games, midterm
reviews)
• Classroom Discussions
Background - Engineering
Faculty of Engineering and Design
• Engineering: 4 departments
• Civil and Environmental Eng., Electronics,
Mechanical & Aerospace Eng., Systems & Computer
Eng.
• 14 programs, including: Mechanical Eng., Electrical
Eng., Engineering Physics, Software Engineering,
Sustainable and Renewable Engineering.
• Engineering programs are accredited.
• Contact hours are important.
Background – ECOR 1606
ECOR 1606 [0.5 credit]
Problem Solving and Computers
Introduction to engineering problem solving. Defining and
modeling problems, designing algorithmic solutions, converting
algorithms to C programs, testing, debugging. Program style,
documentation, reliability. Numeric methods: representation of
data, rounding and truncation errors, root finding, curve fitting.
Lectures three hours a week, laboratory three hours a week.
Background – ECOR 1606 SUMMER
Summer Offering is a Condensed Course
• 13 week course in 6.5 weeks (May/June)
• Schedule: Tuesday and Thursday:
Lectures: 9 AM – 12 noon
Labs: 12:30 PM – 3:30 PM
Department
Mech & Aero
Electronics
Systems
Civil
Course History
1st time
2nd time
>2nd time
Literature Background
“Only a limited amount of empirical research has
addressed the educational advantages and
disadvantages of webcasting and podcasting, although
one good example of such a study was performed by
Brotherton and Abowd (2004).”
(Traphagan et al, 2010)
Impact of class lecture webcasting on attendance and learning
Brotherton et al. (2004) “Lessons learned from eClass:”
Gump, S.E. (2005). The cost of cutting class:
Attendance as a predictor of student success.
von Konsky et al. (2009). Lecture attendance and web
based lecture technologies
Program Logic Model
Program Needs
Eng students need
flexibility,
access,alternative
learning
Failure & drop-out
rates in ECOR 1606
Intermediate Outcomes
Improved Program
Implementation
Increased and more
diverse student enrolment
Increased adoption for
other engineering courses
Immediate Outcomes
Increase student success
in ECOR 1606.
Reliable & Valid
Program Evidence
Long Term Outcomes
Increased student
success/satisfaction/
retention/graduation
Extend Carleton’s
reputation for innovation
Program Activities
Daily Lecture Capture
Daily Lecture Editing &
Posting
Daily physical attendance
check
Mid & Post Surveys
LMS Webcast Monitoring
Program Inputs
Instructors
Media Services –
Equipment &
Personnel (Camera
Operators, Editing
Technicians)
Learning
Management System
(LMS) (Password-
Encoded for Webcast
storage &usage
monitoring)
Financial support
Program Outputs
Course Events Calendar
LMS WebCast Usage
Statistics
Attendance Stats
Grades
Survey Stats
Email Correspondence
and/or Discussion Groups
for Lecture Questions
Library of Course
Materials
Implementation Evaluative Issues
A.1 To what extent is the online captured lectures
relevant to the needs of the engineering students?
In what ways do online captured lectures
supplement their course experience?
A.2 How, and how often, are the online captured
lectures used? When? At what times? What was
their experience with the use and delivery of the
online captured lecture?
A.3 To what extent will online captured lecture have
an impact on the experience of the professors?
Impact Evaluative Issues
B.1. Are a greater percentage of students performing
at passing level with access to webcasts?
B.2 To what extent has motivation, satisfaction and
educational engagement changed as a result of
having access to webcasts?
B.3: What are the unintended consequences of
webcasts ? Are there positive or negative
consequences? For whom?
Data Collection
1. Attendance: Self-reported
2. Moodle LMS: Password protected, usage
statistics
3. Surveys: Likert-scale questions on habits
and perceptions, hosted on getFAST.ca
(Canadian hosting)
4. Final Grades from student records
Data Analysis
Class size: 94 (103 on May 10th)
Participants: 56
Concerns:
1. Sample size
2. Summer Demographics
3. Unreliable Attendance Records
Data Analysis
1. Describe Accesses to the Webcast Lectures
When were the lectures accessed? ( Range of
purposes … what are their needs?)
Continuously over the term.
Sporadically (e.g. Occasional Absence or before
deliverables and exams)
Delta Time
How were the lectures accessed?
View entire lecture or skip through them
Repeated views of same lecture (identify difficult
subject)
Data Analysis
2. Describe Students Who Do/Not Use
• How many students used them? (A Measure of
usefulness, uptake)
• What are the attendance records of those
who did (not) use the webcasts?
• Can we describe two constructs: Physical
(F2F) and/or Virtual Attendance?
• What are the academic standings of those
who did (not) use the webcasts?
ECOR 1606 – Summer 2011
Week Monday Tuesday Wednesday Thursday Friday
1 9 10
Lecture 1
Lab 1
11 12
Lecture 2
Lab 2
13
2 16 17
Assignment 1
Lecture 3
Lab 3
18 19
Lecture 4
Lab 4
20
3 23
Holiday
24
Assignment 2
Lecture 5
Lab 5
25 26
Lecture 6
Lab 6
27
May June
4 30 31
Assignment 3
Lecture 7
Lab 7
1 2
Lecture 8
Lab 8
3
5 6 7
Assignment 4
Midterm
Lab 9
8 9
Lecture 9
Lab Midterm
10
6 13 14
Assignment 5
Lecture 10
Lab 10
15 16
Lecture 11
Lab 11
17
7 20 21
Assignment 6
Review Session
Lab 12
22 23 24
June 26, Exam
A measure of uptake – How often were
the lectures used?
Midterm
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10 11
Nu
mb
er
of
Stu
den
ts W
ho
...
Lecture
Attendance and Number of Viewers – Per Lecture
Attended
Viewed
Which lectures were used the most?
Midterm
Topics
1 Computer Basics
2 Programming Overview
3 Algorithms
4 Variables
5 Conditionals
6 Program Development
7 Loops
8 Functions
9 File I/O
10 Simulation
11 Array Sorting
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 11
Lecture
Viewers and Total Viewings Over Term - Per Lecture
Viewings
Viewers
When are lectures watched?
0
1
2
3
4
5
6
7
8
9
9-M
ay
10-M
ay
11-M
ay
12-M
ay
13-M
ay
14-M
ay
15-M
ay
16-M
ay
17-M
ay
18-M
ay
19-M
ay
20-M
ay
21-M
ay
22-M
ay
23-M
ay
24-M
ay
25-M
ay
26-M
ay
27-M
ay
28-M
ay
29-M
ay
30-M
ay
31-M
ay
1-J
un
2-J
un
3-J
un
4-J
un
5-J
un
6-J
un
7-J
un
8-J
un
9-J
un
10-J
un
11-J
un
12-J
un
13-J
un
14-J
un
15-J
un
16-J
un
17-J
un
18-J
un
19-J
un
20-J
un
21-J
un
22-J
un
23-J
un
24-J
un
25-J
un
26-J
un
Daily Access of Selected Lectures (Num Viewers)
Lecture 2 (May 12) Lecture 4 (May19) Lecture 6 (May 26)
Lecture 7 (May 31) Lecture 8 (June 2)
To what extent are attendance and
viewing related?
0
2
4
6
8
10
12
0 1 2 3 4 5 6 7 8 9 10
Num
ber
of
Lectu
res V
iew
ed
Number of Lectures Attended
Attendance & Viewing
Attend
Correlation=0.08
To what extent do webcasts affect
attendance?
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11
Attendance = {Physical F2F, Virtual}
Attend Only
Attend and View
View Only
Absent
Lecture
Num
ber
of
Stu
dents
who …
To what extent do webcasts affect
attendance?
Lecture
Num
ber
of
Stu
dents
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10 11
Composite Attendance
Absent
Virtual Attendance
Physical Attendance
What kind of student uses webcasts?
Student Code
0
5
10
15
20
25
30
35
100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 147 149 151 154
Nu
mb
er
of
…
Which Students Viewed?
Viewed Lectures
Viewings
Grades (GPA)
0
2
4
6
8
10
12
0 5 10 15 20 25 30
GP
A
Number of Views
Achievement vs Use of Webcasts
Lectures Viewed
Viewings
What kind of students use webcasts?
Correlation Lectures: 0.06
Correlation Viewings: 0.08
0
2
4
6
8
10
12
0 2 4 6 8 10
GP
A
Number of Lectures Attended
Achievement vs Attendance
Physical Attendance
Composite Attendance
What kind of student attends?
Correlation Physical: 0.34
Correlation Composite: 0.22
Mid-Term Survey
1. Have other courses made video recordings of the lectures
available? If 'yes' please state name of the course.
2. Approximately how many lecture videos have you watched?
3. How do you normally watch the videos?
4. How best describes how you use the lecture videos?
5. How best describes why you watch the videos?
6. Do you plan on watching videos of future lectures?
7. Do you plan on re-watching videos you have already viewed?
8. Do you plan on using the videos in preparation for the final exam?
9. On a scale of 1 (lowest) to 5 (highest) please rate how useful you
find having the lecture videos available?
10. Have you missed an ECOR1606 class (not due to sickness)
because you knew it would be available on video?
Response Rate: 35 out of the 56 participants
Post-Term Survey
1. Did you find having the video recordings available useful?
2. Approximately how many lecture videos have you watched?
3. On what device did you normally watch the videos?
4. Check all the different devices you used to watch the videos.
5. How best describes how you used the lecture videos?
6. How best describes why you watched the videos?
7. At what point did you start watching the videos?
8. On a scale of 1 (lowest) to 5 (highest) please rate how useful
you found having the lecture videos available?
9. Did you miss an ECOR1606 class (not due to sickness)
because you knew it would be available on video?
10. How do you agree with the following statement? "All
lectures should be video recorded“
Response Rate: 14 out of the 56 participants
Mid-term Survey - Results
Mid-Term
Post
Mid-term Survey - Results
Mid-Term
Post
Mid-Term Survey – Plans
Post Survey
Comparing Mid and Post Surveys
Mid-Term
Post
Post-Survey - Opinions
Research Lessons Learned
• Ethics Procedure
• Data Collection
Automate Data Collection
Attendance: Self reporting does not work
Surveys: Difficult to get follow-through
• Consider established surveys on
engagement and study habits
Need continual promotion throughout term to
maintain initial interest
Research Questions
Measuring Academic Success
Grades and/or Engagement
Measuring Course Success
Different Learning Supports
Identifying Difficult Topics
Measuring Other Stakeholders
Instructors
Tackling Full Implementation
• Regular Term Offering (Not condensed)
• Longitudinal Offerings (Courses in later years)
References
J.A. Brotherton and G.D. Abowd, “Lessons learned from eClass:
Assessing automated capture and access in the classroom,” ACM
Trans. Comput.-Hum. Interact., vol. 11, 2004, pp. 121-155.
Gump, S.E. (2005). The cost of cutting class: Attendance as a predictor
of student success. College Teaching, 53(1), 21-26.
Traphagan, Tomoko & John V. Kucsera, John V. & Kishi, Kyoko (2010).
Impact of class lecture webcasting on attendance and learning,
Education Technology Research Development (2010) 58:19–37.
von Konsky, B. R., Ivins, J. & Gribble, S. J. (2009). Lecture attendance
and web based lecture technologies: A comparison of student
perceptions and usage patterns. Australasian Journal of Educational
Technology, 25(4), 581-595.
http://www.ascilite.org.au/ajet/ajet25/vonkonsky.html
Bibliography
Barrington and Johnson (2006). The Relationship between Lab
Attendance and Academic Performance in a Computer Information
Systems Course. Information Systems Education Journal, 4 (99).
http://isedj.org/4/99/. ISSN: 1545-679X. (Also appears in The
Proceedings of ISECON 2005: §3573. ISSN: 1542-7382.).
Retrieved October 2011 from http://isedj.org/4/99
Kybartaite, A., Nousiainen, J., & Malmivuo, J. (2009). Evaluation of
Students’ Attitudes towards Virtual Learning Objects for Biomedical
Engineering. IEEE Multidisciplinary Engineering Education
Magazine, 4 (4), December 2009, 102-108.
http://www.ewh.ieee.org/soc/e/sac/meem/index.php/meem/article/vi
ewFile/88/65
Bibliography
Scutter, Sheila & Stupans, Ieva & Sawyer, Tim & King, Sharron
(2010). How do students use podcasts to support learning?
Australasian Journal of Educational Technology 2010, 26(2), 180-
191
Sinclaire, Jollean K. & Simon, Judith C. & Campbell, Charles J. &
Brown, Judith C. (July 2011). Skills versus Concepts: Attendance
and Grades in Information Technology Courses. International
Journal of Education. 2011, Vol. 3, No. 2: E1.
Stodel, E. J., Thompson, T. L., & MacDonald, C. J. (2006). Learners’
perceptions on what is missing from online learning: Interpretations
through the community of inquiry framework. International Review
of Research in Open and Distance Learning, 7(3), 1-24. Retrieved
October 2011 from
http://www.irrodl.org/index.php/irrodl/article/view/325/743
Top Related