Preliminary findings on quantifying how

engineering students use webcasts of class

lectures

Cheryl Schramm

Alan Steele

Introduction

• Goal: Provide systematic evidence to help

assess the use of video-casts as a teaching

tool in order to strengthen and improve

future academic experience of students

• Method: Formative Hybrid [Program]

Evaluation of providing webcast lectures in

engineering courses.

• Implementation Issues: Preliminary run on

ECOR 1606 Summer 2011

• Impact Issues: Ongoing work

Presentation Overview

1. The Context of the Engineering Webcast

Program

2. Literature Background

3. Program Evaluation: Webcasting ECOR 1606

4. Preliminary Data Analysis – Early Results

5. Research Questions

Organizational Context at Carleton

Faculty of Engineering & Design

Faculty of Science

Faculty of Arts & Social Science

Faculty of Public Affairs

Department of Mechanical & Aerospace Eng

Department of Electronics Alan Steele

Department of Civil Eng

Department of Systems & Computer Eng Cheryl Schramm

Interactive Media Services (IMS)

Education Development Centre (EDC) CUOL

Provost & Vice President Academic

Carleton Centre for Research on Engineering Education (CCEE)

Students of ECOR 1606

Associate VP (Teaching & Learning)

CUOL – Carleton University on Line

“…initiative to provide alternative and flexible access to many Carleton University

credit courses. Now in its 33rd year of operation, more than 80 000 students from

the Ottawa area, and around the world have used CUOL distributed courses to learn

and complete a University degree program”

“CUOL courses are recorded on-campus during a regular class, and the lectures are

then made available to local and distance students … CUOL classes are as close

as possible to being in the classroom – you can listen to classroom discussions,

hear questions, and see the lectures just as they are delivered to fellow students.”

http://www2.carleton.ca/cuol/

My Virtual Classroom

Using multiple teaching technologies

• Lecture Materials Powerpoint Slides

• Laptop Demonstrations (Code Development

and Debugging)

• Chalk board problem solving

• Overhead projectors (Card games, midterm

reviews)

• Classroom Discussions

Background - Engineering

Faculty of Engineering and Design

• Engineering: 4 departments

• Civil and Environmental Eng., Electronics,

Mechanical & Aerospace Eng., Systems & Computer

Eng.

• 14 programs, including: Mechanical Eng., Electrical

Eng., Engineering Physics, Software Engineering,

Sustainable and Renewable Engineering.

• Engineering programs are accredited.

• Contact hours are important.

Background – ECOR 1606

ECOR 1606 [0.5 credit]

Problem Solving and Computers

Introduction to engineering problem solving. Defining and

modeling problems, designing algorithmic solutions, converting

algorithms to C programs, testing, debugging. Program style,

documentation, reliability. Numeric methods: representation of

data, rounding and truncation errors, root finding, curve fitting.

Lectures three hours a week, laboratory three hours a week.

Background – ECOR 1606 SUMMER

Summer Offering is a Condensed Course

• 13 week course in 6.5 weeks (May/June)

• Schedule: Tuesday and Thursday:

Lectures: 9 AM – 12 noon

Labs: 12:30 PM – 3:30 PM

Department

Mech & Aero

Electronics

Systems

Civil

Course History

1st time

2nd time

>2nd time

Literature Background

“Only a limited amount of empirical research has

addressed the educational advantages and

disadvantages of webcasting and podcasting, although

one good example of such a study was performed by

Brotherton and Abowd (2004).”

(Traphagan et al, 2010)

Impact of class lecture webcasting on attendance and learning

Brotherton et al. (2004) “Lessons learned from eClass:”

Gump, S.E. (2005). The cost of cutting class:

Attendance as a predictor of student success.

von Konsky et al. (2009). Lecture attendance and web

based lecture technologies

Program Logic Model

Program Needs

Eng students need

flexibility,

access,alternative

learning

Failure & drop-out

rates in ECOR 1606

Intermediate Outcomes

Improved Program

Implementation

Increased and more

diverse student enrolment

Increased adoption for

other engineering courses

Immediate Outcomes

Increase student success

in ECOR 1606.

Reliable & Valid

Program Evidence

Long Term Outcomes

Increased student

success/satisfaction/

retention/graduation

Extend Carleton’s

reputation for innovation

Program Activities

Daily Lecture Capture

Daily Lecture Editing &

Posting

Daily physical attendance

check

Mid & Post Surveys

LMS Webcast Monitoring

Program Inputs

Instructors

Media Services –

Equipment &

Personnel (Camera

Operators, Editing

Technicians)

Learning

Management System

(LMS) (Password-

Encoded for Webcast

storage &usage

monitoring)

Financial support

Program Outputs

Course Events Calendar

LMS WebCast Usage

Statistics

Attendance Stats

Grades

Survey Stats

Email Correspondence

and/or Discussion Groups

for Lecture Questions

Library of Course

Materials

Implementation Evaluative Issues

A.1 To what extent is the online captured lectures

relevant to the needs of the engineering students?

In what ways do online captured lectures

supplement their course experience?

A.2 How, and how often, are the online captured

lectures used? When? At what times? What was

their experience with the use and delivery of the

online captured lecture?

A.3 To what extent will online captured lecture have

an impact on the experience of the professors?

Impact Evaluative Issues

B.1. Are a greater percentage of students performing

at passing level with access to webcasts?

B.2 To what extent has motivation, satisfaction and

educational engagement changed as a result of

having access to webcasts?

B.3: What are the unintended consequences of

webcasts ? Are there positive or negative

consequences? For whom?

Data Collection

1. Attendance: Self-reported

2. Moodle LMS: Password protected, usage

statistics

3. Surveys: Likert-scale questions on habits

and perceptions, hosted on getFAST.ca

(Canadian hosting)

4. Final Grades from student records

Data Analysis

Class size: 94 (103 on May 10th)

Participants: 56

Concerns:

1. Sample size

2. Summer Demographics

3. Unreliable Attendance Records

Data Analysis

1. Describe Accesses to the Webcast Lectures

When were the lectures accessed? ( Range of

purposes … what are their needs?)

Continuously over the term.

Sporadically (e.g. Occasional Absence or before

deliverables and exams)

Delta Time

How were the lectures accessed?

View entire lecture or skip through them

Repeated views of same lecture (identify difficult

subject)

Data Analysis

2. Describe Students Who Do/Not Use

• How many students used them? (A Measure of

usefulness, uptake)

• What are the attendance records of those

who did (not) use the webcasts?

• Can we describe two constructs: Physical

(F2F) and/or Virtual Attendance?

• What are the academic standings of those

who did (not) use the webcasts?

ECOR 1606 – Summer 2011

Week Monday Tuesday Wednesday Thursday Friday

1 9 10

Lecture 1

Lab 1

11 12

Lecture 2

Lab 2

13

2 16 17

Assignment 1

Lecture 3

Lab 3

18 19

Lecture 4

Lab 4

20

3 23

Holiday

24

Assignment 2

Lecture 5

Lab 5

25 26

Lecture 6

Lab 6

27

May June

4 30 31

Assignment 3

Lecture 7

Lab 7

1 2

Lecture 8

Lab 8

3

5 6 7

Assignment 4

Midterm

Lab 9

8 9

Lecture 9

Lab Midterm

10

6 13 14

Assignment 5

Lecture 10

Lab 10

15 16

Lecture 11

Lab 11

17

7 20 21

Assignment 6

Review Session

Lab 12

22 23 24

June 26, Exam

A measure of uptake – How often were

the lectures used?

Midterm

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11

Nu

mb

er

of

Stu

den

ts W

ho

...

Lecture

Attendance and Number of Viewers – Per Lecture

Attended

Viewed

Which lectures were used the most?

Midterm

Topics

1 Computer Basics

2 Programming Overview

3 Algorithms

4 Variables

5 Conditionals

6 Program Development

7 Loops

8 Functions

9 File I/O

10 Simulation

11 Array Sorting

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11

Lecture

Viewers and Total Viewings Over Term - Per Lecture

Viewings

Viewers

When are lectures watched?

0

1

2

3

4

5

6

7

8

9

9-M

ay

10-M

ay

11-M

ay

12-M

ay

13-M

ay

14-M

ay

15-M

ay

16-M

ay

17-M

ay

18-M

ay

19-M

ay

20-M

ay

21-M

ay

22-M

ay

23-M

ay

24-M

ay

25-M

ay

26-M

ay

27-M

ay

28-M

ay

29-M

ay

30-M

ay

31-M

ay

1-J

un

2-J

un

3-J

un

4-J

un

5-J

un

6-J

un

7-J

un

8-J

un

9-J

un

10-J

un

11-J

un

12-J

un

13-J

un

14-J

un

15-J

un

16-J

un

17-J

un

18-J

un

19-J

un

20-J

un

21-J

un

22-J

un

23-J

un

24-J

un

25-J

un

26-J

un

Daily Access of Selected Lectures (Num Viewers)

Lecture 2 (May 12) Lecture 4 (May19) Lecture 6 (May 26)

Lecture 7 (May 31) Lecture 8 (June 2)

To what extent are attendance and

viewing related?

0

2

4

6

8

10

12

0 1 2 3 4 5 6 7 8 9 10

Num

ber

of

Lectu

res V

iew

ed

Number of Lectures Attended

Attendance & Viewing

Attend

Correlation=0.08

To what extent do webcasts affect

attendance?

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11

Attendance = {Physical F2F, Virtual}

Attend Only

Attend and View

View Only

Absent

Lecture

Num

ber

of

Stu

dents

who …

To what extent do webcasts affect

attendance?

Lecture

Num

ber

of

Stu

dents

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11

Composite Attendance

Absent

Virtual Attendance

Physical Attendance

What kind of student uses webcasts?

Student Code

0

5

10

15

20

25

30

35

100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 147 149 151 154

Nu

mb

er

of

…

Which Students Viewed?

Viewed Lectures

Viewings

Grades (GPA)

0

2

4

6

8

10

12

0 5 10 15 20 25 30

GP

A

Number of Views

Achievement vs Use of Webcasts

Lectures Viewed

Viewings

What kind of students use webcasts?

Correlation Lectures: 0.06

Correlation Viewings: 0.08

0

2

4

6

8

10

12

0 2 4 6 8 10

GP

A

Number of Lectures Attended

Achievement vs Attendance

Physical Attendance

Composite Attendance

What kind of student attends?

Correlation Physical: 0.34

Correlation Composite: 0.22

Mid-Term Survey

1. Have other courses made video recordings of the lectures

available? If 'yes' please state name of the course.

2. Approximately how many lecture videos have you watched?

3. How do you normally watch the videos?

4. How best describes how you use the lecture videos?

5. How best describes why you watch the videos?

6. Do you plan on watching videos of future lectures?

7. Do you plan on re-watching videos you have already viewed?

8. Do you plan on using the videos in preparation for the final exam?

9. On a scale of 1 (lowest) to 5 (highest) please rate how useful you

find having the lecture videos available?

10. Have you missed an ECOR1606 class (not due to sickness)

because you knew it would be available on video?

Response Rate: 35 out of the 56 participants

Post-Term Survey

1. Did you find having the video recordings available useful?

2. Approximately how many lecture videos have you watched?

3. On what device did you normally watch the videos?

4. Check all the different devices you used to watch the videos.

5. How best describes how you used the lecture videos?

6. How best describes why you watched the videos?

7. At what point did you start watching the videos?

8. On a scale of 1 (lowest) to 5 (highest) please rate how useful

you found having the lecture videos available?

9. Did you miss an ECOR1606 class (not due to sickness)

because you knew it would be available on video?

10. How do you agree with the following statement? "All

lectures should be video recorded“

Response Rate: 14 out of the 56 participants

Mid-term Survey - Results

Mid-Term

Post

Mid-term Survey - Results

Mid-Term

Post

Mid-Term Survey – Plans

Post Survey

Comparing Mid and Post Surveys

Mid-Term

Post

Post-Survey - Opinions

Research Lessons Learned

• Ethics Procedure

• Data Collection

Automate Data Collection

Attendance: Self reporting does not work

Surveys: Difficult to get follow-through

• Consider established surveys on

engagement and study habits

Need continual promotion throughout term to

maintain initial interest

Research Questions

Measuring Academic Success

Grades and/or Engagement

Measuring Course Success

Different Learning Supports

Identifying Difficult Topics

Measuring Other Stakeholders

Instructors

Tackling Full Implementation

• Regular Term Offering (Not condensed)

• Longitudinal Offerings (Courses in later years)

References

J.A. Brotherton and G.D. Abowd, “Lessons learned from eClass:

Assessing automated capture and access in the classroom,” ACM

Trans. Comput.-Hum. Interact., vol. 11, 2004, pp. 121-155.

Gump, S.E. (2005). The cost of cutting class: Attendance as a predictor

of student success. College Teaching, 53(1), 21-26.

Traphagan, Tomoko & John V. Kucsera, John V. & Kishi, Kyoko (2010).

Impact of class lecture webcasting on attendance and learning,

Education Technology Research Development (2010) 58:19–37.

von Konsky, B. R., Ivins, J. & Gribble, S. J. (2009). Lecture attendance

and web based lecture technologies: A comparison of student

perceptions and usage patterns. Australasian Journal of Educational

Technology, 25(4), 581-595.

http://www.ascilite.org.au/ajet/ajet25/vonkonsky.html

Bibliography

Barrington and Johnson (2006). The Relationship between Lab

Attendance and Academic Performance in a Computer Information

Systems Course. Information Systems Education Journal, 4 (99).

http://isedj.org/4/99/. ISSN: 1545-679X. (Also appears in The

Proceedings of ISECON 2005: §3573. ISSN: 1542-7382.).

Retrieved October 2011 from http://isedj.org/4/99

Kybartaite, A., Nousiainen, J., & Malmivuo, J. (2009). Evaluation of

Students’ Attitudes towards Virtual Learning Objects for Biomedical

Engineering. IEEE Multidisciplinary Engineering Education

Magazine, 4 (4), December 2009, 102-108.

http://www.ewh.ieee.org/soc/e/sac/meem/index.php/meem/article/vi

ewFile/88/65

Bibliography

Scutter, Sheila & Stupans, Ieva & Sawyer, Tim & King, Sharron

(2010). How do students use podcasts to support learning?

Australasian Journal of Educational Technology 2010, 26(2), 180-

191

Sinclaire, Jollean K. & Simon, Judith C. & Campbell, Charles J. &

Brown, Judith C. (July 2011). Skills versus Concepts: Attendance

and Grades in Information Technology Courses. International

Journal of Education. 2011, Vol. 3, No. 2: E1.

Stodel, E. J., Thompson, T. L., & MacDonald, C. J. (2006). Learners’

perceptions on what is missing from online learning: Interpretations

through the community of inquiry framework. International Review

of Research in Open and Distance Learning, 7(3), 1-24. Retrieved

October 2011 from

http://www.irrodl.org/index.php/irrodl/article/view/325/743