CEEA_Paper 2015-04-20_submitted

Proc. 2015 Canadian Engineering Education Association (CEEA15) Conf.

CEEA15; Paper 084

McMaster University; May 31 – June 3, 2015 – 1 of 6 –

TEACHING AN ADVANCED ENGINEERING MOOC:

LESSONS LEARNED

Jeffrey Harris 1, William Heikoop 2, Allison Van Beek 3, and James S. Wallace 1

1 Department of Mechanical & Industrial Engineering, University of Toronto

2 Online Learning Strategies, University of Toronto

3 Faculty of Applied Science & Engineering, University of Toronto

[email protected], [email protected]

Abstract – Massive Open Online Courses (MOOCs)

allow anyone in the public to learn from professors at

universities across the world. An internet connection is

the only requirement to participate in a MOOC. In

engineering, the majority of MOOCs are targeted at self-

learners, and consequently most courses are based on

introductory undergraduate courses. The University of

Toronto offered its first advanced engineering MOOC

entitled, “Wind, Waves & Tides” based on a mixed

fourth-year undergraduate and graduate level course. A

total of 11,723 students registered in the course, and 617

students completed the course in its entirety. The

following paper describes the experience of teaching a

niche interest MOOC and the lessons learned throughout

the endeavour.

Keywords: MOOC, online learning, distance learning,

mechanical engineering, alternative energy, Coursera

1. INTRODUCTION

The prevalence of massive open online courses

(MOOCs) has recently grown as a way of making high-

quality education freely available to the population-at-

large. This new media helps to serve universities’ broad

objective of promoting higher learning and sharing

knowledge.

Early providers of MOOCs included Stanford

University, MIT, and Harvard. Professors from Stanford

developed the for-profit Coursera platform [1], and MIT

and Harvard developed the non-profit edX platform [2].

Some institutions have been cautious about offering

MOOCs because they fear that free MOOCs would

directly compete with their existing courses and therefore

threaten their business model [3-5]. In reality, there has

been no evidence that the proliferation of MOOCs has

resulted in declining university enrollments. In fact, the

majority of participants in MOOCs already have a

university degree. In 32 MOOC sessions offered by

University of Pennsylvania, 83% of participants already

had a post-secondary degree [6]. Similarly, in science,

technology, engineering, and math (STEM) courses

offered by MIT and Harvard, 61% of participants already

had a Bachelor’s degree [7].

The University of Toronto is the midst of its third year

offering MOOCs under the Open UToronto initiative.

The university has partnership agreements with Coursera

and edX platforms, and in its first two years it offered 14

courses with a total of 528,847 registrants [8]. The

majority of the university’s MOOC offerings have been

introductory level courses targeted towards self-learners.

In the fall of 2014, we delivered, Wind, Waves and

Tides: Alternative Energy Systems, the first advanced

engineering MOOC offered by the University of Toronto

and second MOOC offered by the Faculty of Applied

Science & Engineering. The MOOC was based on a

portion of an alternative energy systems credit course

offered as a technical elective for final-year undergraduate

students or beginning graduate students. Therefore, there

were several recommended prerequisites for the MOOC—

an unusual attribute for a MOOC.

Since an advanced engineering course is a niche type

of MOOC, it is interesting to understand who takes this

type of course, how students take the course, the

challenges of teaching the course, and lessons learned

from the experience. We had three concerns prior to the

start of the MOOC: 1) that there wouldn’t be enough

demand for such a specialized topic, 2) that the

prerequisites would limit participation, and 3) that the 7

week course length would be a deterrent, i.e. a larger time

commitment than prospective students would be willing to

make.


CEEA15; Paper 084


2. COURSE DESIGN

Wind, Waves and Tides: Alternative Energy Systems

was delivered over 7 weeks on the Coursera platform.

The University of Toronto’s MOOC design guidelines [8]

were used to help shape the structure of the course.

General best-practices for teaching a MOOC are well-

documented [9-11], and many more guidelines are easily

available with a simple internet search.

2.1 Course structure

Wind, Waves and Tides was divided into three thematic

modules. The Wind and Waves modules were subdivided

into parts, by topic, as shown in Table 1. Each part

started with an introductory video, followed by

approximately 5 additional lectures to explore the topic in

depth.

The MOOC lecture videos were targeted to be roughly

10-15 minutes in length, a duration identified as effective

by previous researchers [12]. Depending on topic,

however, individual videos varied in length from 7 to 26

minutes. The Coursera platform has very comprehensive

analytic features, which made it possible to easily track on

a daily basis, the number of participants who watched

each video or completed an assessment.

Many videos contained embedded multiple choice

quizzes, consisting of quantitative problems that were

adapted from credit course tutorials and homework

assignments. These embedded quizzes were included to

help each student assess whether he/she understood the

lecture material; they were not graded and therefore

served as formative assessment. The advantage is that the

embedded quizzes provide immediate feedback for

comprehension and allow students to monitor the quality

of their own work, without the need for active course staff

participation [13]. The embedded quizzes were set up so

that incorrect answers revealed hints about the error made

or even an explanation (including images or figures) of the

error if the question is more complex. Additionally,

students were encouraged to post on the course discussion

board and relate the lecture material to examples specific

to their local geography.

2.2 Assessment

The course grade was composed of quizzes (60%), a

peer-assessed assignment (15%), and a final exam (25%).

In total, there were 6 graded multiple choice quizzes. The

quiz questions were similar to the practice questions

embedded into the lecture videos, and students were able

to attempt each quiz multiple times. The peer assessed

assignment was designed to connect renewable energy,

Table 1: Outline of course topics

Module 1: Wind Energy

A. Wind sources and influences

B. Fundamentals of wind energy technology

C. Predicting wind turbine output

D. Economic and environmental issues

E. Case studies

Module 2: Wave Energy

A. Wave energy fundamentals

B. Wave energy converters

Module 3: Tidal Energy

planning, and local concerns specific to each student’s

location. It consisted of a short written composition,

approximately 500 words. After the submission deadline,

peers graded each other’s work using the provided rubric.

The final exam consisted of questions similar to those in

the quizzes, and students were allowed up to two attempts.

A grade of 60% or higher was required for students to

earn a Statement of Accomplishment. Note that Wind,

Waves and Tides followed common MOOC practice in

that all assessments were carried out by machine grading

or by peer-assessment due to the large number of students

enrolled. As well, the difficulty of the questions and

problems was substantially reduced, both to facilitate

machine grading and in recognition of the expected wide

range of student backgrounds and ability compared to the

students taking the regular university credit course.

Similarly, the pace was reduced, with the MOOC duration

almost twice that of the time allocated to the material in

the regular credit course.

3. WHO PARTICIPATED

During the session of this course, 11,723 students

registered in the course. As of April 8, 2015, an

additional 1982 students had registered for the course in

“archive mode”, in which they can access the course

materials but are not evaluated. A total of 6,296 different

people had watched at least one lecture.

Registered students were from 174 different countries

(based on IP address), with 44% from emerging

economies. By continent, Europe provided the largest

share of registrants (31%), followed by North America

(26%), Asia (25%), South America (9%), Africa (7%),

and Oceania (2%). By country, the United States

accounted for the largest share of registrants (17%),

followed by India (10%), Brazil (5%), Canada (4%),

Spain (4%), the United Kingdom (4%) and Egypt (3%).

The remaining 53% of the students were from 167

countries, illustrating the tremendous diversity of student

backgrounds.


CEEA15; Paper 084


Table 2: Professional background of course participants

Which of the

following

characterizes

you?

Distribution

of survey

respondents

Distribution

of survey

respondents

who

completed

the course

Completion

rate of

students who

intended to

complete

Student 33.1 % 29.1 % 37 %

Professional 34.6 % 44.3 % 51 %

Researcher 5.9 % 6.6 % 48 %

Academic /

professor 2.3 % 1.6 % 31 %

Lifelong

learner 21.5 % 16.1 % 36 %

None of the

above 2.5 % 2.2 % 36 %

3.1 Course completion rates

Early criticisms of MOOCs focused on their low

completion rates of 5 to 15%, using a similar metric to

“dropout rates” that are used to evaluate university degree

courses [14]. Completion rate is perhaps a metric that

should be weighted less, as MOOC participants do not

take courses with the purpose of earning a degree:

curiosity and job advancement are the top two reasons

why students enroll in a MOOC [6]. Furthermore, many

students intend to audit the course, intend to only

complete a portion of the course, or enroll in a MOOC

before committing to the course [15]. Therefore, more

recently, completion rates are examined in the context of

student intent. In a study of 9 HarvardX courses, 22% of

students who intended to complete a course did so [15].

This metric was more broadly applied to 35 HarvardX and

MITx courses, and the completion rate was 24% for

students who intended to complete the course [7].

Prior to the start of the course, 885 students responded

to a survey. We asked students about their intent to

complete the course. We also asked students their

professional background and whether they had previously

studied engineering to allow us to understand the course

demographics.

In Wind, Waves and Tides, 617 out of 11,723

registrants completed the course, making the completion

rate 5.2%. By tracking the students who responded to the

pre-course survey, we calculated that the completion rate

was 42% for those who intended to complete the course.

This is significantly higher than reported averages [7, 15].

3.2 Characteristics of course participants

In Table 2, we show the distribution of course

participants and completion rate (of students with the

intent of completing the course) by their professional

background. Participants consisted mostly of students,

Table 3: Engineering background of course participants

Have you

studied

engineering

before?

Survey

respondents

Survey

respondents

who

completed

the course

Completion

rate of

students

who

intended to

complete

Yes, I have a

degree in

Engineering or

significant

work

experience

59.2 % 72.5 % 67 %

Yes, I have

taken some

courses in

Engineering or

have some

work

experience

16.5 % 13.3 % 35 %

Yes, one of my

courses had a

unit on

Engineering

5.2 % 5.1 % 38 %

Yes, I have

done some

personal

reading or

exploring on

my own.

5.3 % 3.5 % 34 %

No, not at all 13.2 % 5.4 % 18 %

professionals, and lifelong learners. For participants

intending to complete the course, the completion rates

were substantially higher for professionals and researchers

than they were for other categories.

In Table 3, we show the distribution of course

participants and completion rate (of students with the

intent of completing) by their level of engineering

experience. The majority of participants had an

engineering degree or significant engineering work

experience; however, there was also a substantial number

of students who with no engineering experience. The

MOOC targeted an audience with some engineering

education or experience, and the recommended pre-

requisites included thermodynamics, fluid mechanics, and

heat transfer. Therefore it was expected that participants

with little or no previous engineering education would be

challenged. Indeed, the completion rate was the lowest

for participants with no engineering experience who

intended to complete the course. At the other end of the

spectrum, the completion rate was very high for

participants who intended to complete the course and had

an engineering degree or significant work experience.


CEEA15; Paper 084


4. LESSONS LEARNED

4.1 Advanced courses with pre-requisites can be

successful

Initially, we were curious how the reception would be

for an advanced course with several technical pre-

requisites. With pre-requisites in place, the course

attracted a large number of registrants, and the majority of

participants did have an engineering education. The

students who met the pre-requisites were most likely to

complete the course, and in total, 617 students completed

the course. For comparison the only previous engineering

MOOC offered by the University of Toronto had no pre-

requisites, and 458 students completed that course [8].

On the other hand, approximately 13% of students had

no engineering background and did not likely meet the

prerequisites. Indeed, a number of students self-identified

as non-engineers (e.g. architect, construction company

administrator) on the discussion boards but were

sufficiently interested in the topic to try anyway and asked

for patience from other students. The freedom to take any

course could be considered a MOOC advantage as it

allows students to explore new areas without the academic

consequence of failure. Accordingly, a few students

posted on the discussion board asking for help on very

basic matters such as converting between different units.

These posts, however, were in the minority. What was

especially interesting to note was the highly supportive

response of more knowledgeable fellow students, who in

many cases patiently supplied the missing information in

detail without any hint of condescension. In the absence

of student response, course staff responded to those posts

helpfully. Furthermore, students told us that 7 weeks was

a good length for the course. In a post-course survey, 98

out of 136 course completers said the course length was

“just right”, and only 9 course completers said the course

was too long. Additionally, only 4 out of 21 non-

completers agreed that a shorter course would have made

them more likely to complete the course.

4.2 Make expectations very explicit for peer-

assessed assignments

Peer-assessed assignments are a terrific means of

developing student critical thinking and assessment skills.

For alternative energy systems, it provides an opportunity

to explore non-technical issues, e.g. regulatory and

approval processes, environmental impact, and societal

acceptance. However, the assignment instructions and

marking rubric were based on previous experience with

senior undergraduates and beginning graduate students in

a regular university course and failed to account for the

hugely diverse backgrounds and levels of the MOOC

students.

For future courses, we recommend that expectations

for the peer-assessed assignment be clearly stated in some

detail, especially the expectations for citing references.

With a diverse group of participants in the course, the

expectations for citing references were also diverse. In

some cases, authors described a local energy project in

their own words, and they were penalized during the peer

evaluation because the lack of citations, the absence of

information source documentation, was interpreted as a

sign of poor quality. Adding explicit instructions to the

assignment description and to the marking rubric would

greatly benefit authors and evaluators. Due to the

significant number of students with weak English

language skills, the marking rubric must provide a means

of evaluating communication, e.g. organization, logical

presentation of ideas, while downplaying English

language skills.

In a handful of cases, blatant plagiarism was apparent

and those students received a mark of zero for the peer-

assessed assignment. Coursera does have an Honor Code

to which students agree when they register. However, to

minimize the cases of plagiarism, we recommend

reminding students of the Honor Code in the assignment

instructions. It would also be a good idea to provide a link

to resources that help set expectations. For example, our

Faculty has prepared some excellent online tutorial

material on accurate documentation that could be

referenced in the assignment instructions [16].

4.3 Incorporating MOOCs into the classroom

Through the Open UToronto Initiative, the University

of Toronto aims to leverage the content created for

MOOCs and integrate it into its existing degree-credit

courses [8]. Indeed, all of the course material developed

for Wind, Waves, and Tides, was integrated into the 2015

Winter term offering of MIE 515 Alternative Energy

Systems, a credit course for final year undergraduate and

beginning graduate students. This integration was

facilitated by the intentional design of MOOC slide

templates, logos, etc, that were either generic by topic or

easily rebranded for use in MIE 515. The seven week

Wind, Waves, and Tides MOOC became 3 course modules

that spanned four weeks in MIE 515. An additional 7

course modules were created for MIE 515, using the

techniques and practices that were developed to produce

the material for the MOOC. Due to time and resource

constraints, it was not possible to implement the

embedded quiz feature employed with the MOOC, into

the MIE 515 lecture videos, but this is a goal for the next

MIE 515 offering.

MIE 515 has more complex problem sets and

assignments, including two simulations evaluating

prospective alternative energy systems installations. These

greatly enrich the student learning experience, but they do


CEEA15; Paper 084


require considerable course staff time for evaluation, a

resource not available for MOOCs.

MIE 515 has been offered online since 2011 [17], but

effectively evolved into an online course as an experiment

rather than being designed as an online course. The

MOOC provided a fast-paced opportunity to explore

online delivery and assessment techniques.

The value of peer-assessed assignments was readily

apparent during the MOOC, despite the issues with

assignment instructions and marking rubric design. Many

MOOC students expressed their appreciation on the

discussion boards for the highly constructive and detailed

feedback they received from their peers. Accordingly,

three peer-assessed assignments were implemented in

MIE 515 for the Winter 2005 term to replace marked

participation in online discussion boards. The latter was

ineffective at developing critical assessment skills as

students were reluctant to provide any criticism of their

classmates in a public forum.

The peer-assessed assignments were easily

implemented using the purpose-designed software

peerScholar [18]. There are 3 phases to a peerScholar

assignment: CREATE, ASSESS, and

REVISE/REFLECT. In the CREATE phase, students

develop and submit an initial version of their assignment.

In the ASSESS phase, peerScholar randomly and

anonymously sends the assignment to a specified number

of classmates (3 were used for MIE 515) for comment and

evaluation using the rubric provided by the course

instructor. The comments and evaluations are then

returned anonymously to the original author. The

REVISE/REFLECT phase provides the author with the

opportunity to reflect on the peer assessment and revise

their assignment accordingly. For MIE 515, the final

assignment submitted at the end of this phase was then

marked by course staff. As a result of the peer assessment

process, the quality of the final assignments has been very

high. As of the date of submitting this manuscript, the

course evaluations for MIE 515 have not yet been

released, and these evaluations will provide the student

impressions of the peer-assessed assignments.

5. CONCLUDING REMARKS

Our experience delivering Wind, Waves and Tides,

showed that our three initial concerns were unfounded.

There was substantial demand for a very specialized

advanced engineering course. Indeed, a number of

MOOC participants requested a more advanced follow-up

course or courses on other alternative energy topics.

Second, the pre-requisites were not a limitation Our

experience is consistent with other studies [6,7] that have

shown that MOOC participants are, in general, highly

educated. On the whole, students had the background

knowledge necessary to complete the course. This

advanced engineering MOOC tended to attract

professionals with engineering degrees, and students in

this demographic had the highest completion rates.

Finally, the seven week duration doesn’t appear to

have been a limitation either. Student feedback confirmed

that the course length was suitable. As noted previously,

completion rates for the student who said initially that

they intended to complete the course compare favorably

with data available from other MOOCs.

Acknowledgements

The authors gratefully thank the Faculty of Applied

Science & Engineering, Open Learning Strategies, the

University of Toronto Libraries, and the Open Course

Initiative Fund at the University of Toronto for making the

Wind, Waves and Tides MOOC possible.

References

[1] Coursera http://www.coursera.org

[2] edX http://www.edx.org

[3] B. S. Balaji and A. Chandra Sekhar, “The various faces of

MOOC,” 2013 IEEE International Conference in MOOC,

Innovation and Technology in Education (MITE), pp. 139-

142

[4] Flora S. Tsai, “The state of massive open online courses

(MOOCs) in engineering education: where do we go from

here?” Proc. of the 120th ASEE Annual Conference &

Exposition (Atlanta, USA, 23026 June 2013), 16 pp., 2013.

[5] “The attack of the MOOCs,” The Economist, 20 July 2013.

Available as of 10 April 2015 from

http://www.economist.com/node/21582001/

[6] G. Christensen, A. Steinmetz, B. Bennett, D. Woods, E. J.

Emanuel, “The MOOC phenomenon: who takes massive

open online courses and why?” University of Pennsylvania,

2013. Available as of 10 April 2015 from

http://ssrn.com/abstract=2350964

[7] A.D. Ho, I. Chuang, J. Reich, C. Coleman, J. Whitehill, C.

Northcutt, J. Williams, J. Hansen, G. Lopez, and R.

Petersen, “HavardX and MITx: Two years of open online

courses,” HarvardX Working Paper No. 10, 37 pp., 2015.


http://dx.doi.orgg/10.2139/ssrn.2586847

[8] L. Harrison, “Open UToronto MOOC initiative: Report on

second year of activity” University of Toronto, 2014.


http://www.ocw.utoronto.ca/wp-

content/uploads/2014/09/MOOC-Y2-Report-September-

2014.pdf


CEEA15; Paper 084


[9] “Massively Open Online Courses (MOOC) Design and

Development Process Guidelines” Open UToronto, 2013.


http://www.ocw.utoronto.ca/wp-

content/uploads/2013/05/MOOC-Design-and-Development-

Guidelines-2-Feb-2013.pdf

[10] M. Bali, “MOOC Pedagogy: Gleaning Good Practice from

Existing MOOCs,” MERLOT Journal of Online Learning

and Teaching, vol. 10, no. 1, pp. 44-46, 2014. Available as

of 12 April 2014 from

http://jolt.merlot.org/vol10no1/bali_0314.pdf

[11] A.M.F. Yousef, M.A. Chatti, U. Schroeder, and M.

Wosnitza, “What drives a successful MOOC? An empirical

examination of criteria to assure design quality of MOOCs,”

Proc. of the 2014 IEEE 14th International Conference on

Advanced Learning Technologies (ICALT), (Athens, Greece

7-9 July 2014), pp. 44-49, 2014.

[12] P. Guo, “Optimal video length for student engagement,”

edX blog, 2013. Available as of 16 April 2015 from

https://www.edx.org/blog/optimal-video-length-student-

engagement

[13] D.R. Sadler (1989). “Formative assessment and the

design of instructional systems,” Instructional Science, 18,

119-144, Kluwer Academic Publishers, Netherlands.

[14] T. Daradoumis, R. Bassi, F. Xhafa, S. Caballé, “A review

on massive e-learning (MOOC) design, delivery and

assessment,” Proc. of 2013 Eighth International Conference

on P2P, Parallel, Grid, Could and Internet Computing

(3PGCIC), (Compiègne, France, 28-30 October 2013) pp.

208-213, 2013.

[15] J. Reich, “MOOC completion and retition in the context of

student intent,” Educause Review, 2014. Available as of 9

April 2015 from http://www.educause.edu/ero/article/mooc-

completion-and-retention-context-student-intent

[16] Engineering Communication Centre, Faculty of Applied

Science and Engineering “Accurate Documentation,”


http://www.engineering.utoronto.ca/Directory/students/ecp/

handbook/accurate.htm

[17] J.S. Wallace and H. Dhariwal, “Experience with Online

Delivery of an Engineering Technical Elective Course,”

Third Annual Canadian Engineering Education Association

(CEEA) Conference, Winnipeg, Manitoba, June 17-20,

2012.

[18] PeerScholar http://www.peerscholar.com/

CEEA_Paper 2015-04-20_submitted

Documents

Transcript of CEEA_Paper 2015-04-20_submitted