Do On-Line Students Analyze, Synthesize And Evaluate Better Than Face-To-Face Students?

Preliminary Evidence Drs. Steven C. Myers & Michael A. Nelson

myers@uakron.edu

Department of Economics - The University of Akron

Presented to the

Windows on the Future 2003 Conference

OLN and ITEC-Ohio

March 2, 2003

Online at http://gozips.uakron.edu/~myers/online

Goals for Presentation Talk about our ongoing research

Research Questions Modeling process Results

Talk about the Instructional Design of the Online course

Mastery and Competency Based Learning

Encourage you to contact us

Research Questions(1) Does the mode of delivery (face-to-face or web-

based) have an influence on learning outcomes? (2) Are students in an online environment as likely to

do as well as in face-to-face classes? Will they be able to equal the complex problem solving of the face-to-face students?

(3) Will web-based students develop more favorable attitudes towards economics than the attitudes developed by students in the face-to-face class?

(4) Do student myths about economics affect their learning outcomes and attitudes towards economics?

Acknowledgments Grant from

Carnegie Teaching Academy

Scholarship of Teaching, Assessment and Learning Funds

Institute for Teaching and Learning

The University of Akron

Data collected under signed informed consent from our students subject to the Institutional Research Board for the Protection of Human Subjects at The University of Akron.

Very few failed to give informed consent

Data collection instruments validated by a number of independent reviewers and created by MA grad, Michael Lovette.

Course under study Introduction to Economic Analysis

One-semester principles of economics 3 cr. General education course Required for engineering majors Both face-to-face and on line The online course is similar to a

graduate course offered since Fall 2001

Comparison Face-to-face

Offered by Michael Nelson

Regular class times Lecture and active

learning collaborative techniques

Better than the norm of 83% “chalk and talk” (Becker and Watts)

Some minor web-enhancements such as online gradebook

Online Offered by

Myers (F02) / Nelson (S03)

No face-to-face meetings Mastery & competency

based learning High professor-student

interaction No student to student

interaction Full use of the WebCT

environment

Student Comments / online “The fact that this course was completely

internet based, had no bearing on the level of knowledge I gained from the course. I think I learned as much, or even more, than I would have in a traditional classroom. I contribute this most to the way the instructor organized the course.”

Student Comments / online “I believe that the benefits of web-based

courses … far outweigh any disadvantages such as lack of face-to-face interaction--at least in this particular course. Dr. Myers' course was, of the four I took this semester over the web, truthfully the best organized, most well-adapted to the web (by his efforts) of them all.”

Research design

Student’s success

=f( student’s

initial endowments,

awareness of the economy,

attitudes about economics,

course modality,

student characteristics)

Research design

Student’s success

=f( student’s

initial endowments,

awareness of the economy,

attitudes about economics,

course modality,

student characteristics)

Pre-test scores

total

on recall and recognition

on simple application problems

on complex problems

GPA

Previous attempt of course

Research design

Student’s success

=f( student’s

initial endowments,

awareness of the economy,

attitudes about economics,

course modality,

student characteristics)

Survey of economic facts

20 questions…on trends and facts in the economy.

2 questions:

How does the student collect business and economic news.

How many hours per week does the student spend collecting news

Research design

Student’s success

=f( student’s

initial endowments,

awareness of the economy,

attitudes about economics,

course modality,

student characteristics)

Three Survey Questions:Level of student report of being interested in economics.

Level of student report of liking this economics course.

Level of student report about likelihood of taking more economics courses.

5=strongly agree

4=agree

3=indifferent/no opinion

2=disagree

1=strongly disagree

Research design

Student’s success

=f( student’s

initial endowments,

awareness of the economy,

attitudes about economics,

course modality,

student characteristics)

Online or face-to-face

Major

Level, e.g., freshman

Credits completed

Age

Gender

Ethnicity

Dependent Variable: Student Success Outcome Measures examscore on first 6 chapters

In class exam for face-to-face class Average of first three module quizzes for online class

Final examination (FE, FE_recog, FE_apply, FE_complex) First 3 or 4 levels of Bloom’s Taxonomy

Writing assignment writing2 4th to 6th level of Bloom

Regular Grades in class

Bloom’s Taxonomy (from Carla Lane)

1. Knowledge: arrange, define, duplicate, label, list, memorize, name, order, recognize, relate, recall, repeat, reproduce state.

2. Comprehension: classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, restate, review, select, translate,

3. Application: apply, choose, demonstrate, dramatize, employ, illustrate, interpret, operate, practice, schedule, sketch, solve, use, write.

4. Analysis: analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test.

5. Synthesis: arrange, assemble, collect, compose, construct, create, design, develop, formulate, manage, organize, plan, prepare, propose, set up, write.

6. Evaluation: appraise, argue, assess, attach, choose compare, defend estimate, judge, predict, rate, core, select, support, value, evaluate.

Student Characteristics Online Students

More likely to be To be older To be non-white To be female To have taken more

credits To be decided in their

major More favorably

disposed to economics

Face-to-face StudentsMore likely To be male To be a freshman To be undecided in

their major To spend less time

gathering business and economic news.

The online course design has influenced student success

Grades Distribution

Modules Total Recognition

Conceptual

Analytical

%C&A

2-8 Micro 331 29% 45% 26% 71%

9-15 Macro

563 32% 54% 15% 68%

2-15 Total

894 31% 51% 19% 69%

MC Learning Assessment Breakdown / Online

A B C/D F

Online Course 33% 39% 0% 10%

Face-to-face Course 8% 37% 40% 10%

Results

Students’ Initial Endowments All prior expectations are met Higher pretest and higher GPA are

positive on examscore, writing2 and less so on FE.

Higher complex pretest scores and higher GPA are positive for writing2 and FE_Complex

ResultsEconomic Awareness & Attitudes

“Survey” is weakly positive for examscore and writing2, but has no effect on FE.

No combination of “News gathering” is ever significant.

“Attitudes” are strongly significant for examscore and writing2, but no effect on FE

When attitudes matter take_more econ > interest in econ > will like econ

Results

Student characteristics: Those with an undeclared or undecided major

do much worse than engineering majors for examscore and FE.

Older students do worse on examscore, no effect on writing2 and strongly positive on FE

Females do better in writing2 (strongly significance), but do worse on FE (almost significant)

ResultsOnline Students: Do much better on examscore Show no significance difference in scoring on analytical

and complex reasoning tasks. Insignificant coefficients in writing2 Insignificant and small coefficients in

FE FE_recog FE_Apply FE_complex

There is some evidence of a possible interaction with females

Weak suggestion in the data that Men in online classes do better and Women in online classes do worse all things equal

Design of the online course

Built in modules Modules completed in order Competency based testing Use student feedback

For student learning enhancement For modifying & improving the course

CourseDesign

DL requires planning for contingencies

14 Content Modules

Active Learning vs. Passive Learning

Pre-class and Module 1: Preparing Students to Learn Email me! http://gozips.uakron.edu/~myers/online/

Is Distance Learning for Me? VARK – testing learning styles Orientation Module—‘ How to logon to

WebCT’ Orientation Module—‘ How to Use WebCT’

Syllabus Graduate course Undergraduate course

Pre-class and Module 1:Additional Orientation How to Communicate with Dr.

Myers How to access your online text

http://www.economicsplace.com/econ5e/

Rules of the game Building a relationship –

Breakdown the anonymity Survey “Tell me about yourself”

Content Modules 2-15

Module Introduction & Objectives Chapter Introductions Content Supportive Materials Assessment of Learning - Quizzes

on objectives with multiple trials Evaluation

2 Research & Writing Assignments – some objectives

Practice and experience in reflecting on a topic in the current economy.

Practice in analysis of economic trends.

Gaining of confidence about talking about the economy.

Ability to know and use the resources of economic commentary, prior analysis and data.

Introductions& Content Mostly Passive Learner

Centered Students

progress without intervention

Micro – Modules 2-8

Macro –Modules 9-15

GradedAssessment

Competency based

Everyone strives to get a perfect 10 (Mastery)

Three attempts, 15 min. time limit

Questions a mix of (1) Recognition, (2) Conceptual, &(3) Analytic~70% C&A

Random intervention by Professor

Role of Module Evaluation Planned intervention Forced contact Focus on the

learning Professorial

encouragement Decreases dropout

rates Process repeats

Module EvaluationA Classroom Assessment Technique from Angelo and Cross (1993); tested by Chizmar and Ostrosky (1998)

What comments do you have on this module and your experience in completing it?

What main point have you learned that you did not fully understand before?

What questions … Include any points that still remain muddy or unclear. Do consider posing the muddy points to your fellow students in the discussions.

What recommendations do you have for us as we continue to change and enhance the course?

Do On-Line Students Analyze, Synthesize And Evaluate Better Than Face-To-Face Students?

The answer is NO

and they don’t do any worse either.

Contact: Steve Myers myers@uakron.edu

The University of Akron

Or get this paper and presentation

Online at http://gozips.uakron.edu/~myers/