A Joint MERLOT and European View of Multimedia in Physics

EPS Multimedia Working Group

MERLOT/Physics

MERLOT International Conference 2006Ottawa Canada, 8/10/2006

MIC 2006

Participants

EUPEN Working GroupM. Benedict (Hun), E. Debowska (Pol),B. Eckert (Ger), S. Geiner-Valkier (Neth),H.J. Jodl (Ger), L. Mathelitsch (Aus),I Ruddock (UK), E. Sassi (It), R. Sporken (Bel),

MERLOTT. Bradfield (OK), T. Colbert (GA), L. Keiner (SC), B. Mason (OK), T. Meldor (LA), J. Rauber (SD),S. Sen (NY), P. Sheldon (VA)

MIC 2006

Reviews

Are we measuring something?

And if so, what?

MIC 2006

Comparison...

History

Goals & Methods

Criteria

Results

MIC 2006

EPS History

2002: EUPEN Working Group to Define Quality Standards. Example: Quantum Mechanics (MPTL-7, Parma)

2003: Apply Review Process. Example: Optics (MPTL-8, Prague)

2004: Classical Mechanics (MPTL-9, Graz) 2005: Thermal Physics (MPTL-10, Berlin) 2006: Electricity & Magnetism

(MPTL-11, Szeged)

MIC 2006

MERLOT Review History

In 1996/7 the California State University embarked on a...

MIC 2006

Goals & Methods

Present the high quality stuff, Recognize authors, Identify gaps

EPS –

• Work within one topic

• Collect many resources

• Select the best

• Apply criteria for full review of best

• Write up results

MERLOT –

• Search & submissions

• Rank materials for review

• Apply criteria for full review of best

• Review and publish

MIC 2006

Review Criteria (Rubrics)

EPS –

Motivation Content Method

MERLOT –

Quality of Content Effectiveness as a

Learning Tool Usability

MIC 2006

Review Criteria (Rubrics)

EPS – Motivation

User Friendly: Easy to start, understand, control, and is documented

Attractive: Appealing, interactive, and interesting

Clear Description and Context

Content Method

MERLOT – Quality Effectiveness Usability

Understandable: Runs easily

Intuitive: Attractive, controlled, input/output

Feedback: Clear communication

Documented

MIC 2006

Review Criteria (Rubrics)

EPS – Motivation Content

Relevant: Important topic and media use

Scope: Broad and “Profound” topic

Correct: Content accurate and models indicated

Method

MERLOT – Quality

Correct Models: Numerical, textual, notation

Important Topics: Standard or unique

Conceptual: Understand parameters

Effective Graphics Flexible: Multiple uses

Effectiveness Usability

MIC 2006

Review Criteria (Rubrics)

EPS – Motivation Content Method

Flexible: Broad audience and topics

Matching Target: Correct level, background, objectives

Realization: Media is well used

Documentation: Operation, references, teaching process

MERLOT – Quality Effectiveness

Learner: Level, challenge, control

Relevant Knowledge: Learning goals, application

Experience: Dynamic, flexible, interactive, progress

Feedback: Clear, immediate, positive

Usability

MIC 2006

Review Scales

EPS – +2 to -2 Indicated for each

sub-question Averaged

MERLOT – 1 – Major Problems 2 – Works, but has

errors 3 – Works well,

useful 4 – Excellent, notable

features 5 – Outstanding and

unique

MIC 2006

Agreement?

MERLOT Reviewers: Average Absolute Value of Difference between Reviewers (Scale of 1 – 5, Reviews 3 – 5, 135 items)

Quality: 0.56 (3.8) Effectiveness: 0.60 (3.65) Usability: 0.69 (3.9) Total: 0.57 (3.72)

MIC 2006

Agreement?

MERLOT Reviewers: Average Difference between Reviewers, Time Dependence

Quality: 0.57 (00 – 03), 0.55 (04 – 06) Effectiveness: 0.7 (00 – 03), 0.5 (04 – 06) Usability: 0.7 (00 – 03), 0.69 (04 – 06) Total: 0.6 (00 – 03), 0.53 (04 – 06)

MIC 2006

Agreement?

MERLOT Reviewers: Average Difference between Reviewers, Score Dependence

(N5 = 20, N4 = 61, N3 = 39)

5 4 3

Qual 0.55 0.53 0.55

Eff 0.35 0.62 0.64

Use 0.45 0.66 0.82

Tot 0.35 0.62 0.52

MIC 2006

Agreement?

For MERLOT Reviewers

Average Difference in Total Scores: 0.4 – 0.5

MIC 2006

2002 Quantum Mech

EPS: ~30 items, 7 recommended MERLOT:

2 award winners, 5, 4 (2), 2 not reviewed = 0 (1 in German

Joffre QM (2003 MERLOT Classic) de Raedt QM (No MERLOT Review)

General Recommendations Results from Workshop

MIC 2006

2003 Optics

EPS: ~250 items, 5 Recommended

MERLOT: 5, “4.5” (2), 0 (2)

Paul Falstad Math & Physics Applets

Report & Recommendations

MIC 2006

2004 Mechanics, Different Lists

EPS: ~250 items 7 Recommended Merlot: 5 (2), “4.5” (2), 4, 3, 0 (German)

MERLOT: ~700 items 4 Other Recommendations EPS: Rec (not listed), High quality, 0 (2)

Chaos Collection

Report & Recommendations

MIC 2006

2005 Thermal Physics, Joint

~150 items, 55 Reviewed 1 Highly Recommended EPS & MERLOT

(Engineering) 7 Recommended 1 Disagreement

Expert System for Thermodynamics Stat Mech in Intro Physics

Report & Recommendations

MIC 2006

2006 Elect & Magn, Joint

950 items, 200 Reviewed 6 Highly Recommended EPS & MERLOT 6 Recommended by both 5 Disagreements (Some may not have been

reviewed)

MIT “TEAL” (OCW) Physlet Tutorial Pages

MIC 2006

Comments & Conclusions

Good agreement w/ independent reviews Quality rating for larger collections

Many topics repeated Many Simulations, not video experiments

Instructional aspects rare How to put into a class rarer

MIC 2006

References

http://physik.uni-graz.at/MPTL9/proceedings/ProcSporkenMason.pdf

http://pen.physik.uni-kl.de/w_jodl/MPTL/MPTL10/contributions/mathelitsch/Rep_Recom_Thermodyn_2005.pdf