LAK12 E2Coach presentation
-
Upload
tamckay -
Category
Economy & Finance
-
view
424 -
download
0
description
Transcript of LAK12 E2Coach presentation
04/10/2023 LAK 12: Vancouver
What to do with actionable intelligence: E2Coach as an intervention engine
Tim McKay, University of Michigan Departments of Physics and Astronomy, LSA Honors Program
How learning analytics and computer tailored
communications enable us to provide individualized
feedback, encouragement, and advice to thousands of
students
Mysteries of teaching and learningWhat do students know when they come? What
do they do while in class? What do they learn and remember?
Can we predict outcomes? When? For whom?Can we change outcomes? What should we do?
When? How? For whom?
04/10/2023 LAK 12: Vancouver
Learning Analytics and Knowledge• Education systems generate
a rich stream of increasingly accessible data which can inform teaching and learning
• I’m from big data cosmology, how 100s of millions of galaxies get to be the way they are. We’re exploring what data tells us about how students get to be the way they are
• Learning Analytics: the Better-Than-Expected project for introductory physics– Generates actionable
intelligence• An intervention: adaptation
of PH computer tailored communication in E2Coach– Acts on the intelligence
provided by LA
Our E2Coach application – grown from analytics, aware of general information like identity and goals, reaching into real time, content
specific student performance data
04/10/2023 LAK 12: Vancouver
Better-than-expected in Physics
• LA investigation of two year-long intro physics sequences
• 48,579 students over 14 years
• Institutional data => construct predictions of student outcomes
• Identify those who do better (and worse) than expected: Find out why
• Admissions information– High school GPA– SAT and ACT– State and Country of origin– First generation and SES– Gender
• Internal UM information– Cumulative GPA– Number of credits: UM and
transfer – Exam scores– Homework grades– Final grade in this courseMuch more data is available…
04/10/2023 LAK 12: Vancouver
Essential findings from BTE
• Student grades can be predicted with half letter grade accuracy– Incoming UM GPA the
most powerful predictor– Weak additional
information in SAT/ACT Math
• There is real dispersion: students do better (and worse) than expected
• Significant performance disparities are apparent– Gender: especially strong
in courses where female students are seriously underrepresented
– First generation college students
– Students from low socio-economic status households
04/10/2023 LAK 12: Vancouver
Mean and error on the mean for each bin
One sigma dispersion around the mean for each bin
One-to-one line…
04/10/2023 LAK 12: Vancouver
Exploring BTE/WTE
Gendered performance disparity seen in intro physics courses nationwide
Male students
Female students
Non first-gen students
First-gen students
Performance disparity seen for first-generation college students, also for low SES students…
Learning analytics make all of these explorations possible, even
the qualitative ones. They provide actionable intelligence.
04/10/2023 LAK 12: Vancouver
What to do with actionable intelligence?
• We now have John Campbell’s ‘obligation of knowing’: how we expect them to perform, and what leads to success
• How can we tailor our approaches and interactions to optimize the success of all?
• Options for response:– Tell someone and let
them act: student, instructor, advisors: scale remains a challenge
– Develop tools which act directly in response to student state:• Intelligent tutors etc.• Computer tailored
communication systems
04/10/2023 LAK 12: Vancouver
Tailoring is well established and tested in public health, and has seen major commercial application. An extensive body of peer-reviewed research reports on tests of efficacy in design across interventions ranging from smoking cessation and
diabetes control to cancer treatment decision making and depression. This research provides a strong base for the design of new computer tailored interventions
04/10/2023 LAK 12: Vancouver
MTS was built by the University of Michigan’s Center for Health Communications Research, an established leader in
computer tailored public health interventions. MTS is a mature, fully open-source software system for computer
tailored communication.
04/10/2023 LAK 12: Vancouver
E2Coach: tailored support for
physics students• Used LA and MTS to
construct “E2Coach”: an Electronic Expert coaching system for intro physics courses
• You can find a basic information about the project online:
http://sitemaker.umich.edu/ecoach
• Three groups of players:– Department of Physics– CHCR leadership and
staff– Consultants from across
the campus• Project goals:
– Improved performance and affect for all students
– Reduced disparitiesThe E2Coach team:
Tim McKay, Kate Miller, Jared Tritz, Gus Evrard, Dave Gerdes in Physics
Vic Strecher, Ed Saunders, Holly Derry, Mike Nowak at CHCR
04/10/2023 LAK 12: Vancouver
How does E2Coach work?
MTS
Detailed information
about thousands of students and their current status
Expertise of hundreds of students, dozens of instructors and behavior change experts Individually
personalized messages:
what we all agree we
would say to each
student, if only we could…
The Michigan Tailoring System: a mature open-source software system for
creating content designed specifically for an individual based on data about that
individual
Where the real effort lies
04/10/2023 LAK 12: Vancouver
Expertise and Information
• Structured interviews with faculty
• Survey of 70+ student study group leaders
• Better-than-expected interviews
• Input from students with different backgrounds has extreme relevance!
• Knowledge of each course and its structure
• Real-time input from the course gradebook
• Input from the student– Background, goals and
interests, planned effort, desired and expected grades, self-efficacy, confidence in physics
• Opt-In: 54% (953 total)
04/10/2023
What we provide• Tailored advice on all aspects of the course,
including testimonials from relevant peers
04/10/2023 LAK 12: Vancouver
Performance feedback
04/10/2023 LAK 12: Vancouver
First measures of impact
• First term ended four days ago: final scores for enrolled students 2.3% (4) higher
• Currently examining effects vs. usage
• Disparities on gender, SES, first-gen status
• This is a complex intervention, with many parts: which are key?
• Testing in fall using fractional factorial design
Score differences observed in 104 random samples
Actual measured score difference
04/10/2023 LAK 12: Vancouver
E2Coach in the LA landscape…
SOLAR: Open Learning
Analytics: an integrated & modularized
platform
Siemans, G., et al. July 2011http://solaresearch.org
BTE project and other analytics
Computer tailored
communication
04/10/2023 LAK 12: Vancouver
Learning Analytics at Michigan
• Redesigning for Physics in the fall– Full enrollment– Much tighter approach
• Expanding to situations w/diverse student bodies– Intro Stats: 1800 – Epidemiology: Masters– Freshman orientation
• UM Provost has empowered Learning Analytics Task Force
• Charge:– Improve information
environment for LA– Support LA projects with
three funding cycles– Revise institutional
metrics used for teaching and learning