1/∞ CRESST/UCLA Towards Individualized Instruction With Technology-Enabled Tools and Methods...
-
date post
20-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of 1/∞ CRESST/UCLA Towards Individualized Instruction With Technology-Enabled Tools and Methods...
1/∞CRESST/UCLA
Towards Individualized Instruction With Technology-Enabled Tools and Methods
Gregory K. W. K. Chung, Girlie C. Delacruz, Gary B. Dionne, Eva L. Baker, John Lee, Ellen Osmundson
American Educational Research AssociationAnnual MeetingChicago, IL - April 9-13, 2007
Symposium: Rebooting the past: Leveraging advances in assessment, instruction, and technology to individualize instruction and learning
UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing
2/∞CRESST/UCLA
Structure of Talk
• Problem
• Pre-algebra/Algebra
• Classroom constraints
• Research
• Domain analysis, assessment design, instructional design, screen shots
• Results
• Did it work?
3/∞CRESST/UCLA
Problem
4/∞CRESST/UCLA
Study Context• Why pre-algebra?
• Pre-algebra provides students with the fundamental skills and knowledge that underlie algebra
• Pre-algebra Algebra STEM
• 2001-02 LAUSD 9th grade cohort:
• Only 65% of 9th graders (~28,000) progress to 10th grade (23% retained in 9th grade [~10,000]; 12% leave district)
• Algebra identified as gatekeeper
• In fall 2006, 38% of CSU first-time freshmen needed remediation in mathematics (17,300)
• CSU 5-year graduation rate (STEM): 34%
5/∞CRESST/UCLA
Proportion Wrong
8th(1st semester)
9th (entering)
.23 .23
.51 .10
.38 .08
.34 .33
.86 .70
6th/7th Grade Standards
6/∞CRESST/UCLA
Research
7/∞CRESST/UCLA
Research Questions
• To what extent can students learn from “instructional parcels”—brief slices of instruction and practice?
• To what extent can automated reasoning (i.e., Bayesian networks) be used for automated diagnosis of pre-algebra knowledge gaps?
• What is the architecture for a diagnosis and remediation system?
8/∞CRESST/UCLA
General Idea
Automated Reasoning(Bayes net)
Pre-algebra pretest adding
fractions distributiveproperty
multiplicative identity
Individualized instruction
and practice
Individualized posttest
9/∞CRESST/UCLA
Research Study Overview
• Domain analysis and assessment design
• Identify the key concepts that underlie pre-algebra and the relations among those concepts
• Instruction and practice
• Design based on best-of-breed (worked examples, schema-based instruction, multimedia learning, effective feedback)
10/∞CRESST/UCLA
Domain Analysis
11/∞CRESST/UCLA
Domain Analysis
12/∞CRESST/UCLA
Domain Analysis
13/∞CRESST/UCLA
Sample Assessment Items
DP (OPER)
CP-ADD (CE)
AP-ADD (CE)
DP (CE)
DP (CE)
DP (CE)
DP (OPER)
DP = distributive propertyCP-ADD = commutative property of additionAP-ADD = associative property of addition
CE = common errorOPER = operation
14/∞CRESST/UCLA
Instructional Design
15/∞CRESST/UCLA
Design Assumptions
• To maximize the chance of learning with only brief exposure to content, instruction should:
• Direct the learner’s attention to important content
• Highlight and explain the importance of the content
• Use lay language and 1st/2nd person voice
• Use worked examples with visual annotations, coordinated and complementary narration
• Provide varied examples with different surface features but same underlying concept
• Provide practice on applying the concept
• Provide tailored and explanatory feedback
16/∞CRESST/UCLA
Multiplicative Identity - 4
17/∞CRESST/UCLA
Multiplicative Identity - 5
18/∞CRESST/UCLA
Multiplicative Identity - 6
19/∞CRESST/UCLA
Multiplicative Identity - 7
20/∞CRESST/UCLA
Multiplicative Identity - 8
21/∞CRESST/UCLA
Multiplicative Identity - 9
22/∞CRESST/UCLA
Multiplicative Identity - 10
23/∞CRESST/UCLA
Multiplicative Identity - 11
24/∞CRESST/UCLA
Multiplicative Identity – 12/12
25/∞CRESST/UCLA
Multiple Examples
Stage 1: What is the next step in solving the problem?
Stage 2: What is the result of carrying out the step?
Stage 3: What is the underlying math concept?
26/∞CRESST/UCLA
Feedback is tailored to the specific option selected
Feedback is tailored to the specific option selected
• Knowledge of results (correct/incorrect)
• Explanation—why correct or incorrect
• Knowledge of results (correct/incorrect)
• Explanation—why correct or incorrectWhat to think about to solve problemWhat to think about to solve problem
Animation/video—goal, why, how, andcommon errors
Animation/video—goal, why, how, andcommon errors
27/∞CRESST/UCLA
Method
• 2 group pretest, posttest design
• 113 middle school students
• Instruction vs. no instruction (stratified by concept and high, medium, low knowledge)
• Procedure
• Pretest (84 items, = .89)
• Instruction on concepts
• 4-6 concepts per student (out of 10)
• Individualized posttest
28/∞CRESST/UCLA
Method
• Analysis
• Examined 6 scales (pretest = .61 - .75)
• Adding fractions, distributive property, transformations, multiplicative identity
• Multiplying and adding fractions, rational number equivalence
• Participants dropped an analysis if
• Diagnosed as high knowledge
• Non-compliant (20-50%, depending on scale)
29/∞CRESST/UCLA
Results
Adding fractions
Distributive property
Transfor-mations
Multiplicative identity
Multiply, add fractions
Rationale number
equivalence
max = 8n = 9/22p = .50d = --
max = 8n = 14/18p = .04d = .76
max = 6n = 13/21p = .04d = .77
max = 7n = 7/21p = .06d = .91
max = 6n = 13/12p = .03d = .91
max = 6n = 13/12p = .01d = .50
Instruction
No instruction
30/∞CRESST/UCLA
Student Perceptions
• 46% reported tool the was very useful, 53% reported they were very willing to use
• Tool not for everyone
• “I really understood the way they took step by step to show the problem.”
• “I thought that the video and the practice problems were very easy to understand, but if they were a little more ‘exciting’ it would help make the process more fun.”
• “No, it just made me confused. I like seeing everything on a board, not computer.”
31/∞CRESST/UCLA
Closing Remarks
• Preliminary results promising
• Instruction appears effective, even if brief
• Technical aspects of individualization of instruction and assessment tractable
• DITSy – Dumb Intelligent Tutoring System
• Limitations and next steps
• One instructional day between pretest and posttest; unknown retention effect; low on some scales; novel assessment format
• Refine measures, replicate, validate diagnosis, examine more complex outcomes, examine instructional variables
32/∞CRESST/UCLA
Backup
33/∞CRESST/UCLA
Content
• Ten concepts tested
• Properties (e.g., distributive property)
• Problem solving (e.g., multiplying fractions)
• Time per parcel
• Mean time (mm:ss): 3:45
• Range (mm:ss): 2:05 to 5:19, 10:58
34/∞CRESST/UCLA
Task Sequence
Instruction(worked examples, narration,
visual annotations)
Practice (stage 1)Identify thenext step
Practice (stage 2)Identify the result
Feedback(tailored andexplanatory)
Practice (stage 3)Identify the
math concept
Feedback(tailored and explanatory)
Feedback(tailored and explanatory)
35/∞CRESST/UCLA
Incorrect response feedback: (a) confirmation that the response is incorrect, and (b) a brief explanation of why the response is wrong
Hint on what to think about to solve problem
Feedback is tailored to the specific option selected.
36/∞CRESST/UCLA
Correct response feedback: (a) confirmation that the response is correct, and (b) a brief explanation of why the response is correct
37/∞CRESST/UCLA
Don’t know response feedback: guidance on what the student should be considering
38/∞CRESST/UCLA
animation/video feedback: goal, why, how, and common errors
39/∞CRESST/UCLA
Timing
Concept Default Time
10:58 Adding fractions
5:19 Reducing fractions
4:12 Associative property of multiplication
4:09 Distributive property
3:27 Multiplicative inverse
3:19 Multiplying fractions
3:04 Multiplicative identity
2:27 Associative property of addition
2:14 Commutative property of addition
2:05 Commutative property of multiplication
40/∞CRESST/UCLA
Some Observations
• Classroom / group instruction inefficient
• A lot of time spent on non-instructional activities
• Teacher telling jokes
• Students settling down (pulling out notebook from backpack, opening textbook, getting up to sharpen pencil)
• Teacher writing the equation on the whiteboard
41/∞CRESST/UCLA
Pretest-Posttest (Instruction)
Adding fractions
Distributive property
Transfor-mations
Multiplicative identity
Multiply, add fractions
Rational number
equivalence
max = 8n = 22p = .02d = .50
max = 8n = 18p = .09d = .54
max = 6n = 21p < .001d = .61
max = 7n = 21p < .001d =1.36
max = 6n = 11p = .14d = .37
max = 6n = 12p = .10d = .43
42/∞CRESST/UCLA
Pre-algebra Bayes Net
definitions
definitions
operations
operations
transformations
transformations
common errorscommon errors
43/∞CRESST/UCLA
Diagnosis and Remediation
• Model the domain of pre-algebra with a Bayesian network
• Treats test items as evidence of understanding
• Computes the probability of a student understanding a concept
• Short slices of instruction that are focused on a single concept
• NOT intended to replace classroom teaching
• Intended to support homework, review, wrap-around activities
44/∞CRESST/UCLA
45/∞CRESST/UCLA
Multiplicative Identity - 1
46/∞CRESST/UCLA
Multiplicative Identity - 2
47/∞CRESST/UCLA
Multiplicative Identity - 3