The impact of writing letters to older peers

Shu-Ching Chang, Kun Chen04/21/2010

Research Purpose

• To examine the impact of evidence-based letter-writing tasks on students’ conceptual understanding of some physics concepts.

• In what way students benefit from the task of writing to different audiences.

Question

• Do students who engage in writing letters to older peers perform better than students who do not?

Research Design

• Two-year project.• SchoolClassStudents• Control and treatment groups.• Pre-/post-/delayed tests.• Tests: 20 multiple choice questions based upon 6

core concepts. ( Developed by Horizon Research Inc.)

Core ConceptsCore Concept Content

A (9 items) A force is any push or pull on an object.

B (3 items) A force can change the speed of an object.

C (2 items) Friction is a force that occurs when two surfaces rub together.

D (2 items) If the same force is applied to a lighter vehicle and a heavier vehicle, the speed of the lighter vehicle will change more that the speed of the heavier vehicle.

E (2 items) Energy can be stored in a rubber band and released to turn an axle or spin a propeller to make a vehicle move.

F (2 items) Air resistance is a force that can slow the speed of a moving vehicle.

Writing Task—Three letters exchange between Grade 4 and Grade 11

Grade 11: Design a question to ask based on the concepts that will be needed by the 4th graders to complete the final design challenge.

Grade 4: Make a hypothesis and/or claim to answer the question

Letter 1

Letter 2

Grade 11: This letter will evaluate, critique the rationale, and suggest revisions to the elementary students’ ideas.

Grade 4: Make a new hypothesis and/or claim to answer the question.

Letter 3

Grade 11: Continue to evaluate, critique the rationale, and suggest revisions to the elementary students’ ideas.Grade 4: Same as response 2.

Guidelines for Grade 4• Answer the question in the letter from your high school

partners and explain your thinking by using supporting evidence.

• Evidence can come from your experience and/or from the investigations you are conducting in science class. It can include the data you collect in experiments and your observations.

• Consider different ways to explain your thinking about the concepts of force and motion. How could you use diagrams, pictures, charts or graphs to help explain your ideas?

• If your high school partners suggest an experiment to test one of your ideas, think about how you can control the variable so that you have a fair test.

Data: participants

• 4 elementary Schools in a north-eastern state of the U.S.A• 27 Teachers. Overlap in two years. Not randomly assigned.• Students are different from year to year. 4th graders.

Control Group

Treatment Group Total Number

1st Year 309 145 454

2nd Year 68 316 384

Total 377 461 838

–Method (1: control 2: treatment)– School – Teacher– Student– Pretest Score– Posttest Score– Score type (0:pre 1:post 2:delayed)– Improvement: Post – Pre– Year (Combined)– Delayed test (Not available)

Data: variables

Data Analysis

• ANCOVA (From our client) ---combing two years as a whole ---design structure is totally ignored.

• Split-Plot Analysis --- can we combine the two years’ data? ---schoolteacher/classesstudents

ANCOVA Result

Control Group Treatment Group

n Mean S.D. n Mean S.D.

Pretest 377 7.98 2.874 461 7.93 2.663

Posttest 377 10.79 3.105 461 11.47 2.887

Control Group Treatment Group

n

Adjusted

Mean S.E. n

Adjusted

Mean S.E.

Student

Achievement377 10.77 .14 461 11.477 .127

F(1, 835)= 13.699, p<.000, effect size=0.25

Split Plot Analysis

• Design schoolteacher/classesstudents

• Combine two years’ data? --Students are different in two years. --Tests are standardized. -- Treatments are the same. -- Year effect is not significant.

SPA: Model 1

• Model1 :

studentzteacherrandom

treatmentcontrolmethodschoolfixed

effect

test scorecore - preposttest s diff y

iable: depedent

lkjiwhere

zy kjilklijkijjiijkl

::

):2:1(::

:

)(

var

838,,127,,12,14,,1

,),,()()(

SPA: Model 1

– SAS code:proc mixed data=data1; class School Teacher Student Method; model diff=School|Method ddfm = satterth outp=diags; random Teacher(School Method) Student(Teacher); LSmeans School*Method / pdiff adj=Tukey; LSmeans School;run;

SPA: Model 1

– Result:• there is no significant difference of the score

improvement for treatment and control groups in different schools.

Type 3 Tests of Fixed Effects

Effect NumDf

Den

Df F value Pr > F

SchoolMethod

School*Method

313

34.234.234.2

2.124.330.67

0.11610.04500.5777

SPA: Model 2

• Model2 :

studentzteacherrandom

treatmentcontrolmethodyearfixed

effect

test scorecore - preposttest s diff y

iable: depedent

lkjiwhere

zy kjilklijkijjiijkl

::

):2:1(::

:

)(

var

838,,127,,12,12,1

,),,()()(

SPA: Model 2

– SAS code:proc mixed data=data1; class School Teacher Student Method Year; model diff=Year|Method ddfm = satterth outp=diags; random Teacher(Method) Student(Teacher);run;

SPA: Model 2

– Result:• there is no significant difference of the score

improvement for treatment and control groups for two years.

Type 3 Tests of Fixed Effects

Effect NumDf

Den

Df F value Pr > F

YearMethod

Year*Method

111

43.8109109

1.270.760.08

0.26590.38640.7728

SPA: Model 3

• Model3:

studentzteacherrandom

postprescorecode

trtcontmethodfixed

effects

posttest)retest and scores (py

iabledependent

,l,,k,j,iwhere

zry kjilklikijjiijkl

::

):1:0(:

):2:1(:

:

:var

838,12712121

,)( ),,()()(

SPA: Model 3

– SAS code:proc mixed data=data2; class Teacher Method Student Scorecode ; model Score=Scorecode|Method ddfm = satterth outp=diags; random Teacher(Method) Student(Teacher); LSmeans Scorecode*Method / pdiff adj=Tukey; estimate "12-24" scorecode*method 1 -1 -1 1run;

SPA: Model 3

• From the least square means, holding method fixed, the difference of pre and post tests is significant for control and treatment groups.

Diff of Least Square

Means

Effect M S _M _S Estimate pvalueMethod(M)*Scorecode(S)Method(M)*Scorecode(S)

12

00

12

11

-2.809-3.538

<.0001<.0001

SPA: Model 3

– Results:• the interaction term of method and score-type turns

out to be significant.

Type 3 Tests of Fixed Effects

Effect NumDf

Den

Df F value Pr > F

ScorecodeMethod

Method*Scorecode

111

83637.4836

859.020.89

11.33

< .00010.35270.0008

SPA: Model 3

• the improvement for students in the treatment group is significantly higher than the improvement for students from the control group. • This implies that new method seems to increase the

critical thinking skills.

Estimate

Label Estimate Standard

Erroe DF t value Pr > |t|

12-24 0.7289 0.2166 836 3.37 0.0008

SPA: Model 4

• Model4(add school effect to model 3):

studentzteacherrandom

schoolfscorecodemethodfixed

effects

posttest)retest and scores (py

iable:dependent

lkmjiwhere

zfy mkjilklikmijjimlkji

::

:::

:

var

838,,127,,14,,12,12,1

)( ),,,()()(,,,,

SPA: Model 4

– SAS code:

proc mixed data=data2; class School Teacher Method Student Scorecode ; model Score=Scorecode|Method School/ ddfm = satterth outp=diags; random Teacher(Method) Student(Teacher); LSmeans Scorecode*Method / pdiff adj=Tukey; estimate "12-24" scorecode*method 1 -1 -1 1;run;

SPA: Model 4– Results:• the school effect is not significant . • after adjusting for school effect, the interaction

between method and score-type is significant.

Type 3 Tests of Fixed Effects

Effect NumDf

Den

Df F value Pr > F

ScorecodeMethod

Method*ScorecodeSchool

1113

83634.783634.4

859.021.02

11.330.92

< .00010.31860.

00080.4409

SPA: Model 4

• the improvement for students in the treatment group is significantly higher than the improvement for students from the control group. • This implies that new method seems to increase the

critical thinking skills.

Estimate

Label Estimate Standard

Error DF t value Pr > |t|

12-24 0.7289 0.2166 836 3.37 0.0008

Conclusion & Discussion

• Students who engage in writing letters to older peers performed better than students who do not.

• Teachers are not randomly assigned.• Delayed test.