ExperimentsExperiments

Perfesser PlumPerfesser Plum20122012

The essential feature of the strategy of The essential feature of the strategy of experimental experimental

researchresearch is that you… is that you… CompareCompare two or more situations (e.g., schools) two or more situations (e.g., schools) that are that are as similaras similar as you can make them (e.g., % as you can make them (e.g., % of students who are poor, size of school, quality of of students who are poor, size of school, quality of teaching staff), but these schools are teaching staff), but these schools are different different in in the the few factorsfew factors (e.g., reading materials) (e.g., reading materials) whose whose effectseffects you are trying to find out, or you are trying to find out, or testtest..

IfIf the only main way that the schools differ is the the only main way that the schools differ is the reading materials used, and if one school has reading materials used, and if one school has

consistently consistently higher reading achievement than the others, then higher reading achievement than the others, then it it

isisvery likely that the very likely that the reading materialsreading materials make the make the

difference. difference. What ELSE What ELSE could could it be? After all…. it be? After all….

• The factor that makes the difference in The factor that makes the difference in reading achievement reading achievement can’t becan’t be class size-class size-----because this is because this is the samethe same across all the across all the schools.schools.

• The factor that makes the difference in The factor that makes the difference in reading achievement reading achievement can’t becan’t be teacher teacher qualityquality---because this is ---because this is the samethe same across all across all the schools.the schools.

• The factor that makes the difference in The factor that makes the difference in reading achievement reading achievement can’t be thecan’t be the percentage of students who are poorpercentage of students who are poor------because this is because this is the samethe same across all the across all the schools.schools.Therefore, the factor that makes the Therefore, the factor that makes the difference in reading achievement difference in reading achievement must be must be the ONE major way the schools are the ONE major way the schools are differentdifferent----reading ----reading curriculumcurriculum..

Not all experiments give information that you can Not all experiments give information that you can trusttrust.. The The validity (accuracy)validity (accuracy) of the information is of the information is

weakened by weakened by OTHER factors. So, you have to carry out experiments in a OTHER factors. So, you have to carry out experiments in a

way way that that RULES OUTRULES OUT these other factors. For example… these other factors. For example…1. 1. Maybe students in schools with the highest reading Maybe students in schools with the highest reading

achievement achievement were already reading at a higher levelwere already reading at a higher level.. Therefore, these students don’t have to learn very much Therefore, these students don’t have to learn very much in order to come out first. In other words, maybe in order to come out first. In other words, maybe students’ students’ pre-skills pre-skills (and (and notnot the curriculum materials) the curriculum materials) account for higher achievement.account for higher achievement.

To rule out this possibility, an experiment should have To rule out this possibility, an experiment should have pre-pre-test measurestest measures of the variables whose change we are of the variables whose change we are

interested interested in. For instance, we should measure reading skills in all of in. For instance, we should measure reading skills in all of

the the schools schools BEFOREBEFORE new reading materials are introduced, to new reading materials are introduced, to see if some classes are ahead of other classes.see if some classes are ahead of other classes.

So, if an article says that a program or So, if an article says that a program or method produced more change or is more method produced more change or is more effective than another program or method, effective than another program or method,

ask ask if the researchers did if the researchers did pre-testspre-tests to show that to show that students all students all started at the same levelstarted at the same level.. If they If they did not do pre-tests, then you cannot trust did not do pre-tests, then you cannot trust

the the claim about effectiveness. claim about effectiveness. The group with The group with

the the highest achievement may have started at a highest achievement may have started at a higher level.higher level.

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions? ExperimentExperiment

2. 2. How do you know how much students learned?How do you know how much students learned? The The only way to find out is to do only way to find out is to do postpost-tests-tests (e.g., at the end (e.g., at the end of the semester and year) to see where students’ of the semester and year) to see where students’ reading achievement reading achievement ended upended up. And you have to see . And you have to see how much students how much students changedchanged (learned, achieved) (learned, achieved) betweenbetween the pre-test the pre-test (reading proficiency at the start) (reading proficiency at the start) and the post-testand the post-test (reading achievement at the end). (reading achievement at the end). And you need to And you need to report report how much differencehow much difference there is there is between the pre-test and post-test scoresbetween the pre-test and post-test scores..

If researchersIf researchers do NOT do post-tests, or do not REPORT do NOT do post-tests, or do not REPORT the the

size of differences between the pre-test and post-testsize of differences between the pre-test and post-test, , then you then you

don’t know don’t know HOW MUCHHOW MUCH students learned as a result of students learned as a result of the the

curriculum that was used curriculum that was used betweenbetween the pre-test and the the pre-test and the post-post-

test. test. Therefore, you can’t trust claims that one Therefore, you can’t trust claims that one curriculum curriculum

produced more achievement than the others.produced more achievement than the others.

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions? ExperimentExperiment

3. 3. Were measures objective? Were measures objective? You don’t rely on a physician’s You don’t rely on a physician’s subjective opinionsubjective opinion about about

your your health.health.

““You look fine You look fine to meto me.”.”Maybe the physician is right.Maybe the physician is right.But what if the physician’s opinion is wrong?But what if the physician’s opinion is wrong?Just so, you want Just so, you want objective evidence ofobjective evidence of (in this (in this

experiment) experiment) reading achievementreading achievement. It is nice to know how much . It is nice to know how much

teachers teachers

think think their students learned, but their students learned, but you can’t judge you can’t judge effectiveness based on opinioneffectiveness based on opinion. Reading is a real . Reading is a real

thing. thing. It It cancan be measured be measured objectivelyobjectively---for example, how many ---for example, how many words students read correctly per minute. words students read correctly per minute.

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions? ExperimentExperiment

So, if researchers So, if researchers do NOT use do NOT use objective measuresobjective measures of (for example) of (for example) reading and math achievement (e.g., reading and math achievement (e.g., counting correct words students read counting correct words students read or problems students solve per or problems students solve per minute), but instead use the minute), but instead use the subjective opinionssubjective opinions of teachers of teachers (“Students improved a great deal.”) (“Students improved a great deal.”) then then you cannot trust claims about you cannot trust claims about effectivenesseffectiveness----because the ----because the researchers did not give researchers did not give evidence evidence of of effectiveness. They only gave effectiveness. They only gave evidence of how teachers evidence of how teachers see thingssee things. . But But that that was not the question.was not the question.

4. 4. Were Were instruments instruments and and measurement methodsmeasurement methods tested tested for for validity validity (they measure what they are (they measure what they are supposed to measure) and supposed to measure) and reliability reliability (different (different persons measuring the same thing would come persons measuring the same thing would come up with about the same scores)? up with about the same scores)? Validity.Validity. Do you think a good measure of Do you think a good measure of reading proficiencyreading proficiency is whether students can is whether students can name the parts of a book and can turn the name the parts of a book and can turn the pages? pages?

If your school principal is selecting a new If your school principal is selecting a new reading program, is that the information you reading program, is that the information you want the principal to use?want the principal to use?

““Students taught with Weird Phonics Students taught with Weird Phonics learned to learned to turn the pages. So, I’ll use that turn the pages. So, I’ll use that program with 200program with 200 students in my school.”students in my school.”

I don’t think so.I don’t think so.

Turning pages is not what “READING” Turning pages is not what “READING” means.means.

You want to know if a program You want to know if a program teaches students to: (1) say the teaches students to: (1) say the sounds that go with the letters; (2) sounds that go with the letters; (2) accurately and quickly read single accurately and quickly read single words; (3) accurately and quickly words; (3) accurately and quickly read passages; (4) learn many new read passages; (4) learn many new vocabulary words each week; (5) vocabulary words each week; (5) answer both simple and hard answer both simple and hard comprehension questions about what comprehension questions about what they read.they read.

So, if researchers use instruments and So, if researchers use instruments and measures that measures that do not measuredo not measure what they what they OUGHTOUGHT to measure, then to measure, then the findings have the findings have nothing to do with the research question.nothing to do with the research question.

• The question is whether a curriculum The question is whether a curriculum teaches students to teaches students to readread. But the findings . But the findings have to do with have to do with turning pagesturning pages. . Invalid Invalid measure.measure.

• The question is whether a certain teaching The question is whether a certain teaching procedure increases math procedure increases math achievementachievement. . But the findings have to do with how much But the findings have to do with how much students students enjoyed enjoyed being taught with the being taught with the method. method. Invalid measure.Invalid measure.

• The question is whether peer tutoring is The question is whether peer tutoring is effective at helping students to effective at helping students to spell betterspell better. . But the findings have to do with changes But the findings have to do with changes in in self-esteemself-esteem. . Invalid measure.Invalid measure.

Instruments and measures may measure what they are Instruments and measures may measure what they are supposed to measure (e.g., they measure reading skills) supposed to measure (e.g., they measure reading skills)

but they but they do not provide accurate (valid) informationdo not provide accurate (valid) information on what on what

they they measure. They are like a stop watch that gives the measure. They are like a stop watch that gives the

wrong time. wrong time. Therefore, the findings of research may Therefore, the findings of research may LOOKLOOK as if one as if one curriculum produces higher achievement than another, curriculum produces higher achievement than another,

when in when in fact the information (data, numbers) is incorrect. fact the information (data, numbers) is incorrect.

Therefore, it is essential to use instruments and Therefore, it is essential to use instruments and measures that measures that

have have already been tested and have been shown to already been tested and have been shown to provide provide

accurate informationaccurate information. You cannot trust the claims of . You cannot trust the claims of authors authors

who do who do NOTNOT discuss how their tests, instruments, or discuss how their tests, instruments, or measures measures

were validated.were validated.

5.5. Is the sample size, composition (kinds of Is the sample size, composition (kinds of participants), and selection/allocation of participants), and selection/allocation of participants to groups appropriateparticipants to groups appropriate for the type for the type of study and for how the findings might be of study and for how the findings might be used? used? Poor.Poor. The researcher claims that a teaching method The researcher claims that a teaching method is effective. However, the method was used in is effective. However, the method was used in only only oneone class, or in one school. class, or in one school. There is no There is no way to tell if it would be effective way to tell if it would be effective anywhere anywhere elseelse..Poor.Poor. The researcher claims that a method is The researcher claims that a method is notnot effective. However, it was used in only effective. However, it was used in only one one class, or in one school. There is no way to tell if class, or in one school. There is no way to tell if it would have been effective it would have been effective elsewhereelsewhere..

[More][More]

Poor.Poor. The researcher claims that an The researcher claims that an instructional procedure or curriculum is instructional procedure or curriculum is more effective than another. However, the more effective than another. However, the comparison groupscomparison groups (one class used one (one class used one curriculum; the other class used a curriculum; the other class used a different curriculum) were not created by different curriculum) were not created by random assignmentrandom assignment of children to the of children to the groups, or by groups, or by matchingmatching the groups to the groups to make sure they had, for example, the same make sure they had, for example, the same percentage of boys and girls. percentage of boys and girls.

Therefore, Therefore, the groups may have been the groups may have been different in different in other waysother ways besides the besides the curriculumcurriculum; e.g., more children in one ; e.g., more children in one group got reading instruction at home, group got reading instruction at home, and this (not the curriculum) caused and this (not the curriculum) caused differences in achievement. differences in achievement. [More][More]

So, if research involves comparing (for So, if research involves comparing (for example) example)

achievement from one group/sample (class, achievement from one group/sample (class,

school) to another, but the researchers did school) to another, but the researchers did NOT NOT

make sure (through make sure (through random assignmentrandom assignment of of

students to the samples, or through students to the samples, or through matchingmatching) ) that that

the groups have the the groups have the same characteristicssame characteristics, then , then you you

can’t trust that differences in achievement are can’t trust that differences in achievement are

caused by the curriculum or teaching method.caused by the curriculum or teaching method.

There are several different There are several different experimental designs you can experimental designs you can use.use.

Classic Design: Pre-test, Post-Test, Classic Design: Pre-test, Post-Test, Control Group, with Random Control Group, with Random AllocationAllocation

If the experimental group and control If the experimental group and control group are equivalent in most ways, then group are equivalent in most ways, then differences in outcomes (the amount of differences in outcomes (the amount of change in each group between pre-test and change in each group between pre-test and post-test) is, logically, an effect of the post-test) is, logically, an effect of the intervention.intervention.

EE

C C Pre-test Post-testPre-test Post-test

Time SeriesTime Series

Sometimes, it is not possible to have comparison groups. Sometimes, it is not possible to have comparison groups. But you may have several years of data on the SAME But you may have several years of data on the SAME sample (school, district) before and then after an sample (school, district) before and then after an intervention. If there is a significant change in the intervention. If there is a significant change in the outcome (dependent) variables, such as student outcome (dependent) variables, such as student achievement, it is likely to be the effects of the achievement, it is likely to be the effects of the intervention.intervention.

99 00 01 02 03 04 05 06 07 08 Year99 00 01 02 03 04 05 06 07 08 Year Eclectic Programs Systematic, Explicit Eclectic Programs Systematic, Explicit

Programs Programs Percentage PassingPercentage Passing State Proficiency TestsState Proficiency Tests

Equivalent Time Samples Design. Same Group, Equivalent Time Samples Design. Same Group, Reversal of Treatment and BaselineReversal of Treatment and Baseline

A useful design within a class, for A useful design within a class, for example, involves collecting pre-example, involves collecting pre-intervention data, then instituting the intervention data, then instituting the intervention, then removing it for a intervention, then removing it for a short time, and then re-instituting it. If short time, and then re-instituting it. If student behavior changes along with student behavior changes along with changes in the intervention, it provides changes in the intervention, it provides evidence that the intervention has an evidence that the intervention has an effect. After all, the students are the effect. After all, the students are the same.same.

More More

Equivalent Time Samples. Same Group, Equivalent Time Samples. Same Group, Reversal of Treatment and BaselineReversal of Treatment and Baseline

A = Teacher provides NO praise when students are on A = Teacher provides NO praise when students are on task and participating.task and participating.

B = Teacher provides immediate and enthusiastic B = Teacher provides immediate and enthusiastic praise when students are on task and participating.praise when students are on task and participating.

The outcome (dependent variable) is the number of The outcome (dependent variable) is the number of times during a 30 minute lesson that students times during a 30 minute lesson that students volunteer answers to questions.volunteer answers to questions.

A1 B1 A2 A1 B1 A2 B2 B2