EXPERIMENTAL RESEARCH METHODS · EXPERIMENTAL RESEARCH METHODS ... The
Experimental Research
Transcript of Experimental Research
1Second Language Research Methods by Seliger and Shohamy
What is experimental research?What is experimental research?
An attempt by the researcher to maintain control
over all factors that may affect the result of an
experiment.
What is the role of the researcher?What is the role of the researcher?In doing this, the researcher attempts to
determine or predict what may occur.
2
Experimental Research and Research Experimental Research and Research ParametersParameters
3
1- Synthetic/ Analytic/ Holistic Constituent
2- Heuristic A Priori
3- Level of control and manipulation: Low High
4- Level of explicitness of data collection: Low High EXPERIMENTAL RESEARCH DESIGN
According to the research parameters,
experimental research is analytic and deductive.
The major components of experimental The major components of experimental researchresearch
All experimental approaches involve the control
or manipulation of three basic components:
the populationthe treatmentthe measurement of the treatment
4
The Type and Number of GroupsThe Type and Number of Groups
Experimental research is concerned with studying
the effects of specified and controlled treatments
given to subjects usually formed into groups.
Groups can be formed according to the purpose of
the experiment or they can be natural as they
appear prior to the research. (e.g. Spanish
speakers, males, students in 3-A)
5
It is up to researcher to use pre-existing groups
or to set up a new group for the experiment.
To use already existing group or the one which is
formed for the experiment is one of the
distinguishing points between experimental
designs and quasi-experimental designs.
6
The TreatmentThe Treatment
This refers to anything done to groups in order to
measure its effect.
Treatments are the independent variable in research.
The treatment is a controlled and intentioned
experience. (e.g. a language teaching method
constructed for the experiment)
7
Measurement or ObservationMeasurement or ObservationIt is the evaluation or observation of the effects of the
treatment.
In second language research, the effects of the
treatment(s) can be evaluated by
a language test
a judgment or oral test
a communicative task and etc.
8
Campbell and Stanley’s conventions for the Campbell and Stanley’s conventions for the components of the experimental researchcomponents of the experimental research
XX stands for an experimental treatment.an experimental treatment.
e.g. a teaching method
materials used in a language lab
OO stands for observation or measurement of the observation or measurement of the
effectseffects of the treatment.
RR stands for randomizationrandomization, or the random the random
assignment of subjects to groupsassignment of subjects to groups in order to control
for extraneous variables.9
Experimental Research DesignsExperimental Research Designs One-Shot Design
Single Group Design One Group pre-test+ post-test
Time-Sampling Designs
Static Group or pre-experimental design
Control Group Design
Pre-Post test with randomization of groups
Factorial Design
Quasi-Experimental Design
Separate Sample Design10
11
Single Group DesignsSingle Group Designs
1.1. One-Shot Design: X OOne-Shot Design: X O (Pilot Study Design)
A single treatment is given to a single group or
individual, then they are observed or measured.
It is more appropriate to pilot studies in which
the researcher wishes to try out instruments or
treatments before running a full-scale
experiment.
12
Example of One-Shot Design:
A second language teacher uses a new method
for teaching the communicative use of the target
language. After three weeks of using this method,
a test is administreted to the class. The class
performs well on the test.
X : the new method for communicative use of TL
O : the test applied to the class13
Single Group DesignsSingle Group Designs
Advantages and Disadvantages of One-Shot Advantages and Disadvantages of One-Shot DesignsDesigns
The disadvantage is that there is no way of
knowing the characteristics of the group or the
individual before the treatment.
The advantage is that it correctly identifies what
to avoid in experimental research. Another one is
that it is used for first testing of instruments of
experimental procedures. 14
Single Group DesignsSingle Group Designs
2- One Group pre-test + post-test: O1 X O22- One Group pre-test + post-test: O1 X O2
Subjects are observed or measured twice on the
dependent variable. This design eliminates the
need for a control group design as it uses the
subjects as their own controls.
15
Single Group DesignsSingle Group Designs
Example of One Group pre-test + post-test Design
In a study on the effects of second language
instruction, the researcher wishes to establish that
the group has no knowledge of a grammatical
structure which will be taught in the lesson. In order
to establish this, two tests are constructed which
specifically test for the target structure in a discrete
point test. One test will be used as the pre-test and
the other will become the post-test.
16
Single Group DesignsSingle Group Designs
17
Single Group DesignsSingle Group Designs Advantages and Disadvantages of Advantages and Disadvantages of
One Group Pre-test & Post-test DesignsOne Group Pre-test & Post-test Designs
The disadvantage is that there is no certainty that
the possible differences that appear in O2 are the
result of treatment. The result may be affected by
attrition or maturation.
The other disadvantage is that the pre-test may
sensitize the subjects to specific aspects of the
treatment and thus confound what is measured by
the post-test.
18
Advantages and Disadvantages of One Group Advantages and Disadvantages of One Group Pre-test &Post-test DesignsPre-test &Post-test Designs
This design is useful as it controls a number of
extraneous variables which can affect the
homogeneity of subjects when more than one
group is involved.
As the same group is used for both pre and post
tests, there is no need to match to another group.
Single Group DesignsSingle Group Designs
19
Single Group DesignsSingle Group Designs3- Time Sampling Designs: O1, O2, O3, On…X, 3- Time Sampling Designs: O1, O2, O3, On…X,
On+1 On+1
In this design, a number of samples or
observations are taken over a period of time.
That’s why, in other words, this design refers to
“time-series”.
The treatment is controlled, added after a
number of observations or measurements.
This design can be used to avoid the problems of
One Group pre-test+post test.
Example of Time-Sampling Design
A study wishes to investigate the effectiveness of teaching students the use of relative clauses in English. For the purpose of the research, a treatment is conducted consisted of a lesson which gives the formal features of relative clause formation in English along with practice in using relative clauses in speech and writing. The treatment is preceded by several in-class writing assignments spaced over several weeks. For each writing assignment the number of relative clauses in learners’ writing is tallied and categorized. After the treatment, several similar writing assignments are given and the number of relative clauses are counted and categorized.
20
Advantages and Disadvantages of Time-Sampling Advantages and Disadvantages of Time-Sampling DesignDesign
This design let us ascribe any changes in the subjects’
performance to the treatment with greater assurance.
It discounts the effect of history as in this design the
researcher develops a norm for the population.
In this design, there is no control group, therefore, the
results are obtained from one group.
21
Single Group DesignsSingle Group Designs
To sum up, collecting data over an extended
period before and after the treatment allows us to
establish a normal pattern of performance for the
language or acquisition behavior under
investigation. This allows the researcher to
exclude the possible interaction of incidental
exposure to language material outside the
classroom, or any natural development change
which may take place regardless of instruction.22
Single Group DesignsSingle Group Designs
23
In single group designs, the experimental group acts
as it own control; the comparison is between its
performance with treatment and without treatment.
In designs using control group, there are two or
more groups and the comparison is between them.
The multi-group designs are concerned with
measures to standardize the groups being compared
so that maintain external and internal validity.
24
Control Group DesignsControl Group Designs
1- Static Group or “pre-experimental” Design 1- Static Group or “pre-experimental” Design X O1X O1
O1O1
In this design, a treatment is developed for a
group and its performance is compared with
equivalent group, which does not receive any
treatment.
But, there is no clue if the groups are really equal
to each other before the treatment. 25
Control Group DesignsControl Group Designs
The difference in performance on the dependent
variable may be due to intrinsic group differences
such as:
the first language background
sex
exposure to second language
time of the day which instruction takes place
the level of motivation of groups
the effects of different teachers on groups 26
Control Group DesignsControl Group Designs
There is no need to form groups which is
different from the previous one, to reassign the
teachers again or to rearrange the timetables.
These features can be considered as advantage,
but, if the variables are not controlled, they can
cause invalidity.
27
Control Group DesignsControl Group Designs
When using a static design, to avoid the problems
related to invalidity, the researcher can match subjects
into two groups for various characteristics such as
placement test scores, sex, first language and teacher
rankings to make the groups more comparable.
Matching is a way of increasing the comparability of
the groups when subjects cannot be randomly assigned.
Group A: X O1 Group B: -- O1
28
Control Group DesignsControl Group Designs
A Variation on the Control Group Matching A Variation on the Control Group Matching DesignsDesigns
ESL cloze (o1) acceptability (02)
Film (X)Film (X) ComparisonComparison
NS cloze (01) acceptability (02)
29
Control Group DesignsControl Group Designs
The film is the treatment. ESL refers to experimental subjects while NS refers to control subjects. Following the treatment, the groups take a modified cloze test in which target concepts were left blank. It was followed by an acceptability task in which the same test was presented with the blanks filled in, some with acceptable cognates and others with unacceptable ones. The acceptability performance of ESL learners was compared with that of the native speakers. The cloze test is the pre-test to establish that the ESL learners did have the concepts with which to judge acceptability.Lightbown and Libben (1984)
2- 2- Pre-test/Post-test with Randomization of Pre-test/Post-test with Randomization of GroupsGroups
Group A’ : O1 X O2Group A’ : O1 X O2
Group A R Group A R
Group A’’ : O1 – O2Group A’’ : O1 – O2
Randomization is a procedure with which to
reduce the amount of systematic error that might
result from biases in the assignment of subjects to
groups. 30
Control Group DesignsControl Group Designs
It also provides better control of variables that could affect internal validity.
By randomizing, we are claiming that any effects of extraneous variables occur by chance and that chance is equally distributed between both groups.
An advantage of randomization is that it provides the option of not using pre-test as a method of making groups more comparable. When the subjects are assigned to groups randomly, the effect of subject variables is spread throughout the two groups.
31
Control Group DesignsControl Group Designs
32
Control Group DesignsControl Group DesignsThe larger the number of subjects in each group, the
more likely it is that the subject variables will negate
each other through the randomization process.
Internal Validity and RandomizationInternal Validity and Randomization
Volunteers are solicited for a study on the effects of
computer-assisted instruction on SL achievement.
From the population of volunteers, subjects are
assigned to computer-assisted group first and to the
control group receiving no computer-assisted
instruction second.
Although assigning of subjects to the treatment
or the control group would seem to be random,
the first volunteers may be different from those
who may have been cajoled into participating in
the study. The order of assignment may be a
reflection of motivation at some level and it may
affect the results of the study.
33
Control Group DesignsControl Group Designs
34
Factorial designs are similar to the true experimental
designs and include all of the elements found in those
designs, such as grouping by randomization, pre and
post-testing and treatments.
The difference is that the effects of several
independent variables may be tested at the same time.
35
Factorial DesignsFactorial Designs
In simple experimental designs, the additional
variables are referred to as extraneous variables.
These variables must be controlled if an effect is to be
attributed to a specific independent variable.
However, in factorial designs, more than one variable
may be treated as an independent variable.
36
Factorial DesignsFactorial Designs
Example:
In a study to test for the effect of language
laboratory training on pronunciation, it is also
decided to measure for the effect of such training
on learners of different proficiency levels, as well
as the effect of different kinds of language
laboratory practice.
37
Factorial DesignsFactorial Designs
The effect of practice vs. no practice in language lab. can
be a function of both the proficiency level of the learner
and the type of practice administered in the laboratory.
38
Factorial DesignsFactorial Designs
Proficiency Level Factor: Proficiency Level Factor: Advanced learners will
focus on their pronunciation performance as they are
aware of their problems. Beginners are not aware of
their pronunciation problems as they are dealing with
more basic issues in language learning.
Language Laboratory Factor: Language Laboratory Factor: The type of practice
in the laboratory is another factor. The practice can
be contextualized -practice in communicative
settings- or it can be decontextualized –practice in
little or no communicative settings.
39
Factorial DesignsFactorial Designs
Gb1 O1 Xd O2
R Gb2 O1 Xc O2
Gb3 O1 - O2 No laboratory practice
Ga1 O1 Xd O2
R Ga2 O1 Xc O2 Ga3 O1 - O2 No laboratory
practice
pre-test post-test
Ga: Advanced Proficiency Xd: Decontextualized PracticeGb: Beginning Proficiency Xc: Contextualized Practice
40
Factorial DesignsFactorial Designs
Variations on Factorial Designs in Second Variations on Factorial Designs in Second Language ResearchLanguage Research
A variation of the factorial design can be found in
Flynn (1984) in a study of the ability of Japanese
and Spanish speakers to correctly imitate
sentences with right or left branching adverbial
phrases. Her hypothesis was that the primary
branching direction of the first language would
affect the ability of the learner to imitate such
sentences in English.
41
Factorial DesignsFactorial Designs
In this study, subjects were not randomized as
would be the case in true experimental designs.
Learners were given a pre-test to establish
proficiency level and were then divided into three
levels: beginning, intermediate and advanced. They
were further grouped according to first language.
There was no experimental treatment, but an
experimental task was administered, in which
learners were asked to imitate a set of stimulus
sentences. 42
Factorial DesignsFactorial Designs
In this kind of studies, the treatment took place
before the experiment and consisted of the
subjects’ exposure to the second language. The
purpose of the pre-test in these studies is to
establish levels of ability before the task is
administered. In addition, there is no
randomization in assigning subjects to groups,
because the experimental groups consist of
natural first language groups.
43
Factorial DesignsFactorial Designs
44
Quasi-experimental designs (Campbell and Stanley 1963) are constructed from situations which already exist in the real world, and are probably more representative of the conditions found in educational contexts.
Quasi-experimental research is likely to have external validity as it is conducted under conditions closer to those normally found in educational contexts.
As this designs are less disruptive than others, it is easier to gain access to subject populations and thus easier to conduct such research.
45
Quasi-experimental DesignsQuasi-experimental Designs
For these reasons, quasi-experimental designs are
ideal for teacher-conducted research and for pilot
studies, in which the exploration of a research
idea is the primary goal. But, the greatest problem
is controlling for sources of internal validity.
46
Quasi-experimental DesignsQuasi-experimental Designs
47
Separate Sample pre-test/post-test DesignSeparate Sample pre-test/post-test Design
This design is useful for those situations in which the researcher has access to only one group of subjects at a time.
Example:
As in the previous example, the aim of the
study is to measure the effects of language
laboratory training on the pronunciation of
advanced level class entering every three weeks.
48
Separate Sample DesignsSeparate Sample Designs
In this design, the same experiment is replicated
each time with a different class, the population
characteristics are assumed to be the same.
Group 1: (Week 1) O1 X O2
Group 2: (Week 3) O3 X O4
Group 3: (Week 6) O5 X O6
49
Separate Sample DesignsSeparate Sample Designs
The Advantages of Using This DesignThe Advantages of Using This Design
It allows for a larger population to be treated,
and overcomes the problem of lack of access to
large enough groups of subjects at any one time.
It allows us to control for changes due to history.
The other way to control for history using this
design, is to conduct inter-group comparisons
between post and pre tests.
50
Separate Sample DesignsSeparate Sample Designs