Research design: The backbone of academic inquiry

Peter Neff – Doshisha UniversityMatthew Apple – Nara National College of TechnologyDavid Beglar – Temple University Japan

CUE Forum 2008

Olympic Memorial Youth CenterNovember 1, 2008, 1:15 - 2:50 p.m.

Introduction: The importance of good research design

Approaching the studyDeveloping the study design

Break for Q&A

Designing the right instrumentImplementing your designConclusion

Further Q&A

Overview

Introduction: The importance of good research design

Poor design vs. Good design

Poorly-designed studyHit on an idea, dive right inNo background researchThrow together a survey, give to a

group of unwary participantsCollect data, then ponder how to

analyzeRun to a colleague for helpFish around for most “interesting”

findingsPray to get published

Poor design vs. Good design

Well-designed studyHit on an idea, do background researchFormulate relevant, specific, practical RQsConsider participants, context, data analysis

in advanceDecide/develop instrument; pilot and revise

itDecide on appropriate pre/post-test

instrumentsPlan stages and structure of data collectionPrepare participants adequately…Then carry out the study

The importance of good designA well-designed study provides

many benefits:– Demonstrates researcher

knowledge– Ties the study to an underlying

philosophy– Provides a clear path for the

researcher(s)– Helps avoid mishaps of previous

studies

The importance of good designOther benefits

– Leads to more concrete results, more definitive conclusions

– Improves chances of publication– Raises the status of SLA as a

field of inquiry

A word about mixed methods designsThe great quan-qual debateMixed methods – the “best” of

both worldsAdd a qualitative component to a

quantitatively-oriented study: – Participant interviews– Observational, audiovisual data– Open-ended survey questions

Plan for qualitative analyses (text analysis, response coding)

Part 1Approaching the Study

Approaching the studyHitting upon research ideasReview of the literatureFormulating research questions

Hitting upon research ideasIdentify the topic in a few wordsReflect on “doability” of research

– Can I research this?– Should I research this?– Am I interested in researching this?

Review of the literature can help redefine and revise ideas

Identifying the topic: Hints for starting to narrowPose a short question using “what”

or “how” Write a short title that consists of

one sentence under 12 wordsAsk a friend or colleague to read

your topic and gauge their reactionsDraft research questions to see if

the topic can be adequately explored

A “researchable” topic“Can I do this in my current

situation?”“Does this concern people at other

institutions?”“Does this add to the current body of

research related to this topic?”“Does this study contribute something

from a unique perspective?”

Filtering “probably not so good” ideas: To boldly go where no research has

gone before… (The “Star Trek” idea)My theory is clearly better than X

(The “Steven Krashen is so wrong” idea)My classroom is totally unique

(The “I don’t need theory” idea)This is a really cool technology /

methodology / text book (The “I am primarily a teacher” idea)

Filtering ideas: A few hintsReview research designs and

statistical techniquesReview teaching methods and

overall SLA research resultsEvaluate access to potential study

participantsPlan time for material creation,

study design, and implementation

Review of the literatureRelate the study to continuing

“dialogue” in current researchFinding a “gap” in the literature Provide a framework for the

importance of the study

Review of the Literature:Finding a “gap” in knowledge“We do not enough about X…”“This way of looking at X has never

been done…”“This way of learning about X has

not been duplicated in my context”“Previous research has

inadequately explored X…”

Finding literature: Some hints Google Scholar using key words or

researcher names Scour recent literature review articles Check for “cited” numbers online Get access to university databases Refer to recently published articles

– After the year 2000– During the previous 2 to 3 years

Examine “outside the field” articles

Finding literature: Separating the wheat…“Top tier” journal articlesMost-often-cited articlesRecent articlesResearch articles (not reviews)Books / Edited book-articlesMajor international conference

papersDissertations / dissertation abstracts

…from the chaff“In-house” journal articlesArticles from “proceedings” booksOnline journal articles with only .html

versionsUnedited books from small publishersNewspaper and magazine articlesWeb pagesAnecdotal evidenceYour own previous papers for an MA

course

Research questions: A few useful guidelinesNaturally flow from the literature reviewStrongly connected to the topicAt least two or three (not one)……but not five or six or moreAs specific as possibleDirectly concern variables in the studyDo not contain yes/no question words

RQs: What not to ask“Is X true/false?”“Will X happen if…?”“Does X cause Y?”

“What do participants think of X?”“Why does X happen?”

RQs: What to ask“What differences exist between…”“Compared to X, how does Y…?”“To what degree do X and Y differ…?”“When X is controlled for Y…, how

does Z…?”

“What are underlying patterns among…?”

“To what degree does X predict Y?”

Part 2Developing the Study Design

Research design

Cross-sectional design: A design in which data are collected from a sample at only one point in time.

Longitudinal design: A design in which data are collected at more than one point in time.

Randomized Control-Group Pretest-Posttest Design

Experimental Group 1 T1 Xa (Method a) T2

Experimental Group 2 T1 Xb (Method b) T2

Control Group T1 T2

Randomized Control-Group Pretest-Posttest Design

Reasonably strong conclusions can be reached about the effects of the treatments.

Problem 1: Within session variation (e.g., different teachers or room conditions) may intervene.

The solution? Randomly assigning participants, times, and places to the experimental and control conditions.

Problem 2: The pretest may interact with the treatment. This potential problem is dealt with in the next design.

Randomized Solomon Six-Group DesignPretested (Random assignment) T1 Xa (Method a)

T2

Pretested (Random assignment) T1 Xb (Method b)T2

Pretested (Random assignment) T1T2

Unpretested (Random assignment) Xa (Method a)T2

Unpretested (Random assignment) Xb (Method b)T2

Unpretested (Random assignment)T2

Randomized Solomon Six-Group Design

This design amounts to doing the experiment twice –once with and once without pretesting.

It is possible to know what effects, if any, are associated with pretesting.

If the results of the “two experiments” are consistent, greater confidence can be placed in the findings.

Counterbalanced Design

This design is useful when randomization is not possible and intact groups must be used.

Replication

Xa Xb Xc Xd

1 A B C D2 B D A A3 C A D B4 D C B C

Counterbalanced design The counterbalanced design rotates out

the participants’ differences (e.g., one group has more aptitude or motivation than the other groups) by exposing each group to all variations of the treatment.

Order-of-presentation effects are controlled.

Primary weakness: The possibility of carryover effects from one treatment to the next exists. Allowing time between treatments can alleviate this problem.

Control-Group Time-Series Design

Experimental Group 1 T1 T2 T3 T4 Xa (Method a) T5 T6 T7

T8

Experimental Group 2 T1 T2 T3 T4 Xb (Method b) T5 T6 T7

T8

Control Group T1 T2 T3 T4 T5 T6 T7 T8

Control-Group Time-Series Design

This design allows the researcher to determine growth over time, and the effect of an intervention.

The presence of a control group increases the trustworthiness of the results because the possibility of a contemporary event causing any gains can be determined.

Control-Group Time-Series Design

This design can be extended by exposing the participants to the intervention on multiple occasions.

This approach is more sensitive to partial gains in knowledge and tests the strength of the intervention more than once, thus giving the researcher a more accurate understanding of the effectiveness of the intervention.

Experimental Group 1 T1 T2 Xa T3 T4 Xa T5 T6

Experimental Group 2 T1 T2 Xb T3 T4 Xb T5 T6

Control Group T1 T2 T3 T4 T5 T6

Q&A Break

Part 3Designing the Right Instrument

Instrument Design Commonly used instruments in SLA

research– Scored tests– Rater scores– Surveys– Interviews

Consider your eventual data analysis when developing instruments

Instruments - Scored testsPluses

Quantitative items (M/C, Cloze/C-tests)– Simple to score large

# of participants– Easier to analyze

Qualitative items (short answer, timed essays)– Good complement to

quantitative scores– Can provide more in-

depth assessment of participants’ abilities

Minuses Quantitative items

– Limited to one type of data

Qualitative items– Take more

time/effort to score– Rater bias

Instruments – Performance ratingsAn assessment of participants’

performance in an assigned taskTasks may include presentations,

interviews, written essaysPerformances can be scored using a

Likert-scale, a rubric, or holisticallyUsually scored by at least two

“expert” raters; sometimes also by peers

Performance ratings

Rating criteria should be concretely established with little ambiguity

Avoid including too many (or too few) criteria for one performance task

All raters should undergo a “normative” training session prior to assessment

Use models to train ratersAvoid single-score holistic ratings

Instruments - SurveysOften used for:

– Collecting learner history data (L2 study experience, other background info)

– Assessing participants’ attitudes towards a predetermined construct (language learning motivation, anxiety using the L2)

– Determining reactions to an experimental treatment (teaching methods, innovative learning tasks)

Survey makingFor non-advanced learners – surveys

should be in their L1Build in redundancy - Include multiple

questions for each concept areaQuestions should be simply worded –

avoid negative or confusing wordingDepending on the purpose of the

survey, 20-40 items/session is a good range to shoot for

Survey makingAny survey used in a serious study

should be piloted in advanceIt is acceptable to make

adjustments to an existing instrument

Likert-scale items should usually have between 4 and 6 choices

A few qualitative questions can provide a nice complement to quantitative instruments

Instruments - InterviewsInterviews can provide an

excellent qualitative component to a larger study

It is not necessary to interview all participants – a subsample as small as 10-20% can

be acceptableUse your best judgment on

participants’ language ability– For intermediate-and-above learners,

L2 interviews are often fine

Conducting interviewsInform students they are being

interviewed, obtain consentRecord unobtrusively “Warm up” the participants before

getting into the heart of the interview

Collect more data than you need

Validating Instruments

Instrument ValidityThe construct = The heart of the

matterWhat construct do you wish to

measure? How do you define the construct?What are its component parts? Do

they form a unified whole?

Operationalizing the construct: The items Conceptualize the construct as a continuum: easy—difficult items and less able—more able persons. How have other researchers measured the construct? Write original or adapted items. Cover the estimated range of your participants. Write 50% more items than you intend to use. This will allow you to “cherry pick” the best items as well as items at various levels of difficulty.

Operationalizing the Construct: The ItemsMore able ｜ More difficultpersons ｜ items ｜ x ｜ item 1 xx ｜ item 2 item 3 xxx ｜ item 4 item 5 xxxx ｜ item 6 item 7 item 8 xxxx ｜ item 9 item 10 item 11 xxx ｜ item 12 item 13 xx ｜ item 14 x ｜ item 15 ｜Less able ｜ Less difficultPersons ｜ items

Operationalizing the Construct: The ItemsAfter piloting the items,

statistically analyze the results.Examine dimensionality, item

difficulty, and item content.Select the best items to make an

efficient, highly reliable instrument.

Part 4Implementing Your Design

Implementing the designIncluding other researchersPractical issuesHandling ethical concerns

Including other researchers in the studyThe nature of the researchers

involved– Main researcher plus “helpers”– One researcher plus “other participants”

The nature of the instructors involved– Teaching methods– Students taught– Course goals– University program goals

Working with other researchersWork with people you know and trustEstablish a schedule early Define clear roles for each researcherDecide definite research goals prior

to data collection Keep in regular contactThe “band practice time” principle

Example research rolesHead researcher / contact personData entry specialistStatisticianInterviewerLiterature analystEditor / proofreader

Heading off potential problems Explain study commitments prior to

starting the study Agree on “ownership” prior to data

collection and data entry– Who will keep the data?– Whose name comes first, second, etc.?

Keep everyone aware of deadlines Include everyone in decision-making

processed and data analysis

Things to avoid in group researchPeople you don’t knowResearch groups larger than four

or five membersUsing someone for language skillsInvolving others just to get more

participantsForgetting to thank others for

their assistance

Improving relations with research helpersWrite clear instructionsThank profusely for their time and

effortOffer to send copies of final

research papers and/or resultsOffer to assist in future researchKeep in touch after research ends

Practical IssuesTiming of implementationLearning and research contextParticipant consentFinancial considerations

Timing of the implementationBeginning, middle, or end of semesterDay of the weekTime of dayExams and exam preparation periods“Culture Festivals” or other club-

related events“Open classes” or “parents’ day”

Learning and research contextDiffering course goals (I.e.,

listening class vs. reading class)Different major field of studyGender, age, year in schoolNumber of class meetingsPerception of the value of

research by institution heads

Participant consentAlways allow for “non-participation”

choice from potential participantsWrite clear instructions for

participants asking for their cooperation

Ask co-researchers or helpers to briefly inform participants about their choice

Financial considerationsCopies for questionnaires, exams,

etc.Computer analysis softwareMailing costsInterview travel costsRecording equipment

Reference booksJournal article costs

Heading off lack of cooperation problemsReview requirements of the study

– How many items in the questionnaire? – How many treatments?

Recognize that students are busy, tired, etc.

Plan to get between 30-50% more data than you need

Try to get more data at a later date

Ethical Considerations

Ethical considerations

Students should not be exploited just because they are there

In theory, they should derive benefit (directly or indirectly) from the research

Explain the basic purpose of your research before collecting data– 1-2 sentences should be fine

It is also good to briefly explain this at the beginning of any survey instrument

Ethics - ConsentProvide students with the chance to

opt out of participationVerbal consent is usually acceptable

for surveys, ratings, and test scoresA written consent form may be

necessary for more involved forms of participation (interviews, essay passages)

When in doubt, check recent articles in well-known journals for guidance

Ethics – Other considerationsThe role of this study within your

institutionPotential gender, proficiency or

other issues that may affect your data or conclusions

Have a plan for anonymizing the data (and consider making this conspicuous on your instruments)

Conclusion

In conclusion…There are many factors to consider when

embarking on a serious studySome points to take away…Most studies should fill a place (a “gap”)

within the current academic dialogue Research questions should reflect

continuity with the literature and should be specific

Carefully consider the design setup which will work best with the participant groups and the research and analytical goals

In conclusionFurther points…Different instruments work better in

different circumstances. Choose those which best reflect your aims.

Plan your analyses at the same time you are developing your instruments

Develop and pilot instruments which can cover responses from a range of participants

In conclusionEven more points…Work with others who will be serious

and committed, and then be an organized and conscientious leader

Consider how practical, pedagogical, and institutional concerns may affect your study

Do not forget current ethical guidelines for carrying out participant research

Good luck with your research!

Thank you for listening

Q&A

For a copy of this presentation:

http://jaltcue-sig.org