EMiLExperimental Methods in Linguistics

Halszka Bąk

& Rafał Jończyk

Faculty of English, Adam Mickiewicz University

yellow black

worp vs warp

Research origins

Linguistics: magpie research methods.

Psycholinguistics: psychology.

Neurolinguistics: psychology and neuroscience.

Experimental tasks

Lexical Decision task

CatLafilAim: ortography and semantic processing

Presentation duration: 20ms, 150ms, 200ms, 250ms?

Stimuli placement: centre of the screen? Periphery?

depends on your goals!

?

Semantic Decision task

DoctorHospital?DoctorRose?

The coffee was too hot to… DRINK

The coffee was too hot to…EAT

AIM: Processing of meaning/semantics

Word/Picture naming task

Stroop task

RedBlueGreenAim: investigating interference in the RTs of a task; conflict between word meaning and its colour

Emotional Stroop task: captures attention and RTs to emotional vs non-emotional stimuli

Name the colour:death vs idea

Discrimination task

Which face is more happy?

Failla et al. 2003

Dichotic listening task

Aim:

1. selective attention;

2. hemispheric lateralization of sounds and perception

Free-word recall

Word recognition

Dog leaf

yellow leaflet

punk green

cat skin

dead ideal

PNJA death

nurse hospital

workshop doctor

house stick

dog leaf

yellow leaflet

punk green

cat skin

dead ideal

PNJA death

nurse hospital

workshop doctor

house stick

Task 1

tasksheet, page 2 & 3

Types of data &

methods of interpretation

Quantitative data

Reaction/Response Times (RTs):

measured in miliseconds (ms)

influenced by: priming, emotional states, attention, etc.

Quantitative data

Accuracy rates (ACC) or Error rates (ER):

measured in 1s (correct) and 0s (incorrect)

influenced by: bilingualism, fallacies, cognitivedeficits, etc.

Qualitative data

Facial expression

Prosody (speech contour)

Gestures

Body language

Etc.

Physiological data

Galvanic Skin Response (GSR; also Skin Conductance Response (SCRs))Measures skin conductivity in microSiemens (mS)

influenced by: stimulus modality, intensity, etc.

Physiological data

Heart rate (HR)

measured in beats per minute (BPM)

influenced by: stimuli intensity, participant’s health, etc.

Physiological data

EEG: Event-Related Potentials (ERPs):

measured in microVolts / sec

unrivalled temporal rsolution

Physiological Data

Functional Magnetic Resonance Imaging (fMRI):

blood / water diffusion moleculesmeasured in voxels

great spatial resolution

poor temporal resolution

!!!

Your HYPOTHESES determine what TYPES OF DATA you will COLLECT

Task 2

tasksheet, page 3

Research Questions & Hypotheses

Research question

frames a problem to be addressed & resolved in research

Hypothesis

A prediction about a relationship between

Two (or more) variables

A solid hypothesis is…

...TESTABLE

confirm or falsify the hypothesis

…SUPPORTABLE

provide empirical evidence to confirm/falsify the hypothesis

…GROUNDED

ground your hypothesis in theory/previous research

…RELEVANT

fill a gap in the research, resolve inconsistency, etc.

Special case…

…NULL HYPOTHESIS - Nº

No relationship between the variables

It is always there

fail to reject Nº = cannot draw conclusions from your research

Task 3

tasksheet, page 4

Variables

A variable is…?

…something that changes or varies

e.g. among people (IQ, gender, etc.)

Independent variable

the aspect you manipulate to influence the end results

e.g. instruction, types of stimuli, types of participants

priming, etc.

What is the cause of a given phenomenon?

the cause = independent variable

Dependent variable

the end result (hard data)

e.g. RTs, ACC, ER, ERPs, etc.

e.g. the amount of time you choose to spend doing grammar exercises determines your final score on the PNJA exam in June ;)

summing up

HYPOTHESIS: Positive sentence context makes processing of positive words faster and more accurate.

PREDICTION: faster & more accurate processing of ”+” WORDS in ”+” SENTENCE context.

INDEPENDENT VARIABLE: sentence context, word valence.

DEPENDENT VARIABLE: RTs, ACC/ER

TYPE OF RELATIONSHIP: ”+” SENTENCE causes smaller RTs and greater ACC in responses to ”+”WORDS.

Task 4

tasksheet, page 4

Experimental Design

The simple experiment

1. Define hypothesis

2. Specify and define variables.

3. Participants: experimental vs. control groups.

4. Random assignment of participants to groups

5. Collect dependent variable

6. No significant result – no effect, no interpretation

if you cannot randomly assign participants to your different groups, you cannot do a simple

experiment

2 x 2 design

2 independent variables & 2 levels of each variable

(positive; negative) x (congruent; incongruent)

Gloria accidentally poured boiling water over herself and was BURNT / ROMANTIC*

Their honeymoon in the gorgeous scenery of Paris was very ROMANTIC / BURNT*

Reliability & Validity

Validity

Internal

External

Ecological

Internal validity

A B

Your data reflects the reality under investigation:

independent variables and their manipulation

Threats to internal validity

1. Testing: pre-tests may cause biases

2. Instrumentation: pre- and post-tests must match

3. Regression: biased participant selection

4. Confounding variable:

not properly controlled, manipulated or unnoticed

External validity

Results can be generalized to the wider population

Threats to external validity

1. Participant selection: atypical population sample

2. Setting: the context

3. History: a specific historical context

4. Construct effect: a particular construct exists in one group but not in others

Ecological validity

Experiment conducted in an environment

& in circumstances most similar to or exact to a real-life

situation

Reliability

A reliable experiment can be reliably replicated yielding the same results

Task 5

tasksheet, page 5

Errors & biases

Errors

Murphy’s law: Anything that can go wrong will go wrong

Type 1 error:

You’ve done everything right. Got a significant result. The result is statistically a chance accident.

Type 2 error: You’ve done everything right. DID NOT GET A SIGNIFICANT RESULT. The lack of result is statistically a chance accident.

Demons you can fight:Random errors & biases

Everything done right.

No risk of Type1 or Type 2 error…

Still…the result is WRONG

The result is a chance accident due to a randomly occuring external factor…

This is why you need reliable and replicable experiments!

Solution?

Fight! Increase the number of participants

(random error tends to balance out to zero)!

Biases

2 major offenders, 1 major circumstance

Offender 1

Researcher:

”I’m going to get what I expect & publish it in SCIENCE!”

“It is remarkable how often the first interpretations of new evidence have confirmed the preconceptions of its discoverer."

(John Reader. 2011. Missing Links. Oxford University Press)

Offender 2

Participant A:

”I’ll make you look good/bad!” (obeying demand characteristics)

Participant B:

”I’ll make myself look good!” (social desirability bias)

Non-mitigating Circumstance 1

Day 1:

You’re taking part in an important experiment…

Day 2:

So yeah… this… this thing yeah it’s for my thesis and um…

(Check out Mitchell and Jolley. 2010. Research Design Explained. Seventh edition. Pages 129-143)

Task 6

tasksheet, page 1 & 2

Cause and effect in experimental research

(correlation vs. experimentation)

“Correlation does not imply causation”

Correlation Experiments

Reality observed and measuredmanipulated and measured

Fallaciescum hoc ergo propter hoc(no causality implied in correlated observations)

post hoc ergo propter hoc (when internal validity is poor)

Validity Ecological Internal & external

Task 7

tasksheet, page 6

Original research vs.

Replicating experiments.

Original Replication

Point of origin Theory Existing research

ApproachTheory-based, practically creative

Data-driven, practically repetitive

MethodsSelect from all available and viable

Only choose the one used in the study being replicated

Modifications Compromise validityIncrease validity provided that control group is included!

Stimuli & Participantselection

Participants

Ideal case:

a large randomly selected sample from a homogenous population.

Typical case for MA/BA: a minimally valid, inherently somewhat biased sample

Remedy:

a careful selection of hypothesis, inclusion of relevant control pre- and post-tests.

StimuliA happy case:

stimuli exist, have been validated in a thorough and correct validation procedure, and they are available.

An unhappy reality:

a norming study is an absolute, inevitable MUST.

An unhappier reality still: you must create, validate, and norm your own stimuli.

Remedy:

Meta-analyses of existing research (more later), start early.

The experimenter’s conduct

Beware!

Biases

Errors

Self-presentation

Self-care

Ethical considerations

Human Subject Research(HSR)

Linguistic research is a case of HSR.

Seek permissions to proceed with Internal Review Boards (IRBs) and Ethical Committees.

PhD and senior staff:

strongly advised to take a basic HSR course. E.g. CITI course (http://www.citiprogram.org)

Rule of thumb

Don’t hurt people. Emotionally, psychologically, physically.

Tools & Resources

Meta-analytical tables

ask a single study design question, and search through the top-cited papers in the field to see how others have “done it”.

Example:

Question: How many participants should I use?

Literature: Find top-cited 30-50 papers from the past 10 years on the subject.

Table: Bibliography note, participants number, relevant participant variables.

What to do: abstract > introduction (skim) > methods > participants

Result: Overview of the typical number and type of participant in your discipline.

Existing resources

Affective Sciences databases: SEMAINE, BINED, ILHAIRE, SAVEE, ANEW, IADS, IAPS, etc.

Online corpora: COCA, BNL.

Online services: instant.ly, Monkey Surveys, Google Docs, etc.

Existing experiments for replication: e.g. Englelab (http://www.englelab.gatech.edu).

Linguistlist: http://linguistlist.org/

Courses

Coursera: Free of charge.

MIT courses: Free of charge (Coursera, Edx)

CITI courses: Standard Fee 100$. Requirements: Affiliation with a US Institution (https://www.citiprogram.org/)

Bibliography

Field, Andy. 2009. Discovering statistics using SPSS (and sex and drugs and rock’n’roll). Third edition. London: SAGE.

Mitchell, Mark L. and Janina M. Jolley. 2010. Research Design Explained. Seventh edition. Belmont: Wadsworth CENGAGE Learning.

Further recommended reading

Feynman, Richard. 1974. Cargo Cult Science. Caltech commencement address. (http://neurotheory.columbia.edu/~ken/cargo_cult.html) (Date of Access: 02.04.2014)

Feynman, Richard. 1985. Surely you’re joking, Mr. Feynman! Adventures of a curious character. New York: W. W. Norton.

Feynman, Richard. 1988. What do you care what other people think? Further adventures of a curious character. New York: W. W. Norton.

Sagan, Carl. 1995.The Demon-Haunted World: Science as a Candle in the Dark. New York: Random House.