Research Methods ReviewResearch Methods Review
Advanced Cognitive Advanced Cognitive PsychologyPsychology
PSY 421 - Fall, 2004PSY 421 - Fall, 2004
OverviewOverview The BasicsThe Basics Experimental MethodsExperimental Methods
DesignsDesigns Within-subjectsWithin-subjects Between-subjectsBetween-subjects FactorialFactorial
Statistical ReviewStatistical Review Example of ExperimentExample of Experiment Don’t let me forget a simple Don’t let me forget a simple
assignment to give you before you assignment to give you before you leaveleave
What are Research What are Research Methods?Methods?
Application of the scientific method Application of the scientific method to studying behaviorto studying behavior
Scientific Method = observe, Scientific Method = observe, hypothesize/predict, test, concludehypothesize/predict, test, conclude
Work with structured observationsWork with structured observations Deal with solvable problemsDeal with solvable problems Produce publicly verifiable Produce publicly verifiable
informationinformation Hypotheses, theories, and Hypotheses, theories, and
explanations MUST BE FALSIFIABLEexplanations MUST BE FALSIFIABLE
General Methods – The General Methods – The BasicsBasics Watch people in public settings and drawing Watch people in public settings and drawing
conclusions = conclusions = Observational/DescriptiveObservational/Descriptive Conducting a survey, questionnaire, Conducting a survey, questionnaire,
interview, poll, etc. and comparing interview, poll, etc. and comparing responses on one question to other responses on one question to other questions = questions = Relational/CorrelationalRelational/Correlational
Randomly forming 2 or more groups, treat Randomly forming 2 or more groups, treat them all differently, see how the outcomes them all differently, see how the outcomes differ = differ = ExperimentalExperimental
Compare two or more groups (that are not Compare two or more groups (that are not randomly formed and are different to begin randomly formed and are different to begin with) in a variety of ways = with) in a variety of ways = Quasi-Quasi-ExperimentalExperimental
Experimental MethodExperimental Method This type of method attempts to This type of method attempts to
answer questions about CAUSEanswer questions about CAUSE VariablesVariables
Independent = manipulated or changed Independent = manipulated or changed for various subjects; IVfor various subjects; IV
Dependent = the way to measure a Dependent = the way to measure a change in behavior (or not change); DVchange in behavior (or not change); DV
Control = a potential IV that is held Control = a potential IV that is held constant at one level – all subjects are constant at one level – all subjects are exposed to that one levelexposed to that one level
Manipulation = changeManipulation = change
Manipulating Variables - Manipulating Variables - DesignsDesigns
One variable – multiple levelsOne variable – multiple levels Univariate or one-way designUnivariate or one-way design Level = one aspect of the variable; Level = one aspect of the variable;
conditioncondition ExampleExample
Manipulation TypesManipulation Types Between Subjects = ONEBetween Subjects = ONE
To expose some subjects to one level of the To expose some subjects to one level of the IV and other subjects to another level of the IV and other subjects to another level of the IV – subjects are not exposed to all aspects of IV – subjects are not exposed to all aspects of the variablethe variable
Within Subjects = ALLWithin Subjects = ALL To expose all subjects to all the levels of the To expose all subjects to all the levels of the
IVIV
The BOXThe BOX Boxes with rows and columns – Boxes with rows and columns –
essential for understanding essential for understanding experiment design and statisticsexperiment design and statistics
Level 1
Level 2Level 1
Level 2
Variable A
Variable B
Row 1
Row 2
Column 2Column 1
Factorial DesignsFactorial Designs When more than one variable is When more than one variable is
manipulated in an experimentmanipulated in an experiment 2 Variables = Two way2 Variables = Two way 3 or more variables = multivariate3 or more variables = multivariate Between subjects designBetween subjects design = all variables = all variables
are manipulated between/across the are manipulated between/across the levelslevels
Within subjects designWithin subjects design = all subjects = all subjects receive all the levels of all the variablesreceive all the levels of all the variables
Mixed designMixed design = at least one variable is = at least one variable is manipulated between subjects and at manipulated between subjects and at least one variable is manipulated within least one variable is manipulated within subjectssubjects
Statistical ReviewStatistical Review
Populations and SamplesPopulations and Samples Hypothesis TestingHypothesis Testing Using methods and statistical tests Using methods and statistical tests
togethertogether
Population vs. SamplePopulation vs. Sample Population = everyone that you are interested in Population = everyone that you are interested in
studying or at the very least, generalizing your studying or at the very least, generalizing your results toresults to Ex: Women; children; Men over 40Ex: Women; children; Men over 40 You can’t possibly hope to study the entire populationYou can’t possibly hope to study the entire population
Sample = a subset of the population that contains Sample = a subset of the population that contains the important characteristics of the population; a the important characteristics of the population; a sample is representative of its populationsample is representative of its population Various techniques for sampling from a populationVarious techniques for sampling from a population Ex: PSU Women; children at BFC; Male faculty in Ex: PSU Women; children at BFC; Male faculty in
psychology over 40psychology over 40 Why does this matter?Why does this matter?
Allows research to occur without the impossible task of Allows research to occur without the impossible task of studying everyone studying everyone
Important assumptions for statisticsImportant assumptions for statistics
Hypothesis TestingHypothesis Testing Comparing the Null and Experimental Comparing the Null and Experimental
hypotheses to predict the likelihood hypotheses to predict the likelihood of one being show to be trueof one being show to be true
Null Hypothesis = There is no Null Hypothesis = There is no difference; nothing will change; zerodifference; nothing will change; zero
Experimental Hypothesis = There will Experimental Hypothesis = There will be a difference; something will be a difference; something will change; non-zerochange; non-zero
Hypothesis TestingHypothesis Testing
Null Null HypothesisHypothesis
There is no There is no difference in difference in GRE scores GRE scores between between males and males and femalesfemales
Experimental Experimental HypothesisHypothesis
There is a There is a difference in difference in GRE scores GRE scores between between males and males and femalesfemales
In the real world, the Null is
Your decision
True False
Reject H0
Type I error
Correct Decision
Retain H0
Correct Decision
Type II error
Hypothesis Testing ExampleHypothesis Testing ExampleBarbie and Kendall – Chocolate Eaters
(to be read in class) From this, what could we conclude about this contradiction?
1. According to our assumption, in the real world, H0 is true. Therefore, if Barbie rejected H0 because she thought it was wrong (based on her study's results), what has happened? Did she commit an error or make a correct decision?
2. If Barbie would have retained H0 (and to do that, her study would have resulted in no differences between the mean exam scores from the two groups), and we assume that in the real world, H0 is true, did she commit an error or make a correct decision?
3. If we now assume that Kendall is wrong and in the real world, H0 is false, and Barbie rejected H0because she thought it was wrong (based on her study's results), what has happened? Did she commit an error or make a correct decision?
4. Again, assume that Kendall is wrong, and H0 is false in the real world. If Barbie would have retained H0 (and to do that, her study would have resulted in no differences between the mean exam scores from the two groups), and we assume that in the real world, H0 is true, did she commit an error or make a correct decision?
Hypothesis TestingHypothesis TestingAlpha Beta Effect Size Statistical Power
Definition The probability of committing a Type I error
The probability of committing a Type II error
The size of the effect (difference/
relationship)
The probability of rejecting a false null hypothesis (or the probability of finding an effect if one exists)
When to use Set prior to collecting data
Based on how much power you want (determined before collecting data) or how much power you actually have (determined after collecting data)
To be determined every time you run an inferential statistic (correlation, t-test, ANOVA, chi-square)
Prior to collecting data: estimate power and effect size to determine how many subjects you need to achieve certain level of power. After analyzing data: determine effect size and combine that with sample size to determine power
Interpre-tation
Percent chance of actually committing a Type I error (if the null is true)
Percent chance of actually committing a Type II error (if the null is false)
Small, medium, or large (quantitative values depend on type of effect size test you used)
Percent chance that you will find a statistically significant result given your sample size (N) and effect size (and assuming the null is false)
Example α = .05 means a 5% chance of committing a Type I error
β = .20 means a 20% chance of committing a Type II error
d = .50 means a medium effect size for the effect size corresponding to a t-test
Power = .80 means an 80% chance of finding a statistically significant result given your N, ES, and assumption that the null is false
Scales of MeasurementScales of Measurement Nominal = used to identify a particular Nominal = used to identify a particular
characteristics of the scale; also called categorical characteristics of the scale; also called categorical (categories are mutually exclusive)(categories are mutually exclusive) EX: Sex (M/F); ZIP CodesEX: Sex (M/F); ZIP Codes
Ordinal = numbers indicate whether there is more Ordinal = numbers indicate whether there is more or less of the measured variable; order is or less of the measured variable; order is importantimportant EX: Levels of education (Freshman, Sophomore, Junior, EX: Levels of education (Freshman, Sophomore, Junior,
Senior); Olympic medalsSenior); Olympic medals Interval = numbers correspond exactly to changes Interval = numbers correspond exactly to changes
in the measured variable and there are equal in the measured variable and there are equal distances between numbers that correspond to distances between numbers that correspond to equal changes in the measured variableequal changes in the measured variable EX: IQ; Temperature (Fahrenheit, Celcius)EX: IQ; Temperature (Fahrenheit, Celcius)
Ratio = like an interval scale (equal intervals) but Ratio = like an interval scale (equal intervals) but also includes a true zero point (the absence of the also includes a true zero point (the absence of the measured variable). This allows for multiplication measured variable). This allows for multiplication and division of scale values.and division of scale values. EX: Weight; Height; Temperature (degrees Kelvin)EX: Weight; Height; Temperature (degrees Kelvin)
Descriptive StatisticsDescriptive StatisticsDecide how to summarize and represent data based on Decide how to summarize and represent data based on
the the TYPETYPE of data that you have (its scale of of data that you have (its scale of measurement)measurement)
Measures of Central TendencyMeasures of Central Tendency
1.1. Nominal Scale – ModeNominal Scale – Mode
2.2. Ordinal Scale – MedianOrdinal Scale – Median
3.3. Interval/Ratio Scales - MeanInterval/Ratio Scales - Mean Measures of DispersionMeasures of Dispersion
1.1. Nominal Scale – RangeNominal Scale – Range
2.2. Ordinal Scale – Absolute Deviation from the MedianOrdinal Scale – Absolute Deviation from the Median
3.3. Interval/Ratio Scales – Variance, Standard Error, Interval/Ratio Scales – Variance, Standard Error, Standard DeviationStandard Deviation
Graphical Representations of DataGraphical Representations of Data
1.1. Nominal/Ordinal Scales (Qualitative Data) – Bar Graph, Nominal/Ordinal Scales (Qualitative Data) – Bar Graph, Pie ChartPie Chart
2.2. Interval/Ratio Scales (Quantitative Data) – Line Graph, Interval/Ratio Scales (Quantitative Data) – Line Graph, Frequency Polygon, HistogramFrequency Polygon, Histogram
Inferential StatisticsInferential Statistics
Decide which test to use based on the TYPE of Decide which test to use based on the TYPE of data you have and the KIND of outcome you data you have and the KIND of outcome you
are looking forare looking for
Relationships (Correlations)Relationships (Correlations)1.1. Nominal Scale – Chi-Square Test of Nominal Scale – Chi-Square Test of
Independence, or PhiIndependence, or Phi2.2. Ordinal Scale – Kendall’s Tau or Spearman’s Ordinal Scale – Kendall’s Tau or Spearman’s
rr3.3. Interval/Ratio Scales – Pearson’s Interval/Ratio Scales – Pearson’s rr
DifferencesDifferences1.1. One Independent VariableOne Independent Variable
Between-Subjects manipulationBetween-Subjects manipulation 2 levels2 levels
Nonparametric – Chi-Square Goodness of Fit Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal)(nominal) and Mann Whitney U (ordinal)
Parametric – Independent means t-test or one-way Parametric – Independent means t-test or one-way ANOVAANOVA
3+ levels3+ levels Nonparametric – Chi-Square Goodness of Fit Nonparametric – Chi-Square Goodness of Fit
(nominal) and Kruskal Wallace (ordinal)(nominal) and Kruskal Wallace (ordinal) Parametric – One-way ANOVAParametric – One-way ANOVA
Within-Subjects manipulationWithin-Subjects manipulation 2 levels2 levels
Nonparametric – Chi-Square Goodness of Fit Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal)(nominal) and Mann Whitney U (ordinal)
Parametric – Independent means t-test or one-way Parametric – Independent means t-test or one-way ANOVAANOVA
3+ levels3+ levels Nonparametric – no good testsNonparametric – no good tests Parametric – One way Repeated-Measures ANOVAParametric – One way Repeated-Measures ANOVA
Differences, continuedDifferences, continued2.2. Two Independent VariablesTwo Independent Variables
Between-Subjects manipulationBetween-Subjects manipulation Nonparametric – Wilcoxon-Wilcox Nonparametric – Wilcoxon-Wilcox
and Friedman testsand Friedman tests Parametric – Factorial ANOVAParametric – Factorial ANOVA
Within-Subjects manipulationWithin-Subjects manipulation Nonparametric – Friedman testNonparametric – Friedman test Parametric – Factorial Repeated-Parametric – Factorial Repeated-
Measures ANOVAMeasures ANOVA Mixed manipulation – Mixed Factorial Mixed manipulation – Mixed Factorial
ANOVAANOVA
Putting this all together…Putting this all together… To study behavior, we have to create To study behavior, we have to create
conditions that are controlled enough to be conditions that are controlled enough to be able to predict an outcome in the controlled able to predict an outcome in the controlled conditionsconditions
We have to think about how to study the We have to think about how to study the behavior of interest and how to make it behavior of interest and how to make it change in a predictable fashionchange in a predictable fashion
Experimental methodology allows researchers Experimental methodology allows researchers to control the sample and expose the to control the sample and expose the participants to changes that are predicted to participants to changes that are predicted to influence the outcomeinfluence the outcome
When you construct a particular experiment or When you construct a particular experiment or use a particular research method, a certain use a particular research method, a certain logic applies when choosing the “proper” logic applies when choosing the “proper” statistical method to analyze the statistical method to analyze the results/outcome results/outcome
Experiment ExampleExperiment Example This will be shown in classThis will be shown in class
Signal Detection ExperimentSignal Detection Experiment IV – Presence of TargetIV – Presence of Target Levels: Present or AbsentLevels: Present or Absent Manipulation: Within-SubjectsManipulation: Within-Subjects Measure/DV: Hits, False Alarms, Correct Measure/DV: Hits, False Alarms, Correct
Rejections, MissesRejections, Misses
Top Related