Web vs. Lab John H. Krantz Hanover College. Outline To web or not to web? Deciding to use the web ...
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of Web vs. Lab John H. Krantz Hanover College. Outline To web or not to web? Deciding to use the web ...
Approach
Web is a tool to use in determining best way to ask question
Much as field experiments have become a tool: e.g. Milgram, S. , Bickman, L., & Berkowitz, L.
(1969). Note on the drawing power of crowds of different sizes. Journal of Personality and Social Psychology, 13(2), 79-82.
While this example is trivia it shows an example of a study not possible in the laboratory
Approach
What this implies: For some questions web is a superior method to
lab May depend on your situation as we all use what
we have available
Characteristics of Web vs. Lab Web
Distributed User or remotely
controlled environment and location and some factors are unknowable
Accidental sampling of large population
Potentially anonymous Requires computer
Lab Confined Experimenter controlled
environment or at least determinable
Accidental sampling generally but random sampling of restricted population is possible
Experimenter is known to be present or near
May not have computer
Characteristics of Web vs. Lab Similarities
A certain act of belief is needed to accept self-report responses
Cannot control state and previous state or events of participant This is an issue in control
Random assignment is possible
Reasons for Use – Musch & Reips (2000)
In descending order (7 very important – 1 not important)
Large N 5.5 High statistical power 4.5 Speed data collection/participants
from other countries 3.6 External validity 3.4 Low cost 3.2
Reasons for Use
Possible responses not possible on paper Dynamic and interactive (but any computer would
do) Access to special populations More precise replication Reduction of experimenter effects Comparability to local sample New set of variables to study
Reasons for Concern
Ethical issues Informed consent/debriefing Data security
Data validity Subject falsification/collaboration
Loss of experimental control Equipment/environment Variation
Reasons for Concern
Computers are necessary Lack of ability to clarify instructions and
informed consent New class of variables that may impact
responses Does this last one sound familiar?
Importance of Problems Faced Musch & Reips (2000) In descending order
Lack of control of behavior 3.6 Lack of control of motivation 3.4 Inability to ask questions 3.3 No control over hardware 2.9 Nonrepresentative sample 2.8 Ethical problems 1.5
Issues regarding the sample
Web is stated to be not representative Questions
How many labs have representative sample? Do we know what person variables are significant
or do we control out of the assumption of a possible effect?
The greater diversity of samples (Krantz & Dalal, 2000 and “College Sophomores”) may allow this to become a research question instead of assumption
The Web Study In Context
Relevant Dimensions Methodological Issues Subject-Related Issues Ethical Issues
A Decision Guide
Relevant Dimensions – Method Statistical Power
Refers to noisy data Not in all cases will noise be better on Web
Ruppertsberg et al (2000). Web, n = 151, s = 12.5, SE = 1.02 Control, n = 14, s = 7.4, SE = 1.9
Effect Size or Robustness of the Effect Basically is it easy to obtain the result
Relevant Dimensions – Method Sample Bias
Is the effect sensitive to the characteristics of the sample?
External Validity Does this effect or relationship happen outside of
laboratory Does this effect or relationship happen away from
a computer
Relevant Dimensions – Method Need for Control
Stimulus Problematic Dimensions
Color Luminance Temporal If can keep on one page
Relative spatial dimensions are fine
Example Study: Column Taper Description:
Doric columns appear to lean when vertical Apparently due to taper of column width
Method Three columns. Adjust so out two same position
as inner (vertical or horizontal) Validation Results
Vertical Results
y = 386.88x - 3.36r2 = 0.87
y = 325.75x - 11.72r2 = 0.99
-200
-150
-100
-50
0
50
100
150
200
250
-0.4 -0.2 0 0.2 0.4
Log Column Proportion
Tilt
at
Perc
eiv
ed V
ert
ical (a
rcm
in)
Web Vert
Lab Vert
Linear (Web Vert)
Linear (Lab Vert)
Relevant Dimensions – Method Need for Control
Measurement Need will be sensitive to type of measurement Think of all question order effects Now type of response objects
Environment Effect of Web Known
Has the method been examined on Web Time for data collection
Response time for results
Relevant Dimensions – Subject Special Populations
Ethnic groups Age Education Other targeted populations (Buchanan, 2000)
Self-Monitoring: found groups predicted high and low on scale to validate response
Relevant Dimensions – Subject Interaction with Subjects
Need for Clarification As part of research design
Demand Characteristics Web removes us, but our design might still
provide some hints
Relevant Dimensions – Ethics
Sensitivity of Ethical Issues Deception Potential for harm
Effects of Fraud by Participants on Data Faking responses Participant Collaboration
A Decision Guide - Method
Low High
Statistical Power Either Web
Effect Size Depends1 Either
Sample bias Either Usually Lab2
Ecological Validity
Either Depends3
Sensitivity of Study to/Need:
Decision Guide – Method
Low High
Control Either Lab
Known Valid on Web
Lab at least as check
Web
Time for Data Web Either
Decision Guide – Subject
Low High
Special Populations
Either Web, in most cases
Subjects from other countries
Either Web
Interaction with subjects
Either Lab
Demand Possibility
Either Web
Factors not Included in Decision Guide Did not include ease or cost
Also not important factor in early researchers, Musch & Reips, 2000
These factors should not drive design criteria.
Conclusions
Web and lab distinct environments Each method has strengths Match method to research question Still useful to examine methods
Study locally as well as globally Differences do not necessarily imply web is less
valid than lab Differences could be meaningful Differences could imply problem in lab envirnment