Psychophysical methodsLavanya Sharan

January 26th, 2011

Announcements

• Check class website: http://graphics.cs.cmu.edu/courses/P2P/

• Pick presentations slots for Feb 9th and Feb 14th.

• One CVG and one P slot per person (not on same day)

• Email slot preference to sharan@cs.cmu.edu

• Have to meet instructors 2x before presentation!

Overview

•Basics of designing a perceptual experiment

•Types of experiments

•Analysis of experiments

•IRB

Basics

•Independent variable

•This is what varies

•Can have several levels (i.e. values)

•An experiment may have many indep. vars.

•Dependent variable

•This is what is measured

An example • How well do people recognize objects (as

compared to a favorite CV algorithm)?

Images from the Visual Object Classes Challenge 2010

Independent variable?Dependent variable?

Class participation!

Class participation!

An example

• How well do people recognize objects?

Images from the Visual Object Classes Challenge 2010

Independent variable Object class (20 levels)Dependent variable Accuracy, Reaction time (RT)

Understanding your variables• Discrete or continuous? (e.g., object class is

discrete)

• Measurement scale

- Nominal: there is a difference (e.g., car vs. bus)

- Ordinal: <,=,> make sense (e.g., rank in VOC2010 challenge)

- Interval: size and sign of difference makes sense (e.g., Fahrenheit scale)

- Ratio: all the above and having a 0 makes sense (e.g., percentage correct)

Confounds • Ideally, stimuli only differ in the levels of the

independent variable.

Images from the Visual Object Classes Challenge 2010

Confounding variables Color, shape, size of object, contrast of images, background of objects, location in image, lighting, etc.

Confounds • Ideally, stimuli only differ in the levels of the

independent variable.

Images from the Visual Object Classes Challenge 2010

In practice, impossible to eliminate confounding variables.

A well-designed study minimizes as many confounds as possible.

Dealing with confounds

• Control for confounding factors by:

• Removing confounding variables in stimuli

• Balancing the presence of confounding variables by adding control conditions*

• Separating participants into groups and comparing their responses i.e., control group

Condition = any manipulation of independent variables.

Controls

Images from the Visual Object Classes Challenge 2010Hypothesis: Size of object in image is enough to distinguish the 20 object classes.

Confounding (or control) variable = Visual size of objects

Option 1: Resize and re-crop all the images, never allow size to be an issue.

Hypothesis: Size of object in image is enough to distinguish the 20 object classes.

Confounding (or control) variable = Visual size of objects

Option II: Create second set of images with resized objects, test if size is an issue. Question: Control condition (same

subjects) or control group (different subjects)?

Controls

Images from the Visual Object Classes Challenge 2010

Experimental Designs Between-subjects: Different subjects in different conditions. Minimize learning effects, more subjects

Within-subjects: Same subjects in all conditions. More power, fewer subjects

Mixed design: Some conditions within-subjects and some between-subjects.

Balanced designsEven after adding control conditions and/or groups, we have to worry about order effects

Seeing bikes before buses

Seeing dogs before buses

Balanced designsEven after adding control conditions and/or groups, we have to worry about order effects

Seeing bikes before buses

Seeing dogs before buses

Does it matter? Priming can occur.

Balanced designsEven after adding control conditions and/or groups, we have to worry about order effects

Seeing bikes before buses

Seeing dogs before buses

Solution:

RandomizationCounterbalancing

Balanced designsUse randomization when lots of subjects or each subject sees the same conditions many times.

Use counterbalancing otherwise. E.g., 3 conditions (Dog, Bike, Bus)

Slide content: Aude Oliva, MIT OCW

Dog

Bike

Bus

Order 1 Order 2 Order 3Bus

Dog

Bike

Bike

Bus

Dog

S 1,4,7,10...S 2,5,8,11...S 3,6,9,12...

Overview

•Basics of designing a perceptual experiment

•Types of experiments

•Analysis of experiments

•IRB

Thresholds

Absolute: At which visual size can you detect that there is a bike?

Difference: How much detail is needed to recreate the original shape?

Measuring ThresholdsMethod of Constant Stimuli: Present stimuli at several levels, some below and some above threshold, measure proportion detected.

Method of Limits: Start at one level, if detected decrease else increase level. Converge to an estimate of threshold.

Method of Adjustment: Let subjects adjust the level until they can just detect the stimulus.

Method of Constant Stimuli+ Easy to conduct and analyze

- Need to have an idea of threshold beforehand- Takes longer, more trials

Psychometric function

Image source: http://www.ncbi.nlm.nih.gov/books/NBK11513/Slide content: Lynee Werner, University of Washington

Method of Limits+ Fewer trials+ Don’t need to estimate threshold beforehand- Noise prone

Staircase procedure

Image source: http://www.ncbi.nlm.nih.gov/books/NBK11513/Slide content: Lynee Werner, University of Washington

Method of Adjustment+ Intuitive for subject- Can be unreliable

Image source: http://psychology.wikia.com/wiki/File:Method_of_Adjustment.png

Slide content: Lynee Werner, University of Washington

Overview

•Basics of designing a perceptual experiment

•Types of experiments

•Analysis of experiments

•IRB

Back to detectionWhat about response bias?

Image source: http://www.walker.co.uk/walkerdam/getimage.aspx?id=9781406314403-1&size=webuse

Slide content: Lynee Werner, University of Washington

Subjects might be more prone to saying yes (or no).

Signal detection theory

Slide content: Lynee Werner, University of Washington

Subject said yes Subject said no

Stimulus present Hit Miss

Stimulus absentFalse Alarm

Correct Rejection

‣ d’ is computed from Hits and False Alarms‣ Measure of sensitivity‣ Bias-free

2-AFC design

Image source: http://www.ratracetrap.com/wp-content/uploads/2009/09/Fork-in-the-road-300x237.png

Slide content: Lynee Werner, University of Washington

‣ Instead of asking for yes/no response on each trial, force subject to choose from two options

‣ Another way of removing response bias (interval bias can remain)

How do you know subjects aren’t guessing?

(Bad subjects.)

Image source: http://academic.kellogg.edu/mckayg/buad112/web/pres/coin%20flip.jpg

Need to calculate chance performance for every task

Subjects might have been sleeping, distracted, perverse, or

simply unable to do your task because it is humanly impossible.

Is performance better than chance?

Statistics to the rescue. VOC2010 Challenge example:

20 object classes implies chance = 1/20 = 5%

Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10)

✓ Use independent one-sample t-tests to compare performance in both groups to chance✓ Choose significance threshold (usually p = 0.05)✓ Divide significance threshold by number of tests (Bonferroni correction), here 0.05/2 = 0.025✓ Quote the t-statistic and p-value if less than corrected threshold as showing statistical significance.

Is performance better in one condition than another?

Again, statistics to the rescue. VOC2010 Challenge example:

20 object classes implies chance = 1/20 = 5%

Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10)

✓ Use independent two-sample t-test to compare performance in two groups to each other✓ Choose significance threshold (usually p = 0.05, if combining with two previous tests then 0.05/3 = 0.0167)✓ Quote the t-statistic and p-value if less than threshold as showing statistical significance.

Choosing the right tests is VERY important

Otherwise you don’t know if you are measuring noise or a real effect.

For example, for a within-subjects design, you would use a paired samples t-test.

Get hold of a good statistics book and package and understand precisely what you are doing.

I recommend using SPSS and material from: http://www.statisticshell.com

Overview

•Basics of designing a perceptual experiment

•Types of experiments

•Analysis of experiments

•IRB

IRB and other legalities

You are dealing with human subjects, this means youneed to be very ethical and careful.

CMU has a Regulatory Compliance Office: http://www.cmu.edu/osp/regulatory-compliance/human-subjects.html

You are encouraged to take the online human subjects training: https://www.citiprogram.org/Default.asp?

Image source: http://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpg

IRB and other legalities

For this class:

If you will not use any perceptual data you gather ever again, you don’t need to write an IRB protocol and get approval.

If there is even a tiniest chance, the perceptual data you gather will be show up anywhere, you NEED to write an IRB protocol and get it approved in time. Come talk to me about this.

Image source: http://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpg

Summary

• Figure out your independent and dependent variables.

• Think of all possible confounds.

• Control for confounds, balance your designs.

• Get IRB to run study.

• Analyze data using standard statistical procedures.

• Don’t do all this a day before your deadline!