Timing errors in computer-based paradigms

Richard R. Plant, Garry Turner and Annie Trapp(The Higher Education Psychology Network)

Computer use in experimental settings

• Widespread use of computers for experimental delivery• Ever more complex paradigms designed which look for

smaller effect sizes• Often interoperability with extremely complex third party

hardware and software, e.g. fMRI• Assumption that “anything goes” with today’s hardware –

faster must = better• Undergraduates & postgraduates often share this notion• Easy to deal with large class sizes, e.g. PsychMate, PEAK

Should we be concerned?

• Overuse – do students become hooked on computer-based methods? Do lecturers?

• Are they taught about the potential pitfalls as well as the benefits? Do researchers know about these too?

• Because the “hoops are fewer” is attention to detail laxer today?

• Is research suffering?• Do today’s computer systems produce timing errors?• Should we do something about it?

There’s always a trade-off

• Teaching1. Ease of use (basic

components)

2. Works on basic equipment

3. Good/clear user documentation

4. Simple pre-canned examples

5. Timing not so important

6. Data in easy to use format

7. Cost/licensing

• Research1. Functionality (expandable)

2. Works with custom hardware

3. Good/clear technical documentation

4. Complex technical examples

5. Timing accuracy

6. Data manipulated externally

7. Cost not usually an issue

Research – the timing characteristics of mice

• What kind of contribution can a response device make to timing?

– Examined various brands of mice– Looked at various interfaces, PS/2, USB, Serial– Examined the timing characteristics using a signal generator and

Digital Phosphor Oscilloscope (external to a PC)– Examined the performance of each mouse under a simple

paradigm in E-Prime. Flash a block mid screen then simulate a response at a known offset (collate response times in terms of known versus actual)

• Can you predict response device performance?• What’s the typical contribution?• What effect does the operating system have?• What does the experiment generator contribute?• Does it matter? (Ulrich & Giray 1989)

0

10

20

30

40

50

60

70

80

Trial Number

Res

po

nse

Tar

get

Err

or

(ms)

AMI wheel mouse, PS/2 (ref 6b) OEM mouse, PS/2 (ref 8)

ALPS keyboard, PS/2 (ref KB) PST response box, serial (ref BB1)

Display devices – All created equal right?

Conditional biases - Cross modal priming in the field

So what about benchmarking?Can’t you just tell us which software is best?

What about the real-world?• There is little doubt that the majority of today’s high speed, high spec

hardware and operating systems are capable of real-time data collection (MacInnes & Taylor 2001, Finney 2001).

• Such research, whilst providing a solid baseline, leaves researchers in the field with the fundamental question:

“How does my own paradigm performin the real world?”

• Until now this has been a question that has been extremely difficult to answer.

• Complete real-world paradigms can often be extremely complex making use of both visual and auditory stimuli and requiring complex patterns of responses from subjects.

The only solution seems to be self-validation of ones own timing

• Taking a leaf out of other research cultures where equipment is calibrated yearly

• Independently check presentation, synchronisation and response timing

• State error limits in academic papers• Raise awareness of the issues and their increasing

importance• Make it a requirement from Journals / BPS?• Make it a teaching requirement that students should be

made aware of the issues involved• How easy is it to do?

Our timing toolkit (the Black Box Toolkit) - the notion of a virtual human that checks any paradigm

• Pressing need for researchers to easily and cheaply self-validate their own paradigms in-situ on their own hardware.

• These needs led to the development of the Black Box Toolkit!

Clear results

The beneficiaries

• Researchers in the behavioural sciences (psychology, ergonomics and human-computer interaction) carrying out work involving time-critical measurement

• Practitioners involved in the design and evaluation of equipment for time-critical human performance control

• Lecturers (and their students) who teach experimental design and methodology using software tools

• Software developers of tools for measuring performance timing

• Above all improving the quality and consistency of research findings within the field… At the moment some studies look “suspect” based on our experience to date.