Response Pads - Swansea Universitypsy.swan.ac.uk/staff/freegard/Response Box Report.pdfby Matrix...

SWANSEA UNIVERSITY

Response Pads The design, development and results of a

bespoke response box, and subsequent testing, for comparison purposes, of various devices

used for reaction time measurements in psychological experiments

by Gary Freegard

Department of Psychology 20/8/2012

© Gary Freegard 2012 1

Psychology Response Box

This report supersedes the previous report (dated 1/3/12), it contains a brief history of the design

and development of the Response Box, a comparison of the Response Box against various other

input devices currently used in reaction time measurement in the department and 2 methods to

improve the recorded accuracy. Also to improve the accuracy and reliability of the data a larger

sample was used (normally 10,000 measurements up from 1,000).

This Response Box is shown to have a significantly greater accuracy and precision than all other

options currently available in the Psychology Department (see Conclusion page 9).

Latency is the measure of the time delay experienced in a system. In this report it is the time taken

by a PC to respond after a button has been pressed on a response box or keyboard.

Accuracy is the closeness of measurements to the actual value, the average of the results.

Precision is the repeatability of the measurements, the standard deviation of the results.

All measurements are in millisecond (ms), the vertical axis of all the graphs show the response time

which is the time from button/key press to the output from the parallel port.

The packages used for the testing are as follows:

E-Prime

VB.net

Opensesame

Most of the graphs used here have a vertical axis range from 0 to 40 ms for ease of comparison,

unless otherwise stated.


Design History

Purpose

To design and build a response box that fulfils the following criteria:

use USB standard for communication

‘Plug and Play’ on multiple operating systems (OS)

lower latency than a standard keyboard (8-12ms, Cedrus website)

functionally comparable with our current response boxes (Cedrus)

design limited only by customer specification and by available parts

upgradeable/expandable

Design

To fulfil the design requirements, a micro-controller was required, along with custom software.

Currently, a micro-controller development system called Flowcode is being used in the department

for the Microchip Pic range of controllers; this limits the list of controllers available for the design.

The possible methods of communication are either USB serial (as current units) or USB HID

(Human Interface Device) i.e. keyboard, mouse, joystick etc. The USB HID requires no new drivers

(though the operating system will see it as a new device).

The USB Serial offers two options:

1, to use the micro-controller in USB Serial mode (but availability of drivers for the other

operating systems is a possible limitation)

2, use a separate USB serial device (made by FTDI) which is also used in the Cedrus boxes

(drivers are available for all operating systems)

The USB HID option was chosen as there is no uncertainty with regards drivers.

The micro-controller initially selected was the ECIO (18F4455 Microchip Pic) designed and built

by Matrix Multimedia, the makers of Flowcode. This device is a 40 pin DIL (Dual In Line) with a

USB port attached and with a boot-loader installed. This allows the device to be reprogrammed via

its own USB port and therefore doesn’t need to be removed from the circuit, which aids

development significantly. Since this device is ready to run out of the box, the only other parts

required to fulfil the design specification are the resistors, switches and case.

Response Box Mk1

This was built for proof of concept, and software development.

This consisted of a metal unit with four buttons on the top panel for participant reaction time

measurements plus two buttons on the rear, one to reset the unit to allow reprogramming and the

other to output a ‘\’ for the Mac OS if required. This used only 6 of the 30 available IOs

(Inputs/Outputs).

figure 1


This unit works on Windows, Mac OS and Linux without the need for additional drivers. The

latency was 2.4ms at worst, verified by the results shown in figure 12 and table 1 (page 7).

Response Box Mk2

This version was built to meet a specific requirement of an academic staff member. It has 5

response buttons along the top (1 to 5), 3 along the bottom (a, b and c) and it has 5 buttons in the

centre, 4 of which are arrow keys, these have the same effect as the arrow keys on the keyboard,

and the fifth is an enter button.

Unfortunately the physical size of the ECIO made it unsuitable for this new design, so instead a

18F2455 was used (a 28 pin device with inbuilt USB capabilities with 19 IO pins and

approximately a third of the price). A printed circuit board was designed and built in house.

Using this micro-controller would allow the design of response boxes with up to 90 buttons if so

desired, whilst if it bigger brother (18F4455) was used then it would be possible to create response

box with 160 buttons!

figure 2

figure 3


The Competition:

RB730- part of the RBx30 series of response boxes from Cedrus, makers of SuperLab

There are 4 different models in this range, with varying layout and number of keys, but they

all use the same underlying hardware.

They are limited to a maximum of 8 keys.

They are fitted with an accessory connector that provides 6 configurable inputs/outputs.

They are compatible with the majority of stimulus presentation software.

The website also claims that their pad offers a 1 millisecond reaction time resolution.

SR200A by PST, makers of E-Prime

This box has only 5 buttons and 5 lamps that can be software controlled.

It also has a Microphone input with variable gain and trigger level.

It has an inbuilt Screen refresh detector circuit which uses the optional PST Refresh

Detector.

It offers 0 millisecond debounce period

Uses the serial port though a Serial to USB converter can be used.

This unit is compatible only with software specifically written to use this unit.

Mac Keyboard Dell Keyboard

figure 4

figure 5

figure 6

figure 7


Mouse

IntelliMouse by Microsoft, uses USB communication.

figure 8


Testing the Response Boxes A program was written in VB.net (initially and then other software packages) to respond to button

presses. Once a button press has been received by the Test PC, the test program will produce a 10

millisecond pulse on the parallel port (printer port), the stop signal. Initially a counter timer was

used to measure the time between the button being pressed and the pulse from the parallel port.

There is a latency of approximately 0.050ms (the max is 0.13ms see tables 30/31 page 24) between

the response of the program to a button pressed (on response box) and the port going high.

Initial testing was carried out using the setup shown in figure 9 below. This is a purely manual

approach, the button was pressed manually and the time data was recorded manually.

After several hundred key presses a secondary 18F2455 was programmed to measure and output

(via USB Serial) the times between two events (pulses) on separate inputs. This enabled the

collection of larger sample of timing data and also facilitated the automation of the process (see

figure 10 page 7)

figure 9


The Timer Unit is able to carry out repeated measurements in microseconds (a 1000 microseconds

equals 1 millisecond) and is calibrated to an accuracy of 0.02% + 1 l.s.d.(least significant digit) by

using an external counter timer. The data is sent to a PC by USB Serial and is captured by

HyperTerminal or similar.

Below is a diagram showing the setup for automatic testing, to reduce unnecessary load on the test

PC all timing data was collected by a second PC.

Here is a breakdown of the operation involved in collecting the timing data:

The Press Button signal (sent from the Timer Unit) is used to simulate the pressing of a

button/key in the Response Box/Keyboard

At the same time the Timer Unit internal clock is started

The Response Box/Keyboard sends the button/key data to the Test PC

Upon receipt of the button/key data the Test PC sends a Stop signal via the Parallel Port to

the Timer Unit.

The Timer Unit stops its internal clock and the measured time is sent to the Data PC for

collection

After a random delay (20 to 270ms), the Press Button signal is sent and the process starts

again.

figure 10


Opensesame and VB.net programs required the installation of inpout32.dll to enable the outputting

of data via the parallel port (printer port). Whilst E-Prime comes with a port option built in.

The test programs for Opensesame and E-Prime.

The graph (figure 13) and table 1 below shows the data collected during the testing of the Response

Box using the USB HID interface (keyboard) with a program written in VB.net. The vertical axis (0

to 40 milliseconds) shows the response time of the device under test whilst the horizontal axis

displays the number of key presses (real or simulated).

This graph shows the Time (Response) in milliseconds for 10000 button presses. The results show

the difference between the actual time (zero) and the time taken for the PC to respond to a button

press. The zoomed in area of the graph shows the response times for the first 100 presses, with lines

indicating the average (red), plus 1 standard deviation (green) and minus 1 standard deviation

(purple).

The table (table 1) shows the statistical data for the data shown in figure 13.

From table 1 it can be seen the average (accuracy) is 1.671ms, whilst the standard deviation

(precision) is 0.289ms.

The minimum and maximum delays are 1.122ms and 2.283 respectively.

The ideal response box would have an average of 0ms and a standard deviation of 0ms.

Min 1.122

Max 2.283

Average 1.671

StDev 0.289

table 1

figure 11

figure 13

figure 12


Conclusion Over the next three pages are the conclusions/comparisons of the testing of all the devices, using

three different applications.

All these tests were carried out on a Dell Optiplex 755 (3GHz Duo Core) with Windows XP Sp3

and a laptop to record the timing data.

The compatibility table below shows the device and the application that it was tested with. No

testing of the RB730 or SR200A was carried out with the VB.net application, as I was unaware of

the protocol needed to use these devices.

Compatibility

VB.net E-Prime Opensesame

Response Box USB Hid

Response Box USB Serial 2

Response Box FTDI Serial 2

Response Box MAX232 Serial 1 1 2

Cedrus RB730

SR200A

Mac Keyboard

Dell Keyboard

Mouse

table 2

Notes:

1- Was unable to test automatically as button presses were missed either by the PC or software only 300 presses recorded

2-Opensesame does not come with a serial port option.


E-Prime

The Response Box achieves accuracy and precision that is significantly better than all the other

current response units, some of which come from specialist manufactures.

From this table it can be seen that the Response Box’s accuracy and precision compared to its

nearest rival (SRB200) are 2.78 and 9.52 times better, respectively.

These are the results of the tests on pages 13, 17, 18, 19, 20 and 21.

Total Time Data in ms

Response

Box SRB200 Mac

Keyboard Cedrus RB-x30

Mouse Dell Keyboard

Average 2.503 6.970 8.250 10.977 18.415 27.264

Stdev 0.333 3.181 2.598 4.643 2.372 3.168

table 3

Comparison against Response Box

SRB200 Mac

Keyboard Cedrus RB-530

Mouse Dell Keyboard

Average 2.780 3.290 4.380 7.350 10.980

StDev 9.520 7.750 13.900 7.100 9.490

table 4

figure 14


VB.net

Again the response box achieves an accuracy and precision that is significantly better than the two

other units that are compatible with the VB.net program.

The nearest rival is the Mac Keyboard, which is 4.5 times less accurate and 8.9 times less precise.

These are the results of the tests on pages 13, 19, 20 and 21.

Response

Box Mac

Keyboard Mouse Dell

Keyboard

Average 1.671 7.629 17.774 26.531

StDev 0.289 2.560 2.315 3.114

table 5


Mac

Keyboard Mouse Dell

Keyboard

Accuracy 4.570 10.640 15.880

Precision 8.860 8.010 10.770

table 6

figure 15


Opensesame And again the Response Box comes out on top when compared with the competition.

Once again the nearest rival is the SR200A, which is 2.9 times less accurate and 9.9 times less

precise.

These are the results of the tests on pages 14, 17, 18, 19, 20 and 21.

Response

Box Mac

Keyboard RBx30

Mouse Dell

Keyboard SR200A

Average 2.105 8.013 11.594 18.18 26.762 6.168

StDev 0.296 2.611 4.701 2.339 3.112 2.936

table 7


SR200A Mac Keyboard

RBx30 Mouse Dell Keyboard

Average 2.931 3.807 5.509 8.638 12.716

StDev 9.905 8.810 15.858 7.891 10.498

table 8

figure 16


Response Box Results

A script was written in E-Prime to output a 10ms pulse on the parallel port once a keyboard/button

press was detected. The results are below; this is visibly different to the results from either of the

Keyboard and Serial as the standard deviation is less, the minimum is 0.7ms higher and maximum

is 2ms higher.

This shows that the Response Box works with E-Prime and that the times recorded are comparable

with those from VB.net, though slightly higher but with less variation. From these results it can be

seen that there is a fixed latency of approx 1.8ms and this could be subtracted from all the recorded

reaction times.

Min 1.839

Max 4.372

Average 2.503

StDev 0.333

table 9

Min 1.122

Max 2.283

Average 1.671

StDev 0.289

table 10

figure 17

figure 18


Unfortunately E-Prime does not allow the use of a USB or Normal Serial device as an input device

unless it emulates a PST (E-Prime manufacturer) response box, so the only way to carry out any

tests using Serial mode was by the use of an inline script that would wait for data to be received

before sending a Stop signal to the Timer unit.

With Opensesame it was not possible to test the Serial mode out,

Min 1.526

Max 3.722

Average 2.105

StDev 0.296

table 11

Min 2.031

Max 7.162

Average 2.717

StDev 0.316

table 12

Min 1.13

Max 2.235

Average 1.684

StDev 0.291

table 13

figure 19

figure 20

figure 21


The following tests were carried out to check the methods of communication used by Cedrus and

PST in their response units, by attaching these devices to the Response Box. This required the

modification of the Response Box’s software so that the keypresses were output as Serial data,

which is then sent to the attached device (either FTDI or MAX232).

This data is similar to the data collected for the Cedrus unit (figure 27 and table 19 on page 17), the

Average and Standard Deviation are very close, indicating similar level of performance.

Min 1.799

Max 33.987

Average 10.455

StDev 4.691

table 14

Min 1.114

Max 17.692

Average 9.212

StDev 4.659

table 15

figure 22

figure 23


From the below graph and table it can be seen that using the old RS232 comport would allow

devices to be very precise, this method has the lowest standard deviation of any of the tests that

have been carried out. Unfortunately this port is no longer standard fit on computers, due to it being

superceeded by USB, though it is possible to purchase PCI/PCI-E expansion cards with RS232

ports.

During testing it was not possible to run it automatically, as not all the button presses were received

by the PC, this hi-lights an error in the RS232 hardware/setup, as the Response Box was using the

same software as used in the FTDI test (page 15).

Min 3.696

Max 4.337

Average 3.757

StDev 0.077

table 16

Min 3.827

Max 7.501

Average 3.971

StDev 0.279

table 17

figure 24

figure 25


Cedrus RB-x30 response box results

This device uses the FTDI for its USB Serial communications. It was tested to check if the results

from Response Box were comparable to this device.

The results show that the Response Box is more accurate (5x) and has a higher precision (22x) than

the Cedrus unit when used with E-Prime

This result was totally unexpected, as the belief was that these Cedrus units do record reaction times

accurately and the intention was to benchmark Response Box against the Cedrus.

Min 1.777

Max 19.84

Average 10.977

StDev 4.643

table 18

Min 2.36

Max 20.588

Average 11.593

StDev 4.700

table 19

figure 26

figure 27


SRB200A Response Box

This box is designed by PST specifically to be used with E-Prime. It is a system that was designed

in 1997 is limited to 5 buttons internally (8 buttons via the accessory socket) and requires a serial

port. Though it does have a Microphone input and a Refresh Detector System built in.

This unit achieves better results than the Cedrus unit but it still falls short of the results achieved

with the in house designed unit (Response Box) which has 3.5 times better accuracy and the

precision is 14 times better.

Another surprise is that a better result for both accuracy and precision is achieved using

Opensesame.

Min 1.236

Max 16.523

Average 6.970

StDev 3.181

table 20

Min 1.082

Max 11.438

Average 6.168

StDev 2.936

table 21

figure 28

figure 29


Mac Keyboard

This is a standard Mac keyboard. During the testing this was the surprise device, it was significantly

better than a standard USB keyboard and was better than the RBx30 device.

Min 1.985

Max 14.255

Average 8.250

StDev 2.598

table 22

Min 1.703

Max 13.627

Average 7.629

StDev 2.560

table 23

Min 2.046

Max 14.055

Average 8.013

StDev 2.611

table 24

figure 30

figure 31

figure 32


Mouse

For Windows the polling rate is set automatically to 125Hz, so the expected latency should be 8ms

but as you can see the average latency is approximately 18ms. So this polling rate may only apply

to mouse movement and not the buttons.

The graphs and statistics below show that the mouse’s performance is almost identical for all

software packages.

The only positive point about using this mouse is that it out performs the Dell keyboard.

Min 12.811

Max 23.206

Average 18.415

StDev 2.372

table 25

Min 13.218

Max 22.408

Average 17.774

StDev 2.315

table 26

Min 13.542

Max 22.855

Average 18.180

StDev 2.339

table 27

figure 33

figure 34

figure 35


Standard Keyboard Results The purpose built timer was connected to a keyboard button and again to the parallel port. Below

are the results, as can be seen the results are significantly worse than was expected (8-12ms).

The Response Box accuracy is at least 10 times better and its precision is at least 9 times better than

this keyboard. These results are far worse than was expected (8-12ms) and these keyboards should

only be used when accurate reaction times are not needed across any presentation software.

Min 17.542

Max 33.493

Average 26.531

StDev 3.114

table 28

Min 18.303

Max 34.717

Average 27.264

StDev 3.168

table 29

Min 17.936

Max 34.091

Average 26.762

StDev 3.112

table 30

figure 36

figure 37

figure 38


Additional Information

System Latencies The latencies are caused by the areas shown in the figure 45 (below), these are:

Red– the time it takes the microcontroller to respond to a key press and then send an output

via either the serial or USB interface and then for this interface to transmit this data on to the

PC. Delays here can be caused by using debounce routines and/or keyboard scan routines.

Green– the time for the Input/Output ports and associated hardware to respond to the

incoming data, this is dependent on the method used to acknowledge/transfer the receipt of

data to the operating system; an example is the USB system where data is only transferred

once every 1 millisecond.

Blue – the time for the operating system to pass the data onto the application and then for

this application to respond and then to initiate the stop signal. This is dependent on the

operating system being used (MacOS, Linux or Windows), the speed of the processor,

number of cores, workload etc.

Yellow-the time for the PC hardware to produce an output on the parallel port.

figure 39


PC Port Test

The same method (as used to capture all the Total Response Times) was used to measure the latency

caused by the PC and parallel port (figure 41 below). The Timer Unit (for connections see figure 40

below) was used to produce a Start pulse into the input on the parallel port at the same time its

internal clock was started. Two applications were written in VB.net and E-Prime to respond to an

input on the parallel port by producing an output also on the parallel port, the Stop signal. The

Timer Unit stopped its clock on receipt of the Stop signal. This was done for 1000 trials and the

results are in tables 31/32, and it can be seen that the worst case latency is 0.133ms.

E_Prime

Response Time ms

Min 0.031

Max 0.133

Average 0.048

StDev 0.008

table 31

VB.net

Response Time ms

Min 0.023

Max 0.109

Average 0.049

StDev 0.008

table 32

figure 41 figure 40


USB HID

Explanation of USB HID timing diagram (figure 42)

Upon a button being pressed, the micro-controller attempts to send the data to the PC but this isn’t

possible until the micro-controller receives a USB Polling signal (every 1ms). This is the cause of

the first latency (referred to as Initial Delay in this report) which can vary from 0.1 to 1.3 ms (see

table 13 page 12 : min-1ms and max-1ms).

Once this button data has been sent, the micro-controller has to send an empty data set upon

receiving the next polling signal (1ms later). It is upon receipt of this that the PC will respond to the

button press i.e. the application will receive button data.

The total system latency is made up of the following:

Initial Delay

Sending the empty data

Driving the parallel port

The Initial Delay is variable, but has a direct correlation with the Total Response Time.

The sending of the empty data set is a fixed delay of 1ms

The parallel port is 0.05ms on average (see table 31/32page 23).

So if all the delays are know then it will be possible to adjust the recorded reaction time to the

actual reaction time.

figure 42


Useful Links

18F2455 and 18F4455

http://ww1.microchip.com/downloads/en/DeviceDoc/39632e.pdf

ECIO

http://www.matrixmultimedia.com/ecio.php

Flowcode

http://www.matrixmultimedia.com/flowcode.php

USB

http://www.beyondlogic.org/usbnutshell/usb1.shtml

http://www.usb.org/developers/devclass_docs/HID1_11.pdf

http://en.wikipedia.org/wiki/Universal_Serial_Bus

http://www.usbmadesimple.co.uk/index.html

Cedrus

http://www.cedrus.com/

http://www.cedrus.com/responsepads/why_use.htm)

E-Prime

http://www.pstnet.com/

http://www.pstnet.com/support/kb.asp?TopicID=1835

Empirisoft

http://www.empirisoft.com/Default.aspx?index=0

http://www.empirisoft.com/Hardware.aspx

Opensesame

http://www.cogsci.nl/software/opensesame/

FTDI

http://www.ftdichip.com/

PIC and PICmicro are registered trademarks of Arizona Microchip Inc.

Flowcode and E-Blocks are trademarks of Matrix Multimedia Limited

E-Prime is a registered trademark of Psychology Software Tools, Inc.

Windows is a registered trademark of Microsoft Corporation.

Mac and Mac OS are trademarks of Apple Inc.

http://ww1.microchip.com/downloads/en/DeviceDoc/39632e.pdf

http://www.matrixmultimedia.com/ecio.php

http://www.matrixmultimedia.com/flowcode.php

http://www.beyondlogic.org/usbnutshell/usb1.shtml

http://www.usb.org/developers/devclass_docs/HID1_11.pdf

http://en.wikipedia.org/wiki/Universal_Serial_Bus

http://www.usbmadesimple.co.uk/index.html

http://www.cedrus.com/

http://www.cedrus.com/responsepads/why_use.htm

http://www.pstnet.com/

http://www.pstnet.com/support/kb.asp?TopicID=1835

http://www.empirisoft.com/Default.aspx?index=0

http://www.empirisoft.com/Hardware.aspx

http://www.cogsci.nl/software/opensesame/

http://www.ftdichip.com/

Response Pads - Swansea Universitypsy.swan.ac.uk/staff/freegard/Response Box Report.pdfby Matrix...

Documents

Transcript of Response Pads - Swansea Universitypsy.swan.ac.uk/staff/freegard/Response Box Report.pdfby Matrix...