Virtual Reality for Psychological Assessment in Clinical Practice
Thomas D. Parsons and Amanda S. PhillipsUniversity of North Texas
Clinical psychologists are often asked to make recommendations regarding a client’sability to function in everyday activities (e.g., work, classroom, or shopping). Commonapproaches include a combination of paper-and-pencil (including some computerautomated) assessments, behavioral observations of the client’s performance duringtesting, and self-report measures about their perceived deficits. From this combinationof assessments, observations, and self-reports, the psychologist is expected to makepredictions about the client’s ability to return to the classroom, return to work, andsuccessfully complete other activities of daily living. While there are advantages to thisapproach, there are also some shortcomings—perhaps most notable is the lack ofecological validity. Recent advances in virtual reality technologies allow for enhancedcomputational capacities for administration efficiency, stimulus presentation, auto-mated logging of responses, and data analytic processing. These virtual environmentsallow for controlled presentations of emotionally engaging background narratives toenhance affective experience and social interactions. Within this context virtual realitycan allow psychologists to offer safe, repeatable, and diversifiable assessments of realword functioning. Although there are a number of purported advantages of virtualreality technologies, there is still a need for establishing the psychometric properties ofvirtual reality assessments and interventions. This review investigates the advantagesand challenges inherent in the application of virtual reality technologies to psycholog-ical assessments and interventions.
Keywords: virtual reality, neuropsychology, psychological assessment, ecologicalvalidity
Clinical psychologists are increasingly beingasked to make prescriptive statements aboutevery-day functioning. Unfortunately, resultsfrom many psychological tests are not easilygeneralizable to real-world functioning. Com-mon approaches include a combination of his-tory taking, self-reports, paper-and-pencil cog-nitive assessments, and the psychologist’sobservations of the client’s behavior. From thiscombination, the psychologist is expected tomake predictions about the client’s ability toreturn to the classroom, return to work, and
successfully complete other activities of dailyliving. The limitations inherent in this processhave led to increasing calls for assessmentmethods that provide more generalizable dataabout client functioning (Burgess et al., 2006;Jurado & Rosselli, 2007). Virtual reality (VR)assessments have been developed to provide anenhanced understanding of client functioning inactivities of daily living (Campbell et al., 2009;Matheis et al., 2007).
While VR assessments are often presented astools for neurocognitive assessment in researchsettings, this article aims to provide an introduc-tion to VR assessment for clinical practice.First, current assessment methods are comparedwith assessment in virtual environments. Next,examples of VR assessments with clinical caseexamples are provided. Finally, considerationsfor the adoption of VR technologies in clinicalpractice are explored. We propose that the ad-dition of VR to current psychological assess-ment batteries can improve the generalizabilityof test results and increase the utility and rele-
This article was published Online First August 18, 2016.Thomas D. Parsons and Amanda S. Phillips, Department
of Psychology, Computational Neuropsychology and Sim-ulation Lab, University of North Texas.
Correspondence concerning this article should be ad-dressed to Thomas D. Parsons, Computational Neuropsy-chology and Simulation Lab, National Academy of Neuro-psychology, Department of Psychology, University ofNorth Texas, 1155 Union Circle #311280, Denton, TX76203. E-mail: [email protected]
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Practice Innovations © 2016 American Psychological Association2016, Vol. 1, No. 3, 197–217 2377-889X/16/$12.00 http://dx.doi.org/10.1037/pri0000028
197
vance of psychologists’ recommendations toclients.
Self-Reports and Behavioral Observations
Although traditional assessment methodsprovide valuable information, there are a num-ber of limitations in using this approach (seeTable 1). Determining a client’s functional ca-pabilities requires precise control over the en-vironment and the ability to adjust the potencyor frequency of stimuli (White et al., 2014).This control is difficult to ensure in the tradi-tional assessment environment. Given that psy-chological tests are typically administered one-on-one in a controlled environment; thepsychologist may not receive a clear picture ofthe client’s cognitive functions in everyday ac-tivities. Although psychologists often use infor-mation from parent and teacher reports to get anidea of the patient’s everyday functioning, stud-ies indicate that the agreement between parentsand teachers is modest. For example, the con-cordance between parents and teachers on diag-nosing attention-deficit-hyperactivity disorder(ADHD) varies from .30 to .50 depending onthe behavioral dimensions being rated (Bieder-
man, Faraone, Milberger, & Doyle, 1993; Bie-derman, Keenan, & Faraone, 1990; Mitsis, Mc-Kay, Schulz, Newcorn, & Halperin, 2000;Newcorn et al., 1994; de Nijs et al., 2004).Additionally, there is often not a strong overlapbetween these rating scales and standard tests ofcognitive functioning, suggesting that these as-sessments may be reflective of different aspectsof behavior (see Table 1 for a list of potentialconfounds).
Normative Comparison ofCognitive Performance
For many psychologists, the way to get be-yond these limitations is to emphasize a norma-tive comparison approach to psychological as-sessment. For example, a psychologist can givea continuous performance test (CPT) to assess acognitive construct (e.g., attentional process-ing). Results from the CPT reveal the client’sperformance in areas of inattentiveness, impul-sivity, sustained attention, and vigilance. Thepsychologist can compare the client’s CPT re-sults with reference groups to determinewhether a client is performing as would be
Table 1Comparison of the Advantages and Disadvantages of Traditional Assessment
Advantages Disadvantages
Self-reports Allows identification of areas of clientfunctioning that have been impacted orpreserved.
Controlled environment limits thegeneralizability of results.
Provides information about the client’sperceptions of their performance.
Third-party reports may have poor inter-raterreliability.
Third-party reports provide data about clientfunctioning in various environments.
Third-party reports often fail to stronglyoverlap with cognitive tests.
Potential influence of self-report format onresponses.
Unable to measure automatic processing.Post hoc appraisal of past behavior.
Behavioral observation Provides insights into how the client may reactwhen faced with difficult tasks.
Controlled environment limits ecologicalvalidity of observations.
Normative comparison Allows for measurement of expectedperformance.
Tests are often measure abstract constructsrather than functioning.
Quantifies discrepancies in expectedperformance.
Many cognitive tests were developed for usewith normal populations, not for functionalassessment.
Tests may not be representative of real-worldsituations.
Tests lack generalizability to real-worldsituations.
Test sensitivity and potency cannot be altered.Time-consuming and expensive
administration.
198 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
expected given their achievements and educa-tional attainment.
An unfortunate limitation of this approach isthat the tests used are typically based uponconstruct-driven assessments that do little toreplicate the client’s everyday activities. Fur-thermore, many of the tests commonly usedtoday were never meant for clinical assess-ments. Burgess and colleagues (2006) point outthat cognitive construct measures like theTower of London and the Wisconsin Card Sort-ing Test (WCST) were not originally designedto be used as clinical measures. Instead, thesemeasures were found to be useful tools forcognitive assessment among normal popula-tions and then later found their way into theclinical realm to aide in assessing constructsthat are important to carrying out real-worldactivities. This approach forces the psychologistto rely on measures that were designed for pur-poses other than predictions of real-world func-tioning. Burgess and colleagues (2006) arguedthat we need assessments capable of broadeningour understanding about the ways in which thebrain enables persons to interact with their en-vironment and organize everyday activities. Al-though many cognitive tests do give us someinsight into the client’s everyday performance,they do not provide direct knowledge aboutshortcomings in the functional capabilities ofthe client, which limits the accuracy and utilityof the psychologist’s recommendations (Chay-tor & Schmitter-Edgecombe, 2003; Manchester,Priestley, & Jackson, 2004).
Virtual Environments
One potential answer to the issues raisedabove is the addition of virtual environments to
the psychologists’ cognitive assessment battery,which allow clients to become immersed withina computer-generated simulation (Campbell etal., 2009; Matheis et al., 2007). Potential virtualenvironment use in assessment and rehabilita-tion of human cognitive processes is becomingrecognized as technology advances, and has anumber of advantages (see Table 2). While acomplete listing of all the virtual environment-based psychological assessments is outside thescope of this article, we provide examples vir-tual environments with case examples.
Virtual Classroom
Various virtual classrooms are being vali-dated for assessment of supervisory attentionalprocessing. Virtual Classrooms are simulationenvironments that were designed to assess po-tential attention deficits by embedding cognitivetasks (e.g., CPT; Stroop) into a virtual class-room environment (Díaz-Orueta et al., 2013;Iriarte et al., 2016; Parsons et al., 2007; Rizzo etal., 2006; see Table 3). In these virtual class-rooms the participant is seated at one of thedesks and is surrounded by desks, children, ateacher, and a white board much like theywould be in a real-world classroom. Variouscognitive tasks can be presented on the white-board in the front of the room and the partici-pant performs a task (e.g., Stroop or CPT) withauditory (e.g., airplane passing overhead, avoice from the intercom, and the bell ringing)and visual (e.g., children passing notes, a childraising his hand, the teacher answering theclassroom door, and principal entering theroom) distractors in the background. The num-ber and frequency of these distractors can beadjusted based on age, grade level, or other
Table 2Comparison of the Advantages and Disadvantages of Assessment in a Virtual Environment
Advantages Disadvantages
Precise and consistent presentation of stimuli. Construct-driven assessments will still lack some ecological validity.Clinician control of dynamic stimuli. Lack of guidelines for development, administration, and interpretation.Greater ecological validity. Pediatric and geriatric clients may need additional guidance.Availability of function led assessments. Geriatric clients may have trouble adjusting to the visual environment.Programs administer, record, score, and analyze data. Clients with Autism Spectrum Disorder may become over stimulated.Environmental stimuli and test sensitivity can be
adjusted.Clients with psychiatric disorders or high suggestibility may respond
unfavorably.Environment can be adjusted to reflect clients’
everyday surroundings.
199VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Tab
le3
Rec
ent
Con
stru
ct-D
rive
nSt
udie
s(W
ithi
nP
ast
10Y
ears
)U
sing
aV
irtu
alC
lass
room
Env
iron
men
t
Stud
ySa
mpl
eR
esea
rch
desi
gnT
radi
tiona
lT
ests
Res
ults
Ada
ms
etal
.(2
009)
N�
35bo
ysag
es8–
14ye
ars;
19pa
rtic
ipan
tsw
ithA
DH
Dw
ere
com
pare
dto
16ag
e-m
atch
edco
ntro
ls.
Com
pari
son
ofpa
rtic
ipan
tpe
rfor
man
ceon
the
Con
tinuo
usPe
rfor
man
ceT
est
with
and
with
out
the
Vir
tual
Cla
ssro
om.
•B
ehav
ior
Ass
essm
ent
Syst
emfo
rC
hild
ren
(BA
SC)
•V
IGIL
Con
tinuo
usPe
rfor
man
ceT
est
•G
reat
ersp
ecifi
city
was
foun
dfo
rV
irtu
alC
PT.
•A
lthou
ghdi
ffer
ence
sbe
twee
nth
etw
ogr
oups
wer
eno
tsi
gnifi
cant
,a
stro
ngtr
end
was
obse
rved
for
corr
ect
targ
etid
entifi
catio
nan
dco
mm
issi
oner
rors
.B
ioul
acet
al.
(201
2)N
�36
boys
ages
7–10
year
s;20
part
icip
ants
with
AD
HD
wer
eco
mpa
red
to16
age-
mat
ched
cont
rols
.
AD
HD
and
cont
rols
child
ren
wer
efir
stte
sted
with
the
trad
ition
alco
mpu
teri
zed
CPT
.A
fter
10m
inth
eyw
ere
test
edw
ithth
evi
rtua
lC
PT.
•C
ontin
uous
Perf
orm
ance
Tes
t(C
PTII
)•
Con
ners
’pa
rent
sra
ting
scal
e•
Chi
ldB
ehav
ior
Che
ckL
ist
•St
ate
Tra
itIn
vent
ory
Anx
iety
Inve
ntor
y
•A
DH
Dpa
rtic
ipan
tssh
owed
asi
gnifi
cant
perf
orm
ance
decr
emen
t;de
crea
seof
corr
ect
hits
;an
din
crea
sed
reac
tion
time.
•A
DH
Dch
ildre
npe
rfor
med
wor
seth
anco
ntro
lson
both
the
Vir
tual
CPT
and
the
trad
ition
alco
mpu
teri
zed
CPT
.D
íaz-
Oru
eta
etal
.(2
014)
N�
57pa
rtic
ipan
tsw
ithA
DH
Dbe
twee
nth
eag
esof
6–16
year
s(2
6.3%
fem
ale)
.
Con
verg
ent
valid
ityst
udy
betw
een
both
aV
irtu
alC
PTan
dtr
aditi
onal
com
pute
rize
dC
PT.
Com
pare
dch
ildre
nun
derg
oing
med
ical
trea
tmen
tw
itha
non-
med
icat
edgr
oup
•C
onne
rs’
Con
tinuo
usPe
rfor
man
ceT
est
•W
ISC
-III
(Sel
ecte
dsu
btes
ts)
•B
oth
Vir
tual
CPT
and
the
trad
ition
alco
mpu
teri
zed
CPT
show
edsi
gnifi
cant
corr
elat
ions
.•
The
Vir
tual
CPT
(but
not
Con
ners
’C
PT)
was
able
todi
ffer
entia
tebe
twee
nA
DH
Dch
ildre
nw
ithan
dw
ithou
tph
arm
acol
ogic
altr
eatm
ent
(ina
ttent
ion,
impu
lsiv
ity,
proc
essi
ngsp
eed,
mot
orac
tivity
,an
dat
tent
ion
focu
s).
200 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Tab
le3
(con
tinu
ed)
Stud
ySa
mpl
eR
esea
rch
desi
gnT
radi
tiona
lT
ests
Res
ults
Gilb
oaet
al.
(201
1)N
�54
.29
with
Neu
rofib
rom
atos
isty
pe1
(NF1
);69
%fe
mal
e;m
ean
age
12.2
).25
cont
rols
72%
fem
ale;
mea
nag
e�
12.2
).
Com
pari
son
ofV
irtu
alC
PTan
dth
etr
aditi
onal
test
s.C
ross
sect
iona
lde
sign
.
•C
onne
rs’
Pare
ntR
atin
gSc
ales
-Rev
ised
•Si
gnifi
cant
diff
eren
ces
betw
een
the
NF1
and
cont
rols
onom
issi
oner
rors
and
com
mis
sion
erro
rsin
the
Vir
tual
CPT
.•
Poor
erpe
rfor
man
ceby
NF1
child
ren.
•Si
gnifi
cant
corr
elat
ions
betw
een
num
ber
ofta
rget
sco
rrec
tlyid
entifi
ed,
the
num
ber
ofco
mm
issi
oner
rors
,an
dre
actio
ntim
e.G
ilboa
etal
.(2
015)
N�
76.
41ch
ildre
nag
es8–
16w
ithac
quir
edbr
ain
inju
ry;
35ag
e-an
dge
nder
-m
atch
edco
ntro
ls.
Cro
ss-s
ectio
nal
desi
gn.
•T
est
ofE
very
day
Atte
ntio
nfo
rC
hild
ren
•W
echs
ler
Abb
revi
ated
Scal
eof
Inte
llige
nce
(Mat
rix
Rea
soni
ngan
dV
ocab
ular
y)•
Con
ners
Pare
ntR
atin
gSc
ales
-Rev
ised
•Si
gnifi
cant
betw
een
grou
pdi
ffer
ence
sfo
rnu
mbe
rof
targ
ets
corr
ectly
iden
tified
inth
eV
irtu
alC
PT.
•45
%of
the
child
ren
with
AB
Isu
ffer
edm
arke
dde
ficits
insu
stai
ned
atte
ntio
non
the
Vir
tual
CPT
.•
Atte
ntio
nal
perf
orm
ance
was
foun
dto
bere
late
dto
age,
age
atin
jury
/dia
gnos
isan
dtr
eatm
ent.
Iria
rte
etal
.(2
016)
N�
1282
child
ren
ages
6to
16.
Ano
rmat
ive
stud
y.Si
ngle
appl
icat
ion
with
desc
ript
ive
desi
gn.
Non
e•
Res
ults
wer
ecl
uste
red
into
diff
eren
tca
tego
ries
for
post
erio
ran
alys
is.
•D
iffe
renc
esby
age
and
gend
erw
ere
anal
yzed
,re
sulti
ngin
14gr
oups
,7
per
sex
grou
p.•
Dif
fere
nces
betw
een
visu
alan
dau
dito
ryat
tent
ion
wer
eal
soob
tain
ed. (tab
leco
ntin
ues)
201VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Tab
le3
(con
tinu
ed)
Stud
ySa
mpl
eR
esea
rch
desi
gnT
radi
tiona
lT
ests
Res
ults
Lal
onde
etal
.(2
013)
N�
38ad
oles
cent
sag
es13
–17
year
s.D
escr
iptiv
e/co
rrel
atio
nal
stud
yof
aV
irtu
alC
lass
room
Stro
opta
sk.
Con
verg
ence
valid
ityst
udy.
•D
elis
-Kap
lan
Exe
cutiv
eFu
nctio
nSy
stem
(Tra
ilM
akin
g,T
ower
,T
wen
tyQ
uest
ions
,V
erba
lFl
uenc
y,C
olor
-Wor
dIn
terf
eren
ce)
•B
ehav
ior
Rat
ing
Inve
ntor
yof
Exe
cutiv
eFu
nctio
n•
Chi
ldB
ehav
ior
Che
cklis
t
•V
R-S
troo
pta
skco
rrel
ated
with
D-K
EFS
and
BR
IEF.
•Pe
rfor
man
ceon
the
VR
-St
roop
task
was
corr
elat
edw
ithpa
per–
penc
ilSt
roop
task
.•
VR
-Str
oop
mor
eac
cura
tely
refle
cted
ever
yday
beha
vior
alfu
nctio
ning
.N
olin
etal
.(2
009)
8ch
ildre
nw
ithac
quir
edbr
ain
inju
ry,
ages
8to
12ye
ars.
Rep
eate
dm
easu
res
com
pari
sons
.•
VIG
ILC
ontin
uous
Perf
orm
ance
Tes
t•
No
diff
eren
cebe
twee
nth
eV
irtu
alC
PTan
dth
etr
aditi
onal
com
pute
rize
dC
PTon
tota
lof
omis
sion
s.•
Sign
ifica
ntly
mor
eco
mm
issi
ons
and
long
erre
actio
ntim
esin
the
Vir
tual
CPT
.N
olin
etal
.(2
012)
N�
5025
spor
ts-c
oncu
ssed
and
25m
atch
edco
ntro
lad
oles
cent
s.
Com
pari
son
ofth
etr
aditi
onal
CPT
and
Vir
tual
CPT
was
coun
terb
alan
ced
acro
sspa
rtic
ipan
ts.
•V
IGIL
Con
tinuo
usPe
rfor
man
ceT
est
•V
irtu
alC
PTsh
owed
grea
ter
sens
itivi
tyto
the
subt
leef
fect
sof
spor
tsco
ncus
sion
.•
The
spor
tsco
ncus
sion
grou
pre
port
edm
ore
sym
ptom
sof
cybe
rsic
knes
sth
anth
eco
ntro
lgr
oup.
Pars
ons
etal
.(2
007)
N�
2010
boys
diag
nose
dw
ithA
DH
Dan
d10
mat
ched
cont
rols
.
Inte
rgro
upco
mpa
riso
nof
part
icip
ants
with
AD
HD
and
norm
alco
ntro
ls.
•SW
AN
Beh
avio
rC
heck
list
•C
onne
rs’
CPT
II•
Stro
op•
Tra
ilM
akin
gte
sts
•N
EPS
Y(V
isua
lat
tent
ion,
desi
gnflu
ency
,ve
rbal
fluen
cy)
•W
ISC
-III
(Dig
itSp
an,
Cod
ing,
Ari
thm
etic
,V
ocab
ular
y)•
Judg
emen
tof
Lin
eO
rien
tatio
n
•A
DH
Dgr
oup
exhi
bite
dm
ore
omis
sion
erro
rs,
com
mis
sion
erro
rs,
and
over
all
body
mov
emen
tin
the
Vir
tual
CPT
.•
AD
HD
grou
pw
asm
ore
impa
cted
bydi
stra
ctio
nn
the
Vir
tual
CPT
.•
Vir
tual
CPT
was
corr
elat
edw
ithtr
aditi
onal
AD
HD
asse
ssm
ent
tool
s,be
havi
orch
eckl
ist,
and
trad
ition
alco
mpu
teri
zed
CPT
.
202 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Tab
le3
(con
tinu
ed)
Stud
ySa
mpl
eR
esea
rch
desi
gnT
radi
tiona
lT
ests
Res
ults
Pars
ons
and
Car
lew
(201
6)
Tw
ost
udie
sre
port
ed:
Stud
y1:
50un
derg
radu
ate
stud
ents
(mea
nag
e�
20.3
7;78
%fe
mal
e).
Stud
y1:
•W
echs
ler
Tes
tof
Adu
ltR
eadi
ng•
Vir
tual
Stro
opta
skw
asco
rrel
ated
with
trad
ition
alta
sks
and
elic
ited
anin
terf
eren
ceef
fect
sim
ilar
toth
ose
foun
din
clas
sic
Stro
opta
sks.
Stud
y2:
8st
uden
tsw
ithhi
ghfu
nctio
ning
autis
m(m
ean
age
�22
.88)
and
10m
atch
edco
ntro
ls.
Nor
mat
ive
stud
yco
mpa
ring
Vir
tual
Stro
opto
trad
ition
alta
sks.
•D
elis
-Kap
lan
Exe
cutiv
eFu
nctio
ning
Syst
em:
Col
orW
ord
Inte
rfer
ence
Tes
t•
Stro
opta
skfr
omA
utom
ated
Neu
rops
ycho
logi
cal
Ass
essm
ent
Met
rics
•D
urin
gth
edi
stra
ctio
nco
nditi
onof
the
Vir
tual
Stro
opA
SDgr
oup
perf
orm
ance
decl
ined
.
Stud
y2:
Cro
ssse
ctio
nal
desi
gn.
•W
echs
ler
Abb
revi
ated
Scal
eof
Inte
llige
nce-
Seco
ndE
ditio
nPo
llak
etal
.(2
009)
N�
37bo
ysag
es9–
17ye
ars,
with
(n�
20)
and
with
out
AD
HD
(n�
17).
Cro
ssov
erde
sign
com
pari
ngV
irtu
alC
lass
room
onre
gula
rco
mpu
ter
scre
en.
•T
est
ofV
aria
bles
ofA
ttent
ion
(TO
VA
)•
Shor
tFe
edba
ckQ
uest
ionn
aire
•A
DH
Dgr
oup
perf
orm
edle
ssw
ell
onal
lC
PTta
sks.
•V
irtu
alC
PTsh
owed
effe
ctsi
zes
sim
ilar
toth
eT
OV
A.
•Se
lf-r
epor
ted
pref
eren
cefo
rV
irtu
alC
PT.
Polla
ket
al.
(201
0)N
�27
16bo
ysan
d11
girl
s,w
ithcl
inic
aldi
agno
sis
ofA
DH
D.
Dou
ble-
blin
d,pl
aceb
o-co
ntro
lled,
cros
sove
rde
sign
.
Vir
tual
Cla
ssro
omon
regu
lar
com
pute
rsc
reen
Tes
tof
Var
iabl
esof
Atte
ntio
n(T
OV
A;
Gre
enbe
rg&
Wal
dman
,19
93)
•M
etilp
heni
date
(MPH
)re
duce
dom
issi
oner
rors
toa
grea
ter
exte
nton
the
VR
-C
PTco
mpa
red
toth
eno
VR
-CPT
and
the
TO
VA
,an
dde
crea
sed
othe
rC
PTm
easu
res
onal
lty
pes
ofC
PTto
asi
mila
rde
gree
.•
Chi
ldre
nra
ted
the
VR
-CPT
asm
ore
enjo
yabl
eco
mpa
red
toth
eot
her
type
sof
CPT
.
Not
e.A
DH
D�
Atte
ntio
n-de
ficit/
hype
ract
ivity
diso
rder
;ASD
�A
utis
mSp
ectr
umD
isor
der;
BA
SC�
Beh
avio
rA
sses
smen
tSys
tem
for
Chi
ldre
n;C
PT�
Con
tinuo
usPe
rfor
man
ceT
est;
DK
EFS
�D
elis
–Kap
lan
Exe
cutiv
eFu
nctio
nSy
stem
;B
RIE
F�
Beh
avio
rR
atin
gIn
vent
ory
ofE
xecu
tive
Func
tion;
NE
PSY
�ab
brev
iatio
nof
the
term
“neu
rops
ycho
logy
”;N
F1�
Neu
rofib
rom
atos
isty
pe1;
TO
VA
�T
est
ofV
aria
bles
ofA
ttent
ion;
VR
�V
irtu
alR
ealit
y;W
ISC
-III
�W
echs
ler
Inte
llige
nce
Scal
efo
rC
hild
ren.
203VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
testing needs. In addition, there are a number ofauditory distractors that can be adjusted, such asthe sounds of vehicles passing by and ambientclassroom noise. Other aspects of the virtualenvironment can also be adjusted, includingseating position of the client, the number ofvirtual students, and the sex of the teacher(Rizzo et al., 2006).
The case of G. T. A psychologist receivesa referral for assessment of attentional prob-lems. The client is G. T., a 10-year-old biracialmale who is experiencing difficulty sustainingattention during class and listening to instruc-tions. The information from the parent andteacher versions of the behavior rating scalesare somewhat inconsistent. His teacher endorsesa number of attention problems for G. T. such asfrequent forgetting of instructions before com-pleting schoolwork; G. T. fidgets often; andG. T. inappropriately leaves his seat duringclass. According to his teacher, G. T. also ap-pears to be easily distracted during teacher pre-sentations of class material. However, the par-ent rating reveals only minor difficulties withattention.
As part of a larger psychological battery,G. T. was assented to complete Virtual Class-room CPT and Stroop tasks. The virtual class-room was modified to closely reflect his typ-ical classroom environment by including thesame number of students, assigned desk loca-tion, and same sex of the teacher. While im-mersed in the Virtual Classroom, G. T. sat ata desk in the middle of the virtual classroomthat corresponded with where he sits in hisreal classroom at school. During the Stroopand CPT tasks, head movement tracking sen-sors logged G. T.’s frequent head movementsas he looked away from the blackboard toobserve vehicles passing by outside and vir-tual students moving around in the room.Moreover, the movement sensors logged hisbody movement and revealed that he was veryphysically active during the tasks. He showedpoor performance (compared with norm-reference groups) on the cognitive tasks.
The results of testing indicated G. T. wasshowing several behavioral symptoms ofADHD. Review of the data showed that symp-toms of inattention, such as maintaining atten-tion, are likely because of being easily derailedby classroom and outdoor distractors. This alsoaffected his ability to follow through with in-
structions and listen to the virtual teacher. Ofparticular note are the symptoms of hyperactiv-ity that were directly recorded. He frequentlymoved his legs and arms and squirmed aroundin his seat. He had difficulty sitting still andremaining seated.
Several recommendations were made basedon these observations. G. T. performed betterwhen placed in the front of the virtual class-room, so it was recommended that this adjust-ment be made in his actual classroom. It wasrecommended that classroom windows remainclosed if possible to reduce noise from outsidesources. Additionally, it was recommended thatpeople entering and exiting the room be reducedas much as possible. Finally, the specific dis-tractors were provided to the parent to sharewith G. T.’s teacher, so when these distractorsoccur the teacher will know it is necessary toredirect G. T.’s attention.
In summary, the Virtual Classroom hasseveral advantages as opposed to traditionalpen-and-paper testing. It represents the typi-cal classroom in which students with ADHDand autism often struggle to maintain atten-tion and engagement, which increases ecolog-ical validity compared with traditional testingmethods. In addition, functional impairmentscan be directly observed, which improves thequality of recommendations. The VirtualClassroom provides a record of which distrac-tions caused the client to look away from theboard and how distracting they were (howoften the client looked away). These data caninform recommendations to parents andteachers to remove specific distractors fromthe learning environment. The impact of plac-ing the client in different locations in thevirtual classroom can also be assessed. Theresults of such testing allow for recommen-dations based on direct observation aboutwhere to seat the client in the classroom sothey can perform at their best. The virtualclassroom records the head and body move-ments of the child in real time. Thus, the levelof activity is accurately recorded without thepotential bias that may influence parent andteacher reports (Pas & Bradshaw, 2014; Sayal& Taylor, 2005). Collecting and recordingthese body movement data in an ecologicallyvalid environment rather than relying on par-ent and teacher reports may be a better way ofassessing the “H” in ADHD.
204 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Virtual Shopping Tasks
Another virtual environment approach thatmay be of interest to clinicians is the use of asimulated shopping environment. A number ofthese virtual shopping environments have beendeveloped to assess execution of everyday be-haviors in a virtual shopping center (see Table4). These virtual shopping environments havebeen developed in a manner that allows thepsychologist to systematically vary the informa-tion load (that affects goal maintenance). Forexample, the Virtual Environment GroceryStore (VEGS) uses multiple adaptive trials inthe assessment procedure by creating a pool of“multiple task assignments,” empirically deter-mining their baseline difficulty, and then addingconditions in the environment that affect base-line task difficulty via the manipulation of thedensity of items on shelves, the similarity ofpackaging, and the intensity and types of real-istic irrelevant distractions (e.g., loudness/typeof music in the background and loudspeakerannouncements). The VEGS platform offers arange of difficulties that may be used to makethe tasks sufficiently complex to avoid floor andceiling effects. The use of a simulated environ-ment allows older adults who may be physicallyor behaviorally impaired to be safely assessed.This would not be possible with the traditionalMET, which requires these tasks to be com-pleted in a real-world shopping environment(Parsons, McPherson, & Interrante, 2013).
The case of L. S. A psychologist receives areferral for assessment of memory problems.The client is L. S., a 58-year-old married, His-panic female who sought assessment after mem-ory difficulties appear to have resulted in herfrequently misplacing items around the houseand forgetting to attend appointments. She alsocontinually asks family members to help herwith shopping. Furthermore, she reported that attimes she would get up and go to her kitchenonly to feel confused because she could notremember what she intended to do in the kitch-en. She had a diagnosis of osteoarthritis, whichdecreased movement in her right knee, and de-nied any previous history of head-injury, sub-stance abuse, or psychiatric illness. Data col-lected during the clinical interview suggestedearly onset Alzheimer’s disease, but resultsfrom the neuropsychological assessment wereinconsistent.
To assess her cognitive functioning, L. S. wasasked to complete the VEGS as part of herlarger battery of tests. While immersed in theVEGS, L.S. often had difficulty with sustainedconcentration, which was demonstrated by theindirect route she took to the pharmacy andwhen shopping for items on the shopping list.She also had difficulty remembering how muchmoney was budgeted for purchasing items andcontinually asked her psychologist what shewas supposed to do next. She also performedpoorly on various aspects of the prospectivememory tasks. For example, she had to be re-minded to go to the coupon machine after 5 minof shopping. She also had difficulty recallingthe shopping items and frequently clicked onthe shopping list for a reminder. These difficul-ties were notably amplified when distractorswere present (e.g., announcements over the loudspeaker; people shopping in the same aisle asher).
These results suggest that L.S. is demonstrat-ing behavioral symptoms of suggestive of earlyonset Alzheimer’s disease, including difficultywith concentration, problem solving, and mem-ory for recently learned information. With per-mission from L.S., her caregiver was shown avirtual replay of her activities in the VEGS. Itrevealed areas of difficulty and supported a rec-ommendation that L. S. be provided assistancewith tasks that require sustained attention suchas shopping or household chores. The VEGSreport also revealed that she needed assistancewith tasks that require problem-solving skillssuch as budgeting. It was recommended thatfamily members provide L. S. with frequentreminders of newly learned information such asdates and changes in routines on an easily ac-cessible calendar and in-person.
Virtual Reality Apartment
Virtual Apartments are simulation environ-ments that allow clinicians to see how the clientbehaves in their home environment. The pur-pose of the Virtual Apartment is to assess po-tential cognitive deficits using a simulatedapartment. While a number of virtual apartmentenvironments have emerged, there is a consis-tent emphasis upon (a) ecologically valid rep-resentations of the client’s everyday activities ina living situation; (b) presentation of either tra-ditional cognitive tasks (e.g., CPT; Stroop) or
205VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Tab
le4
Rec
ent
Fun
ctio
n-L
edSt
udie
s(W
ithi
nP
ast
10Y
ears
)U
sing
aV
irtu
alSt
ore
Stud
ySa
mpl
eR
esea
rch
desi
gnT
radi
tiona
lte
sts
Res
ults
Atk
ins
etal
.(2
015)
N�
44he
alth
yyo
ung
and
N�
39he
alth
yol
d.
The
two
grou
psw
ere
com
pare
don
virt
ual
and
trad
ition
alm
easu
res.
•M
AT
RIC
SC
onse
nsus
Cog
nitiv
eB
atte
ry•
UC
SDPe
rfor
man
ce-B
ased
Skill
sA
sses
smen
t-B
rief
•H
opki
nsV
erba
lL
earn
ing
Tes
t•
Bri
efV
isua
lM
emor
yT
est
•E
ach
VR
outc
ome
mea
sure
disp
laye
dsi
gnifi
cant
age-
rela
ted
diff
eren
ces.
•T
radi
tiona
lm
easu
res
ofco
gniti
vefu
nctio
ning
wer
esi
gnifi
cant
lyas
soci
ated
with
VR
perf
orm
ance
acro
ssag
egr
oups
.C
anty
etal
.(2
014)
N�
30se
vere
TB
Ivs
.N
�24
heal
thy
cont
rols
.
Seve
reT
BI
(n�
30)
vs.
heal
thy
cont
rol
(n�
24).
•L
exic
alD
ecis
ion
Pros
pect
ive
Mem
ory
Tas
k•
Hop
kins
Ver
bal
Lea
rnin
gT
est
•T
rail
Mak
ing
Tas
k•
Con
trol
led
oral
wor
das
soci
atio
n•
Hay
ling
Sent
ence
Com
plet
ion
•L
ette
rN
umbe
rSe
quen
cing
•Sy
dney
Psyc
hoso
cial
Rei
nteg
ratio
nSc
ale
•V
irtu
alsh
oppi
ngpe
rfor
man
cedi
ffer
entia
ted
betw
een
TB
Ipa
tient
san
dth
eco
ntro
lgr
oup.
•M
easu
res
ofpr
ospe
ctiv
em
emor
y,ne
uroc
ogni
tive
func
tioni
ng,
and
psyc
hoso
cial
func
tioni
ngw
ere
sign
ifica
ntly
asso
ciat
edw
ithV
irtu
alsh
oppi
ngpe
rfor
man
ceam
ong
TB
Ipa
tient
s.C
arel
li,M
orga
nti,
Wei
ss,
Kiz
ony,
&R
iva
(200
8)
N�
24he
alth
yad
ults
.D
escr
iptiv
est
udy.
n/a
•V
irtu
alsu
perm
arke
tm
aybe
aus
eful
tool
inex
ecut
ive
asse
ssm
ent,
part
icul
arly
due
toits
tem
pora
lan
dac
cura
cym
easu
res.
Ere
z,W
eiss
,K
izon
y,&
Ran
d(2
013)
N�
20ch
ildre
nw
ithT
BI
vs.
N�
20he
alth
yco
ntro
ls.
Com
pari
son
ofT
BI
(n�
20)
tohe
alth
yco
ntro
ls(n
�20
)in
avi
rtua
lm
all.
n/a
•O
utco
me
mea
sure
sof
the
VM
all
succ
essf
ully
diff
eren
tiate
dbe
twee
nch
ildre
nw
ithT
BI
and
heal
thy
cont
rols
.Jo
sman
etal
.(2
006)
N�
20st
roke
patie
nts.
Com
pari
son
ofV
irtu
alSu
perm
arke
tto
trad
ition
alte
sts.
Beh
avio
ral
Ass
essm
ent
ofD
ysex
ecut
ive
Synd
rom
e(B
AD
S)•
Vir
tual
shop
ping
outc
ome
mea
sure
s(#
item
spu
rcha
sed,
#co
rrec
tac
tions
,du
ratio
npa
uses
/sto
ps)
wer
esi
gnifi
cant
lyas
soci
ated
with
the
key
sear
chsu
btes
tof
the
BA
DS.
Josm
anet
al.
(201
4)N
�24
stro
kean
d24
mat
ched
cont
rols
.C
ompa
riso
nof
stro
ke(n
�24
)an
dco
ntro
l(n
�24
)pa
rtic
ipan
ts.
Beh
avio
ral
Ass
essm
ent
ofD
ysex
ecut
ive
Synd
rom
e•
Res
ults
reve
aled
sign
ifica
ntdi
ffer
ence
sin
exec
utiv
efu
nctio
ning
betw
een
post
-str
oke
patie
nts
and
the
cont
rol
grou
pon
virt
ual
shop
ping
outc
ome
mea
sure
s.•
Vir
tual
shop
ping
outc
ome
mea
sure
sw
ere
also
sign
ifica
ntly
asso
ciat
edw
ithth
eB
AD
S.
206 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Tab
le4
(con
tinu
ed)
Stud
ySa
mpl
eR
esea
rch
desi
gnT
radi
tiona
lte
sts
Res
ults
Josm
an,
Sche
nird
erm
an,
Klin
ger,
and
Shev
il(2
009)
N�
30sc
hizo
phre
nia
and
N�
30he
alth
yco
ntro
ls.
Com
pari
son
ofsc
hizo
phre
nia
(n�
30)
tohe
alth
yco
ntro
l(n
�30
)pe
rfor
man
ce.
Beh
avio
ral
Ass
essm
ent
ofD
ysex
ecut
ive
Synd
rom
e•
Vir
tual
shop
ping
outc
ome
mea
sure
sw
ere
sens
itive
todi
ffer
ence
sin
exec
utiv
efu
nctio
ning
betw
een
schi
zoph
reni
apa
tient
san
dco
ntro
lsan
ddi
ffer
entia
ted
betw
een
patie
nts
with
diff
erin
gle
vels
ofex
ecut
ive
func
tion.
•V
irtu
alsh
oppi
ngou
tcom
em
easu
res
wer
ene
gativ
ely
rela
ted
tosy
mpt
oms
ofsc
hizo
phre
nia
and
the
BA
DS.
Kan
get
al.
(200
8)N
�20
stro
kean
dN
�20
mat
ched
cont
rols
.C
ompa
riso
nof
Stro
ke(n
�20
)an
dco
ntro
l(n
�20
)pe
rfor
man
ce.
n/a
•V
irtu
alsh
oppi
ngou
tcom
em
easu
res
cons
iste
ntly
diff
eren
tiate
dbe
twee
nst
roke
patie
nts
(uni
late
ral
brai
nle
sion
)an
dth
eco
ntro
lgr
oup.
Klin
ger,
Che
min
,L
ebre
ton,
and
Mar
ié(2
006)
N�
5Pa
rkin
son’
san
dN
�5
age-
mat
ched
heal
thy
cont
rols
.
Park
inso
n’s
(n�
5)to
age-
mat
ched
heal
thy
cont
rol
(n�
5)pe
rfor
man
ce.
n/a
•V
irtu
alsh
oppi
ngou
tcom
em
easu
res
reve
aled
that
patie
nts
wal
ked
asi
gnifi
cant
lylo
nger
dist
ance
.•
Thi
ssu
gges
tsth
atth
eV
irtu
alSu
perm
arke
tm
aybe
aus
eful
tool
for
asse
ssin
gco
gniti
vepl
anni
ngin
patie
nts
with
Park
inso
n’s.
Oka
hash
iet
al.
(201
3)N
�10
brai
nda
mag
ean
dN
�10
age-
mat
ched
cont
rols
and
N�
10he
alth
y-ol
der
adul
tsco
mpa
red
toN
�10
heal
thy-
youn
g.
Com
pari
son
ofbr
ain
dam
age
(n�
10)
perf
orm
ance
toag
e-m
atch
edco
ntro
l(n
�10
);H
ealth
y-ol
d(n
�10
)an
dhe
alth
y-yo
ung
(n�
10).
Beh
avio
ral
Ass
essm
ent
ofD
ysex
ecut
ive
Synd
rom
e•
Perf
orm
ance
onth
evi
rtua
lsh
oppi
ngta
skw
assi
gnifi
cant
lyas
soci
ated
with
conv
entio
nal
cogn
itive
asse
ssm
ents
.•
Old
erpa
rtic
ipan
tsan
dpa
tient
sw
ithbr
ain
dam
age
scor
edsi
gnifi
cant
lyw
orse
onth
evi
rtua
lsh
oppi
ngta
sks.
Ran
d,B
asha
-Abu
Ruk
an,
Wei
ss,
and
Kat
z(2
009)
N�
9po
st-s
trok
e(n
�9)
and
N�
20he
alth
y-yo
ung
and
N�
20he
alth
yag
ing
Com
pari
son
ofpo
st-s
trok
e(n
�9)
vs.
heal
thy-
youn
g(n
�20
)vs
.he
alth
y-ol
d(n
�20
).
Beh
avio
ral
Ass
essm
ent
ofD
ysex
ecut
ive
Synd
rom
e(Z
ooM
apsu
btes
t)•
Vir
tual
shop
ping
task
ssu
cces
sful
lydi
ffer
entia
ted
betw
een
all
thre
egr
oups
.•
Vir
tual
shop
ping
outc
ome
mea
sure
sw
ere
sign
ifica
ntly
asso
ciat
edw
ithZ
ooM
apsu
btes
t.R
and,
Kat
z,an
dW
eiss
(200
7)N
�14
stro
kean
dN
�23
heal
thy
cont
rol
child
ren
and
N�
44yo
ung
adul
ts;
and
N�
26ol
der
adul
ts.
Com
pari
son
ofst
roke
(n�
14)
vs.
heal
thy
cont
rol
(chi
ldre
nn
�23
;yo
ung
adul
tsn
�44
;ol
der
adul
tsn
�26
).
n/a
•V
irtu
alsh
oppi
ngtim
esu
cces
sful
lydi
ffer
entia
ted
betw
een
post
-str
oke
patie
nts
and
the
cont
rol
grou
psw
ithpo
st-s
trok
epa
tient
sta
king
sign
ifica
ntly
long
erto
com
plet
eth
esh
oppi
ngta
sk.
(tab
leco
ntin
ues)
207VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Tab
le4
(con
tinu
ed)
Stud
ySa
mpl
eR
esea
rch
desi
gnT
radi
tiona
lte
sts
Res
ults
Ras
pelli
etal
.(2
012)
N�
9st
roke
and
N�
10he
alth
y-yo
ung;
and
N-
10he
alth
yag
ing.
Com
pari
son
ofst
roke
(n�
9)vs
.he
alth
y-yo
ung
(n�
10)
vs.
heal
thy-
old.
•T
est
ofE
very
day
Atte
ntio
n•
Iow
aG
ambl
ing
Tas
k•
Stro
opT
est
•Si
gnifi
cant
diff
eren
ces
wer
efo
und
betw
een
all
thre
egr
oups
ontw
oou
tcom
em
easu
res
ofth
evi
rtua
lsh
oppi
ngta
sk(i
.e.,
time,
erro
rs).
•T
heT
EA
(but
not
the
IGT
orSt
roop
)w
assi
gnifi
cant
lyas
soci
ated
with
virt
ual
shop
ping
perf
orm
ance
inpo
st-s
trok
epa
rtic
ipan
ts.
Wer
ner,
Rab
inow
itz,
Klin
ger,
Kor
czyn
,an
dJo
sman
(200
9)
N�
10m
ildco
gniti
veim
pair
men
tan
dN
�30
mat
ched
cont
rols
.
Com
pari
son
ofM
CI
(n�
30)
vs.
cont
rol
(n�
40).
Beh
avio
ral
Ass
essm
ent
ofD
ysex
ecut
ive
Synd
rom
e•
4of
the
8vi
rtua
lsh
oppi
ngou
tcom
em
easu
res
wer
eas
soci
ated
with
perf
orm
ance
onth
eB
AD
S.•
Vir
tual
shop
ping
perf
orm
ance
succ
essf
ully
diff
eren
tiate
dbe
twee
nM
CI
patie
nts
and
the
cont
rol
grou
p.Z
ygou
ris
etal
.(2
015)
N�
34m
ildco
gniti
veim
pair
men
tan
dN
-20
mat
ched
cont
rols
.
Com
pari
son
ofM
CI
(n�
34)
vs.
cont
rol
(n�
21).
•M
ini
Men
tal
•R
ey-O
ster
riet
hC
ompl
exFi
gure
Tes
t•
Rey
Aud
itory
Ver
bal
Lea
rnin
gT
est
•R
iver
mea
dB
ehav
iour
alM
emor
yT
est
•T
est
ofE
very
day
atte
ntio
n•
Tra
ilM
akin
gte
st•
Func
tiona
lR
atin
gSc
ale
for
Sym
ptom
sof
Dem
entia
•Fu
nctio
nal
Cog
nitiv
eA
sses
smen
tSc
ale
•V
irtu
alsh
oppi
ngpe
rfor
man
cew
asm
oder
atel
yco
rrel
ated
with
trad
ition
alne
urop
sych
olog
ical
test
s.•
Vir
tual
shop
ping
perf
orm
ance
was
able
todi
ffer
entia
tebe
twee
nM
CI
patie
nts
and
the
cont
rol
grou
p;ho
wev
er,
itw
asun
able
todi
ffer
entia
teM
CI
subt
ypes
.
Not
e.B
AD
S�
Beh
avio
ralA
sses
smen
tof
Dys
exec
utiv
eSy
ndro
me;
MC
I�
Mild
Cog
nitiv
eIm
pair
men
t;T
EA
�T
esto
fE
very
day
Atte
ntio
n;IG
T�
Iow
aG
ambl
ing
Tas
k;T
BI
�T
raum
atic
Bra
inIn
jury
;U
CSD
�U
nive
rsity
ofC
alif
orni
a,Sa
nD
iego
;V
R�
Vir
tual
Rea
lity;
VM
all
�V
irtu
alM
all.
208 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
everyday activities (without embedded cogni-tive stimuli) in the environment; (c) low andhigh distraction conditions; and (d) logging ofbehavioral metrics (e.g., ambient body move-ment, head/eye gaze).
The ClinicaVR team has extended the virtualclassroom paradigm to a virtual apartment thatsuperimposes construct-driven stimuli (e.g.,Stroop and CPT) onto a large TV. The VirtualApartment Bimodal Stroop (VABS) is a 9.6-mintask. Participants are seated in the living room, infront of a flat-screen TV. A kitchen is located tothe left of the TV and a window is located to theright of the TV. The task builds on the unimodal(visually mediated) Stroop and measures cogni-tive interference using reaction time (RT), com-mission errors and omission errors, and RT vari-ability. The VABS extends the traditional Stroopparadigm via the inclusion of bimodal (auditoryand visually mediated) stimuli.
During the task, distracters appear in differentfield of view locations in the environment. Somedistracters are audio—visual: School Bus passingon the street and SUV viewed through window onthe right; iPhone ringing and vibrating on the table(in front of participant); Toy Robot moving andmaking noise on the floor (center). Auditory dis-tractors included: Crumple Paper (left); Drop Pen-cil (left); Doorbell (left); Clock (left) VacuumCleaner (right); Jack Hammer (right); Sneeze(left) Jet Noise (center). Visual distractors in-cluded: Paper airplane (flying from left to right infront of the participant), Woman walking in thekitchen (center). That condition was designed toassess RTs (simple and complex), selective atten-tion (matching the auditory and visual stimuli),and external interference control (environmentaldistracters).
In a preliminary study, Henry and colleagues(2012) with 71 healthy adult participants foundthat the VR-Apartment Stroop is capable of elic-iting the Stroop effect with bimodal stimuli. Initialvalidation data also suggested that measures of theVR-Stroop significantly correlate with measuresof the Elevator counting with distracters (rangingfrom .38 to .62), the Continuous PerformanceTask (CPT-II; ranging from .32 to .42), and theStop-it task (ranging from .37 to .39). Resultsfrom regression indicated that commission errorsand variability of RTs at the VR-ApartmentStroop were significantly predicted by scores ofthe Elevator task and the CPT-II. These prelimi-nary results suggest that the VR-Apartment Stroop
is an interesting measure of cognitive and motorinhibition for adults.
Virtual reality apartment medicationmanagement assessment. A different Vir-tual Apartment has been developed by Kurtz,Baker, Pearlson, and Astur (2007) to assessesmedication management skills among thosewith schizophrenia. The Virtual RealityApartment Medication Management Assess-ment (VRAMMA) lasts a maximum of 23min. The environment consists of a four-bedroom apartment containing a living room,bedroom, kitchen, and bathroom. During thepractice phase, clients are asked to do a numberof tasks that help build familiarity with thevirtual environment: using a joystick to navi-gate to the living room and turn on the TV,checking the time on the interactive clock ontop of the TV, turning on a light in the bedroom,checking the medication reminder post-it notein the kitchen (with a different prescription thanthe actual trial) after turning on the light andturning off the stove, and opening the medicinecabinet in the bathroom and taking out pills.During the testing phase clients start in theliving room and a message is displayed thatgives them the medications to take and the timethey must be taken. They must then use all theitems from the practice phase to take the correctmedication at the correct time (in 15 min). Sev-eral auditory distractors that simulate real-lifedistractions were included in the virtual apart-ment: a phone ringing, a doorbell ringing, a dogbarking, and a police siren. These auditory dis-tractors are introduced every 3 min. Significantevents, such as turning on lights or checking theclock, and the location and movement of theclient are recorded. The variables recorded dur-ing testing include: quantitative errors, qualita-tive errors, time discrepancy, total distance trav-eled, clock checks, and reminder note checks.The VRAMMA is able to distinguish betweenhealthy controls and those with schizophrenia,who perform worse. Kurtz et al. (2007) con-ducted a validation study with 25 schizophreniapatients and 18 matched healthy controls. Theyfound patients with schizophrenia made morequantitative errors concerning the number ofpills taken (p � .001), were less likely to takepills at the correct time (p � .01), and checkedthe clock less often (p � .001).
The VRAMMA has several advantages com-pared with traditional testing. The VRAMMA
209VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
shows good convergent validity with the Medica-tion Management Ability Assessment (Pattersonet al., 2002), a role-play task involving medicationmanagement that typically would be administeredin an office setting. Clinicians are able to directlyobserve client behavior in a more ecologicallyvalid environment rather than relying of self-report measures, which may be biased by a lack ofinsight from clients with schizophrenia (seeMintz, Dobson, & Romney, 2003 for a meta-analysis) or cognitive and emotional disruptions(Anticevic, Schleifer, & Youngsun, 2015). TheVRAMMA approximates an environment inwhich many clients would likely need to managemedication and includes common tools needed tocarry out this activity, such as a reminder note anda clock. It also allows clinicians to assess when theprocess of performing this task is disrupted andmake specific recommendations from these obser-vations.
The case of J. A. A psychologist receives areferral for assessment of concentration, and aresurgence of visual and auditory hallucina-tions. The client is J. A., a 23-year-old, singleWhite male who was diagnosed with schizo-phrenia in 2013 after hospitalization for a singleepisode of psychosis. He was released from thepsychiatric hospital after 1 month and given aprescription for antipsychotic medication. J. A.continued meeting regularly with a psychiatrist.Several months after being released from thepsychiatric hospital, J. A. began experiencingdifficulty with concentration, and a resurgenceof visual and auditory hallucinations. To ascer-tain whether medication adherence was at issue,J. A. was asked to complete the VRAMMA. Hetook fewer than the required number of pills andtook the pills 3 min after the 15-min time lapsehad expired. While in the virtual apartment,J. A. traveled around the apartment longer thannecessary to complete the task and was dis-tracted by the sound of the barking dog.
Based on these results, it appears that J. A. ishaving difficulty taking the correct number ofpills and using cues in the environment to takepills at the correct time. Positive symptoms arerelated to distance traveled in VRAMMA(Kurtz et al., 2007). J. A. traveled a significantdistance in the virtual apartment, which showshis medication adherence is low. It was recom-mended that J. A. use an automated pill dis-penser that holds the correct number of pills foreach day of the week, reminds J. A. to take
medication if not taken at a scheduled time, andsends a notification to his psychiatrist if he stopstaking medication. J. A. was advised to reduceauditory distractions at home by turning on afan to reduce noise from outside sources.
Considerations in the Adoption of VirtualReality Technologies
As can be seen there are some specific casesin which virtual reality-based psychological as-sessments may offer the psychologist with eco-logically valid assessments of day-to-day activ-ities. That said, there are a number ofconsiderations that go into the decision to addnew technologies to one’s battery of tests (seeStandards for Educational and PsychologicalTesting (American Educational Research Asso-ciation, American Psychological Association, &National Council on Measurement in Educa-tion, 2014). While some of these issues areconcerns related to the current generation ofvirtual reality based assessments, others reflectoutdated concerns from an earlier generation ofplatforms., for example, it used to be the casethat the equipment needed to conduct such as-sessments was bulky and expensive. Recent ad-vances in virtual reality technology have madethe use of simulations in assessment more fea-sible and affordable. Smaller, easier-to-useequipment and reduced cost make virtual envi-ronment assessment a practical tool psycholo-gists can use to gather precise functional dataand to provide customized recommendations toclients (Bohil et al., 2011). Furthermore, vali-dated virtual environments with automatedstimulus presentation, data capture, and scoringare emerging that include sample characteristicsfor norm-referenced assessment. However,there are other concerns that continue to thisday. The dearth of established guidelines for thedevelopment, administration and interpretationof these assessments could lead to importantpsychometric pitfalls. At minimum, all virtualreality-based psychological assessments musthave (and many now do) standardized instruc-tions for administration and methods for scoringand interpreting test results provided in a testmanual. While some virtual environments arebeing designed for limit testing, more workneeds to be done in this area.
210 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Need for VE-Based NeuropsychologicalAssessments to Be Sufficiently Standardized
While the use of virtual environments forassessment is an emerging area of application,adoption will require substantial research anddevelopment to establish acceptable psycho-metric properties and clinical utility. Although areview of VR therapies has revealed statisticallylarge effects on a number of affective domains(Parsons & Rizzo, 2008), findings must be in-terpreted with caution given that some VR stud-ies do not include control groups, and many arenot randomized clinical trials, limiting the con-fidence that the enhancements were caused bythe VR intervention. An important resolution toclinical heterogeneity of outcome measures invirtual environment research is the standardiza-tion of outcomes and the measures used to as-sess these outcomes. The selection of outcomemeasures for standardization need to be relevantto the client’s treatment and health status as wellas psychometrically sound. Another pressingneed among psychologists is the identificationof VE-based assessments that reflect relevantunderlying cognitive and behavioral capacitiesfor assessments of varying degrees of psycho-logical deficits. VE-based assessments mustdemonstrate relevance beyond that which isavailable through simpler means of assessment.As such, there is specific need for VE-basedassessments to be sufficiently standardizedwithin the range and nature of responses avail-able to participants within the virtual environ-ment to allow for reliable measurement.
VE-based assessment studies have oftensought to establish construct validity by dem-onstrating significant associations between vir-tual environments and paper-and-pencil assess-ments (e.g., virtual classroom assessments). Inthe area of function led assessment, multiplecognitive domains may be involved in the sim-ulation of real-world tasks, and associationswith traditional construct driven tests may benecessarily lower than is typically desired toestablish construct validity. In this context, thedegree to which a VE-based model using afunction led approach accurately predicts rele-vant real-world behavior may be more impor-tant than large-magnitude associations with tra-ditional construct driven paper-and-pencil tests(e.g., virtual shopping tasks).
Issues for Use of Virtual Environments inSpecific Patient Populations
In addition to psychometric and technical is-sues, clinicians, researchers, and policymakerswill need to scrutinize emerging VE-based as-sessments to ensure adherence to legal, ethical,and human safety guidelines. The matching ofspecific technologies to the needs and capacitiesof the client will also require careful consider-ation by psychologists. How will virtual envi-ronments be experienced by certain clinicalpopulations? In pediatrics and geriatrics, humanguidance is critical for safeguarding the client’sfull comprehension of assessment use and in-struction. Geriatric clients in particular may findadjusting to virtual platforms, on the whole,difficult (Miller et al., 2014).
Although virtual environments have beensuccessfully applied to the study of age differ-ences in spatial navigation among both healthyand demented elderly, virtual environment-based tasks may be complicated by visual, au-ditory, or motor impairment. In comparisonwith younger controls, aging patients may per-form more poorly on virtual environment-basedtasks simply because of the normative agingprocess or because of lack of experience withcomputers. Maximum effort should be exertedto ensure equitability in sensorimotor capacitiesbetween younger and older adult subjects. Asystematic review by Miller et al. (2014) intro-duced concern regarding the feasibility ofhome-use VE and gaming systems for physicalrehabilitation of older adults. Such systemscould be therapeutic to existing physical impair-ment or could be preventative. A main limita-tion is the low quality of studies investigatingthe effectiveness of these systems in older adultpopulations. Furthermore, some studies citedheightened fall risk, overexertion, and muscu-loskeletal irritation. There is need for more rig-orous research methods including more consis-tent and strenuous reporting of exercise dosagesand adherence. Moffatt (2009) suggests a num-ber of helpful methodological practices in as-sessing older adults in research studies of nav-igation skills, including: (a) allowing agingpatients to practice and ensure maximum famil-iarization with the computer platform, (b) in-cluding measures of computer experience, vi-sual ability, and motor function, and (c)including assessments requiring the same sen-
211VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
sorimotor capacities, but not physical naviga-tion.
A potential barrier to adoption of virtual re-ality technology among clinicians is concernsabout the ability of older adults to use thistechnology. Wandke, Sengpiel, and Sönksen(2012) have outlined several pervasive mythsabout older adults and human-computer inter-action. They state that the myth that older adultsare not interested in computers is not true. Dyckand Smither (1994) conducted a survey ofadults over age 55 and found these older adultswere less computer anxious and had more pos-itive attitudes about computers than adults un-der 30. Wandke et al. (2012) also address themyth that “you can’t teach an old dog newtricks.” While there are decreases in brain plas-ticity in older age, this does not mean thatlearning ceases. It may be the case that someolder adults have had frustrating experiencesthat lead to giving up on learning how to usenew technologies. However, this effect shouldnot be overgeneralized, as older adults are ofteninterested in using newer technologies (Sayago,Sloan, & Blat, 2011).
When assessing older adults for memory per-formance, it is important to avoid invoking stereo-type threat. Chasteen, Bhattacharyya, Horhota,Tam, and Hasher (2005) found that invoking ste-reotype threat about memory abilities in olderadults harms performance on memory tasks, par-ticularly when these adults are aware their mem-ory is being assessed. Subtle and unambiguousage-related stereotypes have also been found toinfluence older adults’ performance on a numberof cognitive and physical tasks (see Lamont,Swift, & Abrams, 2015 for a meta-analysis), suchas and map learning (Meneghetti, Muffato, Suit-ner, De Beni, & Borella, 2015) driving a car(Lambert et al., 2016), and hand grip strength(Swift, Lamont, & Abrams, 2012). It is possiblethat assessment in a virtual environment such asthe VEGS could reduce stereotype threat by ob-scuring the true purpose of the task.
Simulation technology may also be problematicfor individuals with autism spectrum disorder.Given pronounced sensory issues commonlyfound in this population, the head mounted dis-play or even the graphical interface may be expe-rienced as intolerable. Moreover, there is con-cerned that too intense a stimulus presentationmay aggravate sensory processing difficulties.This is an important concern though there is no
evidence from two different studies with studentsdiagnosed with autism that they experience nega-tive effects over and above those experienced bystudents without autism (Parsons & Carlew, 2016;Wallace et al., 2010). However, while these twostudies tend to suggest that negative effects wereself-reported as low, they involved screen-basedvirtual environments. As we adopt newer andmore immersive technologies (i.e., HMDs) it isimportant to consider the potential negative effects(i.e., dizziness, sickness, and displacement) to en-sure that wearable technologies (e.g., HMDs) canprovide an acceptable space for children to use;especially children with disabilities. With thissaid, there is some evidence that suggests childrendo not experience HMDs any more negativelythan screen-based media (Peli, 1998). Althoughmore work is needed in this area, these findingssupport the potential of VR technology for con-tinued greater approximations (Bohil et al., 2011)of cognitive processes in the real world.
Furthermore, individuals with severe psychi-atric conditions that cause limited self-aware-ness, high suggestibility, and/or an altered senseof reality (e.g., hallucinations, delusions) mayrespond undesirably to immersion in a virtualenvironment. There is also the potential of un-intended negative effects of exposure to virtualenvironments—stimulus intensity, if taken toofar, may exacerbate rather than ameliorate adeficit. High-fidelity virtual environments maybe confusing for these individuals and increasenegative behaviors after exposure to the envi-ronment. Flat-screen presentation of virtual en-vironments has proven to be an acceptable al-ternative to full immersion with theenvironment, and may be more appropriate forcertain clinical groups (Attree et al., 1996).
Summary and Conclusions
There are many different types of tests avail-able to psychologists for determining a client’slevel of functioning. The challenge for psychol-ogists is choosing tests that provide accurateinformation for making prescriptive statementsto clients, parents, and teachers based on thebest evidence available. Self-reports can behelpful in collecting data on specific areas offunctioning, but also suffer from lack of agree-ment among informants (Biederman et al.,1993, 1990; Mitsis et al., 2000; Newcorn et al.,1994; de Nijs et al., 2004) potential bias from
212 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
clients (Schwarz, 1999), and provides post hocappraisal of behavior. Normative comparison ofperformances on cognitive assessments allowsthe psychologist to determine if a client’s per-formance is similar or divergent from peers, butprovides limited information about daily func-tioning because they are construct-driven ratherthan function-led.
The addition of virtual reality to a psycho-logical battery provides an opportunity for psy-chologists to obtain more ecologically validdata about client functioning in simulations ofreal-world environments. Virtual environmentsallow the psychologist to have greater control ofdynamic perceptual stimuli and the sensitivityof the test, while also capturing data about clientperformance in activities of daily living (Bohilet al., 2011). The computerized nature of thesetests allows for the accurate capture of neurobe-havioral data, as well as precise recording andscoring of neuropsychological test results(Campbell et al., 2009). Several virtual environ-ments have already been developed to for psy-chologists to use in neuropsychological assess-ment, such as the virtual classroom, virtualgrocery store, and the virtual apartment. Al-though more validation studies need to be con-ducted with virtual reality assessments, the ben-efits of using this technology for understandingdaily functioning are clear. In addition, smallerand more affordable equipment makes virtualreality a viable option for use in psychologicalassessment.
Preferably, virtual assessments will be addedto flexible assessment batteries tailored to eachindividual within the context of the presentingquestion. Thus, traditional construct drivenmeasures should not be abandoned. In somecircumstances, construct driven assessmentsmay be more appropriate in terms of assessing aspecific construct that is generalizable acrossenvironments. For instance, working memorymay be more easily assessed by a simple spantask. The allure of the virtual assessment liesprimarily in enriching stimulus presentation,logging additional variables, and databasebuilding rather than the automation of the entirepsychological battery and the minimization ofhuman interaction.
Virtual environments may add to an existingpsychological battery when the psychologist isattempting to make accurate predictions about aperson’s behavior within the real world. In a
virtual environment, the psychologist can mea-sure functional output of constructs within thecomplexity of a real-world environment. Forexample, in a virtual classroom, selective atten-tion can be measured by conducting tests suchas the CPT in a real world environment. In avirtual environment grocery store, prospectivememory may be assessed using a real-worldtask like remembering to pick up a prescriptionat the pharmacy. Cognitive interference can beassessed in a virtual apartment that includescommon distractors found in an everyday envi-ronment.
Technological innovations, such as virtual re-ality, allow psychologists to expand our meth-ods for designing and implementing assess-ments capable of collecting information thatprovides an accurate picture of client limita-tions. These advances improve the prescriptivestatements psychologists dispense by providingthe opportunity to observe client functioning inreal-world environments—a practice that mightotherwise be infeasible because of clients’ be-havioral and physiological impairments. Byadopting virtual reality as a method for assess-ing clients, psychologists increase the potentialpositive impact of neuropsychological assess-ment for improving the daily functioning ofclients through accurate understanding of neu-ropsychological deficits and directly relevantrecommendations.
References
Adams, R., Finn, P., Moes, E., Flannery, K., & Rizzo,A. S. (2009). Distractibility in Attention/Deficit/Hyperactivity Disorder (ADHD): The virtual real-ity classroom. Child Neuropsychology, 15, 120–135. http://dx.doi.org/10.1080/09297040802169077
American Educational Research Association, Amer-ican Psychological Association, & National Coun-cil on Measurement in Education. (2014). Stan-dards for educational and psychological testing.Washington, DC: American Educational ResearchAssociation.
Anticevic, A., Schleifer, C., & Youngsun, T. C.(2015). Emotional and cognitive dysregulation inschizophrenia and depression: Understandingcommon and distinct behavioral and neural mech-anisms. Dialogues in Clinical Neuroscience, 17,421–434.
Atkins, A. S., Stroescu, I., Spagnola, N. B., Davis,V. G., Patterson, T. D., Narasimhan, M., . . . Keefe,
213VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
R. S. E. (2015). Assessment of age-related differ-ences in functional capacity using the Virtual RealityFunctional Capacity Assessment Tool (VRFCAT).The Journal of Prevention of Alzheimer’s Disease,2, 121.
Attree, E. A., Brooks, B. M., Rose, F. D., Andrews,T. K., Leadbetter, A. G., & Clifford, B. R. (1996,July). Memory processes and virtual environ-ments: I can’t remember what was there, but I canremember how I got there. Implications for peoplewith disabilities. In ECDVRAT: 1st EuropeanConference on Disability, Virtual Reality and As-sociated Technologies, Reading, UK (Vol. 118).
Biederman, J., Faraone, S. V., Milberger, S., &Doyle, A. (1993). Diagnoses of attention-deficithyperactivity disorder from parent reports predictdiagnoses based on teacher reports. Journal of theAmerican Academy of Child & Adolescent Psychi-atry, 32, 315–317. http://dx.doi.org/10.1097/00004583-199303000-00011
Biederman, J., Keenan, K., & Faraone, S. V. (1990).Parent-based diagnosis of attention deficit disorderpredicts a diagnosis based on teacher report. Jour-nal of the American Academy of Child & Adoles-cent Psychiatry, 29, 698–701. http://dx.doi.org/10.1097/00004583-199009000-00004
Bioulac, S., Lallemand, S., Rizzo, A., Philip, P.,Fabrigoule, C., & Bouvard, M. P. (2012). Impactof time on task on ADHD patient’s performancesin a virtual classroom. European Journal of Pae-diatric Neurology, 16, 514–521.
Bohil, C. J., Alicea, B., & Biocca, F. A. (2011).Virtual reality in neuroscience research and ther-apy. Nature Reviews Neuroscience, 12, 752–762.
Burgess, P. W., Alderman, N., Forbes, C., Costello,A., Coates, L. M. A., Dawson, D. R., . . . Channon,S. (2006). The case for the development and use of“ecologically valid” measures of executive func-tion in experimental and clinical neuropsychology.Journal of the International NeuropsychologicalSociety, 12, 194–209. http://dx.doi.org/10.1017/S1355617706060310
Campbell, Z., Zakzanis, K. K., Jovanovski, D.,Joordens, S., Mraz, R., & Graham, S. J. (2009).Utilizing virtual reality to improve the ecologicalvalidity of clinical neuropsychology: An FMRIcase study elucidating the neural basis of planningby comparing the Tower of London with a three-dimensional navigation task. Applied Neuropsy-chology, 16, 295–306. http://dx.doi.org/10.1080/09084280903297891
Canty, A. L., Fleming, J., Patterson, F., Green, H. J.,Man, D., & Shum, D. H. K. (2014). Evaluation ofa virtual reality prospective memory task for usewith individuals with severe traumatic brain in-jury. Neuropsychological Rehabilitation, 24, 238–265. http://dx.doi.org/10.1080/09602011.2014.881746
Carelli, L., Morganti, F., Weiss, P. L., Kizony, R., &Riva, G. (2008). A virtual reality paradigm for theassessment and rehabilitation of executive functiondeficits post stroke: Feasibility study. 2008 VirtualRehabilitation. http://dx.doi.org/10.1109/icvr.2008.4625144
Chasteen, A. L., Bhattacharyya, S., Horhota, M.,Tam, R., & Hasher, L. (2005). How feelings ofstereotype threat influence older adults’ memoryperformance. Experimental Aging Research, 31,235–260. http://dx.doi.org/10.1080/03610730590948177
Chaytor, N., & Schmitter-Edgecombe, M. (2003).The ecological validity of neuropsychologicaltests: A review of the literature on everyday cog-nitive skills. Neuropsychology Review, 13, 181–197. http://dx.doi.org/10.1023/B:NERV.0000009483.91468.fb
de Nijs, P. F., Ferdinand, R. F., de Bruin, E. I.,Dekker, M. C. J., van Duijn, C. M., & Verhulst,D. C. (2004). Attention-deficit/hyperactivity disor-der (ADHD): Parents’ judgment about school,teachers’ judgment about home. European Child& Adolescent Psychiatry, 13, 315–320. http://dx.doi.org/10.1007/s00787-004-0405-z
Díaz-Orueta, U., Garcia-López, C., Crespo-Eguílaz,N., Sánchez-Carpintero, R., Climent, G., & Nar-bona, J. (2014). AULA virtual reality test as anattention measure: Convergent validity with Con-ners’ Continuous Performance Test. Child Neuro-psychology: A Journal on Normal and AbnormalDevelopment in Childhood & Adolescence, 20,328–342.
Dyck, J. L., & Smither, J. A. (1994). Age differencesin computer anxiety: The role of computer expe-rience, gender and education. Journal of Educa-tional Computing Research, 10, 239–248. http://dx.doi.org/10.2190/E79U-VCRC-EL4E-HRYV
Erez, N., Weiss, P. L., Kizony, R., & Rand, D.(2013). Comparing performance within a virtualsupermarket of children with traumatic brain in-jury to typically developing children: A pilotstudy. OTJR: occupation, participation andhealth, 33, 218–227.
Gilboa, Y., Kerrouche, B., Longaud-Vales, A., Kief-fer, V., Tiberghien, A., Aligon, D., . . . PauleChevignard, M. (2015). Describing the attentionprofile of children and adolescents with acquiredbrain injury using the Virtual Classroom. [Ad-vance online publication]. Brain Injury, 29, 1691–1700. http://dx.doi.org/10.3109/02699052.2015.1075148
Gilboa, Y., Rosenblum, S., Fattal-Valevski, A., Tole-dano-Alhadef, H., Rizzo, A. S., & Josman, N.(2011). Using a Virtual Classroom environment todescribe the attention deficits profile of childrenwith Neurofibromatosis type 1. [Advance onlinepublication]. Research in Developmental Disabil-
214 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
ities, 32, 2608–2613. http://dx.doi.org/10.1016/j.ridd.2011.06.014
Henry, M., Joyal, C. C., & Nolin, P. (2012). Devel-opment and initial assessment of a new paradigmfor assessing cognitive and motor inhibition: Thebimodal virtual-reality Stroop. Journal of Neuro-science Methods, 210, 125–131. http://dx.doi.org/10.1016/j.jneumeth.2012.07.025
Iriarte, Y., Diaz-Orueta, U., Cueto, E., Irazustabar-rena, P., Banterla, F., & Climent, G. (2016). AU-LA-Advanced Virtual Reality Tool for the Assess-ment of Attention: Normative Study in Spain.Journal of Attention Disorders, 20, 542–568.http://dx.doi.org/10.1177/1087054712465335
Josman, N., Hof, E., Klinger, E., Marie, R. M., Gold-enberg, K., Weiss, P. L., & Kizony, R. (2006).Performance within a virtual supermarket and itsrelationship to executive functions in post-strokepatients. 2006 International Workshop on VirtualRehabilitation. http://dx.doi.org/10.1109/iwvr.2006.1707536
Josman, N., Kizony, R., Hof, E., Goldenberg, K.,Weiss, P. L., & Klinger, E. (2014). Using the virtualaction planning-supermarket for evaluating executivefunctions in people with stroke. Journal of Strokeand Cerebrovascular Diseases, 23, 879–887. http://dx.doi.org/10.1016/j.jstrokecerebrovasdis.2013.07.013
Josman, N., Schenirderman, A. E., Klinger, E., &Shevil, E. (2009). Using virtual reality to evaluateexecutive functioning among persons with schizo-phrenia: A validity study. Schizophrenia Research,115, 270–277. http://dx.doi.org/10.1016/j.schres.2009.09.015
Jurado, M. B., & Rosselli, M. (2007). The elusivenature of executive functions: A review of ourcurrent understanding. Neuropsychology Review,17, 213–233. http://dx.doi.org/10.1007/s11065-007-9040-z
Kang, Y. J., Ku, J., Han, K., Kim, S. I., Yu, T. W.,Lee, J. H., & Park, C. I. (2008). Development andclinical trial of virtual reality-based cognitive as-sessment in people with stroke: Preliminary study.CyberPsychology & Behavior, 11, 329 –339.http://dx.doi.org/10.1089/cpb.2007.0116
Klinger, E., Chemin, I., Lebreton, S., & Marié, R. M.(2006). Virtual action planning in Parkinson’s dis-ease: A control study. CyberPsychology & Behav-ior, 9, 342–347. http://dx.doi.org/10.1089/cpb.2006.9.342
Kurtz, M. M., Baker, E., Pearlson, G. D., & Astur,R. S. (2007). A virtual reality apartment as a mea-sure of medication management skills in patientswith schizophrenia: A pilot study. SchizophreniaBulletin, 33, 1162–1170. http://dx.doi.org/10.1093/schbul/sbl039
Lalonde, G., Henry, M., Drouin-Germain, A., Nolin,P., & Beauchamp, M. H. (2013). Assessment of
executive function in adolescence: A comparisonof traditional and virtual reality tools. Journal ofNeuroscience Methods, 219, 76–82. http://dx.doi.org/10.1016/j.jneumeth.2013.07.005
Lambert, A. E., Watson, J. M., Stefanucci, J. K.,Ward, N., Bakdash, J. Z., & Strayer, D. L. (2016).Stereotype threat impairs older adult driving. Ap-plied Cognitive Psychology, 30, 22–28. http://dx.doi.org/10.1002/acp.3162
Lamont, R. A., Swift, H. J., & Abrams, D. (2015). Areview and meta-analysis of age-based stereotypethreat: Negative stereotypes, not facts, do the dam-age. Psychology and Aging, 30, 180–193. http://dx.doi.org/10.1037/a0038586
Manchester, D., Priestley, N., & Jackson, H. (2004).The assessment of executive functions: Comingout of the office. Brain Injury, 18, 1067–1081.http://dx.doi.org/10.1080/02699050410001672387
Matheis, R. J., Schultheis, M. T., Tiersky, L. A.,DeLuca, J., Millis, S. R., & Rizzo, A. (2007). Islearning and memory different in a virtual envi-ronment? The Clinical Neuropsychologist, 21,146 –161. http://dx.doi.org/10.1080/13854040601100668
Meneghetti, C., Muffato, V., Suitner, C., De Beni, R.,& Borella, E. (2015). Map learning in young andolder adults: The influence of perceived stereotypethreat. Learning and Individual Differences, 42,77– 82. http://dx.doi.org/10.1016/j.lindif.2015.08.015
Miller, K. J., Adair, B. S., Pearce, A. J., Said, C. M.,Ozanne, E., & Morris, M. M. (2014). Effectivenessand feasibility of virtual reality and gaming systemuse at home by older adults for enabling physicalactivity to improve health-related domains: A sys-tematic review. Age and Ageing, 43, 188–195.
Mintz, A. R., Dobson, K. S., & Romney, D. M.(2003). Insight in schizophrenia: A meta-analysis.Schizophrenia Research, 61, 75–88. http://dx.doi.org/10.1016/S0920-9964(02)00316-X
Mitsis, E. M., McKay, K. E., Schulz, K. P., Newcorn,J. H., & Halperin, J. M. (2000). Parent-teacherconcordance for DSM–IV attention-deficit/hyper-activity disorder in a clinic-referred sample. Jour-nal of the American Academy of Child & Adoles-cent Psychiatry, 39, 308–313. http://dx.doi.org/10.1097/00004583-200003000-00012
Newcorn, J. H., Halperin, J. M., Schwartz, S., Pas-cualvaca, D., Wolf, L., Schmeidler, J., & Sharma,V. (1994). Parent and teacher ratings of attention-deficit hyperactivity disorder symptoms: Implica-tions for case identification. Journal of Develop-mental and Behavioral Pediatrics, 15, 86 –91.http://dx.doi.org/10.1097/00004703-199404000-00004
Nolin, P., Martin, C., & Bouchard, S. (2009). Assess-ment of inhibition deficits with the virtual class-room in children with traumatic brain injury: A
215VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
pilot-study. Studies in Health Technology and In-formatics, 144, 240–242.
Nolin, P., Stipanicic, A., Henry, M., Joyal, C. C., &Allain, P. (2012). Virtual reality as a screening toolfor sports concussion in adolescents. Brain Injury,26, 1564 –1573. http://dx.doi.org/10.3109/02699052.2012.698359
Okahashi, S., Seki, K., Nagano, A., Luo, Z., Kojima,M., & Futaki, T. (2013). A virtual shopping test forrealistic assessment of cognitive function. Journalof Neuroengineering and Rehabilitation, 10, 59.http://dx.doi.org/10.1186/1743-0003-10-59
Parsons, T. D., Bowerly, T., Buckwalter, J. G., &Rizzo, A. A. (2007). A controlled clinical compar-ison of attention performance in children withADHD in a virtual reality classroom compared tostandard neuropsychological methods. Child Neu-ropsychology, 13, 363–381. http://dx.doi.org/10.1080/13825580600943473
Parsons, T. D., & Carlew, A. R. (2016). Bimodalvirtual reality Stroop for assessing distractor inhi-bition in autism spectrum disorders. Journal ofAutism and Developmental Disorders, 46, 1255–1267. http://dx.doi.org/10.1007/s10803-015-2663-7
Parsons, T. D., McPherson, S., & Interrante, V.(2013). Enhancing neurocognitive assessment us-ing immersive virtual reality. Paper presented atthe 17th IEEE Virtual Reality Conference, Green-ville, SC.
Parsons, T. D., & Rizzo, A. A. (2008). Affectiveoutcomes of virtual reality exposure therapy foranxiety and specific phobias: A meta-analysis.Journal of Behavior Therapy and ExperimentalPsychiatry, 39, 250 –261. http://dx.doi.org/10.1016/j.jbtep.2007.07.007
Pas, E. T., & Bradshaw, C. P. (2014). What affectsteacher ratings of student behaviors? The potentialinfluence of teachers’ perceptions of the schoolenvironment and experiences. Prevention Science,15, 940 –950. http://dx.doi.org/10.1007/s11121-013-0432-4
Patterson, T. L., Lacro, J., McKibbin, C. L.,Moscona, S., Hughs, T., & Jeste, D. V. (2002).Medication management ability assessment: Re-sults from a performance-based measure in olderoutpatients with schizophrenia. Journal of ClinicalPsychopharmacology, 22, 11–19. http://dx.doi.org/10.1097/00004714-200202000-00003
Peli, E. (1998). The visual effects of head-mounteddisplay (HMD) are not distinguishable from thoseof desk-top computer display. Vision Research, 38,2053–2066. http://dx.doi.org/10.1016/s0042-6989(97)00397-0
Pollak, Y., Shomaly, H. B., Weiss, P. L., Rizzo,A. A., & Gross-Tsur, V. (2010). Methylphenidateeffect in children with ADHD can be measured byan ecologically valid continuous performance test
embedded in virtual reality. CNS Spectrums, 15,125–130.
Pollak, Y., Weiss, P. L., Rizzo, A. A., Weizer, M.,Shriki, L., Shalev, R. S., & Gross-Tsur, V. (2009).The utility of a continuous performance test em-bedded in virtual reality in measuring ADHD-related deficits. Journal of Developmental and Be-havioral Pediatrics, 30, 2–6. http://dx.doi.org/10.1097/DBP.0b013e3181969b22
Rand, D., Basha-Abu Rukan, S., Weiss, P. L., &Katz, N. (2009). Validation of the Virtual MET asan assessment tool for executive functions. Neuro-psychological Rehabilitation, 19, 583–602. http://dx.doi.org/10.1080/09602010802469074
Rand, D., Katz, N., & Weiss, P. L. (2007). Evaluationof virtual shopping in the VMall: Comparison ofpost-stroke participants to healthy control groups.Disability and Rehabilitation, 29, 1710 –1719.http://dx.doi.org/10.1080/09638280601107450
Raspelli, S., Pallavicini, F., Carelli, L., Morganti, F.,Pedroli, E., Cipresso, P., . . . Riva, G. (2012).Validating the neuro VR-based virtual version ofthe Multiple Errands Test: Preliminary results.Presence, 21, 31– 42. http://dx.doi.org/10.1162/PRES_a_00077
Rizzo, A. A., Bowerly, T., Buckwalter, J. G., Klim-chuk, D., Mitura, R., & Parsons, T. D. (2006). Avirtual reality scenario for all seasons: The virtualclassroom. CNS Spectrums, 11, 35–44. http://dx.doi.org/10.1017/S1092852900024196
Sayago, S., Sloan, D., & Blat, J. (2011). Everydayuse of computer-mediated communication toolsand its evolution over time: An ethnographicalstudy with older people. Interacting with Comput-ers, 23, 543–554. http://dx.doi.org/10.1016/j.intcom.2011.06.001
Sayal, K., & Taylor, E. (2005). Parent ratings ofschool behaviour in children at risk of attentiondeficit/hyperactivity disorder. Acta PsychiatricaScandinavica, 111, 460–465. http://dx.doi.org/10.1111/j.1600-0447.2004.00487.x
Schwarz, N. (1999). Self-reports: How the questionsshape the answers. American Psychologist, 54, 93–105. http://dx.doi.org/10.1037/0003-066X.54.2.93
Swift, H. J., Lamont, R. A., & Abrams, D. (2012).Are they half as strong as they used to be? Anexperiment testing whether age-related social com-parisons impair older people’s hand grip strengthand persistence. British Medical Journal Open, 2,e001064. http://dx.doi.org/10.1136/bmjopen-2012-001064
Wallace, S., Parsons, S., Westbury, A., White, K.,White, K., & Bailey, A. (2010). Sense of presenceand atypical social judgments in immersive virtualenvironments: Responses of adolescents with Au-tism Spectrum Disorders. Autism, 14, 199–213.http://dx.doi.org/10.1177/1362361310363283
216 PARSONS AND PHILLIPS
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Wandke, H., Sengpiel, M., & Sönksen, M. (2012).Myths about older people’s use of information andcommunication technology. Gerontology, 58,564–570. http://dx.doi.org/10.1159/000339104
Werner, P., Rabinowitz, S., Klinger, E., Korczyn,A. D., & Josman, N. (2009). Use of the virtualaction planning supermarket for the diagnosis ofmild cognitive impairment: A preliminary study.Dementia and Geriatric Cognitive Disorders, 27,301–309. http://dx.doi.org/10.1159/000204915
White, S. W., Richey, J. A., Gracanin, D., Bell,M. A., LaConte, S., Coffman, M., & Kim, I.(2014). The promise of neurotechnology in clinicaltranslational science. Clinical Psychological Sci-
ence. Advance online publication. http://dx.doi.org/10.1177/2167702614549801
Zygouris, S., Giakoumis, D., Votis, K., Doumpoula-kis, S., Ntovas, K., Segkouli, S., . . . Tsolaki, M.(2015). Can a virtual reality cognitive trainingapplication fulfill a dual role? Using the virtualsupermarket cognitive training application as ascreening tool for mild cognitive impairment.Journal of Alzheimer’s Disease, 44, 1333–1347.
Received April 25, 2016Revision received July 23, 2016
Accepted July 25, 2016 �
Members of Underrepresented Groups:Reviewers for Journal Manuscripts Wanted
If you are interested in reviewing manuscripts for APA journals, the APA Publicationsand Communications Board would like to invite your participation. Manuscript reviewersare vital to the publications process. As a reviewer, you would gain valuable experiencein publishing. The P&C Board is particularly interested in encouraging members ofunderrepresented groups to participate more in this process.
If you are interested in reviewing manuscripts, please write APA Journals [email protected]. Please note the following important points:
• To be selected as a reviewer, you must have published articles in peer-reviewedjournals. The experience of publishing provides a reviewer with the basis for preparinga thorough, objective review.
• To be selected, it is critical to be a regular reader of the five to six empirical journalsthat are most central to the area or journal for which you would like to review. Currentknowledge of recently published research provides a reviewer with the knowledge baseto evaluate a new submission within the context of existing research.
• To select the appropriate reviewers for each manuscript, the editor needs detailedinformation. Please include with your letter your vita. In the letter, please identifywhich APA journal(s) you are interested in, and describe your area of expertise. Be asspecific as possible. For example, “social psychology” is not sufficient—you wouldneed to specify “social cognition” or “attitude change” as well.
• Reviewing a manuscript takes time (1–4 hours per manuscript reviewed). If you areselected to review a manuscript, be prepared to invest the necessary time to evaluatethe manuscript thoroughly.
APA now has an online video course that provides guidance in reviewing manuscripts. Tolearn more about the course and to access the video, visit http://www.apa.org/pubs/authors/review-manuscript-ce-video.aspx.
217VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE
Thi
sdo
cum
ent
isco
pyri
ghte
dby
the
Am
eric
anPs
ycho
logi
cal
Ass
ocia
tion
oron
eof
itsal
lied
publ
ishe
rs.
Thi
sar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Top Related