MAAD: A Model and Dataset for “Attended Awareness” in Driving

Deepak Gopinath∗ Guy Rosman† Simon Stent† Katsuya Terahata# Luke Fletcher‡

Brenna Argall∗ John Leonard†

∗Department of Mechanical Engineering, Northwestern University, Evanston, Illinois,† Toyota Research Institute, Cambridge, MA,

#Woven Planet Holdings, Tokyo, Japan, ‡ Motional, Cambridge, MA.Corresponding Author: deepakgopinath@u.northwestern.edu

Abstract

We propose a computational model to estimate a per-son’s attended awareness of their environment. We de-fine “attended awareness” to be those parts of a poten-tially dynamic scene which a person has attended to inrecent history and which they are still likely to be phys-ically aware of. Our model takes as input scene infor-mation in the form of a video and noisy gaze estimates,and outputs visual saliency, a refined gaze estimate andan estimate of the person’s attended awareness. In or-der to test our model, we capture a new dataset with ahigh-precision gaze tracker including 24.5 hours of gazesequences from 23 subjects attending to videos of drivingscenes. The dataset also contains third-party annotationsof the subjects’ attended awareness based on observationsof their scan path. Our results show that our model is ableto reasonably estimate attended awareness in a controlledsetting, and in the future could potentially be extended toreal egocentric driving data to help enable more effectiveahead-of-time warnings in safety systems and thereby aug-ment driver performance. We also demonstrate our model’seffectiveness on the tasks of saliency, gaze calibration anddenoising, using both our dataset and an existing saliencydataset. We make our model and dataset available athttps://github.com/ToyotaResearchInstitute/att-aware/.

1. IntroductionWe define “attended awareness” to be those parts of a

potentially dynamic scene which a person has attended toin recent history and which they are still likely to be phys-ically aware of.1 While this measure of a person’s aware-

1Attended awareness will also be referred to as awareness in the rest ofthe paper for the sake of brevity.

Figure 1: Model for Attended Awareness in Driving(MAAD) overview. Our model takes as input a video ofa scene, as seen by a person performing a task in the scene,along with noisily registered ego-centric gaze sequencesfrom that person. The model estimates (i) a saliency heatmap, (ii) a refined gaze estimate, and (iii) an estimate ofthe subject’s attended awareness of the scene. We evalu-ate our model using a unique annotated dataset of third-person estimates of a driver’s attended awareness. By ex-plicitly estimating a person’s attended awareness from noisymeasurements of their gaze, MAAD can improve human-machine interactions. In the driving example, such inter-actions might include safety warnings in situations whereattended awareness is deemed insufficient.

ness is difficult to objectively measure, in certain situationssuch as driving, it is possible to infer, at least to some use-ful degree, the attended awareness of a driver. Driving in-structors routinely assess a driver’s behavior based on theirestimated attended awareness of a given driving scene, andprovide real-time feedback to ensure safety. In the contextof human-machine interaction, inferring human attendedawareness in a given scenario is valuable for machines tofacilitate seamless interaction and effective interventions.Human gaze information can be used by machines for thispurpose and many models have been developed to relatescene understanding and overt attention as saliency esti-mates [7, 10, 15, 22, 41, 42] and objectness measures [1, 6].However, further exploration of the link between visual at-

13426

tention and processes of scene understanding and decisionmaking has been constrained by a limited ability to reasonabout such cognitive processes, which is difficult in generalcontexts [63, 53, 49].

In this work, we propose a model to estimate attendedawareness as a spatial heatmap. The input to our model isa video of the scene under observation, and a noisy esti-mate of the person’s gaze. Our model allows us to (i) com-pute visual saliency for the scene, (ii) leverage this to refinethe noisy gaze estimate, and (iii) combine this informationover time to infer the person’s attended awareness. The con-struction of our model is driven by axiomatic considerationsthat define an image translation network from the scene im-age to the saliency and awareness heat maps. Within alarger human-machine interaction scenario, the model be-comes useful as an inference engine for human awarenessenabling seamless interaction (for e.g., a person operatinga semi-autonomous car or a smart wheelchair). We adopta data-driven approach (as opposed to more limited-scopeanalytical models [30]) to allow for scalable and more com-prehensive modeling of attended awareness that can poten-tially rely on supervision from multiple sources (such as re-sulting actions/self-reporting).

Contributions 1) We propose a learned model that af-fords estimation of attended awareness based on noisy gazeestimates and scene video over time. 2) We further demon-strate how the model affords saliency estimation as wellas the refinement of a noisy gaze signal. 3) We present anew dataset that explores the gaze and perceived attendedawareness of subjects as they observe a variety of drivingand cognitive task conditions. While the dataset is capturedvia a proxy hazard awareness task rather than through realor simulated driving, it serves as a useful starting point tostudy visual saliency and awareness in driving scenes.

2. Related WorkOur work builds on prior work in visual saliency esti-

mation, situational awareness, and driving-specific explo-ration. We briefly summarize related work from these areas.

Visual saliency. Vision scientists have long sought tomodel and understand the mechanisms behind our alloca-tion of attention in a visual scene [63]. Visual salience isa function of many factors, including the spatio-temporalnature of the stimulus itself as well as its relationship toneighboring stimuli and the nature of the visual system per-ceiving it [23]. The evolution of computational approachesto estimating visual image salience has mirrored that ofother popular computer vision problems, with largely hand-engineered models—designed to mirror certain bottom-upand top-down attention processes in the human visual sys-tem (e.g. [8, 23])—giving way to supervised, data-drivenapproaches (e.g. [22, 24]). Unlike image saliency, videosaliency models consider spatial as well as temporal infor-

mation to detect objects of interest in a dynamic way.Spatio-temporal saliency estimation opens inquiry into

how processes of visual attention, situational awareness andtask-related decision making are connected. Previous at-tempts have been made to computationally model situa-tional awareness (see e.g. [4, 38, 59]). Our approach tomodeling situational awareness is unique in that we try toexplicitly estimate the parts of the visual scene to which aperson has attended using a spatio-temporal model for gazeand scene understanding. The three stages of forming sit-uational awareness consists of perception, comprehensionand projection [19] and in our work we focused primarilyon perception. We aim to model, from noisy observationsof a person’s scan path, the set of objects and scene struc-tures which that person is likely to have attended to, andtherefore might be better able to incorporate into their fu-ture decision-making. While we note that peripheral visionalone can achieve visual awareness in many settings [58],we focus on objects of fixation, since we are concerned pri-marily with estimating when drivers fail to notice potentialdriving hazards, which are known to strongly induce fixa-tions [16].

Data-Driven Saliency Modeling and Datasets. Data-driven approaches to image and video saliency rely on state-of-the-art deep learning architectures: CNNs [10, 22, 41],GANs [42] and LSTMs [3, 15, 56], and single image inputsthrough to multi-stream inputs incorporating video, alongwith optical flow, depth or semantic segmentation estimates[33, 40] and even additional modalities such as audio [55].In our work, similar to [39], we adopt a 3D CNN–basedapproach, due to its simplicity and success on other videounderstanding tasks such as action recognition.

While the majority of saliency datasets explore imageand video saliency under controlled viewing conditions(e.g. [32, 37, 56]), in recent years, numerous ego-centricvideo gaze datasets have been developed in which subjectsperform tasks as varied as cooking and meal preparation[34], playing video games [8] and driving [40], in paral-lel to developments in applications of saliency (see, e.g.[51, 45, 11]. On the other hand in the driving domain, in-labdata collection procedures have been extensively adopted asthey have the advantages of high accuracy, repeatability andthe ability to focus on rare scenarios such as critical situ-ations and accidents [61, 21, 17, 2], where understandinghuman perception can inform safety systems approaches.

Our dataset uses an in-lab data collection paradigm,however it differs from prior work for several reasons.Firstly and most notably, we capture multiple subjects ob-serving the same visual stimuli under different cognitivetask modifiers. Our dataset therefore allows for reasoningabout the effect of different cognitive task modifiers on thevisual gaze patterns, given identical visual input. Secondly,we provide annotations for third party estimates of a sub-

3427

ject’s attended awareness, based on observations of theirscan path. For this purpose, we devise a novel reproducibleannotation scheme. Finally, as our dataset is gathered usinga high precision gaze tracker with a chin rest, the precisionof the scan paths is extremely high when compared to thatof eye-glass gaze tracking ego-centric datasets such as [40].

Driving specific applications. Driving is a predomi-nantly visual task. The first studies into driver attention andeye scanning patterns date back over half a century [50].Since then, certain driving behaviors have been well estab-lished and modelled, such as the “perceptual narrowing” ef-fect in which drivers increasingly fixate on the road aheadas task demands increase (e.g. through greater speed, in-creased traffic or lack of familiarity with a route) [20], orthe benefits of understanding driver attention when predict-ing a driver’s future intent [18, 60]. However, to the best ofour knowledge, no models exist with the purpose of quanti-tatively estimating a driver’s spatial awareness of a scene.In recent years, works such as [40, 61] have used high-precision gaze trackers to create video gaze datasets both inthe car and in the lab, allowing for data-driven approaches tomodelling driver attention. While we make use of the road-facing data from [40] in our experiments, our model differsin one key respect. Rather than estimating visual saliencyfrom video alone, we demonstrate how, given access to anoisy gaze estimate of a driver it is possible to simultane-ously estimate scene saliency, a denoised gaze signal andan estimate of the driver’s overall awareness of the scene.

3. MethodThe computational model we propose is guided by sev-

eral assumptions related to human attended awareness andits relation to gaze patterns (Section 3.1). These assump-tions are implemented through a mixture of explicit objec-tives and behavioral regularizations (Section 3.3).

3.1. Assumptions and Supervisory Cues

We use several assumptions about gaze patterns and at-tended awareness to define the priors in training our model:

• A1 Saliency: Gaze tends to focus on specific regions[7], both object and stuff [31].

• A2 Attended awareness: People tend to becomeaware of the objects they look at [52, 12]. Their at-tention is, however, limited in its capacity.

• A3 Awareness decay: Awareness of an object can de-crease (due to forgetting), or increase (when looking atan object) at different rates [46].

• A4 Regularity of gaze, awareness: Gaze and aware-ness maps should be regular and smooth unless other-wise warranted [9, 25].

• A5 Awareness and motion: As an observer movesthrough a dynamic scene, their awareness moves alongwith objects and regions in the scene and exhibits tem-poral persistence. [35, 5].

3.2. Model

We define the Model for Attended Awareness in Driving(MAAD), shown in Figure 2, as a fully convolutional imageencoding-decoding network with shortcut links such as U-Net [47]. Sensor images are encoded and decoded into alatent feature map, M(x, t), from which two convolutionalmodules emit the estimated gaze distribution pG(x, t), andthe awareness image MA(x, t).

The gaze distribution is normalized as a probability den-sity function (via a softmax operator). We note that pG(x, t)is a unified notation for gaze probability maps with andwithout a noisy gaze input from an external gaze tracker. Inthe absence of a noisy gaze input, pG(x, t) is a probabilisticform of saliency. In the rest of the paper we use pG(x, t) todenote both forms to simplify the notation. The individualmodules in the decoder are fed additional information: the(noisy) driver gaze measurement over time in the form ofVoronoi maps, and optical flow maps [40] encoded as twofeature layers for horizontal and vertical motion.

3.3. Loss Function Design

At training time, the gaze and awareness maps are usedto compute several supervisory and regularization terms,whose design is guided by the assumptions in Section 3.1.

3.3.1 Supervisory Terms

Gaze Prediction. We want pG to predict a subject’s gaze asaccurately as possible. This is encouraged via the primarydata term:

LG = −∑t

∑x∈Xg(t)

log pG(x, t), (1)

where Xg(t) are the 2D ground truth gaze points at time t.Perceived Awareness. We include a supervisory source

for awareness estimation. This term surrogates awarenessestimation training by a perceived awareness estimate. Oneapproach to obtain a perceived awareness estimate is to pro-vide a person with the gaze estimate of the driver overlaidon the road scene and query how aware the driver is of spe-cific locations in the image at particular points in time. Thisis further described in Section 4. The cost term reads:

LATT =∑

(x,t)∈labeled

(MA(x, t)− LA(x, t))2, (2)

where the summation is over all annotated samples in loca-tion x at time t, and LA denotes the annotated measure ofawareness in the range [0, 1] as described in Section 4.2.

3428

Decoder Unit

`Optical FlowModule

Input videosequence

ENCODER DECODER

laye

r1

laye

r2

S3D

_1

S3D

_2

Skip connections

Con

v3d

S3D

Gaze TransformModule

S3DBilinear

Upsample

Skipconnection

Output of previous DU

Side

-cha

nnel

info

rmat

ion

Gaze probabilitydensities

Attended awarenessheat mapsLatent Map

Side-channelfeatures

S3D

_3

DU

4

DU

1

DU

2

DU

3

DU

5

Softmax

Conv1d Sigmoid

Conv2d

S3D

Conv3d(1,k,k)

InstanceNorm

ReLUConv3d(k,1,1)

InstanceNorm

ReLU

Spatial Conv Temporal Conv

Conv1d

Side-channel features

Figure 2: Model architecture. During training, our model takes as input a video sequence with associated scan paths from agaze tracker, along with optical flow estimates. The video sequence is processed using a 3D convolutional encoder/decoder,and information from gaze, and optic flow is transformed and injected during the decoding stage. The model outputs twoheat maps: a gaze probability density heat map, pG, which provides a clean estimate of the noisy gaze input, and an attendedawareness heat map, MA, which provides an estimate of the driver awareness, both at the final frame of the input videosequence. For further details, see Section 3.2.

Awareness of Attended Objects. Based on (A2), weadd a term that encourages awareness to be high when aperson is gazing at a scene location:

LAA =∑t

∑x∈Xg(t)

(MA(x, t)− 1)2. (3)

3.3.2 Regularization Terms

Spatial Smoothness. Based on (A4-5), we added regularityterms to both the awareness and gaze maps:

LS,· =

∫|∇ϕ|2√|∇I|2 + ϵ

dx, (4)

where ϕ is MA and pG for LS,A and LS,G respectively andthe integral is computed over all pixels. This regulariza-tion is a variant of anisotropic diffusion [44, 43] with cross-diffusivity based on the scene image I .

Temporal Smoothness. Based on (A3-5), we apply alsotemporal regularization for awareness:

In order to make the map temporally consistent with re-spect to the locations and not just object boundaries, we usea smoothness / decay term based on the image optical flow:

LT =∑x,t

fwOF (MA(x+ vOF(x), t+ 1),MA(x, t)), (5)

fwOF(a, b) = c1((a− wOFb)+) + c2((a− wOFb)−)2,

where vOF(x) is the optical flow computed on the input im-ages and wOF < 1 is a weight factor that is close to 1. ()+and ()− denote positive and negative change respectively inawareness values when going from t to t + 1. Particularly,fwOF(a, b) is set to be an asymmetric loss function that pe-nalizes awareness that increases instantaneously when at-tending to an object less compared to awareness that de-creases rapidly via forgetting. This is accomplished by hav-ing the forgetting term, ()−, to be quadratic.

Awareness Decay. Based on (A3), we expect the levelof awareness to decay over time, which we model via:

LDEC =∑x,t

((1− ϵDEC)MA(x, t)

−MA(x, t+ 1)

)2

, (6)

where ϵDEC is a decay factor.Attention Capacity. Based on (A2), we expect that the

cognitive resources available to the driver do not changeover time. This assumption captures the fact that the over-all awareness should be similar between frames on average.

LCAP =∑t

(∑x

MA(x, t)−∑x

MA(x, t+ 1))2

. (7)

Block-level consistency. We denote by MA(x, t; t1) theawareness estimate at (x, t) that is emitted based on thetraining snippet started at t1, similarly for pG. We definea consistency term [48] between consecutive estimates via

3429

the loss:

LCON,· =∑t1,t

∑x

(ϕ(x, t; t1)− ϕ(x, t; t1 + 1)

)2

, (8)

where ϕ is MA, pG for LCON,A,LCON,G respectively.LCON,· helps to minimize the difference between the pre-dictions of multiple passes of samples at the same times-tamps through the network. The overall training loss is alinear combination of the previously described loss terms.See supplementary material for additional details.

3.4. Training Procedure

We trained our model using PyTorch 1.7 using NV100GPUs. The training was carried out using video input sub-sequences of length 4 seconds, sampled at 5Hz with framesresized to 240×135. Our model was trained using Adamoptimizer [29] with a learning rate of 5× 10−3 and a batchsize of 4. The model weights were updated after the gra-dients were aggregated for a fixed number of steps in orderto reduce the variance in the loss. The batch aggregationsize was set at 8. The first two layers of the encoder (keptfrozen during training) are taken from a pretrained Resnet18implementation in Torchvision [36]. Later spatio-temporalencoding is done by three layers of separable 3D convolu-tion modules denoted as S3D in Figure 2 [62]. To provideoptical flow estimates, we used RAFT pre-trained on Fly-ingChairs and FlyingThings datasets [54].

The decoder consists of stacked decoder units each ofwhich receives input from three sources a) side-channel in-formation b) skip connections from the encoder layer andc) the output of the previous decoder unit, when available.Each decoder unit consists of two submodules: 1) The skipconnections are first processed via an S3D module whoseoutput is then concatenated (channel-wise) with the side-channel information and the output of the previous decoderunit. This concatenated input is processed by another S3Dmodule followed by bilinear upsampling that brings the out-put to the proper resolution for the next decoder unit.

The decoder emits a latent map M which is subsequentlyprocessed by two separate convolutional models to emit agaze probability map, pG and an awareness heatmap de-noted as a MA. The softmax in the gaze convolutional mod-ule ensures that the gaze probability map is a valid proba-bility distribution. More details regarding network architec-ture and training to be found in supplementary material.

4. Dataset DescriptionOur complete dataset comprises approximately 24.5

hours of gaze tracking data captured via multiple exposuresfrom different subjects to 6.2 hours of road-facing videodrawn from the DR(eye)VE dataset [40]. We concentrateour gaze capture on repeated exposures of downtown (as

opposed to highway and rural) driving scenes (77%) anddaylight scenes (90%), since these contain the most diversevisual scenarios. While the original DR(eye)VE datasetcaptured and registered gaze to the road-facing video us-ing a driver head–mounted eye tracker and feature-basedmatching for homography estimation, accumulating signif-icant measurement errors, we opted for an in-lab experi-ment. In-lab experiment offers several advantages such ashigher accuracy and repeatability across subjects and cogni-tive task conditions. Furthermore, models trained on in-labdata has already been shown to be effective when tested onin-the-wild data [61]. We measure gaze to an extremelyhigh precision (0.15◦ − 0.50◦ typical accuracy, 0.01◦ RMSresolution, 0.05◦ microsaccade resolution) using a tower-mounted SR Research EyeLink 1000 Plus tracker. A mainnovelty of the dataset is that in addition to high precisiongaze, we supplement the dataset with third party annota-tions of perceived awareness. The annotation procedure isoutlined in Section 4.2. We recruited 23 subjects (aged 20-55), who each watched a subset of video clips with theirheads mounted in a chin-rest after a 9-point calibration pro-cedure. The subjects all carried US driving licenses and hadat least two years of driving experience. Their primary taskwas to monitor the driving scene as a safety driver mightmonitor an autonomous vehicle. While not a perfect sub-stitute for in-car driving data collection, this primary taskallowed for the capture of many of the characteristics of at-tentive driving behavior. In order to explore the effect of thecognitive task difference (vs. in-car data) on the gaze andawareness estimates, subjects viewed the video under differ-ent cognitive task modifiers, as detailed in Section 4.1 (datacollected with non-null cognitive task modifiers comprise30% of total captured gaze data). Around 45% of videostimuli were watched more than once, of which 11% (40minutes) was observed by 16 or more subjects.

4.1. Cognitive Task Modifiers

Although our dataset is collected in an in-lab experi-ment, we are still interested in subjects’ behavior under adifferent task (in-car driving, engaged as well as distracted).We therefore included in our experiments several secondarycognitive task modifiers to artificially alter participant be-havior in a way which might mimic certain variations in thebehavior of drivers in real conditions. We aimed at modi-fiers that affected visual search patterns, but did not explic-itly instruct the participants toward specific search targets.These modifiers affect the visual patterns by either changingthe goal of visual search or by changing the scene appear-ance. The cognitive task modifiers were as follows:

1. Null condition: The subjects were given the task ofsupervising the driving of the car, looking for and flaggingpossible risky events or obstacles.

2. Blurred condition: Same as 1, but stimulus videos

3430

Figure 3: What is the driver aware of? Our dataset isunique in containing repeated views of the same drivingvideo by 23 subjects performing the same proxy supervi-sory task but under different task modifiers: (i) null (green),i.e. with no secondary condition, (ii) with blurred video(yellow), (iii) with vertically flipped video (red), (iv) fix-ating on the road only (dark blue), and (v) reading occa-sional on-screen text (light blue) (see Section 4.1 for moredetails). Here we overlay a sub-sampled one second gazehistory per subject for a given frame in a video sequence.From observation of the scan paths of subjects, it is evi-dent that some are aware and fixate strongly on the risk inthe scene (e.g. crossing vehicles or pedestrians), while oth-ers fixate fleetingly or, due to the presence of cognitive taskmodifiers, are unaware of the risk.

were blurred with a Gaussian kernel corresponding toN deg of visual field, making scene understanding andtherefore the supervisory task harder, and affecting the vi-sual search pattern.

3. Vertical-flip condition: Same as 1, but stimulusvideos were flipped upside down, making scene understand-ing and therefore the supervisory task counter-intuitive, andaffecting the visual search pattern.

4. Road-only condition: Same as 1, but subjects wereasked to only fixate on road structure and not on dynamicobstacles such as cars and people.

5. Reading-text condition: Same as 1, but stimulusvideos were overlaid with snippets of text of approximatelyeven length at random image locations for P seconds at anaverage interval of Q seconds. Subjects were asked to readeach text snippet while supervising the driving.

4.2. Annotations

We annotated 53,983 sequences of approximately 10seconds sampled randomly from within the data for at-tended awareness. While inferring awareness is difficultand subjective, and probing the subject’s working memorydirectly is impossible, we devised a reproducible annota-tion scheme to explore a third person’s estimate of a per-son’s attended awareness. Our annotation protocol lever-ages the fact that humans are able to develop a theory ofmind of other peoples’ mental states from cues inherent ineye gaze [28]. While the labels provided (as in any manu-ally annotated dataset) are imperfect and have certain limi-tations, they are an important first step towards data-drivenmodeling of awareness. Annotators watched a video snip-pet where the subject’s gaze was marked by two circles cen-tered at the gaze point. One circle (green) size was set to thediameter of a person’s central foveal vision area (2 degrees)at the viewing distance. Another circle (red) was set to adiameter four times the foveal vision circle. At the end ofthe video snippet, a random location was chosen and theannotators were asked whether they believe the subject hasattended to that location on a scale between 1 and 5 (1-no,definitely not aware, 5-yes, definitely aware). Three differ-ent sampling types (object, edges, and non-objects) wereused for sampling the final location. The annotations werelinearly transformed to [0, 1] in LA and provided a super-visory signal that the network (awareness predictor) triedto match. Annotators were asked whether the cursor corre-sponded to a well-defined object, whether they would ex-pect to be aware of the location if they were driving andhow surprised they by the appearance/behavior of the high-lighted region. The annotations had good coverage acrossawareness levels and contained sufficient number of exam-ples of both highly aware as well as unaware examples.Figure 4 shows frames from an example annotation video.More annotation details and statistics in supp. material.

5. ResultsWe now demonstrate the results from our model on sev-

eral tasks of interest such as saliency, gaze refinement andattended awareness estimation. Our model, while appliedhere to a dataset which isn’t strictly egocentric, could bestraightforwardly extended to a true egocentric setting.

5.1. Saliency

In order to confirm that our model is capable of estimat-ing visual saliency, we trained it on the DR(eye)VE datasetimages and splits [40]. As our approach assumes individ-ual gaze samples per frame as the primary supervisory cue,we sampled gaze points from the fixation maps providedin the original dataset for every frame. We generated pG,and compared it against the ground truth gaze map. For the

3431

Figure 4: Annotation example. Left, center: Frames from the sequence with gaze estimates, along with the estimate of thecentral foveal vision area, and a larger red circle to mark what is probably outside the central vision area. Annotators wereguided to treat these as approximations of the subject’s attended gaze. Right: Annotators were asked to say whether thesubject has attended to the red cross region and is aware of it. See the supp. mat. for additional details on the annotation.

standard measures of KL, cross-correlation and informa-tion gain (see [40] for details), we obtained 1.734, 0.565,and −0.0002 respectively, comparing favorably to other al-gorithms tested on that dataset, such as [14, 57].

5.2. Gaze refinement

In the following experiments, we show how our model isable to correct imperfect gaze estimates. In driving, drivergaze estimates can be obtained using camera-based drivermonitoring systems (DMS). However, these estimates canbe highly noisy due to the inherent process noise introducedduring gaze tracking, or biased due to calibration errors inthe DMS [26]. Our hypothesis is that knowledge of wherethe person might look in the scene can help the model re-fine noisy and biased (miscalibrated) coarse estimates. Wedescribe two experiments that were conducted to addressthese typical modes of failure. For all experiments herein,we trained our model with a training dataset encompass-ing 8 urban day-time videos from our dataset with the high-est subject and task coverage and adopted a fixed 80%/20%non-overlapping training/test split.

5.2.1 Gaze Denoising

For this experiment the gaze input to the gaze transformmodule is corrupted by noise, mimicking imperfect knowl-edge of a person’s gaze. We use our model to try to identifythe correct gaze point taking scene saliency into consider-ation. We use a spatially-varying additive white Gaussiannoise model, where the input to the network xnoisy is com-puted according to [27]:

x(i)noisy(t) =x

(i)true(t) + η(i), η(i) ∼ N (0, (σ(i))2), (9)

σ(i) =max(σ(i)n , w ∗ |x(i) − x

(i)0 |)

where η is the additive noise and the standard deviation σ(i)

increases as we get further from the center of the imagealong each coordinate x(i) ∈ {x(1), x(2)}. Our networkdenoises the gaze input by leveraging scene saliency infor-mation as encoded in the network. Fusing the noisy gaze lo-cation allows us to surpass the capability of a pure-saliencybased model. The latter merely finds saliency image peaks

that are close to the noisy gaze location and would be thestraightforward way of incorporating saliency information.We also compare to an approach that relates gaze to thenearest object. In all cases we use a meanshift approach [13]to find nearby objects or peaks, with a standard deviationgiven by σn

√(H2 +W 2), where H and W are the dimen-

sions of the map. The results are summarized in Table 1 anddemonstrate significant improvement with MAAD.

Scenario Raw OBJ SAL MAADσn = 0.05 15.7 62.0 25.7 18.1σn = 0.10 28.1 55.6 25.4 19.8σn = 0.15 40.9 53.3 24.9 21.0σn = 0.20 51.7 53.1 24.9 22.1

Table 1: Mean absolute error (in pixels) of noisy gazerecovery based on object attention: meanshift into nearbyobjects (OBJ), meanshift according to pure saliency map(SAL), and meanshift correction based on the MAAD gazemap, for different noise levels. Our approach improvedupon other alternatives over a wide variety of input noiselevels, far beyond the noise level at train time (σn = 0.03).

5.2.2 Gaze Recalibration

We model imperfect calibration as an affine transformation.For this experiment, the DMS gaze input, xnoisy, to the gazetransform module is given by:

xnoisy(t) = Tcorrect (Tcorrupt (x)) (10)

where Tcorrect, Tcorrupt are both 2D affine transforms. Wemodel the correcting transform, Tcorrect, as an MLP with onehidden layer. The corruption transform is created by addingelement-wise zero-mean Gaussian noise with standard devi-ation σ2

n to the transformation matrix and vector of an iden-tity transform. We show the reduction in average error aftercalibration in Table 2. By leveraging saliency information,we are able to naturally compensate for the calibration er-rors using the model.

5.3. Driver Attended Awareness

In this experiment, we measure the model’s ability to in-fer attended awareness. We do so by measuring the model’s

3432

Scenario Before Afterσn = 0.1 0.23 0.13σn = 0.2 0.58 0.16σn = 0.3 1.12 0.33σn = 0.5 1.23 0.59

Table 2: The error was computed as the sum of the mean-squared error of the elements of the Tcorrect and T−1

corrupt be-fore and after recovery from miscalibration. For each noiselevel the errors were computed by averaging over eight op-timization runs. In Figure 4 (supp. material) we show thegaze map emitted by the model before and after recalibra-tion with miscalibrated gaze input.

Noise level MSE, FG MSE, MAADσn = 0.01 0.357 0.138σn = 0.05 0.359 0.135σn = 0.1 0.425 0.138σn = 0.15 0.464 0.140

Table 3: Mean squared error awareness estimates withspatio-temporal Gaussian with optic flow (FG) and MAAD,as a function of input gaze noise level. Our approach signif-icantly outperforms the baseline and is robust to the noisepresent in the gaze input.

agreement with annotated third person estimate of aware-ness. We compare our approach to the following alternativeof filtered gaze (FG) using a spatio-temporal Gaussian filter.We convolve each of the past gaze samples with a spatialGaussian and utilize optic flow to temporally propagate thegaze information to the subsequent frames and aggregatethem to form an awareness estimate. The optic flow mim-ics the subject’s notion of object permanence under motion,and the spatial Gaussian account for subjective uncertaintyaccumulated over time as well as track limitations with op-tic flow. The results are given in Table 3.

5.4. Ablation Experiments

We performed a series of leave-one out ablations to in-vestigate the impact of the various cost terms on the at-tended awareness estimation task. Both LATT and LAA arecrucial for more accurate awareness estimation (Table 4).

6. Discussion and ConclusionWe have introduced a new model which uses imperfect

gaze information and visual saliency to reason about per-ceived attended awareness within a single model. To trainand validate our model, we generated a dataset which in-cludes both high-accuracy gaze tracks as well as third per-son annotations for estimated attended awareness. MAAD

Ablation Awareness EstimateLATT 0.262LAA 0.199LT 0.159

LCON , · 0.164Full model 0.138

Table 4: Attended awareness estimation (mean squared er-ror) on the test set using different ablations of MAAD. Thetesting noise level was set to be σn = 0.1. More ablationresults are presented in the supplementary material.

Figure 5: Time evolution of gaze (left) and awareness(right) under the reading-text modifier condition. Top:The subject gazes straight ahead and is aware of the car infront. Middle: The subject attends to text block and thegaze map shifts accordingly. The awareness map has tworegions: one for the newly attended and the other for thepreviously attended location. Bottom: As the subject con-tinues to read the text, the gaze map is localized in the bot-tom right, and the awareness map correctly captures howthe subject is unaware of the scene in front.

can easily be extended to work with multi-image streams orscene graphs. Although our subjects viewed pre-collectedvideo stimuli as opposed to being part of a true egocentricvision-action loop, one advantage is that we could acquiremultiple observations of the same video, enabling the studyof distributions rather than single scanpaths of attention forany given video. Our dataset can be compared in manyways to the related egocentric driving dataset from [40].Extending our work and model to study subjects who arein control of real vehicles is a topic for future work.

3433

References[1] Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari. Mea-

suring the objectness of image windows. IEEE Transactionson Pattern Analysis and Machine Intelligence, 34(11):2189–2202, 2012.

[2] Sonia Baee, Erfan Pakdamanian, Inki Kim, Lu Feng, VicenteOrdonez, and Laura Barnes. Medirl: Predicting the visualattention of drivers via maximum entropy deep inverse rein-forcement learning. arXiv preprint arXiv:1912.07773, 2019.

[3] Loris Bazzani, Hugo Larochelle, and Lorenzo Torresani. Re-current mixture density network for spatiotemporal visual at-tention. In International Conference on Learning Represen-tations, 2017.

[4] Thierry Bellet, Pierre Mayenobe, Jean-Charles Bornard, Do-minique Gruyer, and Bernard Claverie. A computationalmodel of the car driver interfaced with a simulation plat-form for future virtual human centred design applications:Cosmo-sivic. Engineering Applications of Artificial Intelli-gence, 25(7):1488–1504, 2012.

[5] Paul J Boon, Jan Theeuwes, and Artem V Belopolsky. Up-dating visual–spatial working memory during object move-ment. Vision Research, 94:51–57, 2014.

[6] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li.Salient object detection: A benchmark. IEEE Transactionson Image Processing, 24(12):5706–5722, 2015.

[7] Ali Borji and Laurent Itti. State-of-the-art in visual atten-tion modeling. IEEE Transactions on Pattern Analysis andMachine Intelligence, 35(1):185–207, 2012.

[8] Ali Borji, Dicky N Sihite, and Laurent Itti. Probabilisticlearning of task-specific visual attention. In IEEE Confer-ence on Computer Vision and Pattern Recognition, pages470–477. IEEE, 2012.

[9] Timothy F Brady and Joshua B Tenenbaum. A probabilis-tic model of visual working memory: Incorporating higherorder regularities into working memory capacity estimates.Psychological Review, 120(1):85, 2013.

[10] Neil DB Bruce, Christopher Catton, and Sasa Janjic. Adeeper look at saliency: Feature contrast, semantics, and be-yond. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 516–524, 2016.

[11] Dario Cazzato, Marco Leo, Cosimo Distante, and HolgerVoos. When i look into your eyes: A survey on computervision contributions for human gaze estimation and tracking.Sensors, 20(13):3739, 2020.

[12] Marvin M Chun. Visual working memory as visual at-tention sustained internally over time. Neuropsychologia,49(6):1407–1409, 2011.

[13] Dorin Comaniciu and Peter Meer. Mean shift: A robust ap-proach toward feature space analysis. IEEE Transactions onPattern Analysis and Machine Intelligence, 24(5):603–619,2002.

[14] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and RitaCucchiara. A deep multi-level network for saliency predic-tion. In International Conference on Pattern Recognition,pages 3488–3493. IEEE, 2016.

[15] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and RitaCucchiara. Predicting human eye fixations via an lstm-based

saliency attentive model. IEEE Transactions on Image Pro-cessing, 27(10):5142–5154, 2018.

[16] David Crundall, Peter Chapman, Steven Trawley, LynCollins, Editha Van Loon, Ben Andrews, and Geoffrey Un-derwood. Some hazards are more attractive than others:Drivers of varying experience respond differently to differenttypes of hazard. Accident Analysis & Prevention, 45:600–609, 2012.

[17] Tao Deng, Hongmei Yan, Long Qin, Thuyen Ngo, and BSManjunath. How do drivers allocate their potential atten-tion? driving fixation prediction via convolutional neuralnetworks. IEEE Transactions on Intelligent TransportationSystems, 21(5):2146–2154, 2019.

[18] A. Doshi and M. M. Trivedi. On the roles of eye gazeand head dynamics in predicting driver’s intent to changelanes. IEEE Transactions on Intelligent Transportation Sys-tems, 10(3):453–462, 2009.

[19] Mica R Endsley. Toward a theory of situation awareness indynamic systems. Human Factors, 37(1):32–64, 1995.

[20] Johan Engstrom, Emma Johansson, and Joakim Ostlund. Ef-fects of visual and cognitive load in real and simulated mo-torway driving. Transportation Research Part F: Traffic Psy-chology and Behaviour, 8(2):97–120, 2005.

[21] Jianwu Fang, Dingxin Yan, Jiahuan Qiao, and Jianru Xue.Dada: A large-scale benchmark and model for driver at-tention prediction in accidental scenarios. arXiv preprintarXiv:1912.12148, 2019.

[22] Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. Sal-icon: Reducing the semantic gap in saliency prediction byadapting deep neural networks. In Proceedings of the IEEEInternational Conference on Computer Vision, pages 262–270, 2015.

[23] Laurent Itti, Christof Koch, and Ernst Niebur. A modelof saliency-based visual attention for rapid scene analysis.IEEE Transactions on Pattern Analysis and Machine Intelli-gence, 20(11):1254–1259, 1998.

[24] Tilke Judd, Krista Ehinger, Fredo Durand, and Antonio Tor-ralba. Learning to predict where humans look. In 2009 IEEE12th International Conference on Computer Vision,, pages2106–2113. IEEE, 2009.

[25] Daniel Kaiser, Timo Stein, and Marius V Peelen. Real-worldspatial regularities affect visual working memory for objects.Psychonomic Bulletin & Review, 22(6):1784–1790, 2015.

[26] Anuradha Kar and Peter Corcoran. Performance evaluationstrategies for eye gaze estimation systems with quantitativemetrics and visualizations. Sensors, 18(9):3151, 2018.

[27] Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Ma-tusik, and Antonio Torralba. Gaze360: Physically uncon-strained gaze estimation in the wild. In Proceedings of theIEEE/CVF International Conference on Computer Vision,pages 6912–6921, 2019.

[28] Saara Khalid, Jason C Deska, and Kurt Hugenberg. The eyesare the windows to the mind: Direct eye gaze triggers the as-cription of others minds. Personality and Social PsychologyBulletin, 42(12):1666–1677, 2016.

[29] Diederik P Kingma and Jimmy Ba. Adam: A methodfor stochastic optimization. In International Conference onLearning Representations, 2015.

3434

[30] Katja Kircher and Christer Ahlstrom. The driver distrac-tion detection algorithm AttenD. In Trent W. Victor MichaelA. Regan, John D. Lee, editor, Driver Distraction and Inat-tention, pages 327–348. Ashgate, 2013.

[31] Alexander Kirillov, Kaiming He, Ross Girshick, CarstenRother, and Piotr Dollar. Panoptic segmentation. In Pro-ceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 9404–9413, 2019.

[32] Matthias Kummerer, Zoya Bylinskii, Tilke Judd, AliBorji, Laurent Itti, Fredo Durand, Aude Oliva, andAntonio Torralba. MIT/Tubingen Saliency Benchmark.https://saliency.tuebingen.ai/.

[33] Congyan Lang, Tam V Nguyen, Harish Katti, Karthik Ya-dati, Mohan Kankanhalli, and Shuicheng Yan. Depth mat-ters: Influence of depth cues on visual saliency. In EuropeanConference on Computer Vision, pages 101–115. Springer,2012.

[34] Yin Li, Zhefan Ye, and James M Rehg. Delving into ego-centric actions. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 287–295,2015.

[35] Tal Makovski and Yuhong V Jiang. The role of visual work-ing memory in attentive tracking of unique objects. Journalof Experimental Psychology: Human Perception and Perfor-mance, 35(6):1687, 2009.

[36] Sebastien Marcel and Yann Rodriguez. Torchvision themachine-vision package of torch. In Proceedings of the18th ACM International Conference on Multimedia, MM’10, page 14851488, New York, NY, USA, 2010. Associa-tion for Computing Machinery.

[37] Stefan Mathe and Cristian Sminchisescu. Actions in the eye:Dynamic gaze datasets and learnt saliency models for visualrecognition. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 37(7):1408–1424, 2014.

[38] Jason McCarley, Christopher Wickens, Juliana Goh, andWilliam Horrey. A computational model of atten-tion/situation awareness. Proceedings of the Human Factorsand Ergonomics Society Annual Meeting, 46, 09 2002.

[39] Kyle Min and Jason J Corso. Tased-net: Temporally-aggregating spatial encoder-decoder network for videosaliency detection. In Proceedings of the IEEE InternationalConference on Computer Vision, pages 2394–2403, 2019.

[40] Andrea Palazzi, Davide Abati, Francesco Solera, Rita Cuc-chiara, et al. Predicting the driver’s focus of attention: the dr(eye) ve project. IEEE Transactions on Pattern Analysis andMachine Intelligence, 41(7):1720–1733, 2018.

[41] Junting Pan, Elisa Sayrol, Xavier Giro-i Nieto, KevinMcGuinness, and Noel E O’Connor. Shallow and deep con-volutional networks for saliency prediction. In IEEE Con-ference on Computer Vision and Pattern Recognition, pages598–606, 2016.

[42] Junting Pan, Elisa Sayrol, Xavier Giro-i Nieto, Cristian Can-ton Ferrer, Jordi Torres, Kevin McGuinness, and Noel EOConnor. Salgan: Visual saliency prediction with adver-sarial networks. In CVPR Scene Understanding Workshop(SUNw), 2017.

[43] Sylvain Paris and Fredo Durand. A fast approximation of thebilateral filter using a signal processing approach. Interna-tional Journal of Computer Vision, 81(1):24–52, 2009.

[44] Pietro Perona and Jitendra Malik. Scale-space and edge de-tection using anisotropic diffusion. IEEE Transactions onPattern Analysis and Machine Intelligence, 12(7):629–639,1990.

[45] Shafin Rahman, Sejuti Rahman, Omar Shahid, Md Tah-meed Abdullah, and Jubair Ahmed Sourov. Classifying eye-tracking data using saliency maps. arXiv, 2020.

[46] Timothy J Ricker and Nelson Cowan. Loss of visual work-ing memory within seconds: The combined use of refresh-able and non-refreshable features. Journal of ExperimentalPsychology: Learning, Memory, and Cognition, 36(6):1355,2010.

[47] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmen-tation. In International Conference on Medical Image Com-puting and Computer-Assisted Intervention, pages 234–241.Springer, 2015.

[48] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.Regularization with stochastic transformations and perturba-tions for deep semi-supervised learning. In NeurIPS, 2016.

[49] Dario D Salvucci. An integrated model of eye movementsand visual encoding. Cognitive Systems Research, 1(4):201–220, 2001.

[50] JW Senders, AB Kristofferson, WH Levison, CW Dietrich,and JL Ward. The attentional demand of automobile driving.Highway Research Record, 1526(195), 1967.

[51] Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato.Calibration-free gaze sensing using saliency maps. In Pro-ceedings of the Twenty-Third IEEE Conference on ComputerVision and Pattern Recognition, pages 2667–2674, 2010.

[52] Benjamin W Tatler. Characterising the visual buffer: Real-world evidence for overwriting early in each fixation. Per-ception, 30(8):993–1006, 2001.

[53] Benjamin W Tatler, Nicholas J Wade, Hoi Kwan, John MFindlay, and Boris M Velichkovsky. Yarbus, eye movements,and vision. i-Perception, 1(1):7–27, 2010.

[54] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. arXiv preprintarXiv:2003.12039, 2020.

[55] Antigoni Tsiami, Petros Koutras, and Petros Maragos.Stavis: Spatio-temporal audiovisual saliency network. InIEEE/CVF Conference on Computer Vision and PatternRecognition (CVPR), June 2020.

[56] Wenguan Wang, Jianbing Shen, and Haibin Ling. A deepnetwork solution for attention and aesthetics aware photocropping. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 41(7):1531–1544, 2018.

[57] Wenguan Wang, Jianbing Shen, and Ling Shao. Consistentvideo saliency using local gradient flow optimization andglobal refinement. IEEE Transactions on Image Processing,24(11):4185–4196, 2015.

[58] Benjamin Wolfe, Jonathan Dobres, Ruth Rosenholtz, andBryan Reimer. More than the useful field: Considering pe-ripheral vision in driving. Applied Ergonomics, 65:316–325,2017.

3435

[59] Bertram Wortelen, Martin Baumann, and Andreas Ludtke.Dynamic simulation and prediction of drivers attention dis-tribution. Transportation Research part F: Traffic Psychol-ogy and Behaviour, 21:278–294, 2013.

[60] Min Wu, Tyron Louw, Morteza Lahijanian, Wenjie Ruan,Xiaowei Huang, Natasha Merat, and Marta Kwiatkowska.Gaze-based intention anticipation over driving manoeuvresin semi-autonomous vehicles. In International Conferenceon Intelligent Robots and Systems, 2019.

[61] Ye Xia, Danqing Zhang, Jinkyu Kim, Ken Nakayama, KarlZipser, and David Whitney. Predicting driver attention incritical situations. pages 658–674. Springer, 2018.

[62] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, andKevin Murphy. Rethinking spatiotemporal feature learning:Speed-accuracy trade-offs in video classification. In Pro-ceedings of the European Conference on Computer Vision(ECCV), pages 305–321, 2018.

[63] Alfred L Yarbus. Eye Movements and Vision. Plenum Press,1967.

3436