Non-invasive Techniques for Human Fatigue

MonitoringQiang Ji

Dept. of Electrical, Computer, and Systems Engineering

Rensselaer Polytechnic Instituteqji@ecse.rpi.edu

http://www.ecse.rpi.edu/homepages/qji

Visual BehaviorsVisual behaviors that typically reflect

a person's level of fatigue include

– Eyelid movement – Head movement – Gaze – Facial expressions

Eyelid Movements Tracking Eyes

Develop techniques that can robustly track eyes under different face orientations, illuminations, and large head movements.

Compute Eye movement parameters PERCLOS Average Eye Closure/Open Speed

(AECS)

Eyes tracking demo

PERCLOS measurement over time

Average Eye Closure Speed Over time

Gaze (Pupil Movements)

Real time gaze tracking No calibration is needed and

allows natural head movements !.

Gaze parameters Spatial gaze distribution overtime Ratio of fixation time to saccade time.

Gaze distribution over time while alert

Gaze distribution over time under fatigue

Head Movement Real time head pose tracking

Perform 3D face pose estimation from a single uncalibrated camera.

Head movement parameters Head tilt frequency over time Percentage of side views (PerSideV)

Facial Expressions Tracking facial features

Recognize certain facial expressions related to fatigue like yawning.

Building a database of fatigue expressions.

Facial expression demo

Fatigue Modeling• Knowledge of fatigue is uncertain and

from different levels of abstraction.

• Fatigue represents the affective state of an individual, is not observable, and can only be inferred.

Overview of Our Approach

Propose a probabilistic framework based on Bayesian Networks (BN) to

model fatigue.systematically integrate various

sources of information related to fatigue.

infer and predict fatigue from the available observations and the relevant contextual information.

Bayesian Networks Construction

• A BN model consists of target hypothesis variables (hidden nodes) and information variables (information nodes).

• Fatigue is the target hypothesis variable that we intend to infer.

• Other contextual factors and visual cues are the information nodes.

Causes for Fatigue

Major factors to cause fatigue include: Sleep quality. Circadian rhythm (time of day). Physical conditions. Working environment.

Bayesian Network Model for Monitoring Human Fatigue

Interface with Vision Interface with Vision ModuleModule

An interface has been developed to connect the output of the computer vision system with the information fusion engine.

The interface instantiates the evidences of the fatigue network, which then performs fatigue inference and displays the fatigue index in real time.

Conclusions

• Developed non-intrusive real-time computer vision techniques to extract multiple fatigue parameters related to eyelid movements, gaze, head movement, and facial expressions.

• Develop a probabilistic framework based on Bayesian networks to model and integrate contextual and visual cues information for fatigue monitoring.