CVPR 2008 June 24 – 26, 2008

Infrared camera: Mid wave: 3.0-5.0 microns Resolution: 640*512 pixels with 14bits Frame rate: 30/60/115 fps Sensitivity: about 25mKTo explore contact-free heart rate and

respiratory rate detection through measuring infrared light modulation emitted near superficial blood vessels or a nasal area.

Ming Yang2, Qiong Liu1, Thea Turner1, Ying Wu2

1 FX Palo Alto Laboratory, Inc., 3400 Hillview Ave., Palo Alto, CA 94304 2 Dept. of EECS, Northwestern Univ., 2145 Sheridan Rd., Evanston, IL 60208

Vital Sign Estimation from Passive Thermal Video

Experiments

Overview of our approach

Ground truth: ADI PowerLab 4/30

Test dataset: Age 20-60, F:8 and M:12

20 subjects for heart rate estimation

7 subjects for respiratory rate estimation

Accurate subject alignment for temporal signal extraction, e.g. involuntary muscular movements are inevitable.

Robust harmonic analysis with low signal-to-noise ratio (SNR) temperature modulation signal, e.g. modulation magnitude 0.1K vs. camera sensitivity 0.025K.

A novel contact-free vital sign measurement method.

Low risk of harm & convenience for quick deployment.

Potential applications: airport heath screening, long-term elder care, workplace preventive care, etc.

N. Sun, M.Garbey, A. Merla, I. Pavlidis. Imaging the cardiovascular pulse. CVPR 2005. (S)

S.Y. Chekmenev, A.A. Farag, E.A. Essock. Multiresolution approach for non-contact measurements of arterial pulse using thermal imaging. CVPR 2006 Workshop.

Goal

Motivations

Challenges

Reference frame

t = 1/ 60 s t = 25/ 60 s t = 26/ 60 s t = 27/ 60 s t = 58/ 60 s t = 59/ 60 st = 0/ 60 s

Frame differences

Pioneering work

Automatic ROI segmentation and alignment

Signal enhancement and outlier removal

Robust harmonic analysis

Vote

Dominant frequency componentRough manualinitialization

Isothermextraction

Optimal thresholdselection

3Dvisualization

Region-of-interests segmentation by thresholding the isotherms and alignment by contour tracking.

Signal enhancement using a non-linear filter, and outlier removal by pixels-of-interests clustering.

Robust harmonic analysis by dominant frequency voting. Subject alignment and

motion compensation

Thermal video captured byan infrared camera

Signal enhancementand denoising

Robust harmonicanalysis

ROIsegmentation

Contourtracking

Pixel ofinterests

clustering

Medianfiltering

Non-linearfiltering

Dominantfrequency voting

time

bpm

( , ) [ ( ) ( , )]j jH f F W t s tx x

Perform N-point (N=1024/2048/4096) FFT of all temperature signals of all pixels using a sliding window:

Non-linear filtering by taking the point-by-point minimum of a rectangle window Wr(t) and a Hamming window Wh(t)

Cluster H(xj, f ) in the band of interest (40-100 bpm for heart rates, and 6-30 bpm for respiratory rates) using K-means, then select the largest cluster to estimate.

( , ) min( [ ( ) ( , )], [ ( ) ( , )])j r j h jH f F W t s t F W t s tx x x

Subject #

fps # of frames

GT bpm

Est. bpm

Diff.

4 60 3000 18 15.8 -2.27 60 3000 17 15.1 -1.9

10 115 5000 11 11.8 +0.811 115 5000 17 16.8 -0.214 115 5000 16 13.9 -2.115 115 5000 15 13.1 -1.917 115 5000 20 18.5 -1.519 115 5000 16 15.2 -0.8

Segment the initial ROI by selecting the isotherm with the sharpest gradient.

Align the ROI by tracking the contour

Extract the temporal signals for individual pixels inside the ROI and denote by

2* 2

( ) ( ( ) ( ))

|| ( ) || || || , 0.001

t

t

t image t ext t

jt j j

E E E

I I

x x x

Respiratory rate estimation results

Heart rate estimation results

Point-by-point comparisonsSubject

#fps # of

framesGT

bpmEst. bpm

Diff. RMSE

1 30 2000 65.3 65.8 +0.5 1.92 30 2000 66.6 63.9 -2.7 3.93 30 1750 65.7 64.7 -1.0 3.34 60 3000 59.8 60.7 +0.9 2.55 60 3500 60.7 60.3 -0.4 3.36 60 2500 66.3 53.0 -3.3 3.97 60 3000 61.1 60.9 -0.2 2.38 115 5000 64.0 65.0 +1.0 3.89 115 5000 78.9 80.1 +1.2 1.9

10 115 5000 65.2 64.4 -0.8 1.711 115 5000 62.8 66.2 +3.4 4.212 115 5000 63.5 62.4 -1.1 3.213 115 5000 73.3 72.6 -0.7 1.814 115 5000 86.6 87.9 +1.3 4.915 115 5000 78.7 76.5 -2.2 3.116 115 5000 75.3 74.7 -0.7 1.917 115 5000 83.1 83.2 +0.1 2.118 115 5000 67.2 68.2 -1.0 1.319 115 5000 67.6 69.3 +1.7 2.820 115 5000 68.7 70.1 +1.4 2.9

The initial ROI segmentation results

( , )js tx

Insensitive to initialization and robust to gentle subject movement and facial expressions.

More stable estimation results compared with the state-of-the-art methods.

Conclusions

t