Abnormal gait detection with rgb d devices using joint motion history features

Abnormal Gait Detection with RGB-D Devices using Joint

Motion History FeaturesAlexandros Andre Chaaraoui, José Ramón Padilla-

López, Francisco Flórez-Revuelta

1st International Workshop on Understanding Human Activities through 3D Sensors (UHA3DS'15) hosted by 11th IEEE International Conference on Automatic Face and Gesture Recognition FG 2015,

May 8, Ljubljana, Slovenia

Outline

1. Motivation

2. Proposal

3. Joint motion history features

4. Classification

5. Experimental results

6. Conclusion

7. Future work

1. Motivation

- Demographic change“(...) by 2060, the ratio between working and retired people will have

passed from four-to-one to two-to-one in the EU.”

- Solutions:- Technological innovation- Assistive and autonomous care services- Low-cost infrastructure

1. Motivation

- Human gait as a metric- Person´s health- Well-being

- Human gait involves a complex mental coordination process.- Anomalies may indicate neurological diseases.

- Physical frailty is associated to an increased risk of cognitive impairment.

2. Proposal

- Providing support to a geriatrician performing gait analysis by means of:- Quantitative evaluation of a person’s gait

- Detection of deviations from the normal gait pattern

- Leading to an improved and earlier detection of such diseases.

- Solution based on:- Marker-less human body pose estimation

- Low-cost RGB-D devices


- RGB-D camera

- Skeletal representation of the human body ( )

- Normalisation ( )- Location- Scale- Rotation

Fig 1. Sample of a skeleton of J = 20 joints obtained with the Microsoft Kinect SDK.


- Skeletons are accumulated using a sliding window of τ frames.

- Joint coordinates are translated to the corresponding coordinates of a fixed size volume.

- Voxels are set to the value r that indicates the recentness of the motion.

Fig 2. This figure shows the superposition of skeletons that is applied to obtain the 3D volume.


- Balance between computational cost and temporal coverage- The feature is obtained each λ frames, .

- Dimensionality reduction- Dimensionality is high since we want to capture low-

level motion

- Axis projection: front, side and bottom views


- Feature fusion

Fig 3. Sample of front (a), side (b) and bottom (c) views that make up a specific instance of the JMH feature.

4. Classification

- BagOfKeyPoses method- https://github.com/DAIGroup/BagOfKeyPoses

- Machine learning method based on a bag of key poses model and sequence alignment

- Requires sequences of features with sequential or temporal relation

- Previously used for human action and gesture recognition

Chaaraoui, A. A., Climent-Pérez, P., & Flórez-Revuelta, F. (2013). Silhouette-based human action recognition using sequences of key poses. Pattern Recognition Letters, 34(15), 1799-1807.

https://github.com/DAIGroup/BagOfKeyPoses

https://github.com/DAIGroup/BagOfKeyPoses

4. Classification

- Most common feature instances (key poses) are learned based on K-means clustering.

- Temporal relation between key poses is learned based on templates of sequences of key poses.

- Substituting each feature with its nearest neighbour key pose.

- Recognition is performed based on sequence alignment and thresholding.

- Only normal samples are learned, since abnormal motion varies and is unknown a priori.


- SPHERE dataset (University of Bristol)- Microsoft Kinect, OpenNI SDK (15 joints)

- Staircase scenario (right leg lead, left leg lead, stop)

- Baseline method for abnormal gait detection based on non-linear manifold learning and Markov assumptions


- Self-recorded DAI gait dataset (DGD)- Microsoft Kinect 2, Microsoft Kinect SDK v2 (20 out of 25 joints)

- Corridor scenario (knee injury, foot dragging)

- Two evaluation methods: Continuous monitorisation (a) and cross-subject validation (b)


- Temporal evaluation- Intel i7-2600 CPU at 3.4 (GHz) with 8 (GB) of RAM

- 9000 FPS

- Compared to frame-wise skeletons:

- Feature extraction: 0.4x

- Matching: 9.3x

- Feature dimensionality: 400x

- Particularly suitable for real-time detection of human gait anomalies

6. Conclusions

- Method for abnormal gait detection- Novel feature for human motion analysis

- Using low-cost RGB-D devices- Support for gait analysis performed by

professionals in care centres or nurseries- Gait monitoring, rehabilitation and diagnosis of

cognitive impairment

- Intrinsic advantages- Low cost and intrusiveness, simple setup

7. Future work

- Consider specific joint motion- Consider joint neighbourhood- Further experimentation with data from

patients- Learn anomaly types and associated degrees of

frailty- Obtain multi-scale JMH to represent complex

behaviours

- Other applications of human motion analysis: ADL recognition

Thank you!

Q & A

Francisco Fló[email protected]

Bag of key poses

Abnormal gait detection with rgb d devices using joint motion history features

Technology

Transcript of Abnormal gait detection with rgb d devices using joint motion history features