CAGR CoachingBusinessBlueprint Transcript 2015...Michelle: Michelle:!!
Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F....
Transcript of Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F....
![Page 1: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/1.jpg)
Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F. Fouhey1
University of Michigan1, Johns Hopkins University2
Sponsored by Procter & Gamble and Nokia Networks
![Page 2: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/2.jpg)
100Days Of Hands Video Dataset
furniture gardening housework packing
diy drink food boardgame
puzzle repair study vlog
131 Days
12 Categories
19.2K Uploaders
27.3K Videos
![Page 3: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/3.jpg)
100K Frame-level Annotations
• Box around hand
• Side (left / right)
• Contact (no / self / other /
portable / furniture)
• Box around object in contact
• Association
![Page 4: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/4.jpg)
Method
Hand
Box
Side
State
Offset
Object
Box
Faster-RCNN
Standard Detection
Classification (if hand)
Classification (if hand)
Regression (if in contact)
Standard Detection
(another class)
[1] Ren et al. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS 2015.
[2] Yang et al. A Faster Pytorch Implementation of Faster R-CNN (https://github.com/jwyang/faster-rcnn.pytorch).
simple greedy matching
on hands and objects
![Page 5: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/5.jpg)
Cross-Dataset Hand Detection
Faster RCNN with Resnet-101 backbone, AP (TP = IoU > 0.5)
100DOH
VLOG
VIVA
Ego
VGG
>80 >70 >60 >40 <40
Train
on
TV+Co
VGG100DOH VLOG VIVA TV+CoEgo
73.990.1 86.4 86.5 65.490.8
21.5
17.4
23.6
40.7
27.7
32.6
90.8
44.9
10.1
8.090.7
56.8
64.678.6 77.5 76.6 59.283.2
79.6
77.4
61.4
56.2
61.7
61.5 74.9
78.8
69.9
66.6
62.4
63.0
![Page 6: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/6.jpg)
Qualitative Results
![Page 7: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/7.jpg)
Method
Hand
Box
Side
State
Offset
Object
Box
L
R
Hasson
et al.
Given location and side, can predict
low-dimensional parameterization of hand
[1] Hasson et al. Learning joint reconstruction of hands and manipulated objects. CVPR 2019.
[2] Romero et al. Embodied Hands: Modeling and Capturing Hands and Bodies Together. SIGGRAPH Asia 2017.
![Page 8: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/8.jpg)
Enabling MANO at Scale
[1] Hasson et al. Learning joint reconstruction of hands and manipulated objects. CVPR 2019.
[2] Romero et al. Embodied Hands: Modeling and Capturing Hands and Bodies Together. SIGGRAPH Asia 2017.
![Page 9: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/9.jpg)
Hand Predictions
Input Prediction Input Prediction
![Page 10: Dandan Shan , Jiaqi Geng *, Michelle Shu *, David F. Fouheyfouheylab.eecs.umich.edu/~dandans/projects/100DOH/file/slides.pdf · Dandan Shan1, Jiaqi Geng1*, Michelle Shu2*, David F.](https://reader034.fdocuments.in/reader034/viewer/2022050503/5f958258291b6a7c735b45b1/html5/thumbnails/10.jpg)
Code and Data Available!
Hand Detection
Model (detectron2)Full Hand State Detection
Model
Mesh Quality Assessment100K labeled Frames
Full Hand State Detection
Model (egocentric)
100DOH Video Dataset
100 Days
Of Hands