Closed Captioning Matters: An Examination of the Use of Captioning for All Students
In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · •...
Transcript of In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · •...
![Page 1: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/1.jpg)
In-Home Daily-Life Captioning Using Radio Signals
Lijie Fan* Tianhong Li* Yuan Yuan Dina Katabi
MIT CSAIL
* denotes equal contribution
![Page 2: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/2.jpg)
How can I make sure grandma is fine?
![Page 3: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/3.jpg)
How can I make sure grandma is fine?Daily Life Captioning
08:30am: Grandma wakes up and leaves bedroom
10:30am: Grandma takes medicine and eats breakfast
02:00pm: Grandma is watching TV
![Page 4: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/4.jpg)
Camera is not acceptable
Camera
![Page 5: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/5.jpg)
How to do Daily Life Captioning?
![Page 6: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/6.jpg)
What about Radio-Frequency(RF) Signals?
RF Device
![Page 7: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/7.jpg)
RF signals are privacy-preserving …
RGB Video RF Signals
![Page 8: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/8.jpg)
RGB Video RF Signals
but are capable of capturing people’s movements and activities
![Page 9: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/9.jpg)
Challenge I. Object Information
![Page 10: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/10.jpg)
Challenge I. Object Information
![Page 11: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/11.jpg)
Challenge I. Object Information
![Page 12: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/12.jpg)
Solution I. Skeleton + Floormap
RF Signal
SkeletonGeneration
Network
Skeleton
![Page 13: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/13.jpg)
Floormap Illustration
Bed
Stove
Sink
TV
RF Device
Fridge
Wardrobe
Shelf
Window
Dish WasherSofa
Solution I. Skeleton + Floormap
X
Y
Table
![Page 14: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/14.jpg)
Challenge II. No Existing RF Captioning Dataset!
Can We Leverage Existing RGB Captioning Dataset?
![Page 15: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/15.jpg)
Solution II. Multi-modal Feature AlignmentRF+Floormap Feature Extraction
FeatureExtractionNetwork
RF Signal Floormap
+ 𝐮𝑃
![Page 16: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/16.jpg)
Solution II. Multi-modal Feature AlignmentRF+Floormap Feature Extraction
Paired Video 𝐗𝑃
VideoEncoder
FeatureExtractionNetwork
RF Signal Floormap
+
Video Feature Extraction
𝐮𝑃
𝐯𝑚𝑃 𝐯𝑛
𝑃
Spatial𝑷𝒐𝒐𝒍𝒊𝒏𝒈
![Page 17: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/17.jpg)
PairedData
Alignment Loss
ℒ𝑝𝑎𝑖𝑟
𝐿2
Solution II. Multi-modal Feature AlignmentRF+Floormap Feature Extraction
Paired Video 𝐗𝑃
VideoEncoder
FeatureExtractionNetwork
RF Signal Floormap
+
Video Feature Extraction
𝐮𝑃
𝐯𝑚𝑃 𝐯𝑛
𝑃
Spatial𝑷𝒐𝒐𝒍𝒊𝒏𝒈
![Page 18: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/18.jpg)
PairedData
Alignment Loss
ℒ𝑝𝑎𝑖𝑟
𝐿2
Solution II. Multi-modal Feature AlignmentRF+Floormap Feature Extraction
Paired Video 𝐗𝑃
VideoEncoder
FeatureExtractionNetwork
RF Signal Floormap
+
Video Feature Extraction
Unpaired Video 𝐗𝑈
VideoEncoder
𝐮𝑃
𝐯𝑚𝑃
𝐯𝑚𝑈 𝐯𝑛
𝑈
𝐯𝑛𝑃
Spatial𝑷𝒐𝒐𝒍𝒊𝒏𝒈
Spatial𝑷𝒐𝒐𝒍𝒊𝒏𝒈
![Page 19: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/19.jpg)
PairedData
Alignment Loss
ℒ𝑝𝑎𝑖𝑟
𝐿2
Solution II. Multi-modal Feature AlignmentRF+Floormap Feature Extraction
Paired Video 𝐗𝑃
VideoEncoder
FeatureExtractionNetwork
RF Signal Floormap
+
Video Feature Extraction
Unpaired Video 𝐗𝑈
VideoEncoder
𝐮𝑃
Unpaired DataAlignment Loss
ℒ𝑢𝑛𝑝𝑎𝑖𝑟𝐷𝑛𝐷𝑚
𝐯𝑚𝑃
𝐯𝑚𝑈 𝐯𝑛
𝑈
𝐯𝑛𝑃
Spatial𝑷𝒐𝒐𝒍𝒊𝒏𝒈
Spatial𝑷𝒐𝒐𝒍𝒊𝒏𝒈
![Page 20: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/20.jpg)
RF-Diary System Structure
![Page 21: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/21.jpg)
RF-Diary can caption people’s daily life in home …
RF Signals
Floormap
RGB Video
RF-Caption
A person enters the kitchen. He takes off his clothes, sits at table and starts playing laptop.
![Page 22: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/22.jpg)
Even when the light is off …
RF Signals
Floormap
RGB Video
RF-Caption
A person walks to the kitchen. He then pours water into a cup and drinks from it.
Not Applicable
![Page 23: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/23.jpg)
Quantitative Results
![Page 24: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/24.jpg)
Summary
• RF-Diary enables captioning people’s daily life in their home.
• RF-Diary uses radio signals as input to address the privacy issues ofcamera.
• RF-Diary achieves comparable results of camera-based captioningand keeps working under poor lighting or occluded scenarios.
![Page 25: In-Home Daily-Life Captioning Using Radio Signalsrf-diary.csail.mit.edu/slides/longtalk.pdf · • RF-Diary enables captioning people’s daily life in their home. • RF-Diary uses](https://reader034.fdocuments.in/reader034/viewer/2022051814/603a76edfa8c7472fe14db4d/html5/thumbnails/25.jpg)
For more information, please visit our webpage:
http://rf-diary.csail.mit.edu