MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson...
-
Upload
reginald-quinn -
Category
Documents
-
view
213 -
download
0
Transcript of MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson...
![Page 1: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/1.jpg)
MIT AI LabViola & Grimson
Variable Viewpoint Reality
Professor Paul Viola & Professor Eric Grimson
Collaborators: Jeremy De Bonet, John Winn, Owen Ozier,Chris Stauffer, John Fisher, Kinh Tieu,Dan Snow, Tom Rikert, Lily Lee,Raquel Romano, Huizhen Yu, Mike Ross,Nick Matsakis, Jeff Norris, Todd AtkinsMark Pipes
![Page 2: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/2.jpg)
MIT AI LabViola & Grimson
The BIG picture: User selected viewing of sporting events.
• show me that play from the viewpoint of the goalie
•… from the viewpoint of the ball
•… from a viewpoint along the sideline
• what offensive plays does Brazil run from this formation
• how often has Italy had possession in the offensive zone
![Page 3: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/3.jpg)
MIT AI LabViola & Grimson
The BIG picture: User selected viewing of sporting events.
• Let me see my son’s motion from the following viewpoint
• Let me see what has changed in his motion in the past year
• Show me his swing now and a week ago
• How often does he swing at pitches low and away
• What is his normal sequence of pitches with men on base and less than 2 outs
![Page 4: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/4.jpg)
MIT AI LabViola & Grimson
A wish list of capabilities
• Construct a system that will allow each/every user to observe any viewpoint of a sporting event.
• Provide high level commentary/statistics– analyze plays
![Page 5: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/5.jpg)
MIT AI LabViola & Grimson
A wish list of capabilities
• Recover human dynamics • Search databases for similar events
![Page 6: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/6.jpg)
MIT AI LabViola & Grimson
VVR Spectator Environment
• Build an exciting, fun, high-profile system– Sports: Soccer, Hockey, Tennis, Basketball, Baseball
– Drama, Dance, Ballet
• Leverage MIT technology in:– Vision/Video Analysis
• Tracking, Calibration, Action Recognition
• Image/Video Databases
– Graphics
• Build a system that provides data available nowhere else…– Record/Study Human movements and actions
– Motion Capture / Motion Generation
![Page 7: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/7.jpg)
MIT AI LabViola & Grimson
Window of Opportunity
• 20-50 cameras in a stadium– Soon there will be many more
• US HDTV is digital – Flexible, very high bandwidth digital transmissions
• Future Televisions will be Computers– Plenty of extra computation available
– 3D Graphics hardware will be integrated
• Economics of sports– Dollar investments by broadcasters is huge (Billions)
• Computation is getting cheaper
![Page 8: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/8.jpg)
MIT AI LabViola & Grimson
For example …
Computed using a single view…
some steps by hand
![Page 9: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/9.jpg)
MIT AI LabViola & Grimson
ViewCube: Reconstructing action & movement
• Twelve cameras, computers, digitizers
• Parallel software for real-time processing
![Page 10: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/10.jpg)
MIT AI LabViola & Grimson
The View from ViewCube
Mul
ti-c
amer
a M
ovie
![Page 11: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/11.jpg)
MIT AI LabViola & Grimson
Robust adaptive tracker
Pixel Consistent
with Background?
Video Frames Adaptive Background Model
X,Y,Size,Dx,DyX,Y,Size,Dx,Dy
X,Y,Size,Dx,Dy
Local Tracking Histories
![Page 12: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/12.jpg)
MIT AI LabViola & Grimson
Examples of tracking moving objects
• Example of tracking results
![Page 13: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/13.jpg)
MIT AI LabViola & Grimson
Dynamic calibration
![Page 14: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/14.jpg)
MIT AI LabViola & Grimson
Multi-camera coordination
![Page 15: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/15.jpg)
MIT AI LabViola & Grimson
Mapping patterns to groundplane
![Page 16: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/16.jpg)
MIT AI LabViola & Grimson
Projecting Silhouettes to form3D Models
3D R
econ
stru
ctio
n M
ovieReal-time 3D
Reconstructionis computed by intersecting silhouettes
![Page 17: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/17.jpg)
MIT AI LabViola & Grimson
First 3D reconstructions ...
3D M
ovem
ent
Rec
onst
ruct
ion
Mov
ie
![Page 18: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/18.jpg)
MIT AI LabViola & Grimson
A more detailed reconstruction…
Model
![Page 19: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/19.jpg)
MIT AI LabViola & Grimson
Finding an articulate human body
Segment
3D Model
VirtualHuman
Human
![Page 20: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/20.jpg)
MIT AI LabViola & Grimson
Automatically generated result:
Bod
y T
rack
ing
Mov
ie
![Page 21: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/21.jpg)
MIT AI LabViola & Grimson
Analyzing Human Motion
• Key Difficulty: Complex Time Trajectories
Complex Inter-dependencies
• Our Approach: Multi-scale statistical models
![Page 22: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/22.jpg)
MIT AI LabViola & Grimson
Detect Regularities & Anomalies in Events?
![Page 23: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/23.jpg)
MIT AI LabViola & Grimson
Example track patterns
• Running continuously for almost 3 years– during snow, wind, rain, dark of night, …
– have processed 1 Billion images
• one can observe patterns over space and over time• have a machine learning method that detects
patterns automatically
![Page 24: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/24.jpg)
MIT AI LabViola & Grimson
Automatic activity classification
![Page 25: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/25.jpg)
MIT AI LabViola & Grimson
Example categories of patterns
• Video of sorted activities
![Page 26: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/26.jpg)
MIT AI LabViola & Grimson
Analyzing event sequencesHistogram of activity over a single day
12am 6am 12pm 6pm 12pm
12am 6am 12pm 6pm 12pm
12am 6am 12pm 6pm 12pm
12am 6am 12pm 6pm 12pm
Resultingclassifier
cars(1564 total
with3.4% FP)
groups of people
(712 total with2.2% FP)
people(1993 total with
.1% FP)
clutter/lighting effects
(647 total with 10.5% FP)
![Page 27: MIT AI Lab Viola & Grimson Variable Viewpoint Reality Professor Paul Viola & Professor Eric Grimson Collaborators: J eremy De Bonet, John Winn, Owen Ozier,](https://reader035.fdocuments.in/reader035/viewer/2022062719/56649ec55503460f94bd0249/html5/thumbnails/27.jpg)
MIT AI LabViola & Grimson
…and this works for other problems
• Sporting events• Eldercare monitoring• Disease progression tracking
– Parkinson’s
• … anything else that involves capturing, archiving, recognizing and reconstructing events!