Large-scale Machine Learning using DryadLINQ
description
Transcript of Large-scale Machine Learning using DryadLINQ
![Page 1: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/1.jpg)
Large-scale Machine Learning using DryadLINQ
Mihai BudiuMicrosoft Research, Silicon Valley
Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop
Stanford, June 11, 2019
![Page 2: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/2.jpg)
2
Goal of DryadLINQ
![Page 3: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/3.jpg)
3
Software Stack
Windows Server
Cluster services
Cluster storage
Dryad
DryadLINQ
Windows Server
Windows Server
Windows Server
Applications
.Net + LINQ
![Page 4: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/4.jpg)
4
Dryad = Execution Layer
Job (application)
Dryad
Cluster
Pipeline
Unix Shell
Machine≈
![Page 5: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/5.jpg)
Collection
.NET objects of type T
LINQ Data Model
![Page 6: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/6.jpg)
6
LINQ Language Summary
Where (filter)Select (map)GroupByOrderBy (sort)Aggregate (fold)Join
Input
![Page 7: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/7.jpg)
7
LINQ
Dryad
=> DryadLINQ
![Page 8: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/8.jpg)
8
DryadLINQ Data Model
Partition
Collection
.Net objects
![Page 9: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/9.jpg)
9
Collection<T> collection;
static bool IsLegal(Key c);
var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};
DryadLINQ = LINQ + Dryad
C#
collection
results
C# C# C#
Code
Dryad job
Data
![Page 10: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/10.jpg)
10
Example: Natal Training
![Page 11: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/11.jpg)
11
Natal Problem
• Recognize players from depth map• At frame rate• Low resource usage
![Page 12: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/12.jpg)
12
Learn from Data
Motion Capture(ground truth)
Classifier
Training examplesMachine learning
Rasterize
![Page 13: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/13.jpg)
13
Running on Xbox
![Page 14: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/14.jpg)
14
Cluster-based training
Classifier
Training examples
Dryad
DryadLINQ
Machine learning
![Page 15: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/15.jpg)
![Page 16: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/16.jpg)
You can have it!
• Dryad+DryadLINQ available for download– Academic license– Commercial evaluation license
• Runs on Windows HPC platform• Dryad is in binary form, DryadLINQ in source• Requires signing a 3-page licensing agreement• http://connect.microsoft.com/site/sitehome.aspx?SiteID=891
![Page 17: Large-scale Machine Learning using DryadLINQ](https://reader034.fdocuments.in/reader034/viewer/2022050723/56816384550346895dd46a5f/html5/thumbnails/17.jpg)
Conclusions
17
Visual StudioLINQDryad
17
=