Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy...
Transcript of Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy...
![Page 1: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/1.jpg)
Managing Director, Microsoft Research Asia
Hsiao-Wuen Hon
Microsoft Research Asia Update
![Page 2: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/2.jpg)
MSRA Overview
People & Organization
Research
Products & Technology Transfer
External Collaboration
![Page 3: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/3.jpg)
Microsoft Research
Research lab locationsRedmond, Washington (Sep, 1991)Cambridge, UK (July, 1997)Beijing, China (Nov, 1998)Silicon Valley, California (July, 2001)Bangalore, India (Jan, 2005)Boston, MA (Sep, 2008)New York City (May, 2012)
Cambridge Asia India
![Page 4: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/4.jpg)
People
New HiresJiang BIAN, Yahoo Research
Karlsson BORJE, Nokia
Xun GUO, MediaTek
Kwantaek KIM, LG
Zhuang LI, MSR SVC
Xin LIU, UC Davis
Tim PAN, GEMFOR Tech
Takaaki SHIRATORI, CMU/Disney
Hong TAN, Purdue – Haptics & HCI
Yingcai WU, UC Davis
Oversea Ph.D’sLiang CHEN (UC San Diego), Koji YATANI(Univ. of Toronto),
15 domestic Ph.D’s & Masters
![Page 5: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/5.jpg)
People and Research Areas
Baining GUO
Visual Computing - Yi MA
Graphics - Baining GUO
Multimedia - Shipeng LI, Feng WU
Theory - Baining GUO
Wei-Ying MAWeb/Data Search & Mining – Jirong WEN, Haixun WANGNatural Language Processing – Ming ZHOU Multimedia Search & Mining – Lei ZHANGKnowledge Mining – Yong RUI, Chin-Yew LINInternet Economics – Tie-Yan LIUMachine Learning – Zheng CHEN
Feng ZHAOSystem - Zheng ZHANG, Lidong ZHOU
Network & Wireless - Yongguang ZHANG
Hardware - Feng-Hsiung HSU
Software Analytics - Dongmei ZHANG
Mobile & Sensing – Feng ZHAO
Hsiao-Wuen HON
HCIHong TAN
SpeechFrank SOONG
Innovation EngineeringJonathan TIEN
Technology StrategyEric CHANG
![Page 6: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/6.jpg)
MSRA Focus Areas
Natural User Interface
CS FundamentalsSearch and Online
Ads
Data-IntensiveComputing
Multimedia
![Page 7: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/7.jpg)
Research GroupsHardware Computing Multimedia Search and Mining
Human Computer Interaction Nature Language Computing
Internet Graphics Software Analytics
Internet Media Speech
Knowledge Mining System
Machine Learning Theory
Media Computing Visual Computing
Mobile and Sensing Systems Web Search and Mining
Innovation Engineering Group Wireless and Networking
Data Management, Analytics and Services
Internet Economics and Computational Advertising
![Page 8: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/8.jpg)
Academic ImpactPublications:
CVPR 2012/11/10/09/08/07 26/13/18/8/15/16 papersSIGIR 2012/11/10/09/08/07 6/8/5/8/8/7 papersSIGGRAPH 2012/11/10/09/08/07 9/4/7/6/8/7 papersWWW 2012/11/10/09/08/07 3/3/6/9/8/6 papersACL 2012/11/10/09/08/07 7/5/4/6/5/4 papersSIGKDD 2012/10/09 8/3/7 papersACM Multimedia 2011/10/09/08/07 15/7/6/8/8 papersSIGMOD 2012/11/10 3/2/5 papersUbiComp 2012/11 2/4 papersCHI 2012/11 7/5 papersSODA 2013/12/11 2/3/3 papersAAAI 2012/11 3/4 papersVLDB 2012/11 4/3 papersOSDI 2012/SOSP 2011 1/1 paperNSDI 201/11 2/1 papersICSE 2012 2 papersFSE 2012 1 paperBest papers - Ubicomp 2012, ICIMCS 2012, Pervasive 2012, Eurosys 2012, MobileHCI2012, PhoneSense 2011, NSDI 2011, IEEE CSVT 2011, CoNEXT 2010, ISSAC 2010, AIRS 2010, UIC 2010, Best student Paper Awards – PODC 2012, ECML PKDD 2010Best demos – Sensys 2011, ACM-MM 2010, SIGCOMM 2010, MMM 2010
![Page 9: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/9.jpg)
Research Updates
Natural User Interface
Kinect User Identity
Kinect Object Digitalization
Avatar KinectKinect Head Pose & Face tracking
Kinect Body Gesture Recognition
![Page 10: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/10.jpg)
Kinect based Object Digitization
Play with your own
gadgets in games
Share your buddy with
others
Natural UI Interaction
w/ Virtual Objects
![Page 11: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/11.jpg)
Kinect based Object Digitization
Simple inputsFront and back snapshots of objects
Smooth 3D reconstructionPoisson surface reconstruction to fill holes and generate watertight smooth surface
Fast speedParallel octree construction using both CPU and GPU
![Page 12: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/12.jpg)
Comparisons
35,862 Triangles36MB Mem20 Seconds
33,263 Triangles28MB Mem5.6 Seconds
7,662 Triangles20MB Mem1.4 Seconds
![Page 13: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/13.jpg)
Comparisons
35,862 Triangles36MB Mem20 Seconds
33,263 Triangles28MB Mem5.6 Seconds
7,662 Triangles20MB Mem1.4 Seconds
![Page 14: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/14.jpg)
Avatar Kinect
Facial expression tracking
3D head pose tracking
3D expression tracking
Capture subtle expressionBrow frown, mouth movement, …
Super FastRGB + Depth sensing
Track 3D Pose
![Page 15: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/15.jpg)
Avatar Kinect
Track 3D Expression
Facial expression tracking
3D head pose tracking
3D expression tracking
Capture subtle expressionBrow frown, mouth movement, …
Super FastRGB + Depth sensing
![Page 16: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/16.jpg)
Gesture Recognition
A lot of applications require accurate understanding of human gestures
Sign language reading and dancing
Some tasks may require the detailed sequence of the gesture action
Kinect Sports
We tackle the problem of gesture recognition + alignment
![Page 17: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/17.jpg)
Xbox Kinect Pipeline: Skeleton Correction & Alignment
Depth Image
Background Removal
Skeleton Extraction
Skeleton Correction & Alignment
![Page 18: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/18.jpg)
Skeleton/Gesture Recognition
Two key observations:
(1) There are systematic errors which enable us to learn alignment function for the correction.
(2) The alignments are nonlinear and non-deterministic.
![Page 19: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/19.jpg)
Skeleton/Gesture Alignment
•The skeletons are aligned w/ numerical values•The numerical values are used to drive the avatar
![Page 20: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/20.jpg)
A Hidden Markov Model
Input Skeletons
Aligned Values
![Page 21: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/21.jpg)
Output Distribution Model
Random forests
A collection of decision trees
Like mixture distribution to model player variation
Ensemble learning (voting) for robustness
Decision tree
Leaf nodes model different poses
![Page 22: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/22.jpg)
Demo
Ground-truth (blue), Input estimation (red), Corrected skeleton (grey)
![Page 23: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/23.jpg)
Qualitative Results
![Page 24: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/24.jpg)
Research Updates
Multimedia
Head Scanning
Hair Modeling
![Page 25: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/25.jpg)
Kinect Based 3D Face Avatar
3D face avatar based on Kinect inputSimple capture – 3 shots
Good results & robust for different people
Depth – provide 3D shape of face, but noisymodeling overall shape and nose
robust alignment and pose estimation against face model
RGBFor texture details (skin color)
For modeling feature regions (mouth, eye)
Could use high-resolution RGB face shot from cameras
![Page 26: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/26.jpg)
Kinect Based 3D Avatar
LaplacianDeformation
Morphable Model
…
3D Face Priors
Dense Correspondence
![Page 27: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/27.jpg)
Kinect Based 3D Face Avatar (Fang Yang will give me new one on Sat.)
![Page 28: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/28.jpg)
Hair in Real World
Important appearance feature
A lot of (~100k) thin fibers
Diverse hairstyles
Complex interaction and dynamics
Hair in Graphics
Important applications
Manual modeling is difficult
Capturing is difficult too
Impractical for normal users
![Page 29: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/29.jpg)
Goal: 3D Hair from Single Image
Single portrait image
3D hair model
![Page 30: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/30.jpg)
Goal: 3D Hair from Single Image
Single portrait image
• Detailed 3D geometry
• Visually plausible
• Simple user interaction
• Enable interesting apps
![Page 31: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/31.jpg)
Result
Single portrait image
+ user strokes
![Page 32: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/32.jpg)
Algorithm
Preprocessing 3D strands2D strands
![Page 33: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/33.jpg)
Strand Tracing
![Page 34: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/34.jpg)
3D Strands: depth estimation
Initial depth w/o region structure w/ region structure
![Page 35: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/35.jpg)
3D Strands: hair volume
Additional strands in the hair volume
![Page 36: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/36.jpg)
Applications
Portrait pop-up
Hairstyle replacement
Hairstyle editing
![Page 37: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/37.jpg)
Portrait Pop-ups: layers completion
![Page 38: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/38.jpg)
Portrait Pop-ups: more results
![Page 39: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/39.jpg)
Hairstyle Replacement
Build pop-up for source & target.
Compensate for head’s shape/pose differences.
![Page 40: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/40.jpg)
Hairstyle Replacement: more results
Original Replaced
![Page 41: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/41.jpg)
Hairstyle Replacement: more results
![Page 42: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/42.jpg)
Hairstyle Editing
Original image Adjust
smoothness
Add geometric
noise
![Page 43: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/43.jpg)
Limitations
Not a complete 3D model of the real hair
Extension to video would be hard
Lighting condition is ignored
![Page 44: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/44.jpg)
Single Image 3D Hair Modeling
![Page 45: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/45.jpg)
3D Hair Replacement
![Page 46: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/46.jpg)
3D Hair Replacement
![Page 47: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/47.jpg)
Research Updates
Search and online AdvertisementLarge Scale Image Search through Query by Image
Building Image Knowledge Base through Mining Billions of Web Images
Keyword auction: Probabilistic Broad Match, Optimal Auction Learning, Bandit Algorithm for Bid Optimization
Attribute based Image Search
Lights!
![Page 48: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/48.jpg)
Attribute based Image Search
![Page 49: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/49.jpg)
What Are Attributes?
Clipart
Linedrawing
![Page 50: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/50.jpg)
What Are Attributes?
Screen Shot
Tiled Images
![Page 51: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/51.jpg)
What Are Attributes?
Simple Background
Portrait
![Page 52: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/52.jpg)
Attribute-based Search Framework
• Advantages– Scalable
– High precision
– Used explicitly or implicitly
Basic Visual Features (color, texture, shape, …), text
Attribute Extractor Attributes
Search Tasks: Compare, Categorize, Rank, Filter, …
![Page 53: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/53.jpg)
Color/Clipart/Linedrawing Filter
![Page 54: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/54.jpg)
Color Filter Examples
dog
![Page 55: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/55.jpg)
Color Filter Examples
60dog + red Red is not a word in the query, but means a color filter
![Page 56: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/56.jpg)
Color Filter Examples
sunset
![Page 57: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/57.jpg)
Color Filter Examples
sunset + blue
![Page 58: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/58.jpg)
Intention Analysis (w/ Query Object)
Flower +
T-shirt +
>
>
![Page 59: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/59.jpg)
Intention Analysis (w/o Query Object)
Sunset +
Forest +
![Page 60: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/60.jpg)
Our Solution
• Salient object detection• Check if an image contains a salient object
• Discover the position of the object
![Page 61: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/61.jpg)
Original Images
![Page 62: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/62.jpg)
Salient Object Detection
Non-object bounding box of salient object
![Page 63: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/63.jpg)
Our Solution
• Salient object detection• Check if an image contains object
• Discover the position of the object
• Learn to naming colors + context-aware color naming
![Page 64: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/64.jpg)
Dominant Color Naming
orangered
green
yellow pink red
red
pink
whiteorange
![Page 65: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/65.jpg)
Our Solution
• Salient object detection• Check if an image contains object
• Discover the position of the object
• Learn to naming colors + context-aware color naming
• Textual adjustment• Promote images whose surrounding texts contain the color
name (e.g., red,…)
![Page 66: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/66.jpg)
BMW
![Page 67: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/67.jpg)
BMW + blue
![Page 68: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/68.jpg)
BMW + pink
![Page 69: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/69.jpg)
BMW+ red
![Page 70: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/70.jpg)
BMW+ white
![Page 71: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/71.jpg)
dog
![Page 72: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/72.jpg)
dog + red
![Page 73: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/73.jpg)
dog + blue
![Page 74: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/74.jpg)
dog + brown
![Page 75: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/75.jpg)
dog + black
![Page 76: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/76.jpg)
dog + white
![Page 77: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/77.jpg)
forest
![Page 78: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/78.jpg)
forest + green
![Page 79: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/79.jpg)
forest + red
![Page 80: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/80.jpg)
forest + yellow
![Page 81: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/81.jpg)
sunset
![Page 82: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/82.jpg)
sunset + blue
![Page 83: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/83.jpg)
sunset + green
![Page 84: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/84.jpg)
sunset + orange
![Page 85: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/85.jpg)
tulips
![Page 86: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/86.jpg)
tulips + pink
![Page 87: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/87.jpg)
tulips + purple
![Page 88: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/88.jpg)
tulips + red
![Page 89: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/89.jpg)
Taylor Swift
![Page 90: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/90.jpg)
Taylor Swift + purple
![Page 91: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/91.jpg)
Taylor Swift + red
![Page 92: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/92.jpg)
Examples of Cliparts
![Page 93: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/93.jpg)
Important Filters
Usage on image search
clipart
photo
line-drawing
face
similar images
show more sizes
Clipart
Linedrawing
![Page 94: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/94.jpg)
Cascaded FilteringWeb Images
![Page 95: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/95.jpg)
Cascaded Filtering
Simple Background Filter:Filter out complex background image and generate foreground mask
# of colors in the image border and corners
rule based classifier
Web Images
Background Filter
![Page 96: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/96.jpg)
Cascaded Filtering
Simple Background Filter:Filter out complex background image and generate foreground mask
# of colors in the image border and corners
rule based classifier
Web Images
Background Filter
![Page 97: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/97.jpg)
Cascaded FilteringWeb Images
Background Filter
![Page 98: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/98.jpg)
Cascaded Filtering
Shape Filter:Prefer images with simple shape
foreground mask shape features, like perimeter, area…
rule-based classifier
Web Images
Background Filter
Shape Filter
![Page 99: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/99.jpg)
Cascaded Filtering
Shape Filter:Prefer images with simple shape
foreground mask shape features, like perimeter, area…
rule-based classifier
Web Images
Background Filter
Shape Filter
![Page 100: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/100.jpg)
Cascaded FilteringWeb Images
Background Filter
Shape Filter
![Page 101: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/101.jpg)
Cascaded Filtering
Texture Filter:Prefer images without complex texture
normalized edge pixel numbers
rule-based classifier
Web Images
Background Filter
Shape Filter
Texture Filter
![Page 102: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/102.jpg)
Cascaded Filtering
Texture Filter:Prefer images without complex texture
normalized edge pixel numbers
rule-based classifier
Web Images
Background Filter
Shape Filter
Texture Filter
![Page 103: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/103.jpg)
Cascaded FilteringWeb Images
Background Filter
Shape Filter
Texture Filter
![Page 104: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/104.jpg)
Cascaded Filtering
Photo Filter:Classify photo and cartoon style
• gradient strength histogram
• saturation histogram• lighting histogram
random forest classifier
Web Images
Background Filter
Shape Filter
Texture Filter
Photo Filter
![Page 105: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/105.jpg)
Cascaded Filtering
Photo Filter:Classify photo and cartoon style
• gradient strength histogram
• saturation histogram• lighting histogram
random forest classifier
Web Images
Background Filter
Shape Filter
Texture Filter
Photo Filter
![Page 106: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/106.jpg)
Cascaded FilteringWeb Images
Background Filter
Shape Filter
Texture Filter
Photo Filter
![Page 107: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/107.jpg)
Cascaded Filtering
Text Filter:Text detection using gradient feature
Web Images
Background Filter
Shape Filter
Texture Filter
Photo Filter
Text Filter
![Page 108: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/108.jpg)
Cascaded Filtering
Text Filter:Text detection using gradient feature
Web Images
Background Filter
Shape Filter
Texture Filter
Photo Filter
Text Filter
![Page 109: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/109.jpg)
Cascaded FilteringWeb Images
Background Filter
Shape Filter
Texture Filter
Photo Filter
Text Filter
Clipart
![Page 110: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/110.jpg)
A problem with filtered results
The results are not always attractive!
![Page 111: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/111.jpg)
Relevance ≠ Clipart Relevance
Ranking should be based on the relevance to query and clipart style, not only the query
Observation - Good clipart images often contain “clipart” or “cartoon” keywords
Solution: visual filter + clipart specific text ranker
![Page 112: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/112.jpg)
Example
crab
keyword: crabVisual attribute: clipart
keyword: crabkeyword: crab, clipartVisual attribute : clipart
Visual filter only
Visual + Text
![Page 113: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/113.jpg)
Results without Clipart Specific Text Ranker
fish
![Page 114: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/114.jpg)
Results with Clipart Specific Text Ranker
fish
![Page 115: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/115.jpg)
Results without Clipart Specific Text Ranker
girl
![Page 116: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/116.jpg)
Results with Clipart Specific Text Ranker
girl
![Page 117: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/117.jpg)
Bing Clipart Filter
• Visual filter + clipart specific text ranker
![Page 118: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/118.jpg)
Linedrawing Filter
![Page 119: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/119.jpg)
Results
![Page 120: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/120.jpg)
![Page 121: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/121.jpg)
![Page 122: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/122.jpg)
![Page 123: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/123.jpg)
Cascaded Filtering
Color Image Filter:Filter out non black-white images
# of non black-white pixels in the image
rule based classifier
Web Images
Color Image Filter
![Page 124: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/124.jpg)
Cascaded Filtering
Text Filter:Filter out images with big text regions (share with clipart filter)
Web Images
Color Image Filter
Foreground Shape Filter 1
Foreground Shape Filter 2
Texture Filter
Text Filter
Linedrawing
![Page 125: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/125.jpg)
Attribute based Visual Quality Improvement
![Page 126: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/126.jpg)
Motivation
Visual quality is important for user experience
![Page 127: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/127.jpg)
Existing SystemText based information (text metawords): • Word hit number• Phrase exactly match number• Static rank• BM25• …
Query textRanker
(decision tree) Ranking resultsInverted file system
Problem:Only used text information, can not consider visual quality
![Page 128: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/128.jpg)
Integrate Visual Features
Color-Hist
HoG
Gist
SIFT
…
Low-level visual features
Relevance Ranker
: text metawords
Hardly directly combine with text metawords
![Page 129: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/129.jpg)
Integrate Visual Features
Color-Hist
HoG
Gist
SIFT
…
Low-level visual features
Relevance Ranker
: text metawords
Mid-level visual features
Image properties/
styles
Image attributes:
: visual metawords
![Page 130: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/130.jpg)
Image Quality Related Attributes
People images
Prefer Not prefer
![Page 131: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/131.jpg)
Image Quality Related Attributes
People related attributes (metawords)
just face head and shoulder human body others
![Page 132: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/132.jpg)
Image Quality Related Attributes
Object images
Not preferPrefer
![Page 133: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/133.jpg)
Image Quality Related Attributes
Object related attributes (metawords)
size: large size: medium size: small
position: center position: off-center position: edge
![Page 134: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/134.jpg)
Image Quality Related Attributes
Image styles
Not preferPrefer
![Page 135: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/135.jpg)
Image Quality Related Attributes
Image style related attributes (metawords)
linedrawing clipart black-white
tiled image - scatter tiled image - grid
![Page 136: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/136.jpg)
Image Quality Related Attributes
General Photos
Not preferPrefer
![Page 137: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/137.jpg)
Image Quality Related Attributes
Attributes (metawords) for general photosPhoto quality
Image color entropy
high quality fair quality low quality
![Page 138: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/138.jpg)
Comparison Results
Old results (query = dog) Current results (query = dog)
![Page 139: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/139.jpg)
Comparison Results
Old results (query = bees) Current results (query = bees)
![Page 140: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/140.jpg)
Comparison Results
Old results (query = Royal Wedding) Current results (query = Royal Wedding)
![Page 141: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/141.jpg)
Comparison Results
Old results (query = Blueberries) Current results (query = Blueberries)
![Page 142: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/142.jpg)
Research Updates
Data Intensive ComputingServer (Software) Switch in DataCenter
NSDI 2011 best paper
Real-time matrix computation & StreamInsight
Eurosys 2012 best paper
Urban Computing
Engkoo Pinyin IME – Next generation cloud-based IME
Beta release: http://pinyin.engkoo.com/
![Page 143: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/143.jpg)
URBAN COMPUTING
![Page 144: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/144.jpg)
Urban Computing
Urban computing is emerging as a concept where every sensor, device, person, vehicle, building, and street in the urban areas can be used to sense city dynamics to enable technologies to
serve people in urban environment .
Urban Computing
Understanding
Mining
Sensing
Improving
![Page 145: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/145.jpg)
City Dynamics
ScopeTraffic flow
Human mobility
Energy consumption
Environment
Economics
Population
……
Data availableMobile phone signal
GPS traces of vehicles and people
Ticketing data in public transportation systems
User-generated content (Tweeter, Yelp)
Map & POI data
Transportation sensor networksCamera and loop sensors
Parking lots
Environmental sensor networkAir quality
Temperature
Radiation
Real estate listings
….
![Page 146: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/146.jpg)
1. GPS trajectories of 33,000 taxis in 2009, 2010, and 2011
![Page 147: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/147.jpg)
2. Online Social Media: Check-in data
![Page 148: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/148.jpg)
Beijing Subway by 2015: The city with the longest distance of subway (561KM)
Two times longer than that of Paris(221.6KM)
3. Ticketing data of public transportations
![Page 149: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/149.jpg)
4. Air quality data
![Page 150: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/150.jpg)
5. Real estate market
![Page 151: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/151.jpg)
6. Beijing road networks 2009-2011
2011: 121,771 nodes and 162,246 segments, 19,524km
32 km
40 km
![Page 152: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/152.jpg)
7. POI Data (2007 – 2012)
![Page 153: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/153.jpg)
Finding Smart Driving Directions
ACM SIGSPAITAL GIS’10 best paper runner-up, KDD’11
Anomalous Events Detection
KDD’11
Passengers-Cabbie Recommender system
Ubicomp’11
Urban Computing for Urban Planning
Ubicomp’11 Best paper nominee
Discovery of Functional Regions
KDD’12
Route Construction from Uncertain Trajectories (a)
(b)
(c) (d) (e)
KDD’12
![Page 154: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/154.jpg)
Discover Regions of Different Functions using Human Mobility and POIs
![Page 155: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/155.jpg)
Goals
Discover regions of different functions in urban areas
Identify the kernel density of a functionality
Functional Regions Functionality Density
![Page 156: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/156.jpg)
Applications
• Calibrating urban planning
• Business planning
• Advertising
![Page 157: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/157.jpg)
Motivation and Challenges
POIs indicate the function
But not enoughCompound
Quality
Human mobilityDifferentiate between POIs of the same category
Indicates the function of a region
Arrival
Leaving
![Page 158: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/158.jpg)
Methodology Overview
Mapping from regions to documentsRegions Documents (R)
Functions Topics (K)
Mobility patterns Words (N)
POIs meta data like keywords and authors
LDA(Latent Dirichlet Allocation)-variant topic model
![Page 159: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/159.jpg)
Territory Identification
Region aggregation
Cluster regions according to topic distributions
Aggregate individual regions into big territories
![Page 160: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/160.jpg)
Annotation of Territories
Emerging residential areas
Regions under construction
Developed residential areas
Developed commercial areas
Areas of historic interests
Nature and parks
Education and science areas
Developing commercial areas
Diplomatic and embassy areas
![Page 161: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/161.jpg)
Evaluation20112010
![Page 162: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/162.jpg)
Research Updates
CS FundamentalsTheory papers in SODA, STOC, FOCS, WINE, …
System & Networking paper in SOSP, SIGCOMM, NSDI, ICSE, Eurosys, Mobicom, Mobisys, Sensys, …
BSGP – High-level GPU programming language
Indoor Navigation
![Page 163: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/163.jpg)
BSGP: Bulk-Synchronous GPU Programming
High-level programming language
Hardware independent
Easy to write and maintain
Well optimized
Easy to debug
![Page 164: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/164.jpg)
GPUs vs. Programmers
Block
Block
Many-threadkernels
Dataflowstreams
?
Low level programming:Direct-compute, CUDA,
OpenCL, AMP …
High level programming:
BSGP !
GPU architecture:stream/kernel
Programmer:data structure/algorithm
![Page 165: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/165.jpg)
Bulk Synchronous Parallel (BSP)
First proposed by Leslie G. Valiant (1990)A bridging model for parallel computation
First CPU implementation by Bill McColl (1997)BSPlib: The BSP Programming Library
Challenges on GPUsMany-thread computational model on GPUs because of the stream-kernel architecture
![Page 166: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/166.jpg)
Research on BSGP (2007~now)
Compiler (ACM TOG 2008)BSGP: Bulk-Synchronous GPU Programming
Debugger (ACM TOG 2009)Debugging GPU Stream Programs Through Automatic Dataflow Recording and Visualization
Applications on GPUs (ACM TOG 2008~2012)KD-tree construction
Micropolygon ray tracing
Hair modeling and rendering
…
![Page 167: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/167.jpg)
Eating Our Own Dog Food (2007~now)
20+ new GPU algorithmsFirst KD-tree/Octree construction on GPUs
Neural network training
Optimization
System RenderAnts
About 50,000 lines of BSGP code, and compiled into about 2,000 CUDA kernels.
![Page 168: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/168.jpg)
Contribution to Community
BSGP download
Research.microsoft.com download sitehttp://research.microsoft.com/en-us/downloads/283bb827-8669-4a9f-9b8c-e5777f48f77b/default.aspx
![Page 169: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/169.jpg)
Fast Bootstrapping of Indoor Localization Services
![Page 170: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/170.jpg)
Motivation
Growing demand for fine-grained indoor localization
Indoor navigation
Mobile advertising
Mobile social network
…
![Page 171: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/171.jpg)
Overall System Architecture
NumerousAlgorithms
AssumedAvailable
Few indoor localization systems actually deployed!
WiFi-based systems Other Sensing Modality
Location Database(WiFi-scans @ locations)
Location Database(signals @ locations)
Location Query Services and APIs(signal location)
Inference
Service
Fingerprinting Alg. (e.g. RADAR)
Model-based Alg. (e.g., EZ)
Modality-specificAlgorithms
Database
NOT true!
![Page 172: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/172.jpg)
Key Problem
How to efficiently build and maintain the location database?
Challenges:
Data collection: high user involvement
Offline processing: noisy or bad data
Database maintenance: frequent environment change
![Page 173: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/173.jpg)
Taxonomy of Data Collection Methods
Supervised: Tap&HoldManually locate oneself on map and collect WiFi fingerprint for that spot
Unsupervised: Walking-basedWalking with a survey plan
For vendors, may tapping at landmarks (e.g., turns)Good for initial data collection
Freestyle walkingFor ordinary users in their normal useGood for database maintenance, via crowdsourcing
![Page 174: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/174.jpg)
Inertial sensor based Dead-Reckoning
Robust step detector
Robust heading estimator
Accurate, personalized step length model
Enabling Walking-based Collection
![Page 175: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/175.jpg)
Walking with a Survey Plan
Problem: Noisy sensor data
Solution: Robust turn detection algorithm
Survey plan Raw sensor data Rectified results
![Page 176: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/176.jpg)
Accuracy of Turn Detection
Path/Trace list # of Effective turns
# of Detected turns
Detected Turn Position Error (mean / min / max)
1st 18 18 1.07, 0.25, 2.24 (seconds)
2nd 9 9 1.17, 0.22, 2.38 (seconds)
3rd 9 9 1.23, 0.41, 2.50 (seconds)
4th 4 4 0.91, 0.58, 2.40 (seconds)
Average step position error: 1.1 m
![Page 177: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/177.jpg)
Comparing Data Collection Methods: End-to-End Evaluation
Training datasets
Tap & hold
45 points in 35 minutes
Walking
1809 points in 35 minutes
Test data set
3288 points
![Page 178: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/178.jpg)
End-to-end Evaluation*
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
0 2 4 6 8 10 12 14Distance Error (meter)
Collection Time 35 min
Tap&Hold
Walking without labeling
* Using model-based WiFi localization method
Plain walking outperforms Tap&Hold, with much less user effort and much higher data yield.
![Page 179: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/179.jpg)
Data Collection with Freestyle Walking
Challenges
Unplanned paths: requiring path detection
Phone usage diversity: error-prone DR results
Solutions:
Trace-back via backward propagation in PF
Location refinement via overall trace rectification
Match traces to map, by landmarks (e.g., turns)
![Page 180: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/180.jpg)
An Illustrative Example
Start
End
Actual start/end point
Ground truth walking path
Backward tracked traces
Tracked start/end point
Forward tracked traces
Continue forward tracking
Trace-back via BP
![Page 181: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/181.jpg)
Accuracy under Freestyle Walking
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 2 4 6 8 10Localization error (meters)
without location refinementwith location refinement
1.8m 6.2m
![Page 182: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/182.jpg)
Conclusion
MSRA follows the same missions after 14 yrs
Advance the state of the art
We are expanding our research areas
Systems & networking, HCI, Haptics,…
We love to collaborate with you as usual
Please visit us often & give us feedback
Let’s work together
![Page 183: Microsoft Research Asia Update€¦ · Innovation Engineering Jonathan TIEN Technology Strategy Eric CHANG. MSRA Focus Areas ... Natural User Interface Kinect User Identity Kinect](https://reader030.fdocuments.in/reader030/viewer/2022041120/5f3267d6969e2b02050fd09d/html5/thumbnails/183.jpg)
THANK YOU