Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng...
Transcript of Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng...
![Page 1: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/1.jpg)
http://poloclub.gatech.edu/cse6242CSE6242: Data & Visual Analytics
Data Integration
Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS AnalyticsMachine Learning Area Leader, College of Computing Georgia Tech
Partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos Faloutsos
![Page 2: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/2.jpg)
What is Data Integration? Combining data from multiple sources to
provide the user with a unified view.
Why is it Important?Think about the apps, websites, and
services that you use every day.
![Page 3: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/3.jpg)
Businesses derive value through data integration.
![Page 4: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/4.jpg)
![Page 5: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/5.jpg)
Apple Siri
![Page 6: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/6.jpg)
![Page 7: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/7.jpg)
More Examples?• Social media (data from users, businesses)
• Facebook: your posts, advertisements, review
• Search engine: Google, Bing, Yahoo, etc.
• Smart assistants: Siri, Cortana, Alexa
• Price comparison: Kayak
• Uber, Lyft: drivers, traffic data, customers
• google maps: users, restaurants, traffic….
�7
![Page 8: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/8.jpg)
How to do data integration?
![Page 9: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/9.jpg)
“Low” Effort Approaches1. Use database’s “Join”! (e.g., SQLite)When does this approach work? (Or, when does it NOT work?)
�9
id name salary111 Smith $40k222 Johnson $60k333 Lee $50k
id name111 Smith222 Johnson333 Lee
id salary111 $40k222 $60k333 $50k
2. Open Refine http://openrefine.org (Video #3 “Reconcile and Match Data”)
![Page 10: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/10.jpg)
IDs are really important, and can simplify data integration!
But who creates the IDs?
�10
![Page 11: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/11.jpg)
Crowd-sourcing Approaches: Freebase
�11
Freebase intro video: https://youtu.be/TJfrNo3Z-DULearn more about Freebase at https://en.wikipedia.org/wiki/Freebase
![Page 12: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/12.jpg)
Freebase(a graph of entities)
“…a large collaborative knowledge base consisting of metadata composed mainly
by its community members…”
�12
Wikipedia.
![Page 13: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/13.jpg)
So what? What can you do with the
Freebase knowledge graph?
Hint: Google acquired it in 2010.
�13
![Page 14: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/14.jpg)
Google Knowledge Graph video: https://youtu.be/mmQl6VGvX-c
![Page 15: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/15.jpg)
Freebase replaced by Google Knowledge Graph API
�15
Example: What does Google know about Taylor Swift?
https://developers.google.com/knowledge-graph/
![Page 16: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/16.jpg)
�16
What does Google know about Taylor Swift? https://developers.google.com/knowledge-graph/
![Page 17: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/17.jpg)
What if we don’t have the luxury of having IDs ?
�17(Screenshot from FreeBase video)
A common problem in academia:Polo ChauDuen Horng ChauDuen ChauD. Chau
![Page 18: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/18.jpg)
Entity Resolution(A hard problem in data integration)
�18
Then you need to do…
![Page 19: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/19.jpg)
Why is entity resolution so difficult?
Let’s understand it through shopping for an iPhone on Apple, Amazon and eBay
![Page 20: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/20.jpg)
![Page 21: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/21.jpg)
![Page 22: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/22.jpg)
![Page 23: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/23.jpg)
D-Dupe Interactive Data Deduplication and IntegrationTVCG 2008University of MarylandBilgic, Licamele, Getoor, Kang, Shneiderman
�23https://linqspub.soe.ucsc.edu/basilic/web/Publications/2006/bilgic:vast06/
![Page 24: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/24.jpg)
![Page 25: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/25.jpg)
Polo
Palo
Alice
Bob
Carol
Dave
![Page 26: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/26.jpg)
Core components: Similarity functions
Determine how two entities are similar.D-Dupe’s approach: Attribute similarity + relational similarity
�26
Similarity score for a pair of entities
![Page 27: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/27.jpg)
�27
Attribute similarity (a weighted sum)
![Page 28: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/28.jpg)
Numerous similarity functions
• Euclidean distanceEuclidean norm / L2 norm
• TaxiCab/Manhattan distance
• Jaccard Similarity (e.g., used with w-shingles)e.g., overlap of nodes’ #neighbors
• String edit distancee.g., “Polo Chau” vs “Polo Chan”
�28
http://infolab.stanford.edu/~ullman/mmds/ch3a.pdfExcellent read:
![Page 29: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/29.jpg)
�29
https://reference.wolfram.com/language/guide/DistanceAndSimilarityMeasures.html
![Page 30: Data Integration - Visualization · CSE6242: Data & Visual Analytics Data Integration Duen Horng (Polo) Chau Associate Professor, College of Computing Associate Director, MS Analytics](https://reader034.fdocuments.in/reader034/viewer/2022051809/601357c9267dbe417a1bc3c8/html5/thumbnails/30.jpg)
Excellent Tutorial on Entity Resolution
http://www.umiacs.umd.edu/~getoor/Tutorials/ER_KDD2013.pdf
by Lise Getoor and Ashwin Machanavajjhala
�30