Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching...
-
Upload
cengage-learning -
Category
Documents
-
view
597 -
download
0
description
Transcript of Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching...
![Page 1: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/1.jpg)
Drinking from the Fire Hose: Tools for Interpreting and Teaching
with Big Data
Mark FrydenbergBentley University
![Page 2: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/2.jpg)
![Page 3: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/3.jpg)
CourseMateEnhanced
Edition
![Page 4: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/4.jpg)
77 Movies and TV Shows!
![Page 5: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/5.jpg)
What's your Bacon Index?
Kevin
Bob
1
Ann2Joe
3
Kim
42 X
![Page 6: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/6.jpg)
![Page 7: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/7.jpg)
![Page 8: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/8.jpg)
APIs
![Page 9: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/9.jpg)
![Page 10: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/10.jpg)
Friend of a Friend
![Page 11: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/11.jpg)
Social Graph
![Page 12: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/12.jpg)
![Page 13: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/13.jpg)
Big Data
'Big data' refers to a collection of tools, techniques and technologies which make it easy to work with data at any scale.
powerof60.com
![Page 15: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/15.jpg)
![Page 16: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/16.jpg)
3 V's
• Volume - amount of data is larger than those conventional relational database infrastructures can handle
• Velocity - the rate at which data is generated, processed and analyzed in (real) time
• Variety – data formats are unstructured and inconsistent
![Page 18: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/18.jpg)
Yottabyte?
![Page 19: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/19.jpg)
Walmart
• Walmart collects more than 2.5 petabytes of data every hour from its customer transactions.
• A petabyte is one quadrillion bytes, or the equivalent of about 20 million filing cabinets’ worth of text.
http://hbr.org/2012/10/big-data-the-management-revolution/ar
![Page 20: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/20.jpg)
Velocity: Drinking from the Firehose
• Scrutinize 5 million trade events created each day to identify potential fraud
• Analyze 500 million daily call detail records in real-time to predict customer churn faster
![Page 21: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/21.jpg)
A Variety of Big Data Sources
![Page 22: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/22.jpg)
McKinsey&Company Report (2011)
• Data is part of every industry and business function.
• Data creates value.• Big data becomes a basis of
competition and growth.• Some sectors will achieve
greater gains.• Shortage of people with
analytical skills.• Need policies related to
privacy, security, ownership.
![Page 23: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/23.jpg)
![Page 24: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/24.jpg)
3000 tweets per seconddata is disorganizedHow does twitter use its data?
![Page 25: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/25.jpg)
Twitter Visualization
![Page 26: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/26.jpg)
![Page 27: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/27.jpg)
Big Data Technologies
• HADOOP: scalable storage, parallel computation
• NoSQL: distributed querying
![Page 28: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/28.jpg)
What this Means
• Change your web page and Google finds it in minutes.
• Ten years ago, you would have to submit a request to Yahoo! to reindex your site.
• All you need is a lot of servers. • Google has a million of them.• No problem.
![Page 30: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/30.jpg)
![Page 31: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/31.jpg)
![Page 32: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/32.jpg)
Collaborative Filtering
![Page 33: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/33.jpg)
Collaborative Filtering
Me You
Black Beauty
Camera TripodThe Black
Stallion
![Page 34: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/34.jpg)
![Page 35: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/35.jpg)
Variety: Semantic Web
![Page 37: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/37.jpg)
Unstructured Data
![Page 38: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/38.jpg)
![Page 39: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/39.jpg)
Health Care
![Page 40: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/40.jpg)
![Page 41: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/41.jpg)
![Page 42: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/42.jpg)
Analyzing Big Data
![Page 43: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/43.jpg)
explore.data.gov
![Page 44: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/44.jpg)
Searching Big Data
![Page 45: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/45.jpg)
Fusion Table Visualizations
![Page 46: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/46.jpg)
Fusion Table Visualizations
![Page 47: Course Tech 2013, Mark Frydenberg, Drinking from the Fire Hose: Tools for Interpreting and Teaching with Big Data](https://reader034.fdocuments.in/reader034/viewer/2022052618/554912aab4c90558458bf69e/html5/thumbnails/47.jpg)
Fusion Table Visualizations