Big Data Tutorial - Marko Grobelnik - 25 May 2012
-
Upload
marko-grobelnik -
Category
Technology
-
view
1.964 -
download
3
Transcript of Big Data Tutorial - Marko Grobelnik - 25 May 2012
![Page 2: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/2.jpg)
Introduction ◦ What is Big data?
◦ Why Big-Data?
◦ When Big-Data is really a problem?
Techniques
Tools
Applications
Literature
![Page 3: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/3.jpg)
![Page 4: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/4.jpg)
![Page 5: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/5.jpg)
‘Big-data’ is similar to ‘Small-data’, but bigger
…but having data bigger consequently requires different approaches: ◦ techniques, tools, architectures
…with an aim to solve new problems ◦ …and old problems in a better way.
![Page 6: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/6.jpg)
From “Understanding Big Data” by IBM
![Page 7: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/7.jpg)
![Page 8: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/8.jpg)
Big-Data
![Page 9: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/9.jpg)
Key enablers for the growth of “Big Data” are:
◦ Increase of storage capacities
◦ Increase of processing power
◦ Availability of data
![Page 10: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/10.jpg)
![Page 11: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/11.jpg)
![Page 12: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/12.jpg)
![Page 13: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/13.jpg)
![Page 14: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/14.jpg)
![Page 15: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/15.jpg)
![Page 16: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/16.jpg)
![Page 17: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/17.jpg)
![Page 18: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/18.jpg)
![Page 19: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/19.jpg)
![Page 20: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/20.jpg)
Where processing is hosted? ◦ Distributed Servers / Cloud (e.g. Amazon EC2)
Where data is stored? ◦ Distributed Storage (e.g. Amazon S3)
What is the programming model? ◦ Distributed Processing (e.g. MapReduce)
How data is stored & indexed? ◦ High-performance schema-free databases (e.g.
MongoDB)
What operations are performed on data? ◦ Analytic / Semantic Processing (e.g. R, OWLIM)
![Page 21: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/21.jpg)
Computing and storage are typically hosted transparently on cloud infrastructures ◦ …providing scale, flexibility and high fail-safety
Distributed Servers ◦ Amazon-EC2, Google App Engine, Elastic,
Beanstalk, Heroku
Distributed Storage ◦ Amazon-S3, Hadoop Distributed File System
![Page 22: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/22.jpg)
Distributed processing of Big-Data requires non-standard programming models ◦ …beyond single machines or traditional parallel
programming models (like MPI) ◦ …the aim is to simplify complex programming tasks
The most popular programming model is MapReduce approach
Implementations of MapReduce ◦ Hadoop (http://hadoop.apache.org/), Hive, Pig,
Cascading, Cascalog, mrjob, Caffeine, S4, MapR, Acunu, Flume, Kafka, Azkaban, Oozie, Greenplum
![Page 23: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/23.jpg)
The key idea of the MapReduce approach: ◦ A target problem needs to be parallelizable
◦ First, the problem gets split into a set of smaller problems (Map step) ◦ Next, smaller problems are solved in a parallel way ◦ Finally, a set of solutions to the smaller problems get synthesized
into a solution of the original problem (Reduce step)
![Page 24: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/24.jpg)
NoSQL class of databases have in common: ◦ To support large amounts of data
◦ Have mostly non-SQL interface
◦ Operate on distributed infrastructures (e.g. Hadoop)
◦ Are based on key-value pairs (no predefined schema)
◦ …are flexible and fast
Implementations ◦ MongoDB, CouchDB, Cassandra, Redis, BigTable, Hbase,
Hypertable, Voldemort, Riak, ZooKeeper…
![Page 25: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/25.jpg)
![Page 26: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/26.jpg)
…when the operations on data are complex: ◦ …e.g. simple counting is not a complex problem
◦ Modeling and reasoning with data of different kinds can get extremely complex
Good news about big-data: ◦ Often, because of vast amount of data, modeling
techniques can get simpler (e.g. smart counting can replace complex model-based analytics)…
◦ …as long as we deal with the scale
![Page 27: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/27.jpg)
Research areas (such as IR, KDD, ML, NLP, SemWeb, …) are sub-cubes within the data cube
Scalability
Streaming
Context
Quality
Usage
![Page 28: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/28.jpg)
A risk with “Big-Data mining” is that an analyst can “discover” patterns that are meaningless
Statisticians call it Bonferroni’s principle: ◦ Roughly, if you look in more places for interesting
patterns than your amount of data will support, you are bound to find crap
Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
![Page 29: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/29.jpg)
Example:
We want to find (unrelated) people who at least twice have stayed at the same hotel on the same day ◦ 109 people being tracked. ◦ 1000 days. ◦ Each person stays in a hotel 1% of the time (1 day out of 100) ◦ Hotels hold 100 people (so 105 hotels). ◦ If everyone behaves randomly (i.e., no terrorists) will the data
mining detect anything suspicious?
Expected number of “suspicious” pairs of people: ◦ 250,000 ◦ … too many combinations to check – we need to have some
additional evidence to find “suspicious” pairs of people in some more efficient way
Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
![Page 30: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/30.jpg)
Smart sampling of data ◦ …reducing the original data while not losing the
statistical properties of data
Finding similar items ◦ …efficient multidimensional indexing
Incremental updating of the models ◦ (vs. building models from scratch)
◦ …crucial for streaming data
Distributed linear algebra ◦ …dealing with large sparse matrices
![Page 31: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/31.jpg)
On the top of the previous ops we perform usual data mining/machine learning/statistics operators: ◦ Supervised learning (classification, regression, …)
◦ Non-supervised learning (clustering, different types of decompositions, …)
◦ …
…we are just more careful which algorithms we choose (typically linear or sub-linear versions)
![Page 32: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/32.jpg)
An excellent overview of the algorithms covering the above issues is the book “Rajaraman, Ullman: Mining of Massive Datasets”
![Page 33: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/33.jpg)
![Page 34: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/34.jpg)
Good recommendations can make a big difference when keeping a user on a web site ◦ …the key is how rich the
context model a system is using to select information for a user
◦ Bad recommendations <1% users, good ones >5% users click
◦ 200clicks/sec
Contextual personalized
recommendations generated in ~20ms
![Page 35: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/35.jpg)
Domain
Sub-domain
Page URL
URL sub-directories
Page Meta Tags
Page Title
Page Content
Named Entities
Has Query
Referrer Query
Referring Domain
Referring URL
Outgoing URL
GeoIP Country
GeoIP State
GeoIP City
Absolute Date
Day of the Week
Day period
Hour of the day
User Agent
Zip Code
State
Income
Age
Gender
Country
Job Title
Job Industry
![Page 36: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/36.jpg)
Log Files (~100M
page clicks per day)
User profiles
NYT articles
Stream of profiles
Advertisers
Segment Keywords
Stock Market
Stock Market, mortgage, banking, investors, Wall Street, turmoil, New York Stock Exchange
Health diabetes, heart disease, disease, heart, illness
Green Energy
Hybrid cars, energy, power, model, carbonated, fuel, bulbs,
Hybrid cars Hybrid cars, vehicles, model, engines, diesel
Travel travel, wine, opening, tickets, hotel, sites, cars, search, restaurant
… …
Segments
Trend Detection System
Stream of clicks
Trends and updated segments
Campaign to sell
segments
$
Sales
![Page 37: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/37.jpg)
50Gb of uncompressed log files
50-100M clicks
4-6M unique users
7000 unique pages with more then 100 hits
![Page 38: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/38.jpg)
Alarms Explorer Server implements three real-time scenarios on the alarms stream:
1. Root-Cause-Analysis – finding which device is responsible for occasional “flood” of alarms
2. Short-Term Fault Prediction – predict which device will fail in next 15mins
3. Long-Term Anomaly Detection – detect unusual trends in the network
…system is used in British Telecom
Alarms Server
Alarms Explorer Server
Live feed of data
Operator Big board display
Telecom Network (~25 000 devices)
Alarms ~10-100/sec
![Page 39: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/39.jpg)
The aim of the project is to collect and analyze most of the main-stream media across the world ◦ …from 35,000 publishers (180K RSS feeds) crawled in
real time (few ~10articles per second)
◦ …each article document gets extracted, cleaned, semantically annotated, structure extracted
Challenges are in terms of complexity of processing and querying of the extracted data ◦ …and matching textual information across the
languages (cross-lingual technologies)
http://render-project.eu/ http://www.xlike.org/
![Page 41: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/41.jpg)
Plain text
Text Enrichment
Extracted graph of triples from text
“Enrycher” is available as as a web-service generating Semantic Graph, LOD links, Entities, Keywords, Categories, Text Summarization
![Page 42: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/42.jpg)
The aim is to use analytic techniques to visualize documents in different ways: ◦ Topic view
◦ Social view
◦ Temporal view
![Page 43: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/43.jpg)
Query
Search Results
Topic Map
Selected group of news
Selected story
![Page 44: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/44.jpg)
Query
Named entities in relation
![Page 45: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/45.jpg)
Query
Topic Trends Visualization
Topics description
US Elections US Budget
Mid-East conflict
NATO-Russia
Result set
![Page 46: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/46.jpg)
Dec 7th 1941
![Page 47: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/47.jpg)
Apr 6th 1941
![Page 48: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/48.jpg)
June 1944
![Page 49: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/49.jpg)
Query
Conceptual map
Search Point
Dynamic contextual ranking based on the search point
![Page 50: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/50.jpg)
Observe social and communication phenomena at a planetary scale
Largest social network analyzed till 2010
Research questions:
How does communication change with user demographics (age, sex, language, country)?
How does geography affect communication?
What is the structure of the communication network?
50 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
![Page 51: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/51.jpg)
We collected the data for June 2006
Log size:
150Gb/day (compressed)
Total: 1 month of communication data:
4.5Tb of compressed data
Activity over June 2006 (30 days) ◦ 245 million users logged in
◦ 180 million users engaged in conversations
◦ 17,5 million new accounts activated
◦ More than 30 billion conversations
◦ More than 255 billion exchanged messages
51 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
![Page 52: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/52.jpg)
52 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
![Page 53: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/53.jpg)
53 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
![Page 54: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/54.jpg)
Count the number of users logging in from particular location on the earth
54 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
![Page 55: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/55.jpg)
Logins from Europe
55 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
![Page 56: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/56.jpg)
6 degrees of separation [Milgram ’60s]
Average distance between two random users is 6.6
90% of nodes can be reached in < 8 hops
Hops Nodes
1 10
2 78
3 396
4 8648
5 3299252
6 28395849
7 79059497
8 52995778
9 10321008
10 1955007
11 518410
12 149945
13 44616
14 13740
15 4476
16 1542
17 536
18 167
19 71
20 29
21 16
22 10
23 3
24 2
25 3 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
![Page 57: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/57.jpg)
![Page 58: Big Data Tutorial - Marko Grobelnik - 25 May 2012](https://reader033.fdocuments.in/reader033/viewer/2022042700/55941bb61a28abf02b8b46c8/html5/thumbnails/58.jpg)
Big-Data is everywhere, we are just not used to deal with it
The “Big-Data” hype is very recent ◦ …growth seems to be going up ◦ …evident lack of experts to build Big-Data apps
Can we do “Big-Data” without big investment? ◦ …yes – many open source tools, computing machinery is
cheap (to buy or to rent) ◦ …the key is knowledge on how to deal with data ◦ …data is either free (e.g. Wikipedia) or to buy (e.g.
twitter)