Big Data Overview 2013-2014
-
Upload
kms-technology -
Category
Technology
-
view
110 -
download
5
description
Transcript of Big Data Overview 2013-2014
1
BIG DATA
Kaushal Amin, Chief Technology OfficerKMS Technology – Atlanta, GA, USA
2
AGENDA
• What is Big Data• Why not RDBMS• NoSQL• NewSQL• Performance Comparison• Case Studies
WHAT IS BIG DATA
4
WHAT IS BIG DATA?
“Big data exceeds the reach of commonly used hardware environments and software tools to capture, manage, and process it with in a tolerable elapsed time for its user population.” - Teradata Magazine article, 2011
“Big data refers to data sets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.” - The McKinsey Global Institute, 2011
Volume and Variety of Data that is difficult to manage using traditional data management
technology
WHAT IS GENERATING BIG DATA?
Homeland Security
Real Time Search
Social
eCommerce
User Tracking & Engagement
Financial Services
5
HOW MUCH DATA?
• 7 billion people• Google processes 100 PB/day; 3 million servers• Facebook has 300 PB + 500 TB/day; 35% of
world’s photos• YouTube 1000 PB video storage; 4 billion
views/day• Twitter processes124 billion tweets/year• SMS messages – 6.1T per year• US Cell Calls – 2.2T minutes per year• US Credit cards - 1.4B Cards; 20B
transactions/year
6
7
LOWER COST OF STORAGEWhat can I buy for $100 (USD) ?
(not adjusted for inflation)
Memory Capacity = 128 GB by 2020
x1420 in 20 years
Disk Capacity =10 TB by 2020
x1000 in 20 years
8
HOW IS BIG DATA DIFFERENT?
• Automatically generated by a machine – (e.g. Sensor embedded in an engine)
• Typically an entirely new source of data– (e.g. Use of the internet)
• Not designed to be friendly– (e.g. Text streams)
• May not have much values– Need to focus on the important part
9
WHO UTILIZES IT?
• Companies and organizations who can leverage large scale consumer produced data – Marketing– Consumer Markets (retail, airlines, hotels, Amazon,
Netflix)– Social Media (Facebook, Twitter, YouTube, LinkedIn)– Search Providers (Google, Yahoo, Microsoft)– People Data Aggregators (LexisNexis, Equifax, Acxiom)
• Other Enterprises are slowly getting into it– Healthcare – Financial Institutes
WHY NOT RDBMS?
TYPE OF DATA
• Structured Data (Transactions)
• Text Data (Web Content)
• Semi-structured Data (XML)
• Unstructured Data– Social Network, SMS, Audio, Video
• Streaming Data – You can only scan the data once as it travels on network
11
WHAT TO DO WITH THESE DATA?
• Aggregation and Statistics – Data warehouse and OLAP
• Indexing, Searching, and Querying– Keyword based search – Pattern matching (XML/RDF)
• Knowledge discovery– Data Mining– Statistical Modeling
12
RDBMS LIMITATIONS
• Very difficult to scale horizontally (more boxes) as the best way to scale is vertically by utilizing bigger box– Physical limited to CPUs, Disk storage, and memory– Large servers are too expensive and still can’t scale
• Requires structure of tables with rows and columns– Does not deal well with unstructured data
• Relationships have to be pre-defined through schema– Difficult to add newly discovered data quickly
13
NOSQL
NOSQL CHARACTERISTICS
• Cheap, easy to implement (open source)– Cluster of cheap commodity servers with cheap storage
• Data are replicated to multiple nodes (therefore identical and fault-tolerant) and can be partitioned– Down nodes can easily be replaced while cluster is operational– No single point of failure
• Easy to distribute• Don't require a schema• Massive Scalability• Relaxed the data consistency requirement
(CAP) – less locking and resource contengency15
16
NOSQL – SEVERAL OPTIONS
• Currently 150 implementations and growing (http://nosql-database.org/)
• Multiple Types based on storage architecture– Key-Value– Document– Column Family– Graph
KEY-VALUE STORE
• Values stored in Key-Value Pairs in hashmap• Distributed across nodes based on key • Simple Operations: insert, fetch, update, and
delete• Best for storing high volume dataset with
low complexity (simple data model)• Some of the market leaders:
– Riak– Amazon Dynamo– Voldermort
17
KEY-VALUE STORE
18
COLUMN FAMILY STORE
• Stores family of columns• Columns are stored as Key-Value pair• A super column is like a catalogue or a collection
of other columns• Columns within a family can be distributed across
nodes• Supports semi-structured data with high
scalability• Some of the market leaders:
– HBase– Cassandra
19
COLUMN FAMILY STORE (HBASE)
20
DOCUMENT STORE
• Supports more complex data model than Key-Value
• Collection of Documents – JSON, XML, other semi-structured formats
• A document is a key value collection• Multi-Index support• Best for storing complex data model but less
scalable• Some of the market leaders:
– MongoDB– CouchDB– SimpleDB
21
DOCUMENT STORE
22
GRAPH DATABASE
• Social Graph with Relationship between Entities• Great for Social Networks
– Facebook friends network– LinkedIn connections network
• Some of the market leaders: – Neo4j– FlockDB– Pregel
23
GRAPH DATABASE - EXAMPLE
24
• Nodes represent entities such as people, businesses, accounts, or any other item you might want to keep track of.
• Properties are pertinent information that relate to nodes such as name, age, DOB, gender.
• Edges are the lines that connect nodes to nodes or nodes to properties and they represent the relationship between the two.
NEWSQL
NEWSQL
• Argument is that Relational Model is not the problem for lack of scalability but the physical implementation limitations
• Development of new relational database products and services designed to bring the benefits of the relational model to distributed architectures
• Three Approaches:– Optimized MySQL storage engines (ScaleDB, MemSQL, Akiban)– New SQL databases (Clusterix, VoltDB, NuoDB)– Sharding Middleware to split RDBMS across nodes (ScaleBase,
Scalearc, dbShards)
26
PERFORMANCE COMPARISON
28
SOURCE AND APPROACH
• Independent testing done by Altoros Systems Inc.• More details at http://
www.networkworld.com/news/tech/2012/102212-nosql-263595.html?page=1
• Using Amazon virtual machines to ensure verifiable results and research transparency (which also helped minimize errors due to hardware differences)
– Riak, a key-value store – Cassandra, a column family store – Hbase, a column family store – MongoDB, a document-oriented database – MySQL Cluster, a NewSQL– Sharded MySQL, a NewSQL
PERFORMANCE ON WRITE
29
30
PERFORMANCE ON READ
CASE STUDIES
32
EXAMPLE: HEALTHCARE
A health care consultancy has made the data coming out of medical practices the focus of its thriving business. The company collects billing and diagnostic code data from 10,000 doctors on a daily, weekly and monthly basis to create a virtual clinical integration model. The consulting company analyzes the data to help the groups understand how well they are meeting the FTC guidelines for negotiating with health plans and whether they qualify for enhanced reimbursement based on offering a more cost-effective standard of care.
It also sends them automated information to better take care of patients, like creating an automated outbound calling system for pediatric patients who weren’t up to date on their vaccinations.
33
EXAMPLE: RETAIL
Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes * of data — the equivalent of 167 times the information contained in all the books in the US Library of Congress.
34
EXAMPLE: UTILITY
With a smart meter, a utility company goes from collecting one data point a month per customer (using a meter reader in a truck or car) to receiving 3,000 data points for each customer each month, while smart meters send usage information up to four times an hour.
One small Midwestern utility is using smart meter data to structure conservation programs that analyze existing usage to forecast future use, price usage based on demand and share that information with customers who might decide to forestall doing that load of wash until they can pay for it at the nonpeak price.
35
GROWTH FORECAST
3636
© 2013 KMS Technology
Q&A