Big Data

Biren Modi

What is Big Data?

Big Data is a blanket term for any collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization.

It is structured or unstructured data and contains all kind of data like text, Semi-structured (XML), streaming, etc…

Dimensions

Volume

Velocity

Variety

(Veracity)

Volume

Enterprises are awash with ever-growing data of all types, easily amassing terabytes—even petabytes—of information.

Turn 12 terabytes of Tweets created each day into improved product sentiment analysis

Convert 350 billion annual meter readings to better predict power consumption

Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few

There are huge volumes of data in the world:

From the beginning of recorded time until 2003,

We created 5 billion gigabytes (exabytes) of data.

In 2011, the same amount was created every two days

In 2013, the same amount of data is created every 10 minutes.

Velocity

Sometimes 2 minutes is too late. For time-sensitive processes such as catching fraud, big data must be used as it streams into your enterprise in order to maximize its value.

Scrutinize 5 million trade events created each day to identify potential fraud

Analyze 500 million daily call detail records in real-time to predict customer churn faster

The latest I have heard is 10 nano seconds delay is too much.

Variety

Big data is any type of data - structured and unstructured data such as text, sensor data, audio, video, click streams, log files and more. New insights are found when analyzing these data types together.

Monitor 100’s of live video feeds from surveillance cameras to target points of interest

Exploit the 80% data growth in images, video and documents to improve customer satisfaction

Private sectors

eBay.com uses two data warehouses at 7.5 petabytes and 40PB as well as a 40PB Hadoop cluster for search, consumer recommendations, and merchandising.

Amazon.com handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and as of 2005 they had the world’s three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.

Walmart handles more than 1 million customer transactions every hour, which are imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.

Facebook handles 50 billion photos from its user base.

Big data software

Aster - Teradata Inc

Datameer - Datameer Inc

FICO® Blaze Advisor® - FICO

Hadoop - Apache Foundation

HP Vertica - HP

MongoDB - MongoDB, Inc

Platfora- Platfora Inc

Spark - Apache Foundation

Splunk - Splunk Inc

Tableau - Tableau Inc

SAP HANA - SAP AG

Examples in real world

Growth of Big Data

Thank You