Hadoop&BigData:Craingthe...

© Avalon Consul,ng, LLC 2014

Hadoop & Big Data: Cra>ing the Enterprise Strategy

Sriram Mohan Senior Consultant, Avalon Consul,ng, LLC

Associate Professor of CSSE, Rose-‐Hulman Ins,tute of Technology


Presenter Overview Who we are •  Consultants providing expert technical integra,ons for

enterprise-‐scale Internet, intranet, and extranet sites •  50+ staff, mostly senior-‐level consultants •  Offices in Dallas and Washington, D.C. Best known for our work in: •  Enterprise search •  Hadoop, big data •  Enterprise content management •  Websites & portals •  E-‐learning •  Unstructured, semi-‐structured content


Presenter Overview Who we are •  Private engineering school in Terre Haute, Indiana •  2,000 students, 12:1 faculty-‐to-‐student ra,o •  10 math, science, and engineering majors Best known for : •  Best undergraduate engineering school in the country •  14 years in a row


Overview

•  Hadoop ecosystem •  Enterprise strategy •  Lambda architecture


What Is Big Data? •  Volume – Large quan,,es (think gigabytes, terabytes of informa,on daily)

•  Velocity – Needs to processed very quickly

•  Variety – Data might be structured, unstructured, varying sources


Overview

•  Hadoop ecosystem •  Enterprise Strategy •  Lambda Architecture


Apache Hadoop

•  Distributed plaaorm for data processing

•  Scalable •  Runs on commodity hardware •  Data & analysis coloca,on


Typical Hadoop Stack


Overview

•  Hadoop Ecosystem •  Enterprise strategy •  Lambda Architecture


How Do You Introduce Hadoop?


Some Tips

•  Start small •  Build a POC •  Replicate an exis,ng system in Hadoop (EDW offload)


What Would an EDW Offload Look Like?

•  How do you bring your data into HDFS? •  How do you analyze the data into HDFS? •  How do you verify the results of the analysis? •  How do you expose the results of your analysis?


How Do You Bring Your Data?

•  Flume •  Sqoop •  Every database/data warehouse has a Hadoop connector


How Do You Analyze the Data?

•  Pig •  Hive •  HBase


How Do You Verify?


How Do You Expose Your Results?

•  BI tools •  Export the data back to a data warehouse


Dealing With Semi-‐Structured Data

•  Naviga,ng the world of NoSQL with Hadoop •  Sample use cases – Batch processing – Search

•  How does Hadoop fit in? – Use Solr/elas,c search – Use HBase


Natural Language Processing •  What is NLP? •  Sample use cases – Adding metadata to emails – Predic,ve models – Forecasts

•  How does Hadoop fit in? – Mahout – Using R with Hadoop


Overview

•  Hadoop Ecosystem •  Enterprise Strategy •  Lambda architecture


Resource

•  Big Data: Principles and best-‐prac4ces of scalable, real4me data systems –  Nathan Marz and James Warren


Why Do We Need This?

•  Compu,ng arbitrary func,ons on an arbitrary dataset in real-‐,me is a daun,ng problem


Batch Layer


Serving Layer


Speed Layer


Handling Queries


Lambda Architecture


Ques,ons?

Hadoop&BigData:Craingthe...

Documents

Transcript of Hadoop&BigData:Craingthe...