Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 1

Experiences in migration of large analytics platform from MPP database to Hadoop YARN

Srinivas Nimmagadda Roopesh VarierTechnical Director, CPE Director, CPE

Agenda

Introduction1

Big Data Needs2

MPP Platform and Challenges3

New Platform based on Hadoop/YARN4

Lessons learned during transition to Hadoop5

2Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier

Overview

• Symantec Cloud Platform Engineering (CPE)– Build consolidated cloud infrastructure and platform services

for next generation data powered Symantec applications.

– Open source components as building blocks• Hadoop and Openstack• Bridge capability gaps and contribute back

• A big data platform for batch and stream analytics integrated with Openstack. – Security, multi-tenancy, and reliability

• Using large scale data analytics for security and data management work loads– Analytics – Reputation based security, Managed Security

Services, Fraud Detection, Dial home application logs

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 3

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 5

Big Data Challenge

• Hundreds of millions of users• Billions of files– File good or not?

• Millions of URLs– URL safe or not?

• Hundreds of thousands of applications– Stable or Crashed

• Constant feed of information – Real time

– Across the global

– From our applications and appliances

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 6

Value from Volume

• Volume of data– Multi-petabyte historical datasets

– Multi-terabyte daily incremental datasets

– Wide variety of input data formats

– How do we manage?

• Variety of workloads– ETL jobs

– Batch applications

– Interactive ad-hoc analysis

• How to extract value from volume near real-time?

Agenda

Introduction1

Big Data Needs2

MPP Platform and Challenges3

New Platform based on Hadoop/YARN4

Lessons learned during transition to Hadoop5

2Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 8

MPP Platform

ETL Cluster Platform Services

Raw Data Store

Data Sources Applications

Batch

Interactive

MPP DB Engine

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 9

Legacy MPP Analytics Solution• Custom Platform Services– Task/Job management (DAG based, Fault-tolerant)

– Functional and performance monitoring

– Automatic data lifecycle management

– Inter cluster data transfers

– Cluster tenancy management

• ETL cluster • RDS (raw data store) on NAS• MPP (Massively Parallel Processing) DB engine at the core

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 10

Key Challenges• Scalability– Supporting rapid data growth

– No support for heterogeneous hardware.

• Operational costs– OpEx and Software licenses

• Supporting new use models– Not Only SQL patterns in analytics (columnar storage, search, streaming)

• Cluster operational challenges– Limited resource management (limits/quotas, utilization throttling)

– Load balancing across multi-mode and multi-tenant workloads

– Integrated secure tenancy services

– HA and DR

Agenda

Introduction1

Big Data Needs2

MPP Platform and Challenges3

New Platform based on Hadoop/YARN4

Lessons learned during transition to Hadoop5

2Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 12

MPP Platform

Raw Data Store ETL Cluster Platform

Services

Data Sources Applications

Batch

Interactive

MPP DB Engine

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 13

7: YARN/HDFS6: DistCP, Falcon5: DAG: Oozie

MPP DB Engine3: HDFS

MPP to Big Data Platform

Raw Data Store

Platform Services

Data Sources Applications

Batch

Interactive

1: Commodity Hardware

2: Hadoop Cluster 4: YARN

ETL

Job Management

State Transfer

Tenancy GuardETL Cluster

Batch

Interactive

Interactive Batch

YARN

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 14

Big Data Platform

Multi-tenant

Data Sources Applications

BatchInteractive

1: Cluster Infrastructure 2: Hadoop 2.x Stack

3: HDFS

5: Oozie

4: YARN

ETLInteractive Batch

Raw Data Store ETL Jobs Batch Interactive

Ad-hoc

workloads

Role-based provisioning Unified Logging

API

Agenda

Introduction1

Big Data Needs2

MPP Platform and Challenges3

New Platform based on Hadoop/YARN4

Lessons learned during transition to Hadoop5

2Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 16

Cluster Build Experiences• Node selection– Single Node SKU, use commodity hardware components

– Memory will be cheap, keep expansion options open

– Spindle-Core-LAN Network ratios (1 : 2.5 : 1.5 Gbs)

• Balance mixed workloads using YARN– Large clusters are better for effective resource utilization

– Balance between ETL, Batch, Interactive jobs with YARN

• Platform features and best practices– Central monitoring, log aggregation, and alerting metrics (ELK stack)

– Role based automated deployment of OS and Hadoop configuration

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 17

Journey to Hadoop• Goals– Open Source platform

– Scalable Distributed Processing

• Existing app base built around SQL• Many technology choices in Hadoop ecosystem– Technology choices: Distributed Query Engines vs. fast MR

– Evaluation with multi-PB data sets using 15 of our representative workloads.• e.g., complex joins (data shuffle), queries with variety of data

– Criteria: Scale, Functionality, Stability, Performance, Integration with other open source ecosystem

– Hive was the only technology able to scale and provide easy migration from our SQL workloads.

– With Tez we had an acceptable performance trade off.

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 18

RDS and ETL Process

• Platform features for ETL– File ingestion and Job management APIs

– Secure tenancy, Replication

• Conversion of 5 GB log file(.gz to .bz2)1. Single node outside Hadoop: ~28 mins

2. In Hadoop, single mapper, parallel read and write approach: ~5 mins

• A parallel RDS and ETL using YARN– Source file ingested from remote location

– Converted to bz2 and stored in HDFS Raw Data Store (Passive data)

– Data is transformed and loaded into Hive (Active data in ORC format)

– Mix “active” and “passive” datasets in HDFS

Use YARN for managing ETL

API

NN

DNDN

DN

DNDN

DN

Local .gz->bz2

MR based .gz->bz2

1

2

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 19

Large Cluster YARN Performance Modeling

• Multi-mode:– ETL jobs: Guaranteed throughput – window computing– Ad-hoc queries – Low latency, fast execution– Batch analytics applications – Throughput

• Multi-level– Departments/Projects, Users

• How do we model and use YARN for above workloads?

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 20

Example

Performance Modeling

ETL

Batch

Ad-hoc

Map Tasks

Reduce Tasks

HDFS Storage

Step 1: Compile your workload model

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 21

YARN Queue Model - 1

ETL Queue

Ad-hoc Queue

Batch Queue

Root Queue

Projects Queues

Jobs

Cluster Utilization:

Avg Latency:

Throughput (jobs):

Step 2: Develop your YARN queue resource allocation hierarchy

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 22

YARN Queue Model - 2

ETL Ad-HocBatch

Root Queue

Project Queues

Jobs

Cluster Utilization:

Avg Latency:

Throughput (jobs):

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 23

YARN Queue Model - 3

ETL Queue

Batch Queue

Root Queue

Ad-hocProject Queues

Jobs

Project Queues

Step 3: Run jobs, iterate thru’ models and pick optimal

Cluster Utilization

Avg Wait Time

Throughput (jobs):

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 24

Right Balance

• Optimal solution is about right balance– Cluster infrastructure

– Use the right software stack from Hadoop ecosystem

– Data management

– Application design and workload balancing with YARN

– Good tools for monitoring and management

• Approach– Start small and iterate faster

– When in doubt, experiment and get data to make decisions.

– Keep up customer use cases in perspective.

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 25

Summary

– Incremental transition from MPP to Big Data– A journey towards open source distributed computing– Uniform Computing!

• Infrastructure building blocks• Single large YARN cluster for variety of compute and storage loads

– Open source – use and contribute• Work with community to address gaps

– Share your ideas

Hadoop Summit 2014 – Srinivas Nimmagadda & Roopesh Varier 26

Q & A