Managing a Multi-Tenant Data Lake

Managing A Multi-Tenant Data Lake

2Copyright 2016 Comcast Corporation. All Rights Reserved

Agenda Timeline for Evolution Why Governance Multi-Tenancy Anti-Patterns / Warning Signs Instituting Governance Managing through Chaos Monitoring/Metrics Environment Tools SLA Management Support and Staffing Demo - Command Center


Timeline – 2013

2013 – “The Experiment” Started with 10 node cluster

Experimentation with batch processing and enrichment of event data

Team assembled from across organization

Primarily solving single use case

30 nodes by end of trial

2 Racks


Timeline – 2014 (H1)

2014 Production “Honeymoon”

Added 70 more nodes along with lower environments (Dev & QA)

Onboard additional ~20 data sets through batch ETL

Supporting a dozen use cases

5 Racks


Timeline – 2014 (H2)

2014 Production “Tiger’s Tail”

Total of 200 nodes to support additional use cases (data science)

Total of ~30 more data sets through batch ETL

Supporting several dozen use cases and ad-hoc exploration

Starting to have difficulty managing resource requests

9 Racks


Timeline – 2015

2015 Production “Cortez”

Adding 250 more nodes to production environment

Fully embraced governance

Supporting 24x7 production use cases

19 Racks


Timeline – 2016

2016 Production “Planetary”

Adding 1300 more nodes to production environment

Standing up separate 500 node data science cluster

Spinning off critical compute to boundary satellite clusters

Reaping benefits from governance and resource planning

48 Racks


Why Governance?

It’s about establishing acceptable behaviors for the benefit of the community

Minimize user/application impact on cluster

Users will do whatever is technically possible Everyone has been conditioned to work “smarter not harder”

Establishing the guardrails not edicts.


Multi-Tenancy Anti-Patterns

Speculative Execution

Optional User Training

Lack of Resource Isolation

Lack of Testing and Measurement

Ad-hoc Communication Channels

Excessive Resource Utilization/Reservation

Informal Service Level Agreements (SLAs)

Public Domain: Plynn9

https://commons.wikimedia.org/wiki/User_talk:Plynn9


Signs of Looming Disaster

Pending Jobs

Queue Fidgeting

Job Rescheduling

Non Predictive Workloads

Cluster Storage Out Of Balance

Public Domain: US DOE


Instituting Governance

Governance is not a technology problem

Governance must be solved using People - Who Processes – What / When / How Policy – Why

Always employ technology to help with enforcement and measurement


Setting Out Governance Standards – Starting Out

Involve the business users to define light-weight policies and processes Onboarding users/applications/tools Resource Utilization Worksheets Deployment checklists Service Level Agreements / Penalties Updates of Governance Standards

You MUST socialize and educate your community on these policies and process

Strive for evolution not revolution


Setting Out Governance Standards – Measurement

Define universally accepted performance measures Storage Compute System Availability Issues and MTTR Average Completion Time Average Pending Apps

Be transparent with results and make them available to entire community

Establish monthly performance reviews with key stakeholders


Setting Out Governance Standards – Enforcement

Lock down as many resources as possible

Monitor resource utilization for compliance

Automate corrective measures

Its all about transitioning from defense to offense and eliminating surprises!


Setting Out Governance Standards – Enforcement

Hadoop provides some base capabilities YARN Queues for compute HDFS Quotas/ACLs for storage

Implement custom solutions for proactive offensive capabilities Job monitoring and migration (Penalty Box) Dynamic Allocation / Queue Flexing Monitor and track leading indicators (Command Center)


Multi-Tenancy: Understanding the Chaos - Monitoring/Metrics

Image Attribution: Pixabay - Creative Commons CC0


Use Case – Extreme Ad Hoc (Data Science)


Challenges? You bet!


Challenges Monitoring and Managing a Multi-tenant Hadoop Environment – Diverse User Community

Div

erse

Use

r Com

mun

ity

Images: Creative Commons


Challenges Monitoring and Managing a Multi-tenant Hadoop Environment - SLAs

Div

erse

SLA

s


Challenges Monitoring and Managing a Multi-tenant Hadoop Environment - Governance



Challenges Monitoring and Managing a Multi-tenant Hadoop Environment – Monitoring & Forecasting



Environment


Our Environment - Tools for Monitoring

Standard Hadoop Monitoring


Environment - Tools for Monitoring

Command Center

Pepperdata


SLA Management

Application Timing



SLA Management

Application Timing

Resource Management



SLA Management

Application Timing

Resource Management

Capacity Management



Support & Staffing



Takeaways for DevOps Model in Hadoop

Train Your Teams (!!!)




Measure, Forecast and Model




Measure, Forecast and Model

Automation and Frameworks


Comcast Command Center


The Command Center: Our Focus

Visualizations & Design


Ease Of Use





Ease Of Use

Extensibility




Ease Of Use

Extensibility

Alerting



The Command Center for Monitoring and Alerting

• Missed SLAs• Guardrails broken

• Definitions• Links

• Containers• Queue capacity

• Status• Measures

• HDFS Usage• Queue Usage

Continuous Evolution

Continuous Engagement


Monitoring and Alerting at Comcast

The Command Center!


Thanks!

Ray HarrisonPrinciple DevOps Architect

Mike FaganPrinciple Big Data Architect

[email protected] [email protected]

We Are Hiring!

Managing a Multi-Tenant Data Lake

Technology

Transcript of Managing a Multi-Tenant Data Lake