2013 11-19-hoya-status

© Hortonworks Inc. 2013

Hoya: HBase on YARNSteve Loughran & Devaraj Das

{stevel, ddas} at hortonworks.com@steveloughran, @ddraj

November 2013


Hadoop as Next-Gen Platform

HADOOP 1.0

HDFS(redundant, reliable storage)

MapReduce(cluster resource management

& data processing)

HDFS2(redundant, reliable storage)

YARN(cluster resource management)

MapReduce(data processing)

Others(data processing)

HADOOP 2.0

Single Use SystemBatch Apps

Multi Purpose PlatformBatch, Interactive, Online, Streaming, …

© Hortonworks Inc.

YARN: Taking Hadoop Beyond Batch

Applications Run Natively IN Hadoop

HDFS2 (Redundant, Reliable Storage)

YARN (Cluster Resource Management)

BATCH(MapReduce)

INTERACTIVE(Tez)

STREAMING(Storm, S4,…)

GRAPH(Giraph)

HPC MPI(OpenMPI)

OTHER(Search)

(Weave…)Samza

Store ALL DATA in one place…

Interact with that data in MULTIPLE WAYS

with Predictable Performance and Quality of Service

IN-MEMORY(Spark)



BATCH(MapReduce)

INTERACTIVE(Tez)


GRAPH(Giraph)

HPC MPI(OpenMPI)

OTHER(Search)

(Weave…)HBaseIN-MEMORY

(Spark)



BATCH(MapReduce)

INTERACTIVE(Tez)


GRAPH(Giraph)

HPC MPI(OpenMPI)

OTHER(Search)

(Weave…)HBase

IN-MEMORY(Spark)

And HBase?

© Hortonworks Inc.

Hoya: On-demand HBase clusters

1. Small HBase cluster in large YARN cluster

2. Dynamic HBase clusters

3. Self-healing HBase Cluster

4. Elastic HBase clusters

5. Transient/intermittent clusters for workflows

6. Custom versions & configurations

7. More efficient utilization/sharing of cluster

© Hortonworks Inc.

Goal: No code changes in HBase

• Today : none

But we'd like• ZK reporting of web UI ports• A way to get from failed RS to YARN container (configurable ID is enough)

© Hortonworks Inc.

Hoya – the tool

• Hoya (Hbase On YArn)– Java tool–Completely CLI driven

• Input: cluster description as JSON–Specification of cluster: node options, ZK params–Configuration generated–Entire state persisted

• Actions: create, freeze/thaw, flex, exists <cluster>• Can change cluster state later

–Add/remove nodes, started / stopped states


YARN manages the cluster

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS

YARN Resource Manager

HDFS

YARN Node Manager

• Servers run YARN Node Managers• NM's heartbeat to Resource Manager• RM schedules work over cluster• RM allocates containers to apps• NMs start containers• NMs report container health


Hoya Client creates App Master

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS


HDFS

YARN Node Manager

Hoya ClientHoya AM


AM deploys HBase with YARN

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HDFS


Hoya Client

HDFS

YARN Node Manager

Hoya AM

HBase Region Server

HBase Region Server

HBase Master


HBase & clients bind via Zookeeper

HDFS

YARN Node Manager

HBase Region Server

HDFS

YARN Node Manager

HBase Region Server

HDFS


HBase Client HDFS

YARN Node Manager

Hoya AMHoya Client

HBase Master


YARN notifies AM of failures

HDFS

YARN Node Manager

HDFS

YARN Node Manager

HBase Region Server

HDFS


Hoya Client

HDFS

YARN Node Manager

Hoya AM

HBase Region Server

HBase Region Server

HBase Master

© Hortonworks Inc.

HOYA - cool bits

• Cluster specification stored as JSON in HDFS• Conf dir cached, dynamically patched before pushing up as local resources for master & region servers

• HBase .tar file stored in HDFS -clusters can use the same/different HBase versions

• Handling of cluster flexing is the same code as unplanned container loss.

• No Hoya code on region servers

© Hortonworks Inc.

HOYA - AM RPC API

//change cluster role countsflexCluster(ClusterSpec)

//get current cluster stategetJSONClusterStatus() : ClusterSpec

listNodeUUIDsByRole(role): UUID[]getNode(UUID): RoleInfogetClusterNodes(UUID[]) RoleInfo[]

stopCluster()

© Hortonworks Inc.

Flexing/failure handling is same code

boolean flexCluster(ClusterDescription updated) { providerService.validateClusterSpec(updated); appState.updateClusterSpec(updated); return reviewRequestAndReleaseNodes();}

void onContainersCompleted(List<ContainerStatus> completed) { for (ContainerStatus status : completed) { appState.onCompletedNode(status); } reviewRequestAndReleaseNodes();}


{ "version" : "1.0", "name" : "TestLiveTwoNodeRegionService", "type" : "hbase", "options" : { "zookeeper.path" : "/yarnapps_hoya_stevel_live2nodes", "cluster.application.image.path" : "hdfs://bin/hbase-0.96.tar.gz", "zookeeper.hosts" : "127.0.0.1" }, "roles" : { "worker" : { "role.instances" : "2", }, "hoya" : { "role.instances" : "1", }, "master" : { "role.instances" : "1", } }, ...}

Cluster Specification: persistent & wire


"roles" : { "worker" : { "yarn.memory" : "256", "role.instances" : "5", "jvm.heapsize" : "256", "yarn.vcores" : "1", "app.infoport" : "0" "env.MALLOC_ARENA_MAX": "4" }, "master" : { "yarn.memory" : "128", "role.instances" : "1", "jvm.heapsize" : "128", "yarn.vcores" : "1", "app.infoport" : "8585" }}

Role Specifications

© Hortonworks Inc.

Current status

• HBase clusters on-demand

• Accumulo clusters (5+ roles, different “provider”)

• Cluster freeze, thaw, flex, destroy

• Location of role instances tracked & persisted

–for placement close to data after failure, thaw

• Secure cluster support

© Hortonworks Inc.

Ongoing

• Multiple roles: worker, master, monitor

--role worker --roleopts worker yarn.vcores 2

• Multiple Providers: HBase + others

– client side: preflight, configuration patching

– server side: starting roles, liveness

• Liveness probes: HTTP GET, RPC port, RPC op?

• What do we need in YARN for production?

© Hortonworks Inc.

Ongoing

• Better failure handling, blacklisting

• Liveness probes: HTTP GET, RPC port, RPC op?

• Testing: functional, scale & load

• What do we need in Hoya for production?

• What do we need in YARN for production?

© Hortonworks Inc.

Requirements of an App: MUST

• Install from tarball; run as normal user• Deploy/start without human intervention• Pre-configurable, static instance config data• Support dynamic discovery/binding of peers• Co-existence with other app instance in cluster/nodes• Handle co-located role instances• Persist data to HDFS• Support 'kill' as a shutdown option• Handle failed role instances• Support role instances moving after failure

© Hortonworks Inc.

Requirements of an App: SHOULD

• Be configurable by Hadoop XML files• Publish dynamically assigned web UI & RPC ports• Support cluster flexing up/down• Support API to determine role instance status• Make it possible to determine role instance ID from app

• Support simple remote liveness probes

© Hortonworks Inc.

YARN-896: long-lived services

1. Container reconnect on AM restart

2. YARN Token renewal on long-lived apps

3. Containers: signalling, >1 process sequence

4. AM/RM managed gang scheduling

5. Anti-affinity hint in container requests

6. Service Registry - ZK?

7. Logging

© Hortonworks Inc.

SLAs & co-existence with MapReduce

1. Make IO bandwidth/IOPs a resource used in scheduling & limits

2. Need to monitor what's going on w.r.t IO & net load from containers apps queues

3. Dynamic adaptation of cgroup HDD, Net, RAM limits

4. Could we throttle MR job File & HDFS IO bandwidth?

2013 11-19-hoya-status

Technology

Transcript of 2013 11-19-hoya-status