Slider: Applications on YARN
-
Upload
steve-loughran -
Category
Technology
-
view
393 -
download
0
description
Transcript of Slider: Applications on YARN
![Page 1: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/1.jpg)
Slider: Applications on YARNSteve Loughran– Hortonworks
stevel at hortonworks.com
@steveloughran
ApacheCon EU, November 2014
![Page 2: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/2.jpg)
log
Modern applications
front end
web
phones
devices
feeds log
stream
processing
front end
front end
log
database
analytics
![Page 3: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/3.jpg)
Fun operational problems
Placement: where to run?
Installation
Configuration & Binding
Client configuration
Lifecycle
Failure handing and recovery
Logging
Upgrading
Metrics & Monitoring
– Start/Stop
– Reconfigure
– Scale up/down
– Rolling-restart
– Decommission/Recommission
![Page 4: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/4.jpg)
Getting your code to work across a set of fairly reliable machines
![Page 5: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/5.jpg)
Apache Slider
Deploying and managingapplications on Apache Hadoop YARN
http://slider.incubator.apache.org/
![Page 6: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/6.jpg)
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
• Servers run YARN Node Managers (NM)
• NM's heartbeat to Resource Manager (RM)
• RM schedules work over cluster
• RM allocates containers to apps
• NMs start containers
• NMs report container health
Background: YARN
![Page 7: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/7.jpg)
Client creates App Master
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
ClientSlider Application Master
![Page 8: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/8.jpg)
Slider AM requests containers
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
Slider Application Master
HBase Master
HBase Region Server
HBase Region Server
![Page 9: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/9.jpg)
Slider: architecture
1. AppMaster
2. AgentProvider
3. Agent
4. AppPackage
5. CLI
Slider
App
Package
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
Agent Component
HDFS
YARN Node Manager
Agent Component
Slider Application Master
Slider CLI
![Page 10: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/10.jpg)
Desired State, agents & scripts
Slider
App
PackageCLI
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
Agent Component
HDFS
YARN Node Manager
Agent Component
1. CLI starts an instance of the AM
2. AM requests containers
3. Containers activate with an Agent
4. Agent gets application definition
5. Agent registers with Slider AM
6. AM issues commands
7. Agent reports back, status,
configuration, etc.
8. AM publishes endpoints,
configurations
Slider App Master
Registry
![Page 11: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/11.jpg)
© Hortonworks Inc. 2014:
App Package: zip XML .py
![Page 12: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/12.jpg)
App package Metainfo XML
Page 12
<service>
<name>HBASE</name>
<version>0.96.0.2.1.1</version>
<exportGroups><exportGroup>
<name>QuickLinks</name>
<exports><export>
<name>org.apache.slider.jmx</name>
<value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx</value>
</export></exports></exportGroup></exportGroups>
<commandOrders><commandOrder>
<command>HBASE_REGIONSERVER-START</command>
<requires>HBASE_MASTER-STARTED</requires>
</commandOrder></commandOrders>
<components><component>
<name>HBASE_MASTER</name>
<minInstanceCount>1</minInstanceCount>
<commandScript>
<script>scripts/hbase_master.py</script>
</commandScript></component></components>
</service>
Application Info
Dependencies
Publish an URI
Contains
components
scripts
![Page 13: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/13.jpg)
deploy: resource.json{
"schema": "http://example.org/specification/v2.0.0",
"global": {
"yarn.memory": "512"
},
"components": {
"HBASE_MASTER": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.memory": "1024",
"yarn.vcores": "1",
},
"HBASE_REGIONSERVER": {
"yarn.role.priority": "2",
"yarn.component.instances": "1"
}
} }
![Page 14: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/14.jpg)
configure: app_config.json
{
"application.def": "/slider/hbase_v096.zip",
"site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
"site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
"site.global.hbase_master_heapsize": "1024m",
"site.global.ganglia_server_host": "${NN_HOST}",
"site.global.ganglia_server_port": "8667",
"site.global.ganglia_server_id": "Application1",
"site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
"site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
"site.hbase-site.hbase.regionserver.port": "0",
"site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}",
"site.core-site.fs.defaultFS": "${NN_URI}",
}
Configurations needed by
Slider
Named variables
Site variables for
application
Named variables
for cluster details
Allocate and
advertise
Variables for the
application scripts
![Page 15: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/15.jpg)
create, start, stop, destroy
$ slider create hbase1 --resources resources.json --template appConfig.json
$ slider list
$ slider status hbase1
$ slider stop hbase1
$ slider start hbase1
$ slider stop hbase1--force
$ slider destroy hbase1
![Page 16: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/16.jpg)
Dynamic application resize
slider flex hbase1--component HBASE_REGIONSERVER 2
![Page 17: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/17.jpg)
Registry lookup
$ slider resolve --path ~/services/org-apache-slider/storm1
{ "type" : "JSONServiceRecord","external" : [ {
"api" : "http://","addressType" : "uri","protocolType" : "webui","addresses" : [ {
"uri" : "http://nn.example.com:46132"} ]
}, {"api" : "classpath:org.apache.slider.publisher.configurations","addressType" : "uri","protocolType" : "REST","addresses" : [ {
"uri" : "http://nn.example.com:46132/ws/v1/slider/publisher/slider"} ]
} } ] }
![Page 18: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/18.jpg)
![Page 19: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/19.jpg)
![Page 20: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/20.jpg)
Failures
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
Slider App Master
HBase Master
HBase Region Server
HBase Region Server
![Page 21: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/21.jpg)
YARN notifies AM
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
Slider App Master
HBase Master
HBase Region Server
![Page 22: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/22.jpg)
request on old node (lax)
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
Slider App Master
HBase Master
HBase Region Server
HBase Region Server
![Page 23: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/23.jpg)
Summary
• XML + python defined app deployment
• Declarative YARN resource specification
• Declarative application configuration
• Slider AM keeps app in desired state
• Lifecycle commands
• Binding and configuration via registry
![Page 24: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/24.jpg)
Slider View for Apache Ambari
http://ambari.apache.org/
![Page 25: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/25.jpg)
Managed Slider
Ambari
Slider
App
Package
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
Agent Component
HDFS
YARN Node Manager
Agent Component
Slider App Master
Registry
• Ambari imports app packages
• Starts the AM
• Interacts with AM to start
containers
• Agents register with Ambari
• Ambari sends commands/
receives results
• YARN manages containers
• Ambari interacts with Registry Registry
![Page 26: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/26.jpg)
![Page 27: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/27.jpg)
![Page 28: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/28.jpg)
Project Status
• In ASF incubation
• 0.60-incubating release this week(!)
• Need more developers!
• Need more application packages!
![Page 29: Slider: Applications on YARN](https://reader033.fdocuments.in/reader033/viewer/2022060121/559447471a28ab0f0d8b45dd/html5/thumbnails/29.jpg)
If you want to run existing appsin a YARN cluster:
Get involved!
http://slider.incubator.apache.org/