© Hortonworks Inc. 2013
YARN and AmbariYARN service management using Ambari
Srimanth Gunturi
September 25th, 2013
Page 1
© Hortonworks Inc. 2013
Agenda
• YARN Overview• Installing• Monitoring• Configuration• Capacity Scheduler• MapReduce2• Future
Page 2Architecting the Future of Big Data
© Hortonworks Inc. 2013
YARN Overview
Page 3Architecting the Future of Big Data
• Yet Another Resource Negotiator (YARN)• Purpose - Negotiating resources (Memory, CPU, Disk, etc.) on a cluster• What was wrong with original MapReduce?
– Cluster not open to non-MapReduce paradigms– Inefficient resource utilization (Map slots, Reduce slots)– Upgrade rigidity
• MapReduce (Hadoop 1.0) -> YARN + MapReduce 2 (Hadoop 2.0)
© Hortonworks Inc. 2013
YARN Overview - Applications
Page 4Architecting the Future of Big Data
• MapReduce1 applications are fully compatible with MapReduce2 – Same JARs can be used– Binary compatibility with org.apache.hadoop.mapred API– Source compatibility with org.apache.hadoop.mapreduce API
© Hortonworks Inc. 2013
YARN Overview - Architecture
Page 5Architecting the Future of Big Data
• ResourceManager• NodeManagers• Containers• Applications (ApplicationMasters)
© Hortonworks Inc. 2013
Installing
Page 6Architecting the Future of Big Data
© Hortonworks Inc. 2013
Monitoring
Page 7Architecting the Future of Big Data
© Hortonworks Inc. 2013
Monitoring – NodeManager Summary
Page 8Architecting the Future of Big Data
NodeManagers Status• Active – In communication with RM• Lost – Not communicating with RM• Unhealthy – Flagged by custom health
check script identified by propertyyarn.nodemanager.health-checker.script.path
• Rebooted – Automatically restarted due to internal problems
• Decommissioned – RM ignoring communications from host. Host placedin yarn.resourcemanager.nodes.exclude-path file.
© Hortonworks Inc. 2013
Monitoring – Container Summary
Page 9Architecting the Future of Big Data
Containers• Allocated – Containers which have
been created with requested resources.• Pending – Containers, whose resources
will become available and are pending creation.
• Reserved – Containers, whose resources are not yet available.
Examples10 GB Cluster• Request three 5GB containers
• 2 allocated, 1 pending.• Request three 4GB containers
• 2 allocated, 1 reserved (2GB)
© Hortonworks Inc. 2013
Monitoring – Applications Summary
Page 10Architecting the Future of Big Data
Applications• Submitted – Application requests made
to YARN.• Running – Application with Masters
which have been created and are running.• Pending – Application requests which
are pending creation.• Completed – Applications which have
completed running. They could havebeen successful, killed or failed.
• Killed – Applications which have beenterminated by user
• Failed – Applications which have failedto run due to internal failures.
© Hortonworks Inc. 2013
Monitoring – Memory Summary
Page 11Architecting the Future of Big Data
Cluster Memory• Used – Memory resource currently
being used across the cluster• Reserved – Memory resources that
are set aside for being allocated.• Total – Memory resource available
across entire cluster
© Hortonworks Inc. 2013
Monitoring – Alerts
Page 12Architecting the Future of Big Data
Service Alerts- ResourceManager health- % NodeManagers alive
Host Alerts- NodeManager health- NodeManager process check
© Hortonworks Inc. 2013
Monitoring – Graphs
Page 13Architecting the Future of Big Data
© Hortonworks Inc. 2013
Configuration
Page 14Architecting the Future of Big Data
© Hortonworks Inc. 2013
Configuration – Capacity Scheduler
Page 15Architecting the Future of Big Data
QueuesRooto Ao Bo C
o C1o C2
o default
• Hierarchical Queues• Capacity Guarantees
• Capacity (%)• Maximum-am-resource-percent (%)
• Elasticity• Maximum-capacity (%)
• Access Control
© Hortonworks Inc. 2013
MapReduce2
Page 16Architecting the Future of Big Data
YARN-321: Generic application history service
© Hortonworks Inc. 2013
Future
Page 17Architecting the Future of Big Data
• Support more YARN applications• Improve per application-type information• Improve Capacity Scheduler configuration• Better health checks
Top Related