Post on 14-Jul-2015
How to monitor the How to monitor the $H!T out of Hadoop$H!T out of Hadoop
Developing a comprehensive Developing a comprehensive open approach to monitoring open approach to monitoring
hadoop clustershadoop clusters
Relevant Hadoop InformationRelevant Hadoop Information
From 3 – 3000 NodesFrom 3 – 3000 Nodes
Hardware/Software failures “common”Hardware/Software failures “common”
Redundant Components DataNode, Redundant Components DataNode, TaskTrackerTaskTracker
Non-redundant Components NameNode, Non-redundant Components NameNode, JobTracker, SecondaryNameNodeJobTracker, SecondaryNameNode
Fast Evolving Technology (Best Fast Evolving Technology (Best Practices?)Practices?)
Monitoring SoftwareMonitoring Software
Nagios – Nagios – – Red Yellow Green Alerts, EscalationsRed Yellow Green Alerts, Escalations– Defacto Standard – Widely deployedDefacto Standard – Widely deployed
– Text base configurationText base configuration– Web InterfaceWeb Interface– Pluggable with shell scripts/external appsPluggable with shell scripts/external apps
Return 0 - OKReturn 0 - OK
CactiCacti
Performance Graphing SystemPerformance Graphing System
RRD/RRA Front EndRRD/RRA Front End
Slick Web InterfaceSlick Web Interface
Template System for Graph TypesTemplate System for Graph Types
PluggablePluggable– SNMP inputSNMP input– Shell script /external programShell script /external program
hadoop-cacti-jtghadoop-cacti-jtg
JMX Fetching Code w/ (kick off) scriptsJMX Fetching Code w/ (kick off) scripts
Cacti templates For HadoopCacti templates For Hadoop
Premade Nagios Check ScriptsPremade Nagios Check Scripts
Helper/Batch/automation scriptsHelper/Batch/automation scripts
Apache License Apache License
Hadoop JMX Hadoop JMX
Sample Cluster P1Sample Cluster P1
NameNode & SecNameNodeNameNode & SecNameNode– Hardware RAIDHardware RAID– 8 GB RAM8 GB RAM
– 1x QUAD CORE1x QUAD CORE– DerbyDB (hive) on SecNameNodeDerbyDB (hive) on SecNameNode
JobTrackerJobTracker– 8GB RAM8GB RAM– 1x QUAD CORE1x QUAD CORE
A Sample Cluster p2A Sample Cluster p2
Slave (hadoopdata1-XXXX)Slave (hadoopdata1-XXXX)– JBOD 8x 1TB SATA DiskJBOD 8x 1TB SATA Disk– RAM 16GBRAM 16GB
– 2x Quad Core2x Quad Core
PrerequisitesPrerequisites
Nagios (install) DAG RPMsNagios (install) DAG RPMs
Cacti (install) Several RPMSCacti (install) Several RPMS
Liberal network access to the clusterLiberal network access to the cluster
Alerts & EscalationsAlerts & Escalations
X nodes * Y Services = < SleepX nodes * Y Services = < Sleep
Define a policy Define a policy – Wake Me Up’s (SMS)Wake Me Up’s (SMS)
– Don’t Wake Me Up’s (EMAIL)Don’t Wake Me Up’s (EMAIL)– Review (Daily, Weekly, Monthly) Review (Daily, Weekly, Monthly)
Wake Me Up’sWake Me Up’s
NameNodeNameNode– Disk Full (Big Big Headache)Disk Full (Big Big Headache)– RAID Array Issues (failed disk)RAID Array Issues (failed disk)
JobTrackerJobTracker
SecNameNodeSecNameNode– Do not realize it is not working too lateDo not realize it is not working too late
Don’t Wake Me Up’sDon’t Wake Me Up’s
Or ‘Wake someone else up’Or ‘Wake someone else up’
DataNodeDataNode– Warning Currently Failed Disk will down the Warning Currently Failed Disk will down the
Data Node (see Jira)Data Node (see Jira)
TaskTrackerTaskTracker
HardwareHardware– Bad Disk (Start RMA)Bad Disk (Start RMA)
Slaves are expendable (up to a point)Slaves are expendable (up to a point)
Monitoring Battle PlanMonitoring Battle Plan
Start With the BasicsStart With the Basics– Ping, DiskPing, Disk
Add Hadoop Specific Alarms Add Hadoop Specific Alarms – check_data_nodecheck_data_node
Add JMX GraphingAdd JMX Graphing– NameNodeOperationsNameNodeOperations
Add JMX Based alarmsAdd JMX Based alarms– FilesTotal > 1,000,000 or LiveNodes < 50%FilesTotal > 1,000,000 or LiveNodes < 50%
The Basics NagiosThe Basics Nagios
Nagios (All Nodes)Nagios (All Nodes)– Host up (Ping check)Host up (Ping check)– Disk % FullDisk % Full
– SWAP > 85 %SWAP > 85 %
* Load based alarms are somewhat useless * Load based alarms are somewhat useless 389% CPU load is not necessarily a bad 389% CPU load is not necessarily a bad thing in Hadoopvillething in Hadoopville
The Basics CactiThe Basics Cacti
Cacti (All Nodes)Cacti (All Nodes)– CPU (full CPU)CPU (full CPU)– RAM/SWAP RAM/SWAP
– NetworkNetwork– Disk UsageDisk Usage
Disk UtilizationDisk Utilization
RAID ToolsRAID Tools
Hpacucli – not a Street Fighter moveHpacucli – not a Street Fighter move– Alerts on RAID events (NameNode) Alerts on RAID events (NameNode)
Disk failed Disk failed
RebuildingRebuilding
– JBOD (DataNode)JBOD (DataNode)Failed DriveFailed Drive
Drive ErrorsDrive Errors
Dell, SUN, Vendor Specific ToolsDell, SUN, Vendor Specific Tools
Before you jump inBefore you jump in
X Nodes * Y Checks * = Lots of workX Nodes * Y Checks * = Lots of work
About 3 Nodes into the process …About 3 Nodes into the process …– Wait!!! I need some interns!!!Wait!!! I need some interns!!!
Solution S.I.C.C.T. Semi-Intelligent-Solution S.I.C.C.T. Semi-Intelligent-Configuration-cloning-toolsConfiguration-cloning-tools– (I made that up) (I made that up) – (for this presentation)(for this presentation)
NagiosNagios
Answers “IS IT RUNNING?”Answers “IS IT RUNNING?”
Text based ConfigurationText based Configuration
CactiCacti
Answers “HOW WELL IS IT RUNNING?”Answers “HOW WELL IS IT RUNNING?”
Web Based configuration Web Based configuration – php-cli tools php-cli tools
Monitoring Battle PlanMonitoring Battle PlanThus FarThus Far
Start With the BasicsStart With the Basics– Ping, Disk !!!!!!Done!!!!!!Ping, Disk !!!!!!Done!!!!!!
Add Hadoop Specific Alarms Add Hadoop Specific Alarms – check_data_nodecheck_data_node
Add JMX GraphingAdd JMX Graphing– NameNodeOperationsNameNodeOperations
Add JMX Based alarmsAdd JMX Based alarms– FilesTotal > 1,000,000 or LiveNodes < 50%FilesTotal > 1,000,000 or LiveNodes < 50%
Add Hadoop Specific AlarmsAdd Hadoop Specific Alarms
Hadoop Components with a Web InterfaceHadoop Components with a Web Interface– NameNode 50070NameNode 50070– JobTracker 50030JobTracker 50030
– TaskTracker 50060TaskTracker 50060– DataNode 50075DataNode 50075
check_http + regex = simple + effectivecheck_http + regex = simple + effective
nagios_check_commands.cfgnagios_check_commands.cfg
Component FailureComponent Failure
(Future) Newer Hadoop will have XML status (Future) Newer Hadoop will have XML status
define command { command_name check_remote_namenode command_line $USER1$/check_http -H $HOSTADDRESS$ -u http://$HOSTADDRESS$:$ARG1$/dfshealth.jsp -p $ARG1$ -r NameNode}define service { service_description check_remote_namenode use generic-service host_name hadoopname1 check_command check_remote_namenode!50070}
Monitoring Battle PlanMonitoring Battle Plan
Start With the BasicsStart With the Basics– Ping, Disk (Done)Ping, Disk (Done)
Add Hadoop Specific Alarms Add Hadoop Specific Alarms – check_data_node (Done)check_data_node (Done)
Add JMX GraphingAdd JMX Graphing– NameNodeOperationsNameNodeOperations
Add JMX Based alarmsAdd JMX Based alarms– FilesTotal > 1,000,000 or LiveNodes < 50%FilesTotal > 1,000,000 or LiveNodes < 50%
JMX GraphingJMX Graphing
Enable JMXEnable JMX
Import TemplatesImport Templates
JMX GraphingJMX Graphing
JMX GraphingJMX Graphing
JMX GraphingJMX Graphing
Standard Java JMXStandard Java JMX
Monitoring Battle PlanMonitoring Battle PlanThus FarThus Far
Start With the Basics !!!!!!Done!!!!!Start With the Basics !!!!!!Done!!!!!– Ping, DiskPing, Disk
Add Hadoop Specific Alarms !Done!Add Hadoop Specific Alarms !Done!– check_data_nodecheck_data_node
Add JMX Graphing !Done!Add JMX Graphing !Done!– NameNodeOperationsNameNodeOperations
Add JMX Based alarmsAdd JMX Based alarms– FilesTotal > 1,000,000 or LiveNodes < 50%FilesTotal > 1,000,000 or LiveNodes < 50%
Add JMX based AlarmsAdd JMX based Alarms
hadoop-cacti-jtg is flexiblehadoop-cacti-jtg is flexible– extend fetch classesextend fetch classes– Don’t call output()Don’t call output()
– Write your own check logicWrite your own check logic
Quick JMX Base Walkthrough Quick JMX Base Walkthrough
url, user, pass, object specified from CLIurl, user, pass, object specified from CLI
wantedVariables, wantedOperations by wantedVariables, wantedOperations by inheritanceinheritance
fetch() output() providedfetch() output() provided
Extend for NameNodeExtend for NameNode
Extend for NagiosExtend for Nagios
Monitoring Battle PlanMonitoring Battle Plan
Start With the Basics !DONE!Start With the Basics !DONE!– Ping, DiskPing, Disk
Add Hadoop Specific Alarms !DONE!Add Hadoop Specific Alarms !DONE!– check_data_nodecheck_data_node
Add JMX Graphing !DONE!Add JMX Graphing !DONE!– NameNodeOperationsNameNodeOperations
Add JMX Based alarms !DONE!Add JMX Based alarms !DONE!– FilesTotal > 1,000,000 or LiveNodes < 50%FilesTotal > 1,000,000 or LiveNodes < 50%
ReviewReview
File System GrowthFile System Growth– SizeSize– Number of FilesNumber of Files– Number of BlocksNumber of Blocks– Ratio’sRatio’s
UtilizationUtilization– CPU/MemoryCPU/Memory– DiskDisk
Email (nightly)Email (nightly)– FSCK FSCK – DSFADMINDSFADMIN
The FutureThe Future
JMX Coming to JobTracker and JMX Coming to JobTracker and TaskTracker (0.21)TaskTracker (0.21)– Collect and Graph Jobs RunningCollect and Graph Jobs Running– Collect and Graph Map / Reduce per nodeCollect and Graph Map / Reduce per node– Profile Specific Jobs in Cacti?Profile Specific Jobs in Cacti?