Nagios Conference 2013 - Leland Lammert - Nagios in a Multi-Platform Enviornment
-
Upload
nagios -
Category
Technology
-
view
1.714 -
download
0
Transcript of Nagios Conference 2013 - Leland Lammert - Nagios in a Multi-Platform Enviornment
Nagios in a Multi-Platform Environment
Leland V. Lammert, PhD
Chief Scientist
Omnitec Corporation
The Problem
Different OSs can require
Different type of connection
Different installation procedure
Nagios checks or an agent require a connection to the remote
machine
(i.e. from the Nagios server to the remote machine)
No issue inside the firewall
There are issues for machines outside the firewall
The Solution SSH
SSH forward connections are a good solution for monitoring inside the firewall
An ssh reverse tunnel is a good solution for monitoring outside the firewall
Active checks can then be used on all systems
No Agents
No Complex Installation
A shell is always available for troubleshooting
Today's Outline
Secure Shell
Connection Process
Credentials
Setting up a Reverse Tunnel
Using checks
Gui's - RDP or X
Troubleshooting
Secure Shell
SSH History
SSH [Secure Shell] is a data exchange protocol that uses an encrypted connection between two network devices
SSH replaced Telnet and other insecure remote shells which send information, most notably passwords, in plaintext
SSH encryption provides confidentiality and integrity of data over an insecure network
Connection Types
Normal [Forward]
Tunnel [Reverse]
SSH Advantages
Secure [ssh2]
Host Identity Verified
User Authorization
Secure Data Transmission
Ubiquitous ('Nix, Windows [Cygwin], OSX)
Reverse Tunnels require no firewall changes
Allows Active Checks
No agent to install or configure
Simplified testing, GUI still available [X or RDP]
SSH Details
Authentication is verified with SSH keys
[Forward] ssh connections work for hosts behind the firewall
[Reverse or Tunnel] ssh connections work for hosts anywhere else
The tunnel passes traffic TO the remote machine FROM the Nagios Server (reverse operation)
SSH Connections can be used to standardize connections to ALL machines!
Forward connections
Requires
Static IP
Open port
A static IP may not available at external locations
An open port for an incoming connection is a BIG security problem
Even when an agent is used [NSCA], there is no access to the remote machine for testing and troubleshooting
Reverse Connections - Tunnels
A Tunnel is packet encapsulation using a network protocol
The data payload protocol is then encapsulated in a delivery protocol.
L2TP (layer 2) Tunneling Protocol)
SSH
Socks
Reverse Tunnels - an ssh connection from client to server, where data is transmitted from server to client (i.e. reverse)
Connection Process
Startup
Start Session
Exchange Server key
Generate Session key
Authentication
Exchange Client key
If ACK, continue
SSH Startup
SSH Authentication
SSH uses Asymetric Encryption
Public/Private Keypair
A Keypair verifies identity for:
Machine
User
Currently keys are 2048 bits in length, usual RSA
Each Key has two parts:
Public and Private
AKA the keypair
Keys are created with ssh-keygen
Credentials
Host and User Credentials
Host Key
Verifies that the host/server at the other end of the connection is the same one seen previously
User Keys
Verifies that the user is authorized for the connection
Why not Passwords?
Requires manual entry
Cannot be automated without storing as plaintext
Much less secure than key
Keys on Unix/Linux
Keys:
/home//.ssh
Private key:id_rsa
Public key:id_rsa.pub
Keys on Windows
Typical Cygwin install location
C:\Program Files\Cygwin
Home directories:
/home//.ssh
Keys: ~/.ssh
Private key:id_rsa
Public key:id_rsa.pub
Keys on OSX
Keys:
/Users//.ssh
Private key:id_rsa
Public key:id_rsa.pub
Setting up a Reverse Tunnel
Basic ssh command
sshBase command
-fRun in background
-XEnable X Forwarding ('Nix)
-nPrevent reading from STDIN
-NNo remote command
-RReverse Operation
ssh -f -X -n -N -R
Ports
REMOTE:127.0.0.1:LOCAL*
REMOTEPort on remote machine [from]
127.0.0.1localhost
LOCALPort on local machine [to]
* Remember, this is being done at the external machine
so the Remote port here is actually on the Nagios server!
Authentication
$USER_NAME@$REMOTE_HOST
USER_NAMEUser ID on remote machine
REMOTE _HOSTHostname or IP of
remote hostt
-p LOGIN_PORT
LOGIN_PORTPort for login on remote machine
-i IDENTITY_FILE
IDENTITY_FILEPrivate key file to use
Three Steps to Create a Tunnel
1. Remote to NagiosConnect remote machine to Nagios server and create Tunnel
2. Make it permanent
cron
autossh
launchctl/launchd
3. Nagios to RemoteSetup access from Nagios user or process to remote machine
Step 1. Remote to NagiosConnect remote machine to Nagios
server
and create Tunnel
Step 1 'Nix
Create keypair on remote system and copy to Nagios Server
ssh-keygen
cat ~/.ssh/id_rsa.pub
ssh
vim ~/.ssh/authorized_keys
G o
:x
test
Step 1 Windows
Install Cygwin (cygwin.com/install.html)
Install in c:\program files\Cygwin
Add autossh, rsync, bash, perl
Create keypair on remote system and copy to Nagios Server [same as 'Nix]
Step 1 OSX
Create keypair on remote system and copy to Nagios Server [same as 'Nix]
Step 2.
Make it permanent
Step 2 'Nix cron
Connection variables
REMOTE_HOST="nagios._______.com"
USER_NAME="________"
REMOTE_PORT="____"
LOCAL_PORT="____" *
LOGIN_PORT="____" *
IDENTITY_KEY="/home/nagios/.ssh/id_rsa"
* NOTE: The ssh port should be changed for security, do NOT use the standard port 22.
Step 2 'Nix cron
Command to create the link
COMMAND="ssh -f -n -N -R \ $REMOTE_PORT:127.0.0.1:$LOCAL_PORT \ $USER_NAME@$REMOTE_HOST \ -p$LOGIN_PORT -i $IDENTITY_KEY"
Step 2 'Nix cron
Running? If not, start it
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
Working? Login to Nagios and check from the other side
ssh -i$IDENTITY_KEY -p$LOGIN_PORT $USER_NAME@$REMOTE_HOST netstat
-an | egrep "tcp.*127.0.0.1:$REMOTE_PORT.*LISTEN" > /dev/null
2>&1
If not working, kill and restart.
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi
Step 2 Windows autossh
Install servicecygrunsrv -I AutoSSH -f "nagios_link" -p
/usr/bin/autossh -a " \
-M : -N -R :127.0.0.1: @"
Use services.msc to set restart options and credentials:
cyg_server
Local Administrator
Step 2 OSX
Install autossh with homebrew
Command/usr/local/bin/autossh -M : -N -R -p 2206 -g 4000:127.0.0.1:22 @ -tt
LaunchctlCan install AutoSSH as System Service
Autostart at boot
Step2 OSX launchctl
Autossh file template
ID=username
HOST=hostname
if [ "X$SSH_AUTH_SOCK" = "X" ]; then
eval `ssh-agent -s`
ssh-add $HOME/.ssh/id_rsa
fi
#AUTOSSH_POLL=600
#AUTOSSH_PORT=20000
#AUTOSSH_GATETIME=30
#AUTOSSH_LOGFILE=$HOST.log
#AUTOSSH_DEBUG=yes
#AUTOSSH_PATH=/usr/local/bin/ssh
export AUTOSSH_POLL AUTOSSH_LOGFILE AUTOSSH_DEBUG AUTOSSH_PATH
AUTOSSH_GATETIME AUTOSSH_PORTautossh -2 -fN -M 20000 -R
2200:localhost:22 ${ID}@${HOST}
Step 3 Nagios to Remote
Setup access from Nagios user or process
to remote machine
Step 3 - 'Nix
Copy Nagios user public key to remote
Note: Nagios may use different UID than the one for testingcat ~/.ssh/id_rsa.pub
ssh
vim .ssh/authorized_keys
G o
:x
Step 3 Windows
Same as 'Nix
Step 3 OSX
Same as 'Nix
What was just created?
Take note at the current time there are working ssh connections to:
Any 'Nix
Any Windows*
OSX
These connections can be used for active checks, as well as troubleshooting and setting up an RDP or X session if needed.
* There sometimes are authentication issues with Windows Domain servers that must be handled uniquely.
Using Checks
Normal [forward] checks
Local checksdefine service{
use local-service
host_name Nagios
service_description Root Partition
check_commandcheck_local_disk!20%!10%!/
}
Checks via ssh connectiondefine host{
use openbsd-server
host_name mx1
alias mail_mx1_server
address 206.197.251.200
icon_image envelope.gif
statusmap_image envelope.gif
check_command check_smtp
}
Reverse Check
Same as other checks, except command format includes additional
datadefine service{
use generic-service
host_name hanley
service_description SSH Check Proc
normal_check_interval 15
retry_check_interval 5
notifications_enabled 1
check_command check_by_ssh_reverse!-p2210!proc
}
Additional data required in command
ssh Private key
ssh Port
Host address
Command Comparison
Local
command_name check_local_disk
command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p }
Forward
command_name check_by_ssh
command_line /usr/bin/ssh -i /.ssh/id_rsa \
nagios@$HOSTADDRESS$ $ARG1$ 2>/dev/null
Reverse
command_name check_by_ssh_reverse
command_line /usr/bin/ssh $ARG1$ -i /.ssh/id_rsa \
nagios@$HOSTADDRESS$ $ARG2$ 2>/dev/null
Building Plugins
Create test machine
VMs work great!
Linux
Windows
OSX
Compile set of plugins for that platform
Maintain platform plugin set
Gui's - RDP or X
X apps run natively
Be sure to include -X in tunnel setup
Connect to remote with ssh -X
Launch app
Possibly explicit invocation options, a la Firefox:
firefox no-remote
Start firefox, but run on the remote machine, forwarding the
display to the Nagios server.
RDP on demand
On remote machine
#
# Reverse ssh link for RDP tunnel
#
ssh -f -n -N -R 3389:127.0.0.1:3389 @ -p
Tunnel 3389 on remote machine to 3389 on Nagios ServerCheck for
open port
netstat -an | grep 3389
Use one connection at a time for sanity!
Troubleshooting
Common Problems
Host Key Verification Failed
The Host Key on the sending side is not the same as seen last time
Possible cause - the remote machine has been rebuilt
Troubleshooting checks
$64K Secret tmux
Very step-by-step
Paremeters often not required
Useful aliases
Hostkey verification failed
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for [storm]:2206 has changed,
and the key for the corresponding IP address [10.0.0.1]:2206
is unknown. This could either mean that DNS SPOOFING is happening
or the IP address for the host and its host key have changed at the
same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
6b:da:2e:50:a9:ea:b0:b1:3d:c1:b8:4a:a3:a5:56:87.
Please contact your system administrator.
Add correct host key in /home/lvl/.ssh/known_hosts to get rid of
this message.
Offending ECDSA key in /home/lvl/.ssh/known_hosts:5
You can use following command to remove all keys for this IP:
ssh-keygen -R storm -f /home/lvl/.ssh/known_hosts
Host key verification failed.
tmux
tmux is a terminal multiplexer
A terminal multiplexer permits switching between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.
Essential for troubleshooting checks!
tmux session
Useful Aliases
alias nagc='cd /etc/nagios'
alias nagh='cd /var/lib/nagios'
alias naghup='ps ax|grep /sbin/nagios |grep -v grep |awk "{print \$1}" | xargs kill -HUP'
alias nagq='netstat -an | grep 127 | grep LISTEN | grep -v 25 | sort --field-separator='\'':'\'' --key=2.1'
alias nagv='/usr/sbin/nagios -v /etc/nagios/nagios.cfg'
Resources
SSH and Authentication
http://en.wikipedia.org/wiki/Secure_Shellhttp://tools.ietf.org/html/rfc4252
SSH Cookbook Tunnels
http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Tunnels
Thank you!
Leland Lammert
www.omnitec.net
[email protected]
Click to edit the outline text format
Second Outline Level
Third Outline Level
Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
Ninth Outline Level
Click to edit the title text format
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level