2

Where I Want to be.....

+

3

What I'll Settle For.....

4

Requirements

Rock solid stability Automated node addition (discovery) Scales horizontally Service dependency models Easy to write plugins Promotes sane workflows Unified front end view Flexible configuration

5

Tool Stack

6

What????but #monitoringsucks and #ihatenagios

How could you?

7

In defense of Nagios

Been around since 1996 Has Service dependencies Easy to write plugins Easy-ish to troubleshoot ROCK SOLID

8

Valid attacks on Nagios

No automated discovery It's complicated to setup Text files – really? Front end won't win any beauty contests Development is slow Stats collection is a PITA

9

Solutions Use Icinga! Use Puppet to auto configure Stats – leave it to graphite. It's really good at that Big boys and girls learn their tools

10

Icinga Fork of Nagios Configurations are compatible More solid architecture ( core, API, Web, IDODB ) Nice front end, nice mobile front end Can use NRPE

11

High Level View

12

Configure Icinga Servers using Puppet Standard Types

13

Things to configure with Standard Types

icinga.cfg (file) => icinga main config file Apache icinga.conf (file) => http access to each server cgiauth.cfg (file) => cgi access cgi.cfg (file) => options, users templates.cfg (file) got lazy => use for basic classes idomod.cfg (template) => template for hostname to DB

14

Configure Icinga using Nagios Types

15

Puppet Nagios Types

nagios_command nagios_contact nagios_contactgroup nagios_host nagios_hostdependency nagios_hostescalation nagios_hostextinfo

nagios_hostgroup nagios_service nagios_servicedependency nagios_serviceescalation nagios_serviceextinfo nagios_servicegroup nagios_timeperiod

16

Configuring Hosts

17

Overview

18

Detailed Overview

19

Store Configs

Store puppet info in a DB Retrieve information from

DB Share info across nodes Use thin_storeconfigs Set up on puppet master

20

Exporting Nagios_host Resources

Export = Save to DB Use facter for dynamic data PRO TIP: use ENC PRO TIP: use targets PRO TIP: hostgroups PRO TIP: use tags

21

PRO TIP: Use your ENC

22

PRO TIP: use targets

Use cfg_dir in icinga.cfg Create a unique file per host

or service Addition and removal are

now super easy Also default dirs are in a

horrible place /etc/nagios

23

PRO TIP: hostgroups Add machines to a

hostgroup Add services to a hostgroup New machines inherit all of

the services associated with a hostgroup

24

PRO TIP: use tags

Tags allow you to filter resources so that you only realize those resources that you need

25

Configuring Services/Commands

26

Icinga Services OR 'Stuff I want to monitor' Associate with a hostgroup Use a target

27

Icinga Commands

OR 'What actually gets run'

Use Macros to set paths in resource.cfg

28

Dependencies

29

PRO TIP: Dependencies

Unreliable services Cut down on the number of alerts Tell me what's really wrong Route alerts accordingly

30

Nagios_servicedependencies

31

NRPE

32

NRPE

Runs on client Secured via SSL Has ACLs Runs as nobody Can run commands Useful for other things...

33

Configuring NRPE

34

NRPE Checks

35

Plugins

36

exchange.nagios.org

37

Writing Plugins

Write in any language Output 1 line to stdout NRPE/Icinga/Nagios all use exit

codes to determine status Run by hand to check

38

Workflows

39

Watching Monitoring

40

Scheduling Downtime

41

Filtering

42

Alerting

43

#monitoringisawesome

REMOVE unreliable checks Just MONITOR – don't bolt on - especially stats TIER your monitoring Use timeperiods for sanity Delegate responses Use dependencies to pin down problems quickly Work smart

44

45

Resources Icinga

http://icinga.org

Puppet

http://docs.puppetlabs.com/references/latest/type.html#nagioscommand

NRPE

http://nagios.sourceforge.net/docs/3_0/addons.html

IRC

##infra-talk, #icinga, #puppet

Contact

christianmague@gmail.com, @maguec, #gaijin (freenode), http://blog.mague.com

Thanks

Yvonne Kong, Michael Catlin, Juan Ortega, Anthony Kong, Puppet Labs, Icinga Team