Turbo Charge your Logs

June 29th 2013https://joind.in/talk/view/8686

Turbo charge your logs

Who?

Ex-pat Englishman, living in Southern Ontario.

Web developer for 5 years, mostly PHP.

Senior Developer at MRX Digital Sports Solutions.

Ex-professional musician.

Why logging?

Of all the things you would come to a conference like this to hear about...

Real time information

At any given moment full of information about how things are performing right now.Mention framework logs, error logs, Apache logs, MySQL logs, system logs.

Confessional time...

Ask how many people have any sort of log monitoring setup.

For those that have, how many are looking at log data in real time?

Ask how many only look at logs during an outage.

Using log data is hard

Not in a human friendly format.

A lot of data.

Many log files.

Potentially many servers.

Many log files generated by many applications/pieces of software.Last time want to be digging through this is in a crisis.

Using your logs pro-actively

How can we stop using log data reactively and start using it pro-actively?

Mention that I can't tell you how to do this.

This talk will introduce some tools that can get you to this point.

Combination of tools will get you to a pro-active log monitoring solution.

Also mention that for each tool I'm talking about there are many alternatives...

Mention closed source alternatives.

Mention that this is being used in production at MRX.

The 'ideal' logging setup

Centralised.

Accepts logs from application code, software and the OS.

Performant.

Scalable.

Easily searchable.

Alarms and alerting.

Of course this will be different for everyone!

Logging from your code

What's wrong with error_log?

Nothing at all but...

It's limited:Have to format the message yourself.

Limited number of destinations.

Doesn't support all logging levels defined in RFC 5424.

Also mention that it's specifically for logging errors, not informational or debug messages.Difficult to format messages.Destinations: file or email.Define log levels in RFC 5425

RFC 5452 logging levels

Debug

Info

Notice

Warning

Error

Critical

Alert

Emergency

Note that notice and emergency are recommended not to be used by Monolog.

Introducing Monolog

PHP 5.3+ logging library by Jordi Boggiano.

Based on Python's Log Book library.

PSR-3 compliant.

Supports logging levels defined in RFC 5424.

Mention that there are many logging libraries but Monolog has seemed to have gained the most traction.Describe what PSR-3 is.

Installing Monolog

Symfony2, Laravel4, Silex and PPI all come with Monolog.

CakePHP and Slim have have plug-ins to use it.

Most easily installed with Composer.

PPI takes pieces of Zend 2, Sf2 and Doctrine2 and mashes them!Silex allows you to register a Monolog provider.

Monolog concepts

Channels.

Handlers.

Formatters.

Processors.

Channels

A channel is a name or category for a logger.

Each logger instance is given a channel when instantiated.

Allows for multiple loggers, each with a different channel.

Channel equates to facility in Syslog.Makes it easy to use different loggers for different parts/functionality in an app.Each logger can have different handlers.

Handlers

Handlers write log messages to a storage medium.

Multiple handlers can be attached to each logger.

Each handler can be configured to handle different log levels and to 'bubble' or not.

Many handlers available or you can write your own.

The handlers constructor accepts the minimum log level that the handler should accept. Defaults differently depending on handler.

Handlers can be shared between multiple loggers.

Needs care when not bubbling! Add more specific handlers later.

Example handlers

Files/SyslogStream Handler

Rotating File Handler

Syslog Handler

NotificationsMail handlers

Pushover Handler

HipChat Handler

DebuggingFirePHP Handler

ChromePHP Handler

Networked LoggingSocket Handler

AMQP Handler

Gelf Handler

Zend Monitor Handler

Rotating File Handler: Creates one file per day but meant as a quick + dirty solution.

Mail handlers include native mail and Swiftmail handlers.

Pushover handler sends mobile notifications through the Pushover API.

HipChatHandler send notification to a HipChat chat room (Rafael Dohms wrote it)

FirePHP and ChromePHP write to FireBug or Chrome consoles. DEV ONLY!!

Formatters

Processes a log message into the appropriate format for a handler.

Each handler has a default formatter to use but this can be overridden.

Use Handler::setFormatter() method to set the formatter for a handler.

Simple example

Mention that logging a message accepts up to two arguments:The message (string) and an array of context.

Using multiple handlers

Leveraging bubbling

Mention that handlers added last are called first.

Processors

Used to amend or add to the log message.

PHP callable, called when a message is logged.

Built in processors available:IntrospectionProcessor

WebProcessor

MemoryUsageProcessor

MemoryPeakUsageProcessor

ProcessIdProcessor

UidProcessor

Mention that this takes away some of the repetition of adding context to each log message.

IntrospectionProcessor: Adds the line/file/class/method from which the log call originated.

WebProcessor: Adds the current request URI, request method and client IP to a log record.

MemoryUsageProcessor: Adds the current memory usage to a log record.

MemoryPeakUsageProcessor: Adds the peak memory usage to a log record.

ProcessIdProcessor: Adds the process id to a log record.

UidProcessor: Adds a unique identifier to a log record.

Processor example

Where does this get us to?

Centralised. Maybe...

Accepts messages from application code, software and the OS.

Performant. Maybe...

Scalable. Maybe...

Easily searchable.

Alarms and alerting. Yes but crude.

We can do better!

Leveraging Syslog

Why Syslog?

Loggable events don't only happen in code!

Many apps/services send messages to syslog.

To get a full picture of what's going on we need to monitor these too.

Mention that system logger can log messages from the OS (e.g kernel) or applications.

Syslog basics

OS daemon to process log messages.

Messages are assigned a facility, such as auth, authpriv, daemon or cron.

Custom facilities of local0 local7.

Messages are also assigned a severity, defined in RFC 5424.

Messages can be sent to files, console or a remote location.

Which Syslog daemon to use?

In part will depend on your OS.

Features:Syslog is the oldest with not as many features.

Syslog-ng is produced under a dual license.

Rsyslog fully featured and open source.

Mention that you can often replace the default syslog daemon in an OS.

Introduction to Rsyslog

Fork of syslog by Rainer Gerhards.

Drop in replacement for syslog.

Many, many features including plugin system for extending.

Default syslogger in Debian, can be installed on other distros too.

Remote logging with Rsyslog

Rsyslog can be configured to work in a client-server setup.One or more machines are setup as clients to forward log messages.

One machine is setup to receive and store them.

Probably want to filter sender on the receiving machine...

Mention that not going into all features of Rsyslog, just focusing on remote logging.Suggest 'man rsyslog' or 'man rsyslog.conf'.Also mention that can use something like Rsyslog or IPTables to filter remote loggers.

Rsyslog client setup

Note this should be added to main rsyslog config file or a file that's included in it.This is for UDP forwarding. TCP would use @@.

Rsyslog server setup

Mention that normally you would need just one of these. Also that the corresponding port needs to be opened in the server config.This would only load the handler for the remote logs. Still needs to be processed with other directives.

Leveling up with Rsyslog

Apache can send all error logs to syslog directly.

Rsyslog can also monitor other log files using the Text File Input module.Example of monitoring Apache access log at https://gist.github.com/joseph12631/2580615

Where does this get us to?

Centralised. Yes.

Accepts messages from application code, software and the OS. Possibly.

Performant. Depends.

Scalable. Depends.

Easily searchable.

Alarms and alerting. Yes but crude.

Note that if all you want is to centralise all of your logs this could be the solution...

Taking it further with Logstash

What is Logstash?

Tool to collect, filter and output log messages.

Currently accepts 34 inputs, has 28 filters and 47 different outputs.

Built in web interface or richer web interface project called Kibana available.

Full information at http://logstash.net/

Kibana web demo at http://demo.logstash.net/

Mention that Logstash is written in Java.

Installing Logstash

Current release is 1.1.13 and can be downloaded from here.

Run from cli, use supervisord or an init.d/upstart script (cookbook entry on how to do this at http://cookbook.logstash.net/).

Inputs, filters and outputs

InputsAMQP/RabbitMQ

Syslog

Varnishlog

FiltersAnonymize

Grok

Geoip

Mutate

OutputsGelf

XMPP

AMQP/RabbitMQ

Nagios

Varnishlog input from Varnishes memory log.Anonymize anonymise fields using a consistent hash.Grok regex library for parsing log messages and processing matches.Geoip add geo data to ip addresses in log messages.Mutate General mutations (rename, remove, replace, modify) to fields.

Logstash config

When starting specify the path to a config file for Logstash to use.

Config file has JSON like syntax.

Three main sections: input, filter and output.

Each section may have multiple instances of each type.

Sample config file

KV filter parses key=value pairs into message parts (eg. key: value).

Where does this get us to?

Centralised. Yes.

Accepts messages from application code, software and the OS. Yes.

Performant. Yes.

Scalable. Yes.

Easily searchable. Possibly.

Alarms and alerting. Yes.

Of course this will be different for everyone!

Introducing Graylog2

What is Graylog2?

Log storage and search application.

Can accept thousands of messages per second and store terabytes of data.

Web interface for searching and analytics.

Built in alerting and metrics.

Discuss advantages and disadvantages to using Graylog or Logstash.

Installing Graylog2

Components:Elasticsearch

MongoDb

Graylog2 server

Graylog2 web interface

Full info on installing at http://support.torch.sh/help/kb

Live demo at http://public-graylog2.taulia.com/login

Mention that graylog server and elasticsearch are written in Java, web interface is a Rails app.Mention login details for the demo username admin or user, password graylog2.

Getting log messages into Graylog2

Can accept log messages in 3 ways:Graylog Extended Log Format (GELF) via UDP .

Syslog via UDP or TCP.

AMQP.

Multiple Graylog2 server instances can be run in parallel to spread processing of logs.

Benefits of UDP 'Fire and forget'.Drawbacks of UDP Lack of acknowledgement of receiving messages.TCP can mitigate packet loss but slower.AMQP guarantees delivery, but more complex to setup and run.GELF is basically JSON. Ideal for sending messages from app code. Libraries in many languages, including a Monolog handler.

Graylog2 web interface

Main view shows all recent log messages and graphs of number of messages received over the last several hours.

Single message can be clicked on to view all details for it.

Dashboard views.

Full search functionality.

Analytics dashboard and metrics.

Web interface view

Details of an individual message

Dashboard view

Searches and streams

Web interface allows fine grained searching by different fields.

Frequently used searches can be saved as streams.

Streams can be marked as favourites by users and can be viewed as dashboards.

Stream alarms

Alarms can be sent for a stream with user defined sensitivity.

Plugins for sending alarms include:Email

PagerDuty

HipChat

Twilio SMS

Jabber/XMPP

You can also write your own plugins.

Where does this get us to?

Centralised. Yes.

Accepts messages from application code, software and the OS. Yes.

Performant. Yes.

Scalable. Yes.

Easily searchable. Yes.

Alarms and alerting. Yes.

Of course this will be different for everyone!

Thanks for listening

Feedback: https://joind.in/8686

Questions?

Putting it all together

A few possible implementations.