| Log management as a service Simplify Log Management

Apache Storm

To Log or Not to Log: Proven Best Practices for Instrumentation

Jon Gifford Founder and Chief Search Officer January 2014

| Log management as a service Simplify Log Management

About Me 15+ years of search experience

§  Lots of different search products §  Lots and lots of machines §  Lots and lots of logs...

Bias: §  Lucene, Java, *nix, Distributed, Realtime, §  Measure, Measure, Measure

| Log management as a service Simplify Log Management

§  Centralized logging and archival

§  Real-time processing, analysis and visualization

§  Monitoring, alerting and troubleshooting

Loggly: Addressing the first big data problem every company faces

| Log management as a service Simplify Log Management

To Log or Not to Log?

Lots of data to solve operational problems faster!

Incredibly painful to grep and inspect yourself + -

| Log management as a service Simplify Log Management

Instrumentation: Your Most Valuable Log Data For us, instrumentation is the act of gathering performance data about your application and surfacing it via logs.

§  Provides a low-impact mechanism for gathering production system performance data

§  Visualization is key to understanding this data

| Log management as a service Simplify Log Management

Best Practices for Instrumentation

| Log management as a service Simplify Log Management

Best Practice #1 Treat instrumentation as an ongoing, iterative process.

§  Start by logging everything at a high level §  Instrument deeper every time you ask, “What

happened here?” §  Instrument deeper in areas that are more variable

or expensive than you thought

| Log management as a service Simplify Log Management

Best Practice #2 Always instrument anything that goes out of process.

§  Can be indicative of larger problems, like the canary in the coal mine §  Example: Instrument calls to

your database, even though they should always be fast

§  Distributed system problems are not well behaved – by definition

| Log management as a service Simplify Log Management

Best Practice #3 Always log unacceptable performance.

§  Log “soft exceptions” (e.g. slow queries) §  “Trust but verify”

| Log management as a service Simplify Log Management

Best Practice #4 If possible, log enough context so you can get the complete picture of what happened from a single log event.

Process

Log this … not this!

Elapsed time End

End Start

| Log management as a service Simplify Log Management

Best Practice #5 View machines, not people, as your logs’ consumer.

| Log management as a service Simplify Log Management

Best Practice #6 Trends tell the story better than data points.

§  Graph performance over time to see daily fluctuations

§  Use rollups where it’s not practical to log every single event

| Log management as a service Simplify Log Management

Best Practice #7 Instrumentation is NOT a substitute for profiling, and vice versa.

Instrumentation Profiling

Your production environment is not the same as development!

| Log management as a service Simplify Log Management

Best Practice #8 Flying slower is better than flying blind.

§  “Speed advantage” is illusory if you can’t understand system performance

§  Our debate: not whether to instrument but how much

Phot

o by

Eri

c Ki

lby

| Log management as a service Simplify Log Management

Implement Instrumentation Best Practices Faster and Better with Loggly.

About Us: Loggly is the world’s most popular cloud-based log management solution, used by more than 5,000 happy customers to effortlessly spot problems in real-time, easily pinpoint root causes and resolve issues faster to ensure application success.

Try Loggly for Free! → http://bit.ly/BestLoggingPractices

Visit us at loggly.com or follow @loggly on Twitter.

| Log management as a service Simplify Log Management

Proven and Trusted by 5,000+ your peers…

“If we didn’t have Loggly, there would be days where our revenue would be affected by as much as 70 percent.”

– Albert Ho, Executive Producer/

Platform Product Manager, Rumble Entertainment