Log Management and Analysis

Logging in Cloud

Vignesh GPrincipal EngineerApril 2016

© 2014 Progress Software Corporation. All rights reserved.2

Agenda

Why do we need logging?

Centralized Logging

What to look for while choosing Logging Tools?

Best Practices

Tools Comparison

Analytics and Performance Management

Saas Logger demo

© 2014 Progress Software Corporation. All rights reserved.3

Need for Logging

Production Monitoring Production troubleshooting Debugging Analytics Security Auditing and Compliance Application Performance Management Alerts

© 2014 Progress Software Corporation. All rights reserved.4

Traditional Logging

© 2014 Progress Software Corporation. All rights reserved.5

Log Aggregator

© 2014 Progress Software Corporation. All rights reserved.6

OpenSource Stack

© 2014 Progress Software Corporation. All rights reserved.7

Saas Tools

© 2014 Progress Software Corporation. All rights reserved.8

Best Practices

Use UUID - Stored in threadLocal (MDC if using slf4j ) AspectJ for logging Obfuscate sensitive information and PII in logs (logentries

datahug, Logstash filters) Capture console logs Log user identity wherever possible

© 2014 Progress Software Corporation. All rights reserved.9

What to look for while choosing Logging Tools?

Website Integrity – Whitelist options User roles and permissions Option to filter data before shipping Where is the data residing Is your data encrypted on the wire? Is the Provider SOC2 compliant? Is Data encrypted on Providers Disk?

© 2014 Progress Software Corporation. All rights reserved.10

Tools Comparison

Splunk• Pros: Feature rich, Market Leader, Excelent Search and charting

tools• Cons: Onpremise, Expensive

SumoLogic• Pros: Feature rich, alerts• Cons: As a developer, if you’re logging sensitive or PII you need to

make sure it’s redacted, Tiem Lag, Upload overhead

PapperTrails• Pros: Affordable• Cons: too basic

LogStash• Pros: Opensource• Cons: It’s a stack of three products

© 2014 Progress Software Corporation. All rights reserved.11

Log Management Landscape

© 2014 Progress Software Corporation. All rights reserved.12

Analytics - Logins

© 2014 Progress Software Corporation. All rights reserved.13

Geo Analytics

_sourceCategory=*demo*tomcat*access | parse regex "(?<remote_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" | lookup latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code from geo://default on ip = remote_ip | count by latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code | sort _count

© 2014 Progress Software Corporation. All rights reserved.14

Response time

© 2014 Progress Software Corporation. All rights reserved.15

APM