Log Management and Analysis for Cloud Applications

Log Management and Analysis

Logging in Cloud

Vignesh GPrincipal EngineerApril 2016

© 2014 Progress Software Corporation. All rights reserved.2

Agenda

Why do we need logging?

Centralized Logging

What to look for while choosing Logging Tools?

Best Practices

Tools Comparison

Analytics and Performance Management

Saas Logger demo


Need for Logging

Production Monitoring Production troubleshooting Debugging Analytics Security Auditing and Compliance Application Performance Management Alerts


Traditional Logging


Log Aggregator


OpenSource Stack


Saas Tools


Best Practices

Use UUID - Stored in threadLocal (MDC if using slf4j ) AspectJ for logging Obfuscate sensitive information and PII in logs (logentries

datahug, Logstash filters) Capture console logs Log user identity wherever possible


What to look for while choosing Logging Tools?

Website Integrity – Whitelist options User roles and permissions Option to filter data before shipping Where is the data residing Is your data encrypted on the wire? Is the Provider SOC2 compliant? Is Data encrypted on Providers Disk?


Tools Comparison

Splunk• Pros: Feature rich, Market Leader, Excelent Search and charting

tools• Cons: Onpremise, Expensive

SumoLogic• Pros: Feature rich, alerts• Cons: As a developer, if you’re logging sensitive or PII you need to

make sure it’s redacted, Tiem Lag, Upload overhead

PapperTrails• Pros: Affordable• Cons: too basic

LogStash• Pros: Opensource• Cons: It’s a stack of three products


Log Management Landscape


Analytics - Logins


Geo Analytics

_sourceCategory=*demo*tomcat*access | parse regex "(?<remote_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" | lookup latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code from geo://default on ip = remote_ip | count by latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code | sort _count


Response time


APM

Log Management and Analysis for Cloud Applications

Internet

Transcript of Log Management and Analysis for Cloud Applications