©2010 Improving Enterprises, Inc.
Logging For Fun and Profit
Jane Prusakova
[email protected]://softwareandotherthings.blogspot.com
Improving Enterprises
Houston JUG
2014
©2010 Improving Enterprises, Inc.
Jane Prusakova
Hard-core developer
Data science geek
Dancer and photographer
©2010 Improving Enterprises, Inc.
Why
How-to
Using the results
So lets talk about logging…
©2010 Improving Enterprises, Inc.
Peek into the soul
Real setup
Real data
Real users
Real workDev environment
Happy path
Minimum load
Limited data
©2010 Improving Enterprises, Inc.
Inside the cloud
No programming-by-coincidence
Replaces debugging
Works on server
©2010 Improving Enterprises, Inc.
Logging [vs Debugging]
Set up once, always ready to use
Minimum disruption to the flow
Works locally and in the cloud
Faster data collection
Collected data is persisted
©2010 Improving Enterprises, Inc.
Logging helps with…
Integration
Multithreading
High load
©2010 Improving Enterprises, Inc.
What else?
Trace what the users do
Features
Sequences
Data flow
©2010 Improving Enterprises, Inc.
When to use logging?
New systems
New features
Bug fixing
©2010 Improving Enterprises, Inc.
Are logs forever?
Permanent
Errors
Extraordinary situations
©2010 Improving Enterprises, Inc.
Some are seasonal
Very temporary
Debugging info
Trace bug fixes
©2010 Improving Enterprises, Inc.
Others last awhile
Medium term
Data flow
Resource utilization
Optimization points
©2010 Improving Enterprises, Inc.
How-to
Using the results
©2010 Improving Enterprises, Inc.
Yes, I want to setup logging!
Pick a framework
Configure logging
Storage and access
Getting value from logs
©2010 Improving Enterprises, Inc.
Log4J SLF4J
Apache MIT license
Logging framework Logging facade
Configuration, log levels and categories,
rotate log files, thread-safe logging.
Support: online documentation, tutorials,
online forums.
©2010 Improving Enterprises, Inc.
Database Files
Easy to parse,
hard to evolve
Very hard to parse,
easy to change
Uses DB
connections
Requires FS
access
Can cause app
crash, loss of logs,
slow down app
Can fill up space
©2010 Improving Enterprises, Inc.
Where to write logs?
Console
File system
DB
Queue mechanisms
Combination
©2010 Improving Enterprises, Inc.
Logging configuration
Java properties-style
XML
Programmatically
Hard-coded
Takes preference
©2010 Improving Enterprises, Inc.
Log Message
What to log with the message
Importance level
Timestamp
Where in the code
Data being processed
System state info
©2010 Improving Enterprises, Inc.
Example layout
log4j.appender.stdout.layout.ConversionPattern=
%d %5p [%t] (%F:%L) - %m%n
- Date
- Log Level
- Thread name
- File and line (slow to retrieve)
©2010 Improving Enterprises, Inc.
Good log messages
Support code
Correctness
Readability
Performance
©2010 Improving Enterprises, Inc.
Better log messages
Relate to generating point
Code
Data
©2010 Improving Enterprises, Inc.
Best log messages
Are formatted for
Readability
Aggregation
©2010 Improving Enterprises, Inc.
Logging objects
.toString()
org.apache.commons.lang.builder.ToStringBuilder
log.info(myDataObject);
©2010 Improving Enterprises, Inc.
What to log
Input and output
Errors and exceptions
Computation results
Integration points
Thread rendezvous
©2010 Improving Enterprises, Inc.
Input and output
public int calculateCost(String user, Flight flight)
{
log.debug(“calculateCost: “ + user + “: “ +
flight);
int cost = … ;
… // cost calculation
log.debug((“calculateCost: cost for “ + user
+ “: “ + flight + “ = “ + cost);
return cost;
}
©2010 Improving Enterprises, Inc.
Errors and exceptions
// do dangerous work
result = DoDangerousWork(user, flight);}
catch (Exception e)
{
log.error(“methodName: exception in
DoDangerousWork for “ + user + “: “ + flight, e);
// recover or re-throw
}
©2010 Improving Enterprises, Inc.
More
Log destination, parameters, outcomesDB calls
SOAP calls
Wait time
Thread joins
Available memory
Memory-intensive operations
Intermediate calculation results
For complicated calculations
©2010 Improving Enterprises, Inc.
Avoid
Little information
… // step 1
logger.info(“did something - XXXX”);
… // step 2
logger.info(“did more work - YYYY”);
foreach (…) {
… // useful work
logger.info(“working hard”);
}
©2010 Improving Enterprises, Inc.
Avoid
Code cluttering
if (logger.isDebugEnabled()) {
logger.debug(“Show Views");
logger.debug("Address:\n" +
view.getAddress());
logger.debug("Email:\n" +
view.getEmail());
}
©2010 Improving Enterprises, Inc.
Watch for logging-caused bugs
Side effects
Logger.info(count++ + “: did something”);
Errors and exceptions
Logger.info(m.GetValue());
©2010 Improving Enterprises, Inc.
DRY principle applies
©2010 Improving Enterprises, Inc.
Growing, growing…
Naming conventions
Rotation
by size
by time
©2010 Improving Enterprises, Inc.
You’ve got logs!
Real-time access
Deal with space limitations
Warning: large datasets
©2010 Improving Enterprises, Inc.
Security
Absolutely
No Sensitive Data
©2010 Improving Enterprises, Inc.
Using the results
©2010 Improving Enterprises, Inc.
Learning from the logs
Never rely on eye-balling
WYS is not WYG
Distinguish common and
rare
©2010 Improving Enterprises, Inc.
Code and logs
©2010 Improving Enterprises, Inc.
Process log
Line by line
No need to load entire file
Aggregate events
Relate to time
©2010 Improving Enterprises, Inc.
Tools
Utilities
grep
sort
uniq
Scripting
Perl
Awk
sed
©2010 Improving Enterprises, Inc.
Learning from the logs
> grep userName myApplication-MMDDYYYY.log
> grep methodName myApplication-MMDDYYYY.log
> grep FATAL myApplication-MMDDYYYY.log
> tail -2000 myApplication-MMDDYYYY.log | grep Exception
©2010 Improving Enterprises, Inc.
Querying logs using Awk
GNU Awk www.gnu.org/software/gawk
Create histograms
Wait times
Number of calls
Exceptions per time period or per user
Compare and relate events
Exception stacks traces
Outcomes by caller or thread
©2010 Improving Enterprises, Inc.
AWK demo
Logs for the demo generously
provided by
http://42graphy.org/
©2010 Improving Enterprises, Inc.
Jane.Prusakova at ImprovingEnterprises.com
SoftwareAndOtherThings.blogspot.com
@jprusakova
Jane Prusakova
http://www.slideshare.net/jprusakova
©2010 Improving Enterprises, Inc.
Logging For Fun and Profit
Jane Prusakova
[email protected]://softwareandotherthings.blogspot.com
Improving Enterprises
Houston JUG
2014
Top Related