Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul...
Transcript of Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul...
![Page 2: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/2.jpg)
The Speaker
open-source developermod_backhandWackamoledaiquiriOpenSSH/SecurIDSpreadetc.
closed-source developerEcelerity MTAEcelerity Clustering
Theo Schlossnagle Pricipal @ OmniTI
![Page 3: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/3.jpg)
Agenda
Understanding the Problem Space
A Survey of Technologies
Implementing Clustered Logging
Understanding New Possibilities
![Page 4: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/4.jpg)
Understanding the Problem Space
![Page 5: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/5.jpg)
The Purpose of Logging
Journalling the fact that a transaction has taken place.
Correlating a series of transactions into a session.
An audit trail.
Forensics.
Activity analysis to understand current trends and predict the future.
![Page 6: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/6.jpg)
Basic Expectations
Logs are reliable.
Events are logged in the order they occur.
They can be partitioned by date.
They can be multiplexed and demultiplexed on demand.
![Page 7: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/7.jpg)
Introducing ClusteringClustering:several machines acting together to provide a single serviceSessions may now be composed of a series transactions that occur on different machines.Ordering is “harder” and more important.
![Page 8: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/8.jpg)
A Survey of Technologies
![Page 9: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/9.jpg)
Traditional LoggingLogs written locally on web serversspace must be allocated
Consolidation happens periodicallycrashes will result in missing dataaggregators must preserve chronologyreal-time metrics cannot be calculated
Monitors must run against log serversmonitors must tail log filesrequires resources on the log servers
![Page 10: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/10.jpg)
Traditional Approach
mon2
web1 web2 web3
Realtime TCP/IP or UDP/IP
Web Clients
log2
storage
Traffic Monitor
C lick-stream Logger
log1
storage
Logging in its infancy
![Page 11: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/11.jpg)
Active Network LoggingLogs written directly to log serversUDP is unreliable and thus not usefulTCP is a point-to-point protocolTwo log server mean double trafficAdd a monitor and that’s triple!
Real-time metrics are possiblemonitors must tail log files still(or publishers must send directly to the monitors... yuck!)
![Page 12: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/12.jpg)
Network Approach
mon2
web1 web2 web3
Realtime TCP/IP or UDP/IP
Web Clients
log2
storage
Traffic Monitor
C lick-stream Logger
log1
storage
Adolescent Logging
![Page 13: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/13.jpg)
Passive Network LoggingLogs constructed from sniffed traffic The players no longer matterWeb servers can be added easily
Drops logs!When tested head-to-head with active logging frameworks we see lossMissing logs is unacceptable
![Page 14: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/14.jpg)
Passive Logging
A lapse in judgement
![Page 15: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/15.jpg)
mod_log_spread LoggingLogs are published over SpreadEfficient reliable network multicastPreserves global ordering of logs
Multiple subscribers at no costwell... almost zero
Extends well beyond ApacheAll logging (enterprise wide) can be utilize this publish/subscribe messaging bus
![Page 16: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/16.jpg)
mod_log_spread
mon2
web1 web2 web3
Web Clients
log2
storage
Traffic Monitor
C lick-stream Logger
log1
storage Spread Ring
Mature Logging
![Page 17: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/17.jpg)
instant aggregationorderingpublish/subscribe modelmultiple subscribersmultiple subscribersmultiple subscribers...
Clustered Logs Provide
![Page 18: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/18.jpg)
Data “feeds”Write them to diskReal-time analysis:popular pagesconcurrent sessions
Who’s online?Understand load-balanced click streams
Multiple Subscriber Magic
![Page 19: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/19.jpg)
Implementing Clustered Logging
![Page 20: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/20.jpg)
So show me!
SpreadApache 1.3 or 2.0mod_log_spreadspreadlogdA spread client API for your favorite language:Perl, Python, CJava, Ruby, PHP,etc.
![Page 21: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/21.jpg)
http://www.spread.org/
A simple /etc/spread.conf:
DebugFlags = { EXIT CONFIGURATION }
EventLogFile = /var/log/spread/mainlogEventTimeStamp
Spread_Segment 10.225.209.255:4913 { # order matters admin-va-1 10.225.209.68 # staging server www-va-1 10.225.209.71 www-va-2 10.225.209.72 www-va-3 10.225.209.73 samwise 10.225.209.240 # logging machines gollum 10.225.209.241 # monitoring machine}
Install Spread
![Page 22: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/22.jpg)
http://www.backhand.org/
A simple httpd.conf:
LoadModule log_spread_module libexec/mod_log_spread.soAddModule mod_log_spread.c#AddModule mod_log_config.cSpreadDaemon 4913
LogFormat "%h %l %u %t \"%r\" %>s %b" common
<VirtualHost coolsiteip:80> CustomLog $coolsite common</VirtualHost>
<VirtualHost slicksiteip:80> CustomLog $slicksite common</VirtualHost>
Install mod_log_spread
![Page 23: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/23.jpg)
; /opt/spread/bin/spuser -s 4913
User: connected to 4913 with private group #user#admin-va-1User> j coolsite============================Received REGULAR membership for group coolsite with 2 members, where I am member 1: #user#admin-va-1grp id is 182571332 1092928408 2Due to the JOIN of #user#admin-va-1User>============================received RELIABLE message from #ap25454#admin-va-1, of type 1, (endian 0) to 1 groups (182 bytes): 68.55.183.91 - - [30/Oct/2004:11:48:51 -0400] "GET /~jesus/ HTTP/1.1" 200 57940 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.5 (KHTML, like Gecko) Safari/125.9"
Verify it is working
![Page 24: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/24.jpg)
http://www.backhand.org/mod_log_spread/
A simple /etc/spreadlogd.conf:
BufferSize = 65536Spread { Port = 4913 Log { RewriteTimestamp = CommonLogFormat Group = "coolsite" File = /data/logs/apache/coolsite/common_log } Log { RewriteTimestamp = CommonLogFormat Group = "slicksite" File = /data/logs/apache/slicksite/combined_log }}
Install spreadlogd
![Page 25: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/25.jpg)
BufferSize = 65536PerlLib /opt/spreadlogd/customPerlUse myloggerSpread { Port = 4913 Log { RewriteTimestamp = CommonLogFormat Group = "coolsite" PerlLog mylogger::log File = /data/logs/apache/coolsite/common_log } Log { RewriteTimestamp = CommonLogFormat Group = "slicksite" File = /data/logs/apache/slicksite/combined_log }}
Spreadlogd:kung-fu (1)
BufferSize = 65536PerlLib /opt/spreadlogd/customPerlUse myloggerSpread { Port = 4913 Log { RewriteTimestamp = CommonLogFormat Group = "coolsite" PerlLog mylogger::log File = /data/logs/apache/coolsite/common_log } Log { RewriteTimestamp = CommonLogFormat Group = "slicksite" File = /data/logs/apache/slicksite/combined_log }}
![Page 26: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/26.jpg)
package mylogger;
use DBI;our $dbh;our $sth;
sub log($$$) { my $sender = shift; my $group = shift; my $message = shift; my ($user, $host) = ($sender =~ /#([^#]+)#([^#]+)/); chomp($message);
$dbh ||= DBI->connect("DBI:mysql:database=weblogs", "logger", "", { RaiseError => 0 }); warn "DBI->connect failed." unless($dbh); if($dbh) { $sth ||= $dbh->prepare(q{INSERT INTO logs (host, group, timestamp, data) VALUES(:1,:2,NOW(),:3)}); $sth->execute($host, $group, $message); }}
Spreadlogd:kung-fu (2)
![Page 27: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/27.jpg)
Understanding New Possibilities
![Page 28: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/28.jpg)
Logs are now streaming in real timeReal-time metricsper server hit rates (traffic)per server hits by response code
relative error serving rateper server document size metrics
detect unexpected bugs do to anomalous traffic
Track deeper datauser habitslength of visit online
All this happens passively
Advances
![Page 29: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/29.jpg)
Stupid Pet Tricks
![Page 30: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/30.jpg)
Credit WhereCredit’s Due
![Page 31: Cl ustered Lo gging - Esoteric Curiolethargy.org/~jesus/misc/Logging AC2005.pdf · T hey can be mul tiplexed and d emul tiplexed o n d emand. Introducing Cl ustering Clustering: ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/6043132957cee57c8607e6be/html5/thumbnails/31.jpg)
The John Hopkins UniversityThe Center for Networking and Distributed Systems
OmniTI Computer Consulting
The Authors and Contributors of Spread:Yair Amir, Michal Miskin-Amir, Jonathan Stanton, Christin Nita-Rotaru,
Theo Schlossnagle, Dan Schoenblum, John Schultz, Ryan Caudy, Ben Laurie,Daniel Rall, Marc Zyngier
The Authors of mod_log_spread and Tools:George Schlossnagle, Theo Schlossnagle, Jonathan Stanton, Yair Amir
Questions?