Python vs JLizard.... a python logging experience
-
Upload
python-ireland -
Category
Technology
-
view
530 -
download
0
description
Transcript of Python vs JLizard.... a python logging experience
![Page 1: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/1.jpg)
Log Management As A Service
Oct 2011
Python vs. JLizard
![Page 2: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/2.jpg)
Outline
• A Logging Story
• About Us
• Data Data Everywhere (i.e. Logs Logs Everywhere)
• LogEntries.com
• Logging in the Cloud & PAAS
• Python vs JLizard
![Page 3: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/3.jpg)
A Logging Story
![Page 4: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/4.jpg)
Can you see the error?
25 log events in this window
![Page 5: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/5.jpg)
Can you see the error?
Consider 10,000 events per second
![Page 6: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/6.jpg)
logentries.com
![Page 7: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/7.jpg)
• UCD spin out company
• Founded by Viliam Holub & Trevor Parsons
• Based on UCD/IBM Research (Enterprise Ireland)
• Participated in National Digital Research Centre’s Launchpad program
(styled on Ycombinator)
• Nova UCD spin out award
• EI Funded Company
• Dogpatch Europe Company
• Logentries Launched in Q3 2011
JLizard & Logentries
![Page 8: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/8.jpg)
• Trevor Parsons
– Tech Background
• Enterprise Software Design, Monitoring Tools, Data Mining
• Java Java Java
• Performance & Testing Tools
– Startups
• Crovan: Training & Consultancy
• JLizard
About Me
![Page 9: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/9.jpg)
Data Data Everywhere (i.e. Logs Logs Everywhere)
• Estimates are that 150 billion gigabytes (exabytes) of data was created in
2005; This year about 8 times that amount (1,200 exabytes) will be created
- The Economist Feb 2010
• “The amount of enterprise data will grow about 650% over the next five
years, the vast majority of it unstructured, or not included in any database. “
- Gartner 2009
• Log data is the fasted-growing data source at large organizations -
CNET magazine 2010
• Many organizations are currently producing terabytes of log data per month
- CIO.Net 2009
![Page 10: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/10.jpg)
100,000 log messages / second x 300 bytes / log
message ~ 28.6 MB
x 3600 seconds ~ 100.6 GB / hour
x 24 hours ~ 2.35 TB / day
x 365 days ~ 860.5 TB / year
x 3 years ~ 2.52 PB
From Anton Chuvakin’s Blog Aug 2010
http://chuvakin.blogspot.com/
Log Maths
![Page 11: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/11.jpg)
Customer Type Log Volumes Events per Second Events per Day
Large Cloud Provider 50 Terabyes per Day 2,000,000 172,000,000,000
Large Social Media
Organisation
25 Terabytes per Day 1,000,000
Telecom Middleware/
Applications
1 Terabyte per Day 50,0000
Large Organisation
(>1000 employees)
300 GB Per Day 15,000
Online Marketing Org 100 GB per day 5,000 432,000,000
Small
Data Centre
10 GBs per Day 500
SAAS Educational
Tools
5Gbs Per Day 250
Single IBM Test Team 2 GBs per Day 100
Online Multimedia 700Mbs Per Day 35
Early Stage Start up 50Mbs Per Day 25 2,000,000
Typical Log Volumes
![Page 12: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/12.jpg)
Consequences
…Hacked System …Slow Web Site …System Crash
“Every minute that Facebook is down costs Zynga $10,000” SocialMediaInfluence.com, September 2010
![Page 13: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/13.jpg)
Logging - A Cloud Requirement
Server to Admin Ratio Increasing Significantly, from 10:1 to 500:1 Computerworld.com, July 2010
Compliance and Security Legislation Driving Requirement for Log Management
Where Is My Elastic Log Management Service? securecloudreview.com & networkworld, September 2010
![Page 14: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/14.jpg)
PAAS Logging
• Limited Logging Infrastructure
– App Engine – limited log buffers
– Heroku – 500 events for free
• No/Limited access to File System
• No Log Storage
• How to debug without any logs????
![Page 15: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/15.jpg)
Python vs. JLizard
Python @logentries
• Logentries Agent
• Web interface – Django
• App Engine Plug ins
![Page 16: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/16.jpg)
Logentries Python Agent
Why Python?
• Light weight agent
• Runs on Linux Out Of The Box
• Windows support
– created an exe
![Page 17: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/17.jpg)
![Page 18: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/18.jpg)
Available on github:
www.github.com/logentries
What it does:
– Communicates with Logentries API
– Uses SSL
– Data compressed on the wire
Logentries Python Agent
Features:
- Authenticate with Logentries Account
- Register your machine
- Configure your logs
- Monitor Logs
- Export Logs
- Push Logs
- Filter Logs
- Configure your logs (clusters /apps)
- Navigate and Manage your account
- Windows Event Logs
Other Stuff:
– Run as daemon
– Run as windows service
![Page 19: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/19.jpg)
Django
• Web Site
• Notifications
• User Authentication
• Landing Pages/Sign up
• Billing (Recurly Python Interface)
• Application UI
![Page 20: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/20.jpg)
App Engine Logging
Logging on App Engine:
• Nice interface
• Limited Buffers
• Can download logs periodically
• No Trend Analysis
• No Business Intelligence
• App Engine Roadmap - Improvements
![Page 21: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/21.jpg)
App Engine Logging
Logging to Logentries:
• In-Process Logging
• Push-Queue Logging
• Pull-Queue Logging.
![Page 22: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/22.jpg)
Use Cases & Platforms
• Windows/Linux/Mac
• Syslog Forwarding
• Developer Libs
• Ruby, Java, C#, Python,
Node.js
• Engineyard
• Heroku
• App Engine
• Cloudbees
• Mobile Devices
• Android
• Prevent System Crash
• Reduce Log Analysis Time
• Business Analytics
• Upload Saved Logs
• Instant Access to All Logs
• Mobile Device Logs
• Secure Log Storage
![Page 23: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/23.jpg)
Logentries
![Page 24: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/24.jpg)
Logentries
![Page 25: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/25.jpg)
Logentries
![Page 26: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/26.jpg)
Logentries
![Page 27: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/27.jpg)
Logentries
![Page 28: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/28.jpg)
Logentries
![Page 29: Python vs JLizard.... a python logging experience](https://reader033.fdocuments.in/reader033/viewer/2022052303/555a2f13d8b42a900d8b4e9c/html5/thumbnails/29.jpg)
Questions
Get a free account: www.logentries.com