Cloudlytics - Analyze S3 & CloudFront Logs
-
Upload
cloudlytics -
Category
Technology
-
view
123 -
download
1
description
Transcript of Cloudlytics - Analyze S3 & CloudFront Logs
![Page 2: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/2.jpg)
Wh
y L
og
Pro
ce
ssin
g
1. Hidden Information in Logs
2. Traditional Way
3. With Cloudlytics
Use
Ca
se
s
1. ISVs
2. e-Learning
3. Media
4. Other Content Distribution Re
po
rts
1. Geographical Distribution
2. Popular Downloads
3. SPAM Attacks
4. Streaming Pattern
Co
min
g S
oo
n
1. Customizable Reports
2. SPAM Attack Alerts
3 .External Plugins
![Page 3: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/3.jpg)
Information Hidden in S3 & CloudFront Logs
3
-
Object Details Download Status
Download /Streaming Time
Number of
Bytes
Transferred
Details about Edge Locations
IP Address of the Requester
Referrer Link Time Taken to Download Object(S3)
Details about Play, Pause, Stop (Streaming Content on CloudFront)
![Page 4: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/4.jpg)
Uncover the Hidden Information
• Generating logs for Amazon S3 &
CloudFront logs is optional
• Log files are stored in S3 buckets
• CloudFront Log files are compressed
and stored in .gz format
• A log file is generated every hour, but we
have seen varied patterns with multiple files
generated every hour
• No ready solution from AWS to process
these log files
4
S3 logs CloudFront
logs
Logs Stored
in S3
Logs Analyzed by
CLOUDLYTICS
![Page 5: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/5.jpg)
Image Courtesy: www.fao.org
Traditional Log Processing
• Extract data from the source using an ETL tool
• Transform data and load in a data-warehouse
• Takes days to process a few GBs of
log files using traditional hardware
• Alternately use a Hadoop distribution
to process logs
• But maintaining a Hadoop cluster is a huge overhead
5
![Page 6: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/6.jpg)
Log Processing with Cloudlytics
• Cloudlytics - Analyze your Amazon S3 &
CloudFront Logs
• Detailed analysis of your S3 & CloudFront
access patterns
• Dynamic Graphs to get a 360 degree
perspective
• Scalable & Reliable service built using
Amazon EMR & RedShift
• Pay as you go
6
![Page 7: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/7.jpg)
Log Processing – Big Data Approach
• Cloudlytics extracts log files stored
in S3 buckets
• Processes the log files to
transform information
• Stores the processed data in
a data-warehouse
• Graphical and tabular reports generated from data-warehouse
7
![Page 8: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/8.jpg)
8
Cloudlytics Use Cases
![Page 9: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/9.jpg)
Independent Software Vendors (ISVs)
• ISVs distribute downloadable software to
end users across the globe
• ISVs need to ensure that downloads are
fast, helps improve user experience
• ISVs need to track each download for
success and failure
• Identify broken links on the website, helps improve user
experience
• Identify the most popular downloads, focus on popular products
• Identify spam attacks, help reduce bandwidth costs
9
![Page 10: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/10.jpg)
E-Learning Companies
• E-Learning companies distribute educational
content in multiple formats
(ebooks, audio, video etc.)
• Figure out the most popular content
• Figure out end user engagement by querying
number of events per request (Play, Pause, Stop)
• Get a breakup of requests by Operating System and Devices,
develop content creation for specific platforms
10
Image Courtesy: http://www.elifescience.in/images
![Page 11: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/11.jpg)
Media Organizations
• Large number of media assets
available online
• Content does not go any changes
during it’s life cycle
• Some content is extremely popular
while others do not get any views
• Identify the most popular content,
set caching mechanism
• Figure out end user engagement by querying number of events per
request (Play, Pause, Stop)
• Identify the edge locations from which the content is downloaded
the most, optimize billing using CloudFront pricing Class
11
![Page 12: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/12.jpg)
Cloudlytics REPORTS
12
Which Reports Resonate
with Your Business Needs?
![Page 13: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/13.jpg)
Geographic Reports
13
![Page 14: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/14.jpg)
Browser & OS Statistics
14
![Page 15: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/15.jpg)
Detail IP Monitoring
15
![Page 16: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/16.jpg)
Timeline Charts
16
![Page 17: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/17.jpg)
The TOP 10
17
![Page 18: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/18.jpg)
Edge Location Traffic
18
![Page 19: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/19.jpg)
Get Started in 3 Easy Steps
19
Configure your Log
buckets
Register for Free
Analyze & Generate
Reports
![Page 20: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/20.jpg)
Pricing
• No upfront costs
• Register for free and analyze upto 25MB logs/month FREE
• Pay only for the amount of logs you subscribe
20
![Page 21: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/21.jpg)
To sum it Up
21
Identifying Popular
Downloads & Streams
Get Geographical Distribution
of Downloads & Streams
Improve User Experience by
Calculating & Optimizing Latencies
Identify Edge Location
Traffic and optimize AWS Billing
Identify Spam
Attacks
Get Streaming Pattern
for Video Content
![Page 22: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/22.jpg)
Our Global User Reach
22
80+ Users and
Counting!
![Page 23: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/23.jpg)
Coming Soon!
23
Image Courtesy: http://www.trophies.com/coming-soon/
![Page 24: Cloudlytics - Analyze S3 & CloudFront Logs](https://reader034.fdocuments.in/reader034/viewer/2022042613/54c6ad5b4a79595e6c8b4575/html5/thumbnails/24.jpg)
Visit :Cloudytics.com
Contact: [email protected]
Call: +1 212 960 8532
Follow Us On :
Our Blog : http://blog.blazeclan.com/
Thank You