Scalable Log Analysis with WSO2 BAM
-
Upload
anjana-fernando -
Category
Software
-
view
141 -
download
0
Transcript of Scalable Log Analysis with WSO2 BAM
![Page 1: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/1.jpg)
Scalable Log Analysis with WSO2 BAM
Anjana Fernando
Senior Technical LeadWSO2 Inc.
![Page 2: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/2.jpg)
Agenda
• Logging Requirement• Thinking Big• Log Publishing• Log Storage• Log Analysis• Log Search• Custom Dashboards / Alerts• Demo• Future Improvements
![Page 3: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/3.jpg)
Logging Requirement
• Record interested events of a system• Analyse the log events• Take appropriate actions with the analysis
![Page 4: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/4.jpg)
Thinking Big
• Not your typical logger• WSO2’s logging solution is made
from ground up for large deployments
• Utilizes a big data architecture for logging
• WSO2 BAM as the framework
Source: fishingforsoul.wordpress.com
![Page 5: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/5.jpg)
Log Publishing
• Asynchronous• Thrift protocol based event streams• Custom Log4J pattern layout for capturing:
• Tenant information• Server information• Application information
• Not only for Java applications, you can write your own log publisher using any language,
![Page 6: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/6.jpg)
Log Storage
• Multi-level storage• Most recent logs stored in Cassandra• Archived logs store in HDFS
Source: www.carolinasit.com
![Page 7: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/7.jpg)
Log Analysis
• Hive/Hadoop based log summarisation and archiving
• Log archiving daily stored as a hierarchy of tenants and applications
• Custom analytics possible via custom Hive scripts and CEP integration
![Page 8: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/8.jpg)
Log Search
• Search the most recent logs• System Logs
• View logs from all applications, search by log level, keywords• System Logs
• View logs from a specific application, search by log level, keywords
![Page 9: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/9.jpg)
Custom Dashboards / Alerts
• Google Gadgets / Jaggery application based dashboards• Alerting support using in-built CEP features in BAM
• Supports output adaptors such as Email, SMS, HTTP, JMS with message types text, XML, JSON etc..
![Page 10: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/10.jpg)
WSO2 Distributed Logging Architecture
![Page 11: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/11.jpg)
Demo
![Page 12: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/12.jpg)
Future Improvements
• Out of the box support for many other log formats• A mechanism to define custom formats
• Improved searching capabilities• Better indexing for archived logs
• Support for additional data stores, i.e. not only Cassandra, but MongoDB, RDBMS etc.., required for better embeddability with other products
![Page 13: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/13.jpg)
Questions?
![Page 14: Scalable Log Analysis with WSO2 BAM](https://reader036.fdocuments.in/reader036/viewer/2022062319/5586dafad8b42a51718b46e3/html5/thumbnails/14.jpg)
Thank You