Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open...

12
Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013

Transcript of Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open...

Page 1: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Log Analysis using Open Source Scalable SystemsGurvinder SinghVerktøysamling Oct'2013

Page 2: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Motivation

● Distributed Systems

● In the moment of heat

● Centeralized interface to logs

● Easier access

10/2/13 SLIDE 2

http://www.themeparkreview.com/tatsumediaday/tatsumediaday57.jpg

http://www.themeparkreview.com/tatsumediaday/tatsumediaday57.jpg
Page 3: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Challenges

● Almost every component generates logs

● Different Formats and logging methods

● Different requirements for processing

● Dashboards

● Alerts

10/2/13 SLIDE 3

http://img72.imageshack.us/img72/3885/nephew2logs.jpg

http://img72.imageshack.us/img72/3885/nephew2logs.jpg
Page 4: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Components

● Logstash

● Redis

● Elasticsearch

● Kibana

● Beaver

● lumberjack

● .....

10/2/13 SLIDE 4

Page 5: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Architecture

Page 6: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Architecture

Page 7: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Architecture

Page 8: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Architecture

Page 9: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Architecture

Page 10: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Demo

Page 11: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Possible Input/Processing/Output Options

10/2/13SLIDE

11

Page 12: Log Analysis using Open Source Scalable Systems - …samling:2013:log... · Log Analysis using Open Source Scalable Systems Gurvinder Singh Verktøysamling Oct'2013. ... gemfire filters

Thanks:-)

Questions?

10/2/13SLIDE

12


Scalable training of L 1 -regularized log-linear models

Scalable training of L 1 -regularized log-linear models

Översiktskatalog över Register över Wilskmans samling

Översiktskatalog över Register över Wilskmans samling

Using vRealize Log Insight - vRealize Log Insight 8 · 2020. 7. 23. · vRealize Log Insight provides scalable log aggregation and indexing for the vCloud Suite, including all editions

Using vRealize Log Insight - vRealize Log Insight 8 · 2020. 7. 23. · vRealize Log Insight provides scalable log aggregation and indexing for the vCloud Suite, including all editions

ScaleFFS: A Scalable Log-Structured Flash File System for ...

ScaleFFS: A Scalable Log-Structured Flash File System for ...

Forest Plantation Management Plan - Samling Group FPMP Public Summary 4 Nov 16.pdf · SST Management Plan LPF/0014 SEGAN Edition 3 ... Samling uses the MTCS principles and criteria

Forest Plantation Management Plan - Samling Group FPMP Public Summary 4 Nov 16.pdf · SST Management Plan LPF/0014 SEGAN Edition 3 ... Samling uses the MTCS principles and criteria

Scalable database, Scalable language @ JDC 2013

Scalable database, Scalable language @ JDC 2013

Scalable Vector Graphics (SVG) 1 · 10/27/2004  · Scalable Vector Graphics (SVG) 1.2 G.2 Registration of Media Type image/svg+xml Appendix H: References Appendix I: Change log Authors

Scalable Vector Graphics (SVG) 1 · 10/27/2004  · Scalable Vector Graphics (SVG) 1.2 G.2 Registration of Media Type image/svg+xml Appendix H: References Appendix I: Change log Authors

Scalable Vector Graphics (SVG) 1 · 27/10/2004  · Scalable Vector Graphics (SVG) 1.2 G.2 Registration of Media Type image/svg+xml Appendix H: References Appendix I: Change log Authors

Scalable Vector Graphics (SVG) 1 · 27/10/2004  · Scalable Vector Graphics (SVG) 1.2 G.2 Registration of Media Type image/svg+xml Appendix H: References Appendix I: Change log Authors

 · – avd. för Teknisk Geologi, Lunds tekniska högskola . PROGRAM . 09.00 Registrering & samling, kaffe/te . 09.30 Välkomsthälsning. Mikael Hellsten, Stiftelsen Bergteknisk

 · – avd. för Teknisk Geologi, Lunds tekniska högskola . PROGRAM . 09.00 Registrering & samling, kaffe/te . 09.30 Välkomsthälsning. Mikael Hellsten, Stiftelsen Bergteknisk

Utdrikningslag Koder: Gamle sanger ... H-Musikk Rev C.pdfKoder: Gamle sanger – DVD samling – CDG Samling – Norsk Karaoke NR Tittel Artist Ext Nr HM Nr 1. (EVERYTHING I DO) I

Utdrikningslag Koder: Gamle sanger ... H-Musikk Rev C.pdfKoder: Gamle sanger – DVD samling – CDG Samling – Norsk Karaoke NR Tittel Artist Ext Nr HM Nr 1. (EVERYTHING I DO) I

Webinar: Lateral Movement€¦ · Data Scientists. Log-Likelihood Ranking. Normality Classifier (MIVB) Scalable Implementation (Spark, GraphX) Computer Scientists. Deployable Workflow

Webinar: Lateral Movement€¦ · Data Scientists. Log-Likelihood Ranking. Normality Classifier (MIVB) Scalable Implementation (Spark, GraphX) Computer Scientists. Deployable Workflow

Can Scalable Development Lead to Scalable Execution? · Can Scalable Development Lead to Scalable Execution? Damian Rouson Scalable Computing R&D Sandia National Laboratories Sponsors:

Can Scalable Development Lead to Scalable Execution? · Can Scalable Development Lead to Scalable Execution? Damian Rouson Scalable Computing R&D Sandia National Laboratories Sponsors:

Scalable Transactions for Scalable Distributed Database ...digitalassets.lib.berkeley.edu/etd/ucb/text/Pang_berkeley_0028E_15408.pdf · Scalable Transactions for Scalable Distributed

Scalable Transactions for Scalable Distributed Database ...digitalassets.lib.berkeley.edu/etd/ucb/text/Pang_berkeley_0028E_15408.pdf · Scalable Transactions for Scalable Distributed

MRNet: From Scalable Performance to Scalable Reliability

MRNet: From Scalable Performance to Scalable Reliability

Big Data Meets HPC Log Analytics: Scalable …Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale Byung H. Park , Saurabh Hukerikar , Ryan

Big Data Meets HPC Log Analytics: Scalable …Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale Byung H. Park , Saurabh Hukerikar , Ryan

…A Scalable Systems Initiative Scalable Microsoft ...€¦ · A Scalable Systems Initiative Scalable Microsoft Application Re engineering Technology Scalable Systems Inc Web: systems.com

…A Scalable Systems Initiative Scalable Microsoft ...€¦ · A Scalable Systems Initiative Scalable Microsoft Application Re engineering Technology Scalable Systems Inc Web: systems.com

Scalable Web Softwareweb.stanford.edu/class/cs193s/CS193S-1-07.pdf · The Web is Just Right • Many basic computer science ... • They are so called because in a log/log plot they

Scalable Web Softwareweb.stanford.edu/class/cs193s/CS193S-1-07.pdf · The Web is Just Right • Many basic computer science ... • They are so called because in a log/log plot they

Maritime transport- The next step in e-mobility Samling ... transport... · Samling for Verneombudene Konsernsjef Jannicke Hilland Maritime transport- The next step in e-mobility

Maritime transport- The next step in e-mobility Samling ... transport... · Samling for Verneombudene Konsernsjef Jannicke Hilland Maritime transport- The next step in e-mobility

Assisting with Scalable Scalable Vector Graphics and ... · SVG Scalable Vector Graphics [6] SSVG Scalable Scalable Vector Graphics [10] LWA Live Website Annotate [See Section 4]

Assisting with Scalable Scalable Vector Graphics and ... · SVG Scalable Vector Graphics [6] SSVG Scalable Scalable Vector Graphics [10] LWA Live Website Annotate [See Section 4]

Big Data Meets HPC Log Analytics: Scalable Approach to ......Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale Achievement: Designed and

Big Data Meets HPC Log Analytics: Scalable Approach to ......Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale Achievement: Designed and