Splunk Architecture overview

16
1 SPLUNK OVERVIEW ALEXANDER FOK BIG DATA ARCHITECT FEBRUARY 2017

Transcript of Splunk Architecture overview

Page 1: Splunk Architecture overview

1

SPLUNK OVERVIEW

ALEXANDER FOK

BIG DATA ARCHITECT

FEBRUARY 2017

Page 2: Splunk Architecture overview

2

• What is Splunk?

• Splunk Main Functionality

• Why Splunk?

• Demo

Agenda

Page 3: Splunk Architecture overview

3

• Real Time logs collection, indexing and data analytics

• Time Series data management

• Search query language

tail –f ALL_LOGS*| grep “WHATEVER YOU NEED”

tail –f SOME_LOGS| grep “WHATEVER YOU NEED”| count by InterestingField

Commands Pipe

tail –f SOME_LOGS| grep “WHATEVER YOU NEED”| count by InterestingField

3

Splunk Main Functionality

Page 4: Splunk Architecture overview

4

• Presentation level – graphs, tables, etc

• Historical analysis

• Automation capabilities

• APIs

REST

Command line

Data Exploration and Visualization Capabilities

Page 5: Splunk Architecture overview

5

• Rolls Royce in the field

• Proven field success

• Flexible, User friendly, modern tool

• Enterprise grade – users access management, security, multitenant platform, data retention management policy

•Reach Echo system

•Splunk appstore –splunkbase.splunk.com

5

Why Splunk?

Page 6: Splunk Architecture overview

6

•Strong Visualization Capabilities – reports, dashboards

•Infinite scale – up to hundreds of TB logs per day

•Strong Post processing capabilities - Calculated and Extracted Fields

•Various Optimizations

Frequent Reports precalculation

Lookup tables

Field Tags

•Advanced Data Models - CIM

Why Splunk?

Page 7: Splunk Architecture overview

7

sourcetype=mysql_config OR sourcetype=remedy_changeticket

| dedup _raw, User

| transaction TicketId, User

| eval hasTicket = if(eventcount > 1, "Yes", "No")

| rename PrevPropValue as "Original_Value", NewPropValue

as "New_Value", hasTicket as "Change_Ticket"

| fields _time, User, Property, "Original_Value",

"New_Value", "Change_Ticket"

Page 8: Splunk Architecture overview

8

• Web logs• Log4J, JMS, JMX• .NET events• Code and scripts

• Configurations• syslog• SNMP• netflow

• Configurations• Audit/query logs• Tables• Schemas

• Hypervisor• Guest OS, Apps• Cloud

• Configurations• syslog• File system• ps, iostat, top

• Registry• Event logs• File system• sysinternals

Logfiles Configs Messages Traps Alerts

Metrics Scripts TicketsChanges

Linux/UnixWindows NetworkingDatabasesApplicationsVirtualization

& Cloud

• Click-stream data• Shopping cart data• Online transaction

data

Customer Facing Data

Outside the Datacenter

• Manufacturing, logistics…

• CDRs & IPDRs• Power consumption• RFID data• GPS data

No predefined schema, no custom connectors, no RDBMS, no need to filter/forward.

Splunk – The Big Picture

8

Page 9: Splunk Architecture overview

9

Splunk Architecture

Page 10: Splunk Architecture overview

10

Splunk’s MapReduce-based Architecture10

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Chunk 1

Chunk 2

Chunk 3

Chunk 4

Search Head

map

map

map

map

map

map

map

map

map

Answer

reduce

Server 1 Server 2 Server N

time

Page 11: Splunk Architecture overview

11

•Events, Indexes, Fields – key value pairs, columns

•Index Time

events are processed, classified, time stamp is extracted indexed

Predefined Fields are extracted

events can be enriched

Events can trigger logic -> alerts, reports, dashboards updates etc

•Search Time

events are searched

fields are extracted or calculated

transactions are closed

Visualizations can be built

Splunk Typical WorkFlow

Page 12: Splunk Architecture overview

12

•Show events counts by SFlow

•SFlow|stats count by SFlow

•| transaction SAUPID startswith="Product Start" endswith="Product End"

Demo

Page 13: Splunk Architecture overview

13

What is An App?

• Terminology

• Apps – A workspace that solves a specific use case with a navigable view

• Add-on – A reusable Splunk component that does not contain a view

• Example

• Splunk for Cisco Security is an App

• The collection of field extractions/sourcetypes/transforms/eventypes thatmap raw firewall logs is an Add-on

Page 14: Splunk Architecture overview

14

•CIM – Common Information Model

•Domain centric data models – OSSEC, networking, ticket management

•Data normalization

•Validation

•Visualization

•Action generation

Splunk as SIEM

Page 15: Splunk Architecture overview

15

Marathon Tel Aviv 2017 – See you tomorrow

Page 16: Splunk Architecture overview

16

Alexander Fok, Big Data Architect

THANK YOU


SPLUNK Capabilities Overview - GTRI · Overview GTRI is the only ELITE Splunk integrator with certified Splunk architects and engineers on staff. In addition to ... SPLUNK-Overview

SPLUNK Capabilities Overview - GTRI · Overview GTRI is the only ELITE Splunk integrator with certified Splunk architects and engineers on staff. In addition to ... SPLUNK-Overview

Splunk 6.2.0 Overview

Splunk 6.2.0 Overview

Virtualizing Splunk on Nutanix - Midland Information Systems · Virtualizing Splunk on Nutanix 2. Introduction | 6 2. Introduction 2.1. Audience This reference architecture is intended

Virtualizing Splunk on Nutanix - Midland Information Systems · Virtualizing Splunk on Nutanix 2. Introduction | 6 2. Introduction 2.1. Audience This reference architecture is intended

Splunk Architecture Guide

Splunk Architecture Guide

NetFlow Analytics for Splunk · NetFlow Analytics for Splunk App User Manual 3 Introduction Overview NetFlow Analytics for Splunk App is designed to deliver next generation, real-time,

NetFlow Analytics for Splunk · NetFlow Analytics for Splunk App User Manual 3 Introduction Overview NetFlow Analytics for Splunk App is designed to deliver next generation, real-time,

Referent / Redner Benjamin Tiggemann · 2015-07-08 · Splunk for Exchange Splunk for Enterprise Security* Splunk for Vmware Splunk for PCI Compliance* Splunk for Windows Splunk for

Referent / Redner Benjamin Tiggemann · 2015-07-08 · Splunk for Exchange Splunk for Enterprise Security* Splunk for Vmware Splunk for PCI Compliance* Splunk for Windows Splunk for

Overview of the Log Correlation Engine Splunk Clientstatic.tenable.com/prod_docs/Overview_of_the_LCE_Splunk_Client.pdf · The Log Correlation Engine Splunk Client has one purpose:

Overview of the Log Correlation Engine Splunk Clientstatic.tenable.com/prod_docs/Overview_of_the_LCE_Splunk_Client.pdf · The Log Correlation Engine Splunk Client has one purpose:

FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE or new to Splunk, you want to design a solution that will not go through peaks and valleys of performance and scale as Splunk takes

FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE or new to Splunk, you want to design a solution that will not go through peaks and valleys of performance and scale as Splunk takes

Splunk Architecture Workshop - Aplura€¦ · Who Are You? You Know Splunk is Right for You You Know Key Splunk Concepts You Require Purpose-Built Architecture You Have Scale Big

Splunk Architecture Workshop - Aplura€¦ · Who Are You? You Know Splunk is Right for You You Know Key Splunk Concepts You Require Purpose-Built Architecture You Have Scale Big

Splunk 6.2.3 Overview

Splunk 6.2.3 Overview

Splunk overview Russian

Splunk overview Russian

AboutUs* - SplunkConf · AboutUs* Splunk*Architecture*@BoschMSI SplunkinBosch IoTBusiness Splunk*Monitoring*@BoschMSI 1 2 3 4 5 Pastand*Future*Challenges* BoschSoware&Innovaons Agenda

AboutUs* - SplunkConf · AboutUs* Splunk*Architecture*@BoschMSI SplunkinBosch IoTBusiness Splunk*Monitoring*@BoschMSI 1 2 3 4 5 Pastand*Future*Challenges* BoschSoware&Innovaons Agenda

Dell EMC Ready Solutions for Splunk Solution Overview · Solution overview Dell EMC Ready Solutions for Splunk Harness machine data with simplified deployment of optimized solutions

Dell EMC Ready Solutions for Splunk Solution Overview · Solution overview Dell EMC Ready Solutions for Splunk Harness machine data with simplified deployment of optimized solutions

Full*Stack** Splunk*Development or* “How*to*Build*a Splunk ... · Agenda! Team* Idea Implementaons* Architecture* Challenges* Final*SoluIon* Tips’n’Tricks Future* 3

Full*Stack** Splunk*Development or* “How*to*Build*a Splunk ... · Agenda! Team* Idea Implementaons* Architecture* Challenges* Final*SoluIon* Tips’n’Tricks Future* 3

TR-4260: NetApp Architecture for Splunkstatic.ziftsolutions.com/files/8a16c27a4e0e96df014e1c4760e85e56.pdf · Technical Report NetApp Architecture for Splunk Walter Schroeder, Matt

TR-4260: NetApp Architecture for Splunkstatic.ziftsolutions.com/files/8a16c27a4e0e96df014e1c4760e85e56.pdf · Technical Report NetApp Architecture for Splunk Walter Schroeder, Matt

Splunk Integration Guide - files.observeit.comfiles.observeit.com/docs/Splunk Integration Guide.pdf · 1 Overview This document describes the integration of ObserveIT with Splunk

Splunk Integration Guide - files.observeit.comfiles.observeit.com/docs/Splunk Integration Guide.pdf · 1 Overview This document describes the integration of ObserveIT with Splunk

Introducing*the*Splunk*Web*Framework* · Agenda! Splunk*Developer*Plaorm*Overview* Whatis*the*Splunk*Web*Framework*and*Why?* Customer*Experience*– CloudPassage* Demos Deploymentand*AddiJonal*Resources*

Introducing*the*Splunk*Web*Framework* · Agenda! Splunk*Developer*Plaorm*Overview* Whatis*the*Splunk*Web*Framework*and*Why?* Customer*Experience*– CloudPassage* Demos Deploymentand*AddiJonal*Resources*

FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE … · FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE A Framework for Deploying Splunk® Enterprise at Scale June 2017

FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE … · FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE A Framework for Deploying Splunk® Enterprise at Scale June 2017

FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE · 2021. 5. 6. · RED HAT ENTERPRISE LINUX 7 ... SPLUNK AND CISCO UCS – BENEFITS ... FLASHSTACK INTRODUCTION The Pure FlashStack

FLASHSTACK CI FOR SPLUNK REFERENCE ARCHITECTURE · 2021. 5. 6. · RED HAT ENTERPRISE LINUX 7 ... SPLUNK AND CISCO UCS – BENEFITS ... FLASHSTACK INTRODUCTION The Pure FlashStack

Reference Architecture: Splunk Enterprise with ThinkSystem ... · Reference Architecture: Splunk Enterprise with ThinkSystem Servers Describes reference architecture for Splunk Enterprise

Reference Architecture: Splunk Enterprise with ThinkSystem ... · Reference Architecture: Splunk Enterprise with ThinkSystem Servers Describes reference architecture for Splunk Enterprise