How Netskope Mastered DevOps with Sumo Logic

How Netskope Mastered DevOps with Sumo Logic

Kumar Saurabh,

Co-Founder & VP Engineering

12/3/2014

Sumo Logic Confidential

Agenda

• Sumo Logic Overview

• Running Sumo on Sumo

• How Netskope mastered DevOps with Sumo Logic

• Q&A

Sumo Logic Confidential2

Search

Visualize

Predict

Applications

Mobile

Internet of Things

Network and Server

Transforming Machine Data Into Meaningful Insights


On-Prem Data Centers

Cloud Sources

Collector

Collector

Powerful Architecture, Effortless Deployment


Hybrid Data Sources

Private Public

PaaSIaaS

SaaS

Hosted Collector

The Analytics Engine


Core

Analytics


LogReduce™- Transform Logs Into Meaningful Patterns

Reduce 100,000+ log messages into 20 relevant patterns

Search across multiple sources and timeframes

Discover “the needle in the haystack”

Drill down into results for additional context

Annotate results, influence future ranking with human context

6

02/15/2014 10:03:16 UTC Health status check: zim-5 is OK

02/15/2014 10:03:11 UTC Health status check: gir-3 is OK

02/15/2014 10:03:07 UTC Health status check: gir-2 is TIMED OUT

02/15/2014 10:02:45 UTC Health status check: dib-1 is OK

A log line and is its LogReduce Signature


$DATETIME Health status check: **** is ****

printf(“%s Health status check: %s is %s”, timestamp, hostid, hoststatus)

LogReduce

02/15/2014 10:03:16 UTC Health status check: zim-5 is OK

A Baseline and its Underlying Signatures


02/15/2014 10:39:16 UTC Health status check: asm-5 is OK02/15/2014 10:38:16 UTC module=xyz reported a TimeoutException02/15/2014 10:37:16 UTC Health status check: gar-3 is OK02/15/2014 10:37:16 UTC Health status check: gar-7 is OK02/15/2014 10:35:16 UTC database connection established: ora3302/15/2014 10:29:16 UTC Health status check: abd-3 is TIMED OUT02/15/2014 10:37:16 UTC Health status check: gar-3 is OK02/15/2014 10:27:16 UTC pix firewall: tcp denied: 10.12.2.1002/15/2014 10:22:16 UTC Health status check: dim-5 is OK02/15/2014 10:21:16 UTC database connection established: ora123302/15/2014 10:20:16 UTC Health status check: fim-5 is SLOW02/15/2014 10:19:16 UTC pix firewall: tcp accept: 111.12.2.1002/15/2014 10:18:16 UTC Health status check: zim-5 is OK02/15/2014 10:15:16 UTC database connection established: ora124302/15/2014 10:13:16 UTC module=adyz reported a NullPointer02/15/2014 10:12:16 UTC Health status check: z21-5 is OK02/15/2014 10:09:16 UTC Health status check: rsm-5 is OK02/15/2014 10:06:16 UTC database connection established: ora121302/15/2014 10:05:16 UTC pix firewall: tcp accept: 111.12.22.1002/15/2014 10:04:16 UTC Health status check: 5gm-5 is OK

$DATETIME Health status check: **** is ****

$DATETIME Module=**** reported a ****

$DATETIME pix firewall: tcp ****: ****

$DATETIME database connection established: ****

S1:S2:S3:S4:

SIGNATURE %(COUNT)

S1 55%(11)

S2 10%(2)

S3 20%(4)

S4 15%(3)

Baseline

How does AD find events using the baseline?


Baseline

SIG 12:20 12:25 12:30 12:35 12:40 12:45 12:50 12:55 13:00 13:05 13:10 13:15 13:20 13:25 13:30 13:35 13:40 13:45 13:50 13:55 14:00 14:05 14:10 14:15

55% S1 54% 52% 20% 52% 58% 58% 57% 56% 54% 48% 57% 57% 54% 55% 58% 56% 56% 52% 40% 54% 55% 54% 53% 57%

10% S2 11% 9% 14% 11% 10% 9% 30% 11% 10% 8% 9% 11% 10% 0% 10% 9% 10% 11% 10% 10% 11% 10% 9% 11%

20% S3 18% 23% 38% 20% 17% 19% 5% 19% 23% 18% 20% 19% 23% 26% 15% 20% 18% 23% 35% 21% 20% 20% 23% 19%

15% S4 17% 16% 28% 17% 15% 14% 8% 14% 13% 11% 14% 13% 13% 19% 17% 15% 16% 14% 15% 15% 14% 16% 15% 13%

S5 15%

S1

S2

S3

S4

Event 2

S1

S2

S3

S4

Event 1

S1

S2

S3

S4

S5 ✪

Event 3

S1

S2 ✖

S3

S4

Event 4

S1

S2

S3

S4

Event 5

Up

Down

✖ Gone

✪ New

Each event is a UNIQUE combination of changes from baseline

0%

50%

100%

S1

S2

S3

S4

S5


Anomaly Detection: Expose Unknown Events

10

Powerful Visualization of Transactional Relationships

APPLICATIONSFOR ENTERPRISE SYSTEMS

APP MANAGEMENT SECURITY AND COMPLIANCE IT OPERATIONS

How We Use Sumo Logic

IN ONE DAY,

SUMO LOGIC

Analyzes

5.8petabytes

of data

Scans

14trillion records

Examines

247years

of data

Overview of Sumo Logic Deployment


Production(2000+ nodes)

Pre-Production(For Internal Use Only)

Logs2TB+ a day

Logs

https://service.sumologic.com

Dev QAPerformance

TestsBuild

SystemsIT

Logs LogsLogs

DevOps Philosophy

Instrument Everything, Monitor KPIs

RCA: Flexible query language to ask any question

Turn incidents into actionable knowledge

– Alerts and Monitors

Peace for the paranoid: Anomaly Detection

– Proactive response better than reactive

Key to Running a Successful DevOps Shop


Monitoring Production KPIs


Scheduled Queries in last 24h by user

Drill downs from KPI


How We Use Sumo Logic

Abhay Kulkarni – Vice President of Engineering

© 2014 Netskope. All Rights Reserved. 20

Discovery, Visibility, and

Granular Control for Safe

Cloud Enablement

• Real-time, granular control of any

cloud app, sanctioned or not

• Deep contextual visibility

• Comprehensive end-point

coverage and flexible deployment

The Netskope Active Platform™

© 2014 Netskope. All Rights Reserved.

Discover Apps and Mitigate Risk

21

• Discover enterprise cloud apps

• Get their enterprise-readiness

score and details

• Understand your risk based on

your usage of those apps


Visibility About Usage, Data, and Anomalies

22

• Drill into usage details

• See “Who’s sharing content?” or

“Who’s uploading PCI?”

• Detect anomalies like excessive

logins or downloads


Granular Control for Data Protection and Compliance

23

• Enforce activity-level policies

like “no sharing outside of the

company”

• Prevent loss of sensitive data

with DLP policies like “no

downloading of PII to mobile”

• Coach users with automated

messages to build awareness

How Netskope uses Sumo Logic



Key Use Cases

25

• Track usage of Netskope Platform – gain insights into customer behavior

• Track application performance and throughput – measure for growth

• Track application errors – identify issues before customers do


Netskope Infrastructure

26

• 1000s of servers across the globe

• Each server has Sumo Logic agent deployed

• Automated bootstrapping of new servers using Ansible

• Use Sumo Logic partitions and field extraction rules to automatically

detect regions and data centers along with categories

• Take time to design your collectors – source, source category etc.


Track User Behavior

27

• Netskope instrumented its platform to emit key anonymous statistics

• Queries in Sumo Logic aggregate these stats

• Run daily reports to gain customer insights

• Use Sumo Logic built-in apps to create automatic dashboards – we use

Nginx and Apache


Sumo Logic Nginx Dashboard

28


Sumo Logic Nginx Visitor Access Dashboard

29


Understanding Application Performance and Throughput

30

• Netskope platform is built to analyze large amounts of data every day

• Tracking analysis pipeline is key to great customer experience

• Instrument code for measuring time taken

• Sumo Logic aggregates logs


Tracking Application Errors and Exceptions

31

• Instrument code to emit exceptions and errors in a standardized format

• Set-up real time alerts to send email notifications to operations team

• Track error volume for continuous improvement

Thank you


How Netskope Mastered DevOps with Sumo Logic

Technology

Transcript of How Netskope Mastered DevOps with Sumo Logic