Virtual SplunkLive! for Higher Education Overview/Customers

Copyright © 2014 Splunk Inc.

WELCOME

VIRTUAL SPLUNKLIVE! FOR HIGHER EDUCATION JANUARY 28, 2015

DAVE SCHWARTZ Director of Business Development, Splunk

(ALL TIMES EASTERN US TIME ZONE) 1:00 Welcome 1:10 Splunk Overview [Monzy Merza, Splunk] 1:45 Internet2 NET+ Splunk Offering [Andrew Kea_ng, I2] 2:00 Ohio State University [Mark Runals] 2:30 Baylor University [Jon Allen, Keith Schonenfield] 3:00 University of Washington [S. De Vight, P. Michaud] 3:30 Splunk Cloud [Nick Pavlovich, Splunk] 3:50 10 minute break 4:00 Breakout Sessions

Gecng Started Security IT Opera_ons

TODAY’S AGENDA

500+ Educa;onal Ins;tu;ons Buy Splunk

4

5

Safe Harbor Statement During the course of this presenta_on, we may make forward looking statements regarding future events or the expected performance of the company. We cau_on you that such statements reflect our current expecta_ons and es_mates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in this presenta_on are being made as of the _me and date of its live presenta_on. If reviewed ager its live presenta_on, this presenta_on may not contain current or accurate informa_on. We do not assume any obliga_on to update any forward looking statements we may make. In addi_on, any informa_on about our roadmap outlines our general product direc_on and is subject to change at any _me without no_ce. It is for informa_onal purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obliga_on either to develop the features or func_onality described or to include any such feature or func_onality in a future release.

Disrup;ve Approach to Unstructured Data

Structured RDBMS

SQL Search

Schema at Write Schema at Read

1980-‐2010 2010+

ETL Universal Indexing

Unstructured

Volume | Velocity | Variety

7

Make machine data accessible, usable and valuable to everyone.

7 7 7

COLLECT DATA FROM ANYWHERE

SEARCH AND ANALYZE EVERYTHING

GAIN REAL-‐TIME OPERATIONAL INTELLIGENCE

The Power of Splunk

8

9

Why Splunk?

FAST TIME-‐TO-‐VALUE

ONE PLATFORM, MULTIPLE USE CASES

VISIBILITY ACROSS STACK, NOT JUST SILOS

ASK ANY QUESTION OF DATA

ANY DATA, ANY SOURCE OR DEPLOYMENT MODEL

10

Turning Machine Data Into Business Value Index Untapped Data: Any Source, Type, Volume

Online Services Web

Services

Servers Security GPS

Loca_on

Storage Desktops

Networks

Packaged Applica_ons

Custom Applica_ons Messaging

Telecoms Online

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-‐ Premises

Private Cloud

Public Cloud

Ask Any Ques;on

Applica;on Delivery

Security, Compliance and Fraud

IT Opera;ons

Business Analy;cs

Industrial Data and the Internet of Things

Phases of Opera;onal Intelligence

Reac;ve

Search and

Inves_gate

Proac_ve Monitoring and Aler_ng

Opera_onal Visibility

Proac;ve Real-‐_me Business Insight

IT Opera_ons

Applica_on Delivery

Developer Plamorm (REST API, SDKs)

Business Analy_cs

Industrial Data and Internet of

Things

12

Delivers Value Across IT and the Business

Security, Compliance, and Fraud

Why Domino’s uses Splunk for Applica;on Management and Business Analy;cs

Understand device and app usage trends for

orders

Real-‐;me revenue

insights from store data

Visibility into online and

mobile coupon redemp;on

Refine campaigns for higher conversion

13

14

Apps & Capabili;es for Business Analy;cs

Apps, Features & Partners •  DB Connect •  Stream •  ODBC Driver •  Data Models •  Pivot

IT Opera_ons

Security, Compliance, and Fraud

Applica_on Delivery

Developer Plamorm (REST API, SDKs)

Business Analy_cs

Industrial Data and Internet of

Things

15

Delivers Value Across IT and the Business

Building Smarter Transporta;on

Improving Safety

Reducing Fuel Costs

Improving On-‐Time Opera_ons

Over $1 Billion in Poten;al Savings

16

17

Apps & Capabili;es for Industrial Data & Internet of Things

•  DBConnect

•  REST API and SNMP Modular Inputs

•  Universal Forwarder for Raspberry Pi

Apps, Features & Partners

REST

Splunk Products: What’s New?

18

19

What’s New in Splunk Enterprise 6.2

Gecng Data In Advanced Field Extractor

Instant Pivot Event Paqern Detec_on

Prebuilt Panels

Search Head Clustering Distributed

Management Console

Powerful Analy;cs for Broader Number of Users

Faster Data Onboarding

Breakthrough Scalability and

Centralized Mgmt.

Unparalleled Cloud Service for Machine Data

100% Up;me SLA

Hybrid Plaform

Secure and Reliable

Instant Access

20

21

What’s New in Hunk 6.2

Hunk Sandbox Data Explorer

Faster to Deploy and Gain Value

Instant Pivot Event Paqern Detec_on

Prebuilt Panels

More Powerful Analy;cs for Everyone

AWS Hunk Service Hunk Apps

Extend Exploratory Analy;cs

Extending Opera;onal Intelligence to Mobile Apps

Deliver Beqer Performing, More Reliable Apps

Deliver Real-‐Time Omni-‐Channel

Analy_cs

End-‐to-‐End Performance and Capacity Insights

22

New Data Sources

Universal Forwarder on z/Linux

Syncsort Ironstream on z/OS

Mainframe

Kepware

Industrial Data

23

Splunk App for Stream

Wire Data

Mainframe Data

VMware

Plamorm for Machine Data

Easy to Adopt Splunk

Exchange PCI Security

DB Connect Mobile Forwarders Syslog / TCP / Other

Sensors & Control Systems

Rich Ecosystem of Apps

Across Data Sources, Use Cases & Consump;on Models

Stream

24

Dev.splunk.com 40,000+ ques;ons and answers

600+ apps Local User Groups and

SplunkLive! events

25

Thriving Community

Educa;on

Healthcare

Technology

Energy and U;li;es

Manufacturing

Telecommunica;ons

Cloud and Online Services

Government

Retail

Financial Services and Insurance

Media

Travel and Leisure

26

Proven at 8,400+ Customers in 100 Countries Over 3/4 the Fortune 100

FREE ONLINE SANDBOX

FREE DOWNLOAD

FREE AMAZON MACHINE

IMAGES (AMI)

27

Easy to Try & Get Started

1 3 2

Thank you

ANDREW KEATING Program Manager, Internet2

ROB REED Worldwide Educa_on Evangelist, Splunk

On-‐premise, Splunk Enterprise is an Internet2 NET+ Offering

ALL US-‐based Higher Educa_on Ins_tu_ons benefit from: –  Pre-‐nego_ated contract –  Educa_on-‐only pricing (3 year term, payable

in annual installments)

30

31

More than 45 universi;es signed up… Smallest license…….20 gb Largest license………1 terabyte Average purchase…100 gb

32

Contact [email protected] •  How much Splunk do you need? •  How much can you get with the budget you have?

Thank you

01.28.2015 Splunk Live

Mark Runals

35

Ø  OSU Environment

Ø  General Thoughts

Ø  Recent Security Work

Agenda

36

About Me

IT Security in some fashion for 12+ years

At OSU for 2 ½ years

Using Splunk for 2 ½ years (direct correlation)

Other LM/SIEM Space •  Managed a medium size ArcSight deployment

•  Used Symantec’s MSSP

Splunk Apps: •  Data Curator, Forwarder Health, Change Tracker/Config Mgmt

37

Large Place 64k Students; 43k Staff; 175 Undergraduate Programs; ~200k IPs

Distributed 100+ IT groups; 30 CIOs; 7 Campuses; 1,245 Buildings; own zip code

Technology You name it we probably have it (somewhere)

OSU Environment

38

1.7 TB data per day 430B events in the system 10k+ Devices 12 types of firewalls Multiple OS 90+ teams with data in Splunk 700+ different types of data 350+ users

Splunk After 2+ Years

39

Lessons Learned

Don’t boil the ocean •  Have a data rollon / data definition process •  Start leveraging a Common Information Model (CIM)

Check out Splunk’s

There are different work streams •  Data Management – getting data in •  Knowledge Management – getting data out

Data Curator app •  Designed to help with previous point

40

Splunk – First Steps

1.  If you have firewall data make an interactive dashboard that helps teams identify blocks.

2.  Go out and buy a 30” or 40” TV and display something on it •  Splunk v6.x embedded reports •  Huge ROI

41

Don’t Display…

Top 5 Countries Attacking Us 1.  China 2.  US 3.  Romania 4.  Somewhere 5.  Somewhere Else

Top 5 Authentication Locations 1.  Columbus, OH 2.  Ohio (other) 3.  US 4.  etc 5.  etc

42

IDS – Last 24hrs

Use built in Splunk map if you must; doesn’t display numbers /sigh

43

Authentication – Last 24hrs

Eye candy = budget

44

Incident Life Cycle

Detection

Response

Collect Data

Content Creation

Alert

Typical MSSP Demarcation

Triage/ Tune

Log Forensics Investigation Remediate

45

Recent Security Work Leveraging Splunk

•  Investigating accounts sending spam •  Grade changes •  Library proxy abuse •  Detecting cheating on LMS

46

Accounts Sending Spam

1.  Alert sourcetype="MSExchange:2010:MessageTracking” original_client_ip=* | iplocation original_client_ip | eval Country = if(cidrmatch("128.146.0.0/16",original_client_ip) OR cidrmatch("140.254.0.0/16",original_client_ip) OR cidrmatch("164.107.0.0/16",original_client_ip), "OSU Address", Country) | stats sum(recipient_count) as recipient_count values(Country) as sending_countries by sender message_subject | where recipient_count > 15000 OR (like(sending_countries,"%Nigeria%") AND recipient_count>10) | sort -recipient_count

2.  Dashboard for investigation

Search is leveraging Splunk Exchange sourcetype definition. App v 2.1.2

47


48


49


sourcetype = snort [sourcetype = msexchange_data sender=$user$ original_client_ip=* | dedup original_client_ip | rename original_client_ip as src_ip | fields src_ip] | …

Pass the user name token (red) to the subsearch (blue) which pulls out the associated IPs and renames them according to the field snort uses

50

Grade Change

•  Investigation kickoff evidence – lockpick stuck in lock •  Many logs useful

•  Learning Management System •  Various authentication logs •  Wireless

51

Library Proxy Abuse

OSU pays for online resources

Student falls for phishing

Malicious site leverages account creds and library proxy

Notification by vendor that there was an issue •  Had user name – how can we identify malicious behavior?

52

Recent Security Work Leveraging Splunk

User Agent string looks interesting!

Often the malicious actors will setup a website that leverages the compromised creds. The number of source IPs will be very low.

53

Cheating on LMS Tests

Online test taking will only grow What can we use to spot anomalies?

Ø  Multiple tests from same IP Ø  Time elements from tests (ie time taken vs avg time)

54


55


[utma_cookie_extracts] REGEX = __utma=(?<utma_domain_hash>[^\.]+)\.(?<utma_systemid>[^\.]+)\.(?<utma_first_visit>[^\.]+)\.(?<utma_last_visit>[^\.]+)\.(?<utma_current_visit>[^\.]+)\.(?<utma_session>\d+) [utmb_cookie_extracts] REGEX = __utmb=(?<utmb_domain_hash>[^\.]+)\.(?<utmb_session>[^\.]+)\.(?<utmb_cookie>[^\.]+)\.(?<utmb_current_visit>\d+) [utmz_cookie_extracts] REGEX = __utmz=(?<utmz_domain_hash>[^\.]+)\.(?<utmz_current_visit>[^\.]+)\.(?<ut_mz_session>[^\.]+)\.(?<utmz_campaign>[^\.]+)\.utmcsr=(?<utmz_campaign_source>[^\|]+)\|utmccn=(?<utmz_campaign_name>[^\|]+)\|utmcmd=(?<utmz_campaign_medium>[^\|]+)\|utmctr=(?<utmz_campaign_terms>[^;]+); [ga_cookie_extracts] REGEX = _ga=(?<ga_version>GA\d+)\.(?<ga_cookiepath>\d+)\.(?<ga_systemid>\d+)\.(?<ga_current_visit>\d+)

Google Analytics Transforms

56

Summary

Going from a data repository to an engine takes time You have a data lake full of black swans

•  Use use cases to drive your efforts / start somewhere •  Don’t wait for perfect

57

Email: [email protected] Blog: runals.blogspot.com

Contact Info


Splunk@BaylorUniversity

Keith Schoenefeld Senior Information Security Analyst

Jon Allen Assistant Vice President &

Chief Information Security Officer

59

About Baylor •  Private faith based ins_tu_on •  Founded in 1845

•  16,260 students

•  Over 2,900 faculty/staff

60

Jon Allen

•  Over 15 years at Baylor University

•  Started the informa_on security group

•  M.S. Computer Science

61

Keith Schoenefeld

•  15 Years in Higher Educa_on Informa_on Security •  Vulnerability Management •  Log Management (ng-‐syslog, rsyslog, Splunk) •  Splunk Cer_fied Architect by the end of February.

62

Enhancing Security Infrastructure

•  PCI compliance •  Gaining vision into high volume log sources

–  Ac_ve Directory –  Firewalls –  IDS/IPS

•  Build a new service within IT that has security advantages

63

Ini_a_ve Buy In

•  Great security wants us to do what

•  Push the opera_onal benefits

•  Find one or two early wins

64

Cluster Master

Cluster Members Dedicated Search Head

Splunk Forwarders

.

.

.

65

Technical Specifica_ons

•  Dedicated Search Head (x1) –  48 cores –  64G RAM

•  Cluster Members (x3) –  Clustered for High Availability and Faster Searching –  Each has:

ê  3.3 TB local storage, configured in RAID 10 (~2000 iops) ê  10 TB SAN storage (~ 700 iops) ê  32 cores ê  64G RAM

66

Networking Group •  Firewall •  IPS •  IAS •  DHCP •  Networking Devices •  Windows Servers •  Linux Servers

Servers •  Ac_ve Directory •  Exchange •  Linux Servers

PCI •  Firewall •  IPS •  Ac_ve Directory

Client Services •  AV

Items in RED are logs we could not previously access effec_vely.

67

Proven Effec_veness

•  Servers ê  User Login troubleshoo_ng

–  Cuts troubleshoo_ng _me from 3 hours to 10 minutes each ê  Email flow troubleshoo_ng

–  Cuts troubleshoo_ng _me from 1 hour to 10 minutes each ê  Server Performance sta_s_cs

–  Exchange Volumes

68

Proven Effec_veness

•  Security –  Lost/Stolen Device tracking –  Event tracking –  Faster incident detec_on –  Anomalous user login detec_on

69

Robust Toolset

•  Raw logs to knowledge in minutes •  Use visuals to explain complex issues

•  Link disparate data sources

70

Shellshock Time Ac;on Device Source IP Dest IP Dest

Port Dest Net

Tue Oct 21 04:33:56 2014 ids bro 89.121.161.232 129.62.aa.bb 80 DC

Tue Oct 21 04:34:02 2014 reset-‐both PAN 89.121.161.232 129.62.aa.bb 80 DC

Tue Oct 21 04:40:05 2014 ids bro 188.10.85.113 129.62.cc.dd 80 Dept. A

Tue Oct 21 04:40:11 2014 reset-‐both PAN 188.10.85.113 129.62.cc.dd 80 Dept. A

Tue Oct 21 04:40:23 2014 ids bro 188.10.85.113 129.62.cc.ee 80 Dept. A

Tue Oct 21 04:40:28 2014 reset-‐both PAN 188.10.85.113 129.62.cc.ee 80 Dept. A

Tue Oct 21 04:40:30 2014 ids bro 188.10.85.113 129.62.cc.ff 80 Dept. A

Tue Oct 21 04:40:35 2014 reset-‐both PAN 188.10.85.113 129.62.cc.ff 80 Dept. A

71

DNS Amplifica_on Aqacks

75

Messaging Visual

76

Account Compromise

77

Building Apps

78

Lessons Learned

•  There is never enough license

•  Be prepared for rapid adop_on

•  Go big or go home on hardware


Ques_ons

Jon Allen Assistant Vice President & Chief Information Security

Officer

Keith Schoenefeld Senior Information Security Analyst

Thank You


Web Applica_on Monitoring and Analy_cs

University of Washington

82

Stephen De Vight Web Applica_on Engineer

83

Agenda "   About us "   Splunk at the University of Washington "   Suppor_ng an exis_ng service "   Providing data to UX with client-‐side instrumenta_on

84

Academic and Collabora_ve Applica_ons "   A division within UW-‐IT focused on building student facing Web applica_ons

"   Must develop new applica_ons while maintaining legacy applica_ons with limited resources

"   Facts and figures –  Small team of 6 engineers –  Maintain ~15 applica_ons –  Support over 140,000 users across 3 campuses –  Support 9 groups on campus running their own Splunk instances via our license

master

85

What We Maintain

86

My Background and Role "   Stephen De Vight

–  With the UW since 2006 –  Current Role: Web Applica_on Engineer, 2011 –  Mission: To support teaching and learning on campus through the development

of interac_ve Web and mobile applica_ons

87

Splunk Enterprise at UW -‐ 2012

aca-‐log

Universal Forwarders

88

Splunk Enterprise at UW -‐ 2014

splunk-‐search01 splunk-‐license

splunk-‐index01 splunk-‐index02

Universal Forwarders

‘External’ Splunk instances

89

Suppor_ng an Exis_ng Service

•  Homegrown suite of academic applica_ons

•  Currently consists of 8 dis_nct tools •  Released in 1999

90

Our Needs –  Situa;on: Legacy database logging system reached end of life, was not scaling

well, and was too costly to directly replace

–  Struggling with: Finding a solu_on that is both easy to build and maintain as well as being able to scale to our needs

–  Wanted: An easy to use, UI-‐driven, applica_on to search our log data

–  Enter Splunk: Splunk Enterprise allowed us to build a custom searching app as well as a dashboard for monitoring service status

91

Catalyst Log Search

•  Advanced XML view •  Search form negates the need for

users to learn Splunk search language or understand our log formacng and structure

•  Support can analyze user ac_vity to provide insight into incident reports

Screenshot here

92

Catalyst Dashboard

•  Gauge current level of ac_vity at a glance

•  Examine last day of ac_vity for anomalous usage

•  Targets slowest loading URLs for performance improvement

93

Data Driven User Experience

•  Mobile Web version of our student portal

•  Focused on providing _mely, ac_onable informa_on to our students

•  Based on a student's situa_on and the _me of the quarter we dynamically display, hide, move, and reorder content

94

Our Needs –  Situa;on: UX needs a way to validate their assump_ons around what content is

relevant to a student at various points in the quarter

–  Struggling with: Correla_ng user ac_vity with ins_tu_onal data (e.g. class standing, campus, etc.)

–  Wanted: A self-‐driven means for UX and business analysts to analyze log data

–  Enter Splunk: Splunk, along with our client-‐side logging solu_on, allows us to correlate user ac_vity with certain ins_tu_onal aqributes we log

95

Client-‐Side logging •  Google Analy_cs did not get us

everything we needed •  Using logger4javascript to collate

events and POST to a REST interface •  Events are bundled to reduce network

overhead •  Events are wriqen to file by REST

server

hlp://www.log4javascript.org/

96

Working with Client Logs

•  Link Log –  Link loca_on –  Target URL –  Ac_on (view, click)

•  Card Log –  Card loca_on URL –  Card name –  Card posi_on –  Ac_on (load, view, expand, collapse)

INFO 21 22:25:31

{

"level": "INFO", "url": "https://my.uw.edu/mobile/landing/",

"timestamp": 1421907930962,

"logger": "link",

"session_key": "xc63940325jlo3dsdfcgtt3126b",

"message": { "href": "http: //gmail.uw.edu/",

"action": "click"

}

} [link]

97

Simple Query

index=myuw_production sourcetype=myuw_link_log action=click |stats count by target_url

98

Server-‐Side Session Log

•  Session Log –  Graduate or undergraduate –  Class standing –  Campus

INFO 21 22:21:20

{

"is_grad": false, "netid": "javerage",

"is_ugrad": true,

"class_level": "FRESHMAN",

"session_key": "xc63940325jlo3dsdfcgtt3126b",

"campus": "seattle" }

[session]

99

Evenqypes and Transac_ons

index=myuw_production (sourcetype=myuw_link_log OR sourcetype=myuw_session_log)

Build an evenqype that contains both link and session logs

100

Session Ac_vity with Transac_ons

index=myuw_production eventtype=link_event |transaction fields=session_key maxspan=8h |search target_url=*dars.asp AND action=click |stats count by target_url

•  Create a transac_on based on session_key

•  Find transac_ons that contain a link click to ‘*dars.asp’

•  Get count of other URL targets clicked within that transac_on

101

Combining Logs with Transac_ons

index=myuw_production eventtype=link_event |transaction fields=session_key maxspan=8h |search action=click |stats count by class_level

•  Create a transac_on based on session_key

•  Find link events that have a click ac_on

•  Using the session log, determine how many link clicks were made by each class level

102

What’s Next "   Add more of our applica_on’s logs to Splunk

–  Deploying forwarders via Ansible to our hosts "   Get addi_onal people up to speed with querying in Splunk "   Reach out to addi_onal campus partners who want to buy into the license

103

Top Takeaways "   Building a search form makes Splunk simple to use "   Determine your analysis needs before crea_ng your logging scheme "   Client side logging can provide valuable insight into user behavior "   Transac_ons make combining logs easy

Thank You

SPLUNK CLOUD

NICK PAVLOVICH – AVP CLOUD SALES

KYLE HOURIHAN – CLOUD SPECIALIST

Apps and data moving to

cloud

Cloud data can remain in cloud

No data silos

Desire to consume

Splunk as a service

Cloud and Your Business

Search Head(s)

Indexer(s)

Search Head(s)

Indexer(s)

On Premises Private Cloud Public Cloud On Premises Private Cloud Public Cloud

Full Featured

Enterprise Ready Easy

What We Built

High availability across Indexers &

Search Heads

Multiple AWS availability zones

Dedicated Cloud environments

-  Secure -  10x Bursting

Splunk Cloud fully monitored using Splunk Enterprise

Built for 100% Uptime

Forward data Search Monitor

Get value fast

What You Do Hardware setup

Storage Scaling

Monitoring

What We Do

Hybrid Search

Search Head(s)

Indexer(s)

Search Head(s)

Indexer(s)

On Premises Private Cloud Public Cloud On Premises Private Cloud Public Cloud

Single Pane of Glass Visibility

Get Started From Home Page

112

Click here

Free Download or Online Sandbox

113

Commonwealth Bank Cloud Discussion 22/10/14

TECHNICAL DISCUSSION

Dedicated Deployments

Clustered Indexers & Search Heads

Mul_ple Data Centers

Proac_ve, con_nuous monitoring

Orchestra_on Layer

Mul_-‐region Opera_ons

Processes for data and customer protec_on

SSL Encryp_on

Splunk Cloud – Technical Overview

Opera;onal Excellence

Security Support

Enterprise grade support

Architecture

115

Architecture Diagram

Customer Stack

Opera;onal Monitoring

Users Searching via HTTPS

Forwarders over SSL

S3 backup

Orchestra;on Layer •  Chef •  Ansible •  Jenkins

Amazon VPC

116

Mul;-‐AZ, Clustered Search Heads

Mul;-‐AZ, Clustered Indexers

…

Master Nodes

…

Behind-‐firewall Forwarder

Management

Any Data Input Correlated with Exis;ng Data Sources

Scripted/Modular inputs TCP/UDP Local files Rest-‐API

117

UF or LWF or Heavy

THANK YOU!!

Virtual SplunkLive! for Higher Education Overview/Customers

Technology

Transcript of Virtual SplunkLive! for Higher Education Overview/Customers