Copyright © 2015 Splunk Inc.

Splunk atThe Vancouver Clinic

Davin Studer Systems Analyst

2

Agenda

About me and The Vancouver Clinic

How we started

Splunk deployment

Splunk use cases at The Vancouver Clinic

Best practices

3

Vancouver Clinic Overview

Serving the Southwest Washington since 1936

Locally owned and governed

Comprehensive and high quality of patient care

4

My Background and Role

IT team supports all clinic’s IT operations needs– Infrastructure monitoring and sizing– Root cause analysis

System analyst at The Vancouver Clinic– Integration of medical systems– Improvement of business processes

5

How We Got Started

Needed real-time solution for event logging and proactive monitoring across the entire IT infrastructure– Predicting failures and understanding performance of the systems– Before Splunk, slow and manual process of collecting event data from

multiple client machines– Centralized logging for PCI compliance

Started with Splunk two years ago for medical records privacy monitoring– Pioneered using Splunk for patient privacy monitoring in PNW– Huge interest from other clinics and hospitals in this use case

6

Splunk at The Vancouver Clinic Today

6

Splunk data types: Firewall logs, DNS lookups, application logs, Windows events and performance logs, MS SQL logs, Infrastructure syslog, SAN metrics, etc.

IT operations team is the main user of Splunk

Active users #: 15

Splunk Apps deployed: Windows Infrastructure App, DB Connect, Splunk on Splunk, Palo Alto, Citrix, Symantec

2 search heads

2 indexers

>1500 forwarders

7

Planning for Expansion

• Estimating capacity growth and proactive expansion plans

• Disk latency and IOPS monitoring– Identifying causation

• Disk Group Usage balancing

8

Securing the Network• Intrusion detection

• Outbound activity monitoring

• Switch hardware issues– Misconfiguration– Hardware failure

• Ensuring network link redundancy

9

Capacity Planning and Database Optimization

• Proactive capacity planning and estimating database growth

• Trending changes in load times

• Identifying anomalous load timings

• Visibility into how long SQL queries are taking

• Correlation of High CPU usage to poorly written SQL queries

10

Servers and Applications

• Monitoring VMWare Clients and Hosts– CPU– Memory– Disk Usage/Performance

• Exchange Performance

• EMR– BLOB storage

• Citrix PVS

11

AHA! Moment

Don’t limit yourself to just log monitoring Splunk can do much more!

12

Patient Privacy Monitoring• Splunk helps us comply with

patient privacy laws

• Highlights anomalous patient record access– Employee accessing medical records

without authorization– Prior to Splunk lack of visibility

• Other “turn-key” tools we evaluated were expensive and less flexible– Still required huge time investment

• Interest from other regional hospitals and clinics

Splunk’s Value for The Vancouver Clinic

• Splunk easier to use and more cost effective

• Splunk is flexible and we can modify reportsSaved over 50K

• We are able to catch problems proactively before they happen

• Increased confidence and satisfaction toward our IT teamProactive Monitoring

• Our data is centralized

• Less need for hunting in various locations for log data

• Ability to see trends/patterns in our logsFaster Support Response

14

Lessons Learned

Value of Splunk community– Users on answers.splunk.com are very helpful

Make your custom logs more Splunk friendly– Easier to index key/value pairs

Trust your Splunk data– Hard to break out of old habits of going to the source.– Much easier to correlate disparate data within Splunk.

Re-evaluate your Splunk data every once in a while

15

What’s Next

Extending Splunk deployment for proactive monitoring– Building more alerts and dashboards

Creating executive dashboard and reports

Look into the SDK’s and REST API