NHS Choices: Managing complex infrastructure to deliver critical online services

Presented by Andy Callow, Head of Technology Delivery

2

About HSCIC

3

The Health and Social Care Information Centre (HSCIC) was formed in April 2013 as an executive non-departmental public body

and the national provider of information, data and IT systems for patients, service

users, clinicians, commissioners, analysts and researchers in health and social care.

About HSCIC

4

• We have responsibility for a number of national systems, including Spine, NHSMail, Electronic Referrals and GPs Systems of Choice and NHS Choices to name just a few.

• April 2016 - Announced moving to new operating name NHS Digital

HSCIC - What We Do

5

We provide information, data and IT Systems for commissioners, analysts and clinicians in health and social care. Our work includes…

• Setting standards that protect patient’s confidential information, reduce bureaucracy and improve data quality

• Operating essential technology services that support the health and care system

• Collecting, analysing and publishing national data and statistical information that helps inform decision making

• Developing the next generation of national data and information systems

About NHS Choices

6

• UK’s biggest health website• More than 14,000 regularly

updated articles and hundreds of videos and

interactive tools• Compare health services in

England

6

About NHS Choices

7

About NHS Choices

8

Our Build

9

Our Original Challenge

10

Good coverage

48MVisits/ Month

Massive Logs

Nov 2010

• Tried out free version

June 2011

• 5GB/day license

Today• Multiple use

cases

Future• Wider

use

Our Splunk Journey

11

Live Performance Stats on Display

Live Performance Stats on Display

• Public Health England campaigns dashboard:

13

Live Performance Stats on Display

14

Live Performance Stats on Display

15

Live Performance Stats on Display

16

Internet

CDN

Web Application Firewall/Reverse Proxy

Cloud Provider

Search Provider (SaaS)

Legacy Network (IaaS)

Load Balancers

Windows Web

Servers

Linux Web Servers

Database Clusters

Cloud Web Services (PaaS)

Cloud Database Services (PaaS)

Splunk

Database Load Balancers

Indexers

Single Search Head

Searching

Splunk TopologyReporting Alerting Dashboards

NHS Choices

Data segmented by user

Our Data Sources

19

Online Services

Web Server Logs

CDN (Akamai)

Windows Event Logs

Mail Server Logs

Performance Counters from Windows & LINUX

RSS Logs

Searching Reporting Alerting Dashboards

Our Use Cases

Troubleshooting

Monitoring Impact of

Changes to the Website

Managing Unpredictable

Traffic

Reporting to Management

20

Troubleshooting

21

Troubleshooting: Root Cause Analysis

DDOS attack blocked by CDN

Reviewed data for previous year

Investigation = no previous attack

Answer the same day

Troubleshooting: Root Cause Analysis

Page rating averages dropping

Used Splunk to investigate

Identified set of IP addresses in Germany

Blocked IPs & introduced dashboard

Troubleshooting: Root Cause Analysis

24

Troubleshooting – using alerts

25

Monitoring Impact of Changes

26

Managing Unpredictable Traffic

27

Managing Unpredictable Traffic

28

Reporting to Management

29

MonitoringData

combined over period

GroupedResults

grouped into interest areas

Availability Actual vs Target

Future Plans

30

Analytics team: Monitor real time

searches

Partner team: Track usage of content

provided to syndication

partners

Product Owners: Build product

specific dashboards

DevOps team: ingest data from

other popular tools

• Make data available to the business

Value from Splunk

More informed decisions

Identifying business value in the data we already have

Alerts and reports free up

time

Correlate previously

unconnected events

Real-time and historical analysis

31

Thank you

32