CWP Meetup - Managing websites and emergencies

Managing Websites and Emergencies - Lunch & Learn

CWP Meetup – 20 Dec 2017

Here today from CASS IT & MCDEM

• Gavin HamiltonPrincipal Advisor, Web and Publishing team, CASS IT

• Leighton CorbanSystems & Security Architect, Architecture team, CASS IT

• Andrew Hammond-TookeSenior Communications Advisor, Digital Channels, Communications team, MCDEM

Covering• Civil Defence website on CWP • Its use in emergencies • Emergency Mobile Alert (EMA) project and campaign • Preparation for EMA and its impact on the Civil Defence

website • EMA test on 26 November • Lessons learned: website and EMA campaign • Application to your sites on CWP

CASS• Central Agencies Shared Services, unit of the Treasury • IT, Finance, HR functions for SSC, Treasury, DPMC • 26 websites • CMSs: Drupal, SilverStripe, Plone, static (bespoke),

Squarespace • Hosting: CASS itself, SilverStripe CWP (7 instances),

external, cloud

Ministry of Civil Defence and Emergency Management

• A unit of DPMC since 2014 − DPMC oversees national security system − "4Rs" of risk reduction, readiness, response and recovery − MCDEM is a lead agency (geological, meteorological, infrastructure

hazards) − Sarah Stuart-Black, Director

• 6 websites − Civil Defence, Happens, GetThru, Shakeout, What's the Plan Stan,

Takatu • Comms team of 7: Anthony Frith, Manager

− Public education; social media & web content − Public information management during emergencies; manage 6 duty

webmasters − BAU comms support for MCDEM & CDEM sector & Ministers

Civil Defence website hosting• Large instance on CWP • Active DR (disaster recovery, load balancing

Akld/Wgtn) • Premium WAF/CDN (Incapsula) • Additional site for training

Civil Defence website support• DNA: Application support, software development • SilverStripe: 24/7 support, code warranty, stress testing • CASS IT:

− Duty systems engineer/Duty IT helpdesk technician − Strategy/planning − Funding for 'keeping the lights on', planned work − Vendor relationships − Solution architects/security specialists − Content support

• MCDEM Comms − Comms strategy − Primary content managers − Emergency dashboard − Duty Webmaster roster and training

Civil Defence website in emergencies• Declared emergencies notified on home page − Yellow for emergencies − Blue for awareness messages

• All key information on home page • Reduced page size in KB (fewer images) • Sub-pages refer to home page • Dashboard for Duty Webmasters • RSS feeds from CDEM regions & manual curation

Civil Defence website in ‘peacetime’ > No declared emergencies

Emergency checker on

every page – updated every

minute

Banner images

displayed

Banner images

displayed

logo.png

Civil Defence website in emergencies > Declared emergencies notified on home page > Yellow for emergencies

Yellow emergency

content

Bannerimages

removed – reduce page

load times

logo.png

Civil Defence website in emergencies > Declared emergencies > Sub-pages > Emergency checker refers to home page

https://mcdem2-uat.cwp.govt.nz

Yellow emergency checker – refers to

home page for

emergency content

Civil Defence website in emergencies > Dashboard for managing regional and national emergencies and awareness messages

One active emergency –

Bay of Plenty region

Civil Defence website in emergencies > Dashboard > Review (manage) an emergency

Civil Defence website in emergencies > Dashboard > Review (manage) an awareness message

Emergency Mobile Alert (EMA)

• Alerts to all EMA-capable phones reachable from cell towers on 2degrees, Vodafone, Sparks networks

• Funding 2016/2020: $18.9m • System from Netherlands company: one2many.eu • “Serious hazards that involve threats to life, health or

property, or in some cases for test purposes" • MCDEM (coordinator) + Police, Fire and Emergency New

Zealand, MPI, MoH

EMA campaign• Current campaign: Mid-Oct (Get Ready Week) - Dec • TV, radio, motorway signs, social media • List of capable phones • FAQs/fact sheets (multiple languages) • Promotional resources: posters, videos, graphics/brand

guide • Short URLs/redirects using htaccess on CWP instance − civildefence.govt.nz/ema − civildefence.govt.nz/emergency-mobile-alert

EMA campaign

EMA challenges for Civil Defence website

• 2.5m smart phone handsets in NZ • Incomplete data from telcos eg cell tower time lag, 3G vs

4G, EMA-capable phones’ functionality & configuration

• How many will get the alert? • Will alerts be staggered? • Can/should we include a URL in the message? Will it be

clickable? • Will the platform handle the traffic generated? • Is our website design/code optimised?

Preparing website(s) for EMA• DNA/SilverStripe/CASS IT/MCDEM involved • Aug – Nov

− SilverStripe/DNA analysing performance of site code − SilverStripe stress testing (test site on UAT, Incapsula/Revera involved) − DNA implemented priority performance improvements − CASS IT benchmarking

• 4 Oct – Accidental test − Refined stress testing scenario (Android devices auto loading) − Needed ema.civildefence.govt.nz microsite

• 26 Nov – Real nationwide EMA test − Monitoring, analyzing results − Survey Monkey feedback, Colmar Brunton survey

Analysing website performance• Range of tools used

− www.webpagetest.org (free) − Chrome/Firefox developer tools (free) − Light stress testing also identifies issues

• Focus on − Optimising caching (look at caching directives) − Errors − Requests for HTML, CSS, JS, images, fonts

• How many • Size in bytes

• www.civildefence.govt.nz − Emergency mode (lighter)

− Peacetime (no declared emergencies) mode (heavier)

Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site summary


Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site waterfall view


logo.png

Google fonts, analytics

references

2.0 sec ‘Document complete’ page load

39 requests to fully load

page

Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site performance review

logo.png

Ratings

39 requests to fully load

page

Preparing website(s) for EMA > Analysing website performance > www.webpagetest.org > Current UAT site performance review

Feedback/suggestions

for optimisation

of images and caching

Preparing website(s) for EMA > Analysing website performance > Chrome developer tools > Check cache directives in headers

Headers for logo.png

Age in cache1 year =

31536000 secs

Served from Incapsula

Preparing website(s) for EMA > Analysing website performance > Chrome developer tools > Check site session cookies

Incapsula session cookies OK

Priority Civil Defence website performance improvements

• Goals − Maximise content delivery by Incapsula CDN (the cache) − Minimise ‘avoidable load’ ie requests to origin CWP server

• Implemented before EMA nationwide test − Redeveloped the ‘emergency checker’ − No 404s (error messages can’t be cached) − Redundant session cookies removed − No mixed http and https references in site code − Effective and targeted caching directives

• ‘Time to live in the cache’: home page & emergency content 30 secs, sub-pages 2 mins, unchanging images/assets 1 year

• Addressing in 2018 − References to Google fonts − Size of SVG images/maps

Two websites strategy for 26 Nov EMA testwww.civildefence.govt.nz • SilverStripe CMS • Hosted on CWP • Incapsula CDN • Stress tested by

SilverStripe • URL not in test alert

message • Traffic from social media

or direct • Permanent

ema.civildefence.govt.nz • Microsite, static HTML • Hosted by CASS • Cloudflare CDN • Stress tested by

Integration QA • URL in test alert message • Traffic from message

• Temporary only

Scenarios for website stress testing

• 1. Real-world possibility – 2 major national emergencies − Permanent site: www.civildefence.govt.nz − Major earthquake in Hikurangi trench/subduction zone triggers major tsunami

affecting East coast of NI and SI − 2M EMA-capable phones/handsets now, 3M in 2 years − Use psychology of Kaikoura earthquake behavior on site − Modelling of tests based on: requests to load page(s) * number of users *

psychology of user behavior (including duration) • 2. Real-world campaign activity • 3. Impact of the EMA test

− 350,000 users following a link to a Civil Defence website − Spikes with Android phones automatically loading link to the website

Civil Defence website stress testing

Scenario Requests

Hour User target count/hr First load Reload Total/hour Reqs/min Reqs/sec GB/sec

Emergency events – notified by national EMA message with link

1 1,500,000 3,000,000 195,000,000 198,000,000 3,300,000 55,000 83.2

2 2,500,000 5,000,000 325,000,000 330,000,000 5,500,000 91,667 138.7

3 100,000 200,000 13,000,000 13,200,000 220,000 3,667 5.5

Campaign, no emergency eg ‘Happens’, ‘Stop What You’re Doing’

1 200,000 5,200,000 4,800,000 10,000,000 166,667 2,778 2.9

2 100,000 2,400,000 2,400,000 4,800,000 80,000 1,333 1.4

3 50,000 1,200,000 1,200,000 2,400,000 40,000 667 0.7

Recent and ongoing stress testing of www.civildefence.govt.nz is based on these scenarios driving traffic to the site

Also tested ema.civildefence.govt.nz to these levels

Civil Defence website stress testing• Conclusions from testing by SilverStripe and

Integration in QA − High levels of confidence − Leighton’s view

• Challenges during testing − Rate limiting by Revera − Test accounts and their DDoS protection limits − Testing tool constraints eg servers to generate load − Access to Incapsula and Cloudflare technical staff

The EMA Test – what happened?• Sun 26 Nov – staggered test

during news hour 6pm-7pm − 98% of cell towers successfully

broadcast the alert − Colmar Brunton poll: 49% of

NZers received alert/were near someone who did

The EMA Test – what happened?• Both websites performed very well • ema.civildefence.govt.nz − Peak 25,000 concurrent visitors − 45,000 unique visitors 6pm-7pm

• www.civildefence.govt.nz − Was in (light) emergency mode (message about the test) − 92,000 unique visitors – whole day − Referrals from news sites, direct, Survey Monkey

• 25,000 Survey Monkey responses

EMA test – what happened? > ema.civildefence.govt.nz Cloudflare > 500K cached requests vs 0.5K uncached

EMA test – what happened? > www.civildefence.govt.nz > Peak 25,000 users shortly after 6pm; 92,000 users whole day

EMA test – what happened? > www.civildefence.govt.nz > 94% of requests/sec served from cache (Incapsula)

EMA test – what happened? > www.civildefence.govt.nz Incapsula > Test and day after

Grey line: Requests/sec passed to origin

server

Green line: Incapsula serving bulk from

cache

Dotted line: Total requests

peak 340 requests/sec We tested for

92,000 requests/sec

EMA test – what happened? > CWP host (WLG) serving www.civildefence.govt.nz

Spike in requests

handled by origin server at 6.15pm – peaked at

1.5K requests/min

Same data supplied by SilverStripe for AKL host

No problems with origin

server response

times

Lessons learned - websites• Focus on performance/caching profile of pages • Vigilance: changes in technology stack, impact of

software upgrades/development • Value of stress testing • Do regular spot performance tests (10 key pages) • Shared documentation/understanding between vendors/

IT/MCDEM

Lessons learned – EMA test & campaign

• Handset EMA capability and configuration highly unpredictable

• Largest ever mass broadcast alert in NZ − But telcos can’t provide data on who received it

• Accidental test on 4 October − MCDEM took responsibility for the mistake swiftly − Spent the entire day apologising to every complaint − Ended up with an additional 1000 Facebook followers

• Planned for public response to the actual test − MCDEM people responding on all social channels and phones − Survey Monkey feedback very useful, one-way was okay

What next?• More performance improvements • Switch from Incapsula to Cloudflare when available on

CWP • Cache control module • More testing! • ‘Better Responses’ ministerial review findings • Digital strategy for MCDEM sites • Microsites for campaigns

Questions• Gavin Hamilton

Principal Advisor, Web and Publishing team, CASS IT

• Leighton CorbanSystems & Security Architect, Architecture team, CASS IT

• Andrew Hammond-TookeSenior Communications Advisor, Digital Channels, Communications team, MCDEM

END - Some extra slides for reference follow

Accidental test EMA alert message - 4 Oct

Source: https://www.stuff.co.nz/national/97522387/early-morning-civil-defence-mobile-alerts-anger-sleepinterrupted-kiwis

CWP Meetup - Managing websites and emergencies

Technology

Transcript of CWP Meetup - Managing websites and emergencies