Draft Sector Black Sky Playbook V3.0 - WaterISAC · Draft/Sector Steering Committee Reviewed on dd...
Transcript of Draft Sector Black Sky Playbook V3.0 - WaterISAC · Draft/Sector Steering Committee Reviewed on dd...
0
WATERSECTORBLACK
SKYPLAYBOOKV3.0
Draft/SectorSteeringCommitteeReviewedonddmmm
2017.
ABSTRACTThisSectorBlackSkyPlaybookreflectscollectiveinput
fromnumerouspartners,aswellasoperationalindustry
technicalpersonnel,asarecommendedframeworkfor
planningresilienceinvestments,restorationplanningand
cross-sectorcoordinationneededforlongduration,multi-
regionpoweroutages.Thispeer-revieweddocumentis
designedasaresourcefortheelectricsubsector,
addressingcriticalBlackSkyresilienceneedsfor
Preparation/Mitigation,Response,Restorationand
Recoverymeasures.
JohnOrganek,PE
V3.0
1
Table of Contents RoleoftheEPROSectorBlackSkyPlaybook................................................................................................3
SectorBackground...................................................................................................................................3
SectorBlackSkyEnvironment..................................................................................................................4
SectorModelOverview................................................................................................................................5
SectorModelGraphic..............................................................................................................................6
SectorBlackSkyStrategicMissionStatement.............................................................................................7
SectorBlackSkyStrategicMissionPrioritiesMatrix................................................................................8
BlackSkyAssumptions...............................................................................................................................11
DHSPowerOutageIncidentAnnex:.....................................................................................................12
NationalInfrastructureAdvisoryCouncilWaterSectorResilience:....................................................12
BlackSkyDecisionsOverview....................................................................................................................12
BlackSkyDecisionsMatrix.....................................................................................................................12
SectorBlackSkySituationalAwarenessOverview.....................................................................................13
PriorityInformationRequirementsMatrix............................................................................................13
SectorInitialActions..................................................................................................................................14
SectorInitialActionsMatrix(V3)...........................................................................................................14
InternalSectorRequirements....................................................................................................................15
InternalSectorRequirementsMatrix.....................................................................................................22
ExternalandCrossSectorDependenciesOverview...................................................................................23
ExternalandCrossSectorRequirementsMatrix....................................................................................28
SectorSpecializedResourceRequirementsOverview...............................................................................29
SectorCommoditySpecificListMatrix...................................................................................................29
SectorBlackSkyCommunicationsOverview.............................................................................................30
SectorCommunicationsMatrix..............................................................................................................30
SectorBlackSkyAssessmentTool(s)Overview.........................................................................................31
SectorBlackSkyPlanningRequirements(On-going).................................................................................32
SectorBestPracticesMatrix(On-going)................................................................................................33
IntegratedandSharedPlanningActions(3.5/V4)......................................................................................34
2
PlanningandCoordinationActionsMatrix............................................................................................35
SectorBlackSkyResilienceConsiderationsOverview(3.5/V4).................................................................36
ResilienceInitiativesMatrix...................................................................................................................36
SectorBlackSkyRegulatoryImpactsandIssuesOverview(On-Going).....................................................37
SectorRegulatoryMatrix.......................................................................................................................37
SectorBlackSkyEssentialCriticalInfrastructure(MC)Overview(3.5/V4)................................................38
SectorCriticalInfrastructureMatrix(V3.5/V4)......................................................................................38
SectorBlackSkySpecializedSkillTrainingRequirementsOverview(V3.5/V4/V5)....................................39
SectorSpecializedSkillTrainingRequirementsMatrix(3.5/V4/V5)......................................................39
AnnexA–Assessments(On-going)............................................................................................................40
SectorOverallResilienceAssessment....................................................................................................40
AnnexB–RegulatoryIssuesDetailStatements(On-Going)........................................................................2
IssueStatement1:TestingEmergencyGenerators.................................................................................2
IssueStatement2:ReducedWaterQualityLevels..................................................................................2
AnnexC–CommunicationsRequirements(V3/V4).....................................................................................2
CommunicationsRequirement1:InternalCoordination.........................................................................2
CommunicationsRequirement2:ExternalSupportRequirements.........................................................2
CommunicationsRequirement3:StatusofServices...............................................................................3
CommunicationsRequirement4:DamagetoComponents.....................................................................3
CommunicationsRequirement5:RequestforPersonnel........................................................................3
CommunicationsRequirement5:RequestforResupplyofTreatmentChemicals..................................4
AnnexD:ResilienceRequirementsbyLayer................................................................................................5
3
Role of the EPRO Sector Black Sky Playbook TheapproachtobuildingthisPlaybookisdesignedtocontinuouslyengageWaterSectorutilitymembers
tocontributetoanevolvingframeworkofrecommendedguidelinestomanagerisksoflongduration,
multi-regionpoweroutagesassociatedwithemerging“BlackSky”hazards.
ThisPlaybookwillbecontinuallyupdatedandreviewedusingtheEPROSECTORsteeringcommittee
process,throughconsultationwithWaterSectorutilityprofessionalsandmanagers.Itrepresentsthe
consolidatedrecommendationsofthesesectormanagersandprofessionalstoaddresstheunique
challengesposedbywidearea,longdurationoutages.Itprovidesguidelinestohelpindividualentities
strengthentheirownresiliencemeasures,developfocusedoperationalplansandidentifyvitalexternal
supportneededintheeventoftheseseverehazardscenarios.
Sector Background Waterisalifelinesectorthatservesbusinessesandcommunitiesonadailybasisandbringsthemback
tonormalafteradisaster,whichmakesmaintainingwaterservicesandquicklyrestoringthemapriority.
Maintaininganddistributinganadequatesupplyofbothpotableandwaterusedforfirefighting,isvital
toensuringsuccessofashelter-in-placestrategy.
• Roughly85percentofallwaterandwastewatersystemsarepubliclyownedandoperatedby
municipalitiesandmostaresmall;morethan80percentofcommunitywatersystemsand
publiclyownedtreatmentworksservepopulationsoflessthan3,300.
• MostStateandmunicipaldecision-makersareconstrainedbylong-heldexpectationsby
customersthatwaterisalow-cost,affordableserviceandthatthecostpaidbycustomersdoes
notaccountfortruelife-cyclecosts.
• Nearlyallwaterinfrastructureassetsareoutofpublicsightandhistoricallyreliable,leadingto
anunderappreciationofthecriticalityofwaterservicesandtheinfrastructurethatdelivers
them.
• Likeothersectors,waterhasanaginginfrastructurethatrequiresmassivereinvestmentto
upgradepipes,mains,andequipment.Manyassetsarenearingorbeyondtheirexpected
lifespan,leadingtoroughly240,000watermainbreaks,andbetween23,000and75,000
sanitarysewageoverflowsperyearintheUnitedStates.Theestimatedinvestmentgapranges
fromabout$400billiontonearly$1trillion,justtomaintaincurrentlevelsofwaterservice.
• Communitywatersystemsarenotconnectedtoadjacentsystems,unlikeelectricityand
transportationinfrastructure,whichareinterconnectedintonationalnetworks.Thismeansthat
watersystemscannottakefulladvantageofnetworkeconomiesthatenablemutualassistance
anditmeansthatvirtuallyeachsystemmustbetreatedindividually,ona‘oneof’basis.
• UnliketheEnergyandTransportationSectors,whicheachhaveaFederaldepartmentand
Cabinetpositiondedicatedtotheirsectorsandinfrastructure,theWaterSectorhasno
correspondingFederaldepartmentdedicatedtoitssector.TheU.S.EnvironmentalProtection
Agency(EPA),whichservesastheSector-SpecificAgency(SSA)fortheWaterSector,regulates
andenforcestheCleanWaterActandtheSafeWaterDrinkingAct.Whileithasprograms
designedtoimprovethesecurityandresilienceoftheNation'sdrinkingwaterandwastewater
4
infrastructure,itsprimarymissionisensuringwaterqualityfromaregulatoryperspectiveand
notwithaperspectiveofoveralloperationandresilienceofthesystems.
Figure1:WaterSectorSnapshot(Source:EPA,2015WaterandWastewaterSystemsSector-SpecificPlan,(2015SSP),2016)
Sector Black Sky Environment WaterSectorutilitiesrequireaconsiderableamountofenergytoeffectend-to-endflow,processingand
distribution/collectionofwaterandwastewater.ThismakestheWaterSectorutilitiesparticularly
vulnerabletoBlackSkyhazardsandincreasestheneedtobuildinresiliencetothesehazards.Beingable
tomaintainsocietalintegrityfollowingaBlackSkyeventwillbevitallydependentonmaintaining
5
sufficientwatersupplyanddistributionsystemintegritytomeetrequirementsforpotable,firefighting
andotherusesofwater,suchasforbusinesses.Inordertoachievethegoalofshelter-in-place,the
WaterSectorutilitiesmustabletooperatesystemswhileoffoftheElectricGridandmeetminimal
sustainablelevelsofservice.Tosupportevenminimalservices,alargenumberofemergencypower
unitswillbeneedtobeacquiredand/orcapableofbeingrelocated,andtheiraccompanyingsupply
chainsforfuelandtechnicalsupportadjustedaccordingly.Neededchemicals,particularlyfor
disinfection,willbequicklydepletedandtheirresupplywillrequireamajorlogisticseffort,overa
deterioratedtransportationnetwork.TheSectormustquicklypivottoBlackSkyprotocolsthatwould
enablesustainingminimalservicelevelstothelargestnumberofpeople.
Figure2:WaterSectorImpactfromLossofElectricity(Source:NIACWaterSectorResilience:FinalReportand
Recommendations)
Sector Model Overview TheWaterSectorisdividedintotwomajorsetsofservices,WaterandWastewater.Waterservices
includesupportfor:drinking/cooking/bathing;firefighting;industrialuse;nationaldefensemission
assurance;andwastewatersystems.Wastewaterservicesincludesupportfortreatinghouseholdand
industrialwaste,aswellasforstormflow,inthecaseofacombinedsewersystem.
Waterissourced,treated,distributed(includingstorage),managedandreusedincompliancewith
standards,e.g.,qualityandpressure,toassuresocietalwellbeing.Wastewateroperationsinclude
sanitaryandstormwatercollectionandtreatment,aswellasthetreatmentanddisposalofthesludge
(biosolids)producedbytheremovalprocesses.
Figure3describestheend-to-endWaterSectorprocess,acrossbothwaterandwastewater.
6
Waterinfrastructurecomprisesthephysicalandcyberassetsofdrinkingwaterandwastewatersystems,
asdefinedbyHomelandSecurityPresidentialDirective7(HSPD-7),the2013NationalInfrastructureProtectionPlan(NIPP2013),andthe2015WaterandWastewaterSystemsSector-SpecificPlan(2015SSP).
Sector Model Graphic
Figure 3: End to End Overview of Water/Wastewater Process (reference: NIAC Water Sector Report)
TheotherLifelineSectorsalsodependheavilyontheWaterSectorutilitiesforbeingabletomaintain
theirsectoroperations,andsocietyasawholedependsontheeffectiveinteroperabilityamongallof
theseLifelineSectors.Figure4,fromtheNationalInfrastructureAdvisoryCouncilReportonWater,
depictsthedependencythatotherLifelineSectorshaveontheWaterSector.
7
Figure4:LifelineSectorInterdependencywithWaterSector(NIACWaterSectorResilience:FinalReportand
Recommendations,2016)
Sector Black Sky Strategic Mission Statement Intheseverelydisrupted,multi-region,long-durationpoweroutagesassociatedwithBlackSky
scenarios,themostcriticalgoalsforpublichealthandsafety,andsocietalcontinuitywillbesystematic,
timelyandwell-prioritizedpowergridrestoration,andmaintainingthelargestpossiblenumberof
peopleina“shelter-in-place”statusduringthatmulti-weekorlongerrestoration.
FortheWaterSectorutilities,thesecondofthesetwogoalsrepresentstheprimarymissionforthe
utility’sBlackSkyPlaybook:
Toachievethismission,minimalservicelevelsmustbedefined,especiallyforutilitiesservinglargecities
andcriticalfacilities,becausepopulationslivingindenseurbanenvironmentswillbetheleastableto
shelter-in-placeifwaterserviceisseverelydisrupted.The410verylargewatersystemsintheU.S.
(thoseserving100,000ormorecustomers)offeranespeciallyleveragedopportunitytomaximizethe
WaterSectorBlackSkyMission:Maintainingatleastminimalwaterandwastewaterservicefor
thelargestpossiblenumbersofresidentsandcriticalfacilities,inaffectedregions,forlong
durations(atleast30days).
8
impactofBlackSkypreparednessinitiatives.Ifthewaterutilitiesinmajorurbanareascansustain
reducedlevelsofservicetoenablemostoftheircustomerstoshelterinplace,thebenefitsgainedfrom
avertingunplannedmassmigrationswouldbeenormous.Focusingonthesmallnumberofverylarge
systemsalsoboundstheplanningandlogisticsproblemsthatWaterSectorutilitypartnerswillconfront.
Forgovernmentagenciesandprivatecontractorsresponsibleforsupplyinggenerators,fuel,and
chemicalsinBlackSkyevents,beingabletotargettheiroperationsonafewhundredofthelargest
systems(versusthemorethan40,000smallandverysmallsystems)wouldsignificantlynarrowtheir
challenge.Whileallwatersystemsandtheircustomersareimportant,thelimitedresourcesandthe
enormouschallengesofBlackSkyenvironmentsrequirethatresilienceeffortsfocusonthe(currently)
greatestvulnerabilitytounplannedmassmigration.
PreparingtoimplementtheseservicelevelsinBlackSkyscenariosrequires,inturn,developmentof
expandedresilienceandoperationalplans,aswellasspecificexternalsupportmeasuresfrom
governmentandmass-careNGOpartners.
TheseBlackSkyservicelevelsandtherequirementsthatcharacterizetheexpandedsectorplanningand
externalsupportneededtoprepareforthem,maybesummarizedinthreelevels—NearNormal,
Reduced,andNoService--andaredescribedindetailbelow.
Sector Black Sky Strategic Mission Priorities Matrix Meetingthesector’smissionrequirement–maintainingatleastminimalwaterandwastewaterservice
forthelargestpossiblenumberofresidentsandcriticalfacilities,forlongdurations–requirescareful
definitionofserviceprioritiesasafunctionofcustomertype,geographicregionandrelated
characteristics.DecisionsonBlackSkyservicelevels—madeincoordinationwithregulatorybodies--
alsovaryfromutilitytoutility.
Figure5:WatersystemservicelevelsinBlackSkyemergencies
Agivenutility,dependingontheoverallsystemserviceleveltheychoose,maydefinedifferent
servicesforcustomers,basedondifferentprioritycriteria.
Overallsystemservicelevelswillvaryasafunctionoftimeandmilestone.Forexample,
immediatelyafteraregionalblackoutbegins,waterutilitieswilltypicallybeginoperatingaccording
9
tonormal“GraySky1”playbooks,optimizedfornormaloperationduringshortdurationpower
outages.TofacilitateplanningforemergencydrinkingwatersupplyduringaGraySkyevent,EPA
andAWWAjointlydeveloped“PlanningforEmergencyDrinkingWaterSupply”in2011,foruseby
watersystemsofallsizes.Amongthemanyexternalrequirementsdefinedbelow,apriorityforthe
WaterSectorutilitieswillbetoreceive,asearlyaspossible,adefinitiveestimateoftheoutageclass
(i.e.,GraySky/BlackSky)toenablethemtoconvertovertoBlackSkyPlaybookprotocolsand
operationsattheearliestpossibleopportunity.
OncewaterutilitiesbecomeawarethattheoutagerepresentsalongdurationBlackSkyscenario,
eachutilitywouldthenbeginoperatingaccordingtoitsuniqueimplementationoftheBlackSky
Playbookrecommendations.AsshowninFigure5,thiswouldtypicallyinvolveshifting,notto
reducedwaterservicelevelsassociatedwithconventionalscenarios,buttolevelsassociatedwith
BlackSkyscenarios.Asreviewedbelow,thesewouldincludepreplannedchangestooperateat
minimumpressurelevels,alongwithapotentialforgeographically-variedservice,andreductionsin
waterquality.
a. System-widequalityreductions:Insteadofmaintainingdistributionsystempressuresandother
functionsnecessarytoprovidewaterthatmeetsnormaldrinkingwaterqualitystandards,
systemsshoulduselowerquality,still-potablewater,orprovidewaterthatcanbemade
drinkablebycustomers,givenadequatecommunitypreparation,suchasprovidinginstruction
about‘boiledwateradvisory’.Becausepowerandnaturalgaswillnotbegenerallyavailableto
customersinthesescenarioshowever,theprimarymeanswillincludeuseofbleach,iodineor
otherdisinfectants,ormanualfilters.
Asanessentialelementofthisproposedrequirement,recommendedemergencypurification
meanswillbecoordinatedwithgovernmentandNGOpartners,aswellasdirectlytocustomers,
toensureamaximumdegreeofadvancepreparationbythecommunity.EPAhasrecently
updatedguidanceonemergencydisinfection:(SeeEPAinstructionsonemergencydisinfection
ofdrinkingwater,whichshouldbedistributedtoallcustomers):
https://www.epa.gov/sites/production/files/2015-11/documents/epa816f15003.pdf
b. Geographically-variedservicelevels:Ratherthanseekingtomaintainasetservicelevelovera
utility’sentiredistributionsystem,utilitiesshoulddevelopplansthatallowforselectively
targetingareas,forexample,pressurezones,forsustainedservicebasedonpopulationdensity,
topography,criticalcustomers,orothercriteria.
Asastartingpoint,thiswouldinvolveassessmentofwhetheritismoreeffectivetooperatewith
reducedwaterqualityacrossthewholeserviceareaortoscalebackserviceinselectedareasif
pressurezoneisolationisfeasible.Utilitiesobtainingwaterfrommultiplewatersourcesmay
needtoexamineeachsourcetodeterminewhichneedstheleastamountofpowerand
chemicalstotreatanddeliver.
Ifgeographiccutbackswouldbeoptimumforaparticularutilityinthesescenarios,thesearea-
basedservicelevelreductionswouldbeidentifiedbyconductingadvancedplanningand
exercisesforthesescenarios.Theseoperationalplanswouldrequireadvancecoordinationwith
1Theterm“graysky”isusedhereasacounterpointtoBlackSkyevents.Itisnotintendedtobeadiminutiveterm.
Grayskyeventscanalsobeverysevere–suchasSuperstormSandyortheJapanesetsunamithatprecipitatedthe
Fukushimadisaster–buttheyaremorelocalized(sub-regional)andofshorterduration(lessthanonemonth)than
BlackSkyevents.
10
sectorpartnerssince,itmaybenecessarytomoveaportionofthepopulationwithinthe
utility’sservicearea.Inputsintotheplanningprocesswouldinclude:
1. Assessmentofwhetheronlycertainzonescanhavewaterserviceforthedurationofthe
event,orwhetherpressurezonescanbealternatedforseveralhoursatatime.
2. Assessmentoftheviabilityofusingemergencycontingencyequipment,suchasmobile
pumpingstations,toprovideintermittentservicetopressurezones.
c. System-widepressurereduction:Currently,20poundspersquareinch(PSI)ofpressureisa
typical(nearlycanonical)minimumstandardrequirementforpressureinwatersystems.Black
Skyplanningshouldconsiderthepossibilityofoperatingatevenlowerpressures—in
consultationwithregulators.Utilitiesshouldanalyzetheiroperationsanddeterminewhethera
lowerpressureforBlackSkyoperationscanbefeasiblyachievedandwhetheritwouldstillbe
safe(fordrinking)andfunctional(forfirefightingandsewage),andnotnegativelyimpactthe
physicalintegrityoftheirsystems.
d. Preplanningtoreducedemand:Meetingdemandwillbeeasierifdemandisitselfreduced.
Whilesomenaturaldemandreductionwilllikelyoccurduringemergencies,utilitiesshouldwork
withgovernmentandNGOpartnerstoprovidetheircommunitieswithadvanceinformation,
periodicmediapublicserviceannouncements,andothermeanstoencouragetheircustomers
toactivelyreducewateruseduringemergencies.2
Advanceplanningforreduced-usageagreementswithhigh-usecriticalfacilitiesthatwouldstill
beoperatinginthesescenariosisalsoapriority.Forlargewatercustomers,suchashospitals,
HVACsystemsarealargewaterconsumer.Pre-negotiatedplansforalternatewatersupplyfor
thesesystemsmaybeanoption.
e. Non-PotableWaterPlanning:Incaseswhereutilitiesareunabletoprovidereducedpressure,
minimal-puritypotablewater–orwaterthatconsumerscanmakepotable–maintainingthe
flowofwaterforfiresuppression,sewerageandothernon-drinkingpurposeswillbevitalfor
avoidingtheunplannedmigrationtippingpointinaBlackSkyevent.Utilitiesshouldensurethat
asafallbackplan,theyareabletosustainwaterflowssufficienttomeetnon-drinkingneeds.
f. Preplanningforcontrolledwaterandwastewatersystemshutdownandrestoration:Despite
implementingallofthesePlaybookoptions,someresidualriskof“completefailure”willremain.
Toprovideforaworstcasescenario,utilitiesshoulddevelopplanstoacceleratecomplete“dry
start”restorationofservicewhenthepowergridreturnstooperation.
U.S.waterutilitieshavelittleexperienceinrestoringwaterflowsafteracompleteservice
shutdown.Giventhedifficultiesofdoingso,especiallyintermsofreplacingpipesthatwill
breakoncewaterpressureislostortooquicklyrestored,itwillbeimperativeforsystemsto
avoidtotalshutdownbyseverelyreducing(andharshlyprioritizing)levelsofserviceinBlackSky
events.Nevertheless,utilitiesmaywanttohelpemergencymanagersandotherresilience
stakeholdersunderstandrestorationplans,priorities,andtimelines,andconsiderinfrastructure
investmentsthatmighthelpaccelerateservicerestoration.
g. Alternativehydro-systems:Ahydro-systemnetworkprovidesapotentialcapabilityfor
emergencymanagementagencies,inpartnershipwithwaterutilities,toprovideemergency
2ReductionindemandplanningandpublicoutreachisthesubjectofexistingguidanceissuedbyAWWAinthe
formof“M60:DroughtPreparedness&Response”,designedtoassistautilityindeterminingtriggerlevelsfora
demandreductionprogram.
11
potablewatersuppliestopopulationsthatcannotbeservedbythewaterdistributionsystem.
Analternativepotablewaterdistributionsystem,thesehydro-systems(equipmentand
operations)couldbemodeledafterIsrael’sconceptofoperationsforfurnishingpotablewater
duringwatercrises.
Althoughthiswouldnotprovideanadequatesolutionforlargecities,forsmallercommunities–
whicharemorelikelytofacesuchcompletewaterserviceshutdowns–thisapproachcould
provideavalidbackupoption.Thesealternativedistributionsystemswouldbedesigned,
configured,operatedandmaintainedtoprovidepotablewaterunder“NoWaterService”or
“FireandSanitationServicesOnly”conditions.Note:otheremergencywaterdistributiontactics,
suchasbottledorbulkwater,areaddressedunderSection3.,ExternalRequirements.
h. KeyResources:WARNandEmergencyWaterSupplyGuidance
1. TheAmericanWaterWorksAssociationhasdevelopedtheWater/WastewaterAgency
ResponseNetwork(WARN),amutualassistancenetworkfortheWaterSectorutilities
toprepareforandrespondtodisastersandotheremergencies.3
2. TheEnvironmentalProtectionAgencyandAWWAhavedevelopedguidancetofacilitate
planningandconsiderationofneedsunderemergencyconditions.4
EPROHandbookII,Chapter2(Water)andthisPlaybookprovideasolidfoundationforemergency
planningtoaddresstopicsa.throughh.aboveandcanbeexpandedupontoaddressBlackSky
scenarios.Missionprioritiesbyphasearelistedbelow.
Black Sky Assumptions BecauseoflimitedexperiencewithBlackSkyenvironments,planningintheWaterSectormust
necessarilybebasedonassumptions.Theprincipalassumptionisoneofanelectricgridoutageacrossa
verylargefootprintandthatlastsoveramonthormore.Thoughwegenerallyconsideralargefootprint
torepresentoneoftheinterconnections,itisconceivable,inacyberattackforexample,thatmultiple
3http://www.awwa.org/NationalWARN.org
4 Planning for an Emergency Drinking Water Supply http://www.awwa.org/Portals/0/files/resources/water%20knowledge/rc%20emergency%20prep/Emergencywater.PDF
Phase Priority Missions
Mitigation 1 Conductcross-sectorplanning
Mitigation 2 Participateincross-sectorBlackSkyexercises
Response 2 Reducequality,systemwide
Response 3 Varyservicelevelsgeographically
Response 4 Reducepressuresystem-wide
Response 1 Reducedemand
Response 5 Conduct“BoilWater”operations
Recovery `1 CoordinatewithElectricSubsector
Recovery 2 Restartlong-startupfacilities,suchassludgedigestors
Recovery 2 ExecutecoordinatedrestartwithElectricSubsector
12
interconnectionsmightbeaffected,ifnotdirectly,thenasaresultofbeingrestrainedfromproviding
resourcestootherinterconnections.
Tosharpentherelevanceoftheseassumptions,thefollowingassumptionsaretakenfromtwosources:
DHS Power Outage Incident Annex: • CauseAgnostic
• MultipleStates/Regions;millionsofpeoplewithoutpowerforlongperiod
• Largeportionofpopulationrequiressustainedmasscaresupport
• Lossofcriticallifelinefunctionsresultsinriskstohealth,personalsafety,nationalsecurity,and
economicviability
• ImpactstootherCriticalInfrastructuresectorscausessignificantlossofservicesorfunctionsand
createsneedforsustainedoperationalcoordination
National Infrastructure Advisory Council Water Sector Resi l ience: • BlackSkyeventsarehigh-impact/uncertainprobability
• UtilitieshavebuiltsomeresilienceforBlackSkyevents,butadditionalpreparednessisneeded
• PowerfailureisthelargestriskdependencytotheWaterSector
• Availablegenerators:
o Maynotbelargeenoughtoaddresspowerrequirements
o Requireextensivequantitiesoffuelandcontinuousmaintenancetosustainoperations
Black Sky Decisions Overview Waterandwastewatermanagersandoperatorswillberequiredtomakenumerouscriticaldecisions
followingaBlackSkyevent,mostlikelyunderaseverelylimitedcommunicationsenvironment.Detailed
planningandexerciseofinternaloperations,aswellascoordinationwithexternalpartners,would
mitigatetherisksassociatedwithbeingunabletoobtaincriticalinformationfollowingaBlackSkyevent.
PreparationsshouldfocusonreducingthetimerequiredtoeffectdecisionsinthePlan-Do-Check-Act
cycle.OnemethodistodevelopStandardOperatingProcedures,basedontheBlackSkyConceptof
Operations,toidentifydecisionsandtoderiveandminimizecriticalinformationrequirements.In
addition,utilitiescouldusehydraulicandcross-sectormodeling,insupportofBlackSkyplanning,to
identifyvariantscenariosanddevelopcontingencyplanswhichcouldthenbeexecutedcentrallywith
minimalinformation,orautonomously,bottomup,attheoperatorlevel.
Black Sky Decisions Matrix DecisionsaffectingWaterSectorutilitieswilltakeplaceat3basiclevels,ifaBlackSkyeventoccurs:
acrosssystems(Regional/State/Federal);atsystem;andbelowsystem.Furthermore,whilehierarchical
decisionmakinghasbeenproventobemostefficientapproachduringemergencies,thedecisions
acrosstheselevelswillbeaffectedbythedegreetowhichinformation,authority,expertiseand
capability-to-executeexistatanymomentintime.Asarule,decisionsduringcrisesshouldbemadeat
thelowestlevel,whileeachincreasinglevelshouldbefocusedoncoordinatingandfacilitatingthe
actionstakingplacebelowtheirlevel.Such‘federated’approachtodecisionmakingshouldbewell
coordinated,plannedandexercised.
Intheeventofamajorlossofelectricalpower,awaterutilitymustfirstdeterminewhetheraBlackSky
eventhastakenplaceandwhetherthesystemoperatormustinitiateResponseundertheirpre-
13
determined,pre-plannedBlackSkyProtocols.Ifcommunicationshavefailed,thisdecisionmayrevert
downtoindividualsubcomponentsofthesystem.
Sector Black Sky Situational Awareness Overview FollowingaBlackSkyevent,WaterSectorutilitiesmustdeterminethefollowingandincorporatethe
informationintheirEmergencyResponsePlan(ERP):
• WHAT:servicesmustIprovideandatwhatservicelevels
• WHERE:doIprovideand/orshutservicesdown,bothexternallytothepublicandinternally,to
thecomponentsofthesystem
• WHO:doIprovideservicestoandobtainservicesfrom
• WHEN:doIinitiateBlackSkyProtocols
• HOW:doIsustainminimalistservicelevels
Priority Information Requirements Matrix
Phase Priority Decision
Response 1 WhethertoinitiateBlackSkyServiceLevels
Response 2 Whatchangesaretobemadetosystemoperationstosustainservicesas
longaspossible
Mitigation,
Response
3 Wheretoemployemergencypower
Response 4 Whentorecallandwheretopositioncriticalpersonnel
Information Source Priority ConfidenceLevel
StatusofSystem SCADAandon-site
visits
1 LOW,initially,untilall
componentshavebeen
inspected
HIGH,subsequently
Consequences,e.g.,numbers
andareasaffectedbysystem
Impactanalyses
conductedby
individualsystems
andtheir
components
2 LOW,initially,untilall
componentshavebeen
inspected
HIGH,subsequently
StatusofSystemsacross
Region
IndividualSystems 3 LOW,initially,untilall
componentshavebeen
inspected
HIGH,subsequently
Consequencesofoutage
acrossRegion
WARN,with
informationfrom
individualsystems
4 LOW,initially,untilall
componentshavebeen
inspected
HIGH,subsequently
StatusofSystemsacross
OutageArea
WARN,with
informationfrom
individualsystems
5 LOW,initially,untilall
componentshavebeen
inspected
14
Sector Initial Actions WaterSectorutilityactionsinresponsetoaBlackSkyeventwillvaryovertime,followingtheevent.
Theywillalsovaryoverindividualsystems,whichdiffermarkedlyinhowtheysource,treatand
distributewaterandwastewater.Generally,utilitieswillactivatetheirEmergencyResponsePlans,
beginningwithstartingemergencypowerunitsandcontinuetoprovidewater/wastewaterservices
accordingtopre-plannedpriorities.Whilemaintainingnormalservicesundercurtailedpowersupport,
systemandsub-systemoperatorswillconductasituationalassessment,sothattheycandetermine
whereandtowhatdegreetoreduceservicelevels.Effectivepre-planning,underavarietyofscenarios,
wouldgreatlyimprovetheoutcomeoftheseinitialactions.
Sector Initial Actions Matrix (V3)
HIGH,subsequently
Consequencesofoutage
acrossOutageArea
WARN,with
informationfrom
individualsystems
6 LOW,initially,untilall
componentshavebeen
inspected
HIGH,subsequently
Priority InitialAction Desired/RequiredOutcome
1 AssessCapabilitytoMaintainCriticalServiceLevelsfor
systemandsub-systems
Accuratesituationawareness
andestimateoftimeto
servicerestoration(ETS)
2 ContactElectricUtilitytodetermineDate/Timeof
Restoration
Situationawarenessand
determinationofwhetherto
gotoBlackSkyprotocol
3 AssessConsequencestoPopulation/AreaServed Maintain,orrestorein
minimaltime,serviceto
maximumnumberofpeople
4 SwitchtoBlackSkyProtocolsforOperations Maintainsustainablelevelof
services
5 SwitchtoEmergencyPower Maintainservicesatpre-
designatedlevels
6 RecallKeyPersonnel ProvideResponseand
Recoverycapability
7 EstablishSectorLiaisontoEmergencyOperationsCenters Sharesituationawareness
acrossLifelineSectors
8 CommunicateRecoveryStatustoEOC Sharesituationawareness
acrossLifelineSectorsand
provideinformationusedto
coordinaterecoveryactions
15
Internal Sector Requirements InBlackSkyoutages,eventhebest-preparedwaterandwastewatersystemswillbeunabletooperate
alloftheirwaterpressurezonesattheirtypicalqualitystandards,andtoprovidenormalwastewater
flowandtreatmentlevels,foraperiodofweekswithoutexternalpower.BlackSkyrequirementswill
typicallyincludehardwareresilienceinvestmentsandcomprehensiveoperationalplanningtoenable
thesesystemstooperateindegradedpressureandtreatmentmodes,forlongdurations,with
infrequentanduncertaindeliveriesofreplacementdieselfuelandtreatmentchemicals.
A. WATERANDWASTEWATERSYSTEM:RECOMMENDEDBLACKSKYHARDWAREANDFACILITYRESILIENCE
REQUIREMENTS
1. EmergencyPower
Waterandwastewaterutilitiesshouldassumethattheywillneedtheirownemergencypower
suppliesinordertofunctionduringaBlackSkyevent,andthatevenmultiplepowerfeedswill
proveinadequatetohelpthemsustainservice.
DuringemergencypowerplanningforBlackSkywaterandwastewatersystemoperations,
plannersshouldemployhydraulicmodelingtoolsandavarietyofscenariostodeterminethe
mosteffectiveallocationofemergencypowerunits,andthendevelopoperationsplansto
execute,accordingtotheemergencypowerunitallocation.
Emergencypowerconsiderationsincludeacquisitionofsufficientinventoriesofemergency
generatorstosupportwatertreatmentandpumpingrequirementsforadesignatedportionof
theareaserved,andincluderequirementsforsparegenerators,toreplacethosethatbecome
inoperable.Whengeneratorsarenewlyacquired,procurementspecificationsshouldcallfor:
long-durationoperation,EMPandCyberprotection,provisionsformobility,andcapabilitiesto
enableeithercontinuous,shortdurationorperiodicoperation.
a. Planforadequate,onsiteemergencygeneration
“Justintime”strategiesforacquiringemergencypowerwillneedthoroughrevampingand
reassessment.WaterSectorutilitiesshouldbuildstrategiestoaddresstheenormous
mismatchthatwillemergeinBlackSkyeventsbetweenthenumberofwaterand
wastewatersystemsrequestingjust-intimeinstallationofemergencygenerators,available
generatorsuppliesandinstallationtechniciansfamiliarwiththeinstallationrequirements
specifictothatsite.
Utilitiesshouldplantoacquire,ataminimum,generatorsforwatersupplyandminimal
treatmentrequirements,forliftstations,andtoensureatleastonegeneratorisavailable
forpumpingtoeachpressurezonetomaintainthetargetservicelevel,asdeterminedby
theutility.Asstatedabove,pre-planningandfundingrestraintsmightdisclosetheneedto
shutdownoneormorepressurezones,andtoplanformigrationofthepopulationto
anothersupportablelocation.
b. Preparationsandinformationneededtosupport(limited)emergencymanagement
agencygeneratoracquisition
Whilemajorwaterutilities–particularlythoseservinglargecities,shouldworkwith
regulatorybodiestoensureadequateemergencygeneratorsareinstalledandavailable
onsite,manysmallerutilitiesmaybeunabletoacquiresufficientgenerators.Ifsuchutilities
16
anticipaterequestingreal-timedeliveryandinstallationofemergencygeneratorsduringa
BlackSkyoutage,prearrangedconnectionsfortheappropriatefacilitieswillbeessential.
RegistrationofneedsintheUSArmyCorpsofEngineers’EmergencyPowerFacility
AssessmentTool(EPFAT)ishighlyrecommended.
UntilbetterdataexistsonWaterSectorutilities’emergencypowerplansandcapabilities,it
willbeimpossibleforemergencymanagementagenciestodevelopstrategiestomitigate
shortagesingeneratorsuppliesandinstallationtechnicians.WaterSectorutilities,
associationsandtheirpublicsectorpartners(includingtheEPA)shouldconductsurveysto
determinetheexistingversusrequiredemergencygeneratorinventoryofU.S.waterand
wastewatersystemsintheUnitedStates.Informationwillberequiredontheavailableand
requiredgeneratorinventory–numbersandsizesofgenerators–infourcategories:
1. Utilitiesthathaveadequateon-siteemergencypowergeneratorsandfuelstorage
tanks,orplantoacquirethem,includinghavinggeneratorinventoryinformation
(numbersandsizes)onhand.
2. Utilitieswithliftsorotherfacilitiesrequiringemergencygeneration,preconfigured
forgeneratorconnection.
3. Utilitieswithneithergeneratorsnorpreconfiguredconnectionpoints.
4. Utilitieswithamixofpumps,liftsandfacilitiesrequiringgenerators–somewith
generators,somepreconfigured,somewithneither.
Anassessmentshouldbemadeofthisdatatoidentifypotentialshortfalls,bylocation,and
toidentifypotentialpointsofcompetitionforresources,especiallysinceaBlackSkyevent
willgeneratewidespreaddemandacrossalllifelinesectors,andsuppliersgenerallydonot
stockenoughassetstocoverpeakdemand.
Inaddition,acomprehensiveassessmentshouldbemadeacrossallregulatorybodyrulesto
determinehowoperationsplanswillbeimpacted,preparingforandoperatingunderaBlack
Skyenvironment.Primeareasinclude:full-loadtesting,transportationrestrictions,access
restrictions.
Utilitieswithhydraulicmodelingcapabilitiescanusethesetoolstomodel‘whatif’scenarios
thatvarybythenumberandlocationofemergencypowergeneratorsacrosstheirsystem.
Runninganumberofscenarioswithonlypartialemergencypowercoveragewill:enable
optimalpre-placementoflimitedresources;familiarizepersonnelwithpotential
vulnerabilitiesforresilienceinvestment;andpromoteadaptabilitybycreatingavarietyof
operationalcontingenciesthatcouldbeusedtocutdowndecisiontimeduringresponseand
recovery.
c. Generatorrequirementsshouldtakeintoaccountsparingrequirements
Mostbackuppowerunitsarenotdesignedtoruncontinuously,underload.Utilitiesshould
developdynamic(versusstatic)assessmentsoftheirgeneratorrequirements,andaccount
forlikelyreplacementrequirementsovertime.Watersystemsthatexperiencedgenerator
failuresduringSandy,forexample,haveincreasedtheirstockpilesofreplacement
generatorssincethatevent.Otherutilitiesshouldconsideradoptingasimilarapproach.
17
Whencontemplatingtheacquisitionofspares,utilitiesshouldevaluatethealternativeof
procuringamoreexpensivegeneratorcapableofcontinuousoperation.
d. Procurementguidelinesforemergencygenerators
Owners/Operatorsofsomefacilitieswithoutemergencybackupgeneratorsmaynotknow
exactlywhattheiremergencybackuppowergeneratorrequirementsare.Key
determinationsinclude:5
• Generatorsizerequired(inkWorMW)
• Doesthegeneratorneedtoprovidefullload,orpartialload?Ifpartial,itis
importanttolockoutotherbreakerssoasnottooverloadthegenerator.
• GeneratorOutputVoltageRequired(e.g.,480,120/208,4160)
• GeneratorPhaseOutput(singleorthree-phase)
• FacilityElectricalConnectionPoint:Isthereanexistingmanualtransferswitch?Will
thegeneratorconnecttothemainelectricalpanel,oranotherconnectionpoint?
Cantheconnectionpointbepre-installed?
• BasicOrderofMaterials(BOM)requiredtophysicallyconnectthegenerator:
Cable/Conductor(size,length,numberofruns);ConnectionLugs;GroundingPoint
(existingorneedstobeinstalled?);GroundingCablesizeandlength
• CRITICAL:Facility’sCommercialPowerGridIsolationPointLocation(importantfor
safetyofanyelectricallinemenoffsite,andalsotoensurebackuppoweris
dedicatedtothefacility)
• BlackSky-compatibleenhancementsthatneedtoberequestedwhenprocuring
units:Designedforlongduration,continuousoperation,withexpandedfueltank,
anddesignedwithenhancedEMPandcyberprotection.
UtilitiesshouldassessandregistertheiremergencypowerrequirementsbyusingtheCorps
ofEngineers’EmergencyPowerFacilityAssessmentTool:http://epfat.swf.usace.army.mil/
e. Specialoptionsforlowpowerequipment
Fortelemetrysystemsandotherlower-powerconsumptioncomponents,someutilities
havebegunusingsolarpowerandbatterybackupsystems.TheuseofsensorsinWater
SectorutilitiesisexpandingveryrapidlyandEISCouncilisresearching
sensor/communicationscapabilitiesthataffordBlackSky-securedsituationawareness,atan
affordablecost.
f. Planningforemergencypowertoensurewastewaterthroughput
Waterandwastewaterutilities--whetherseparateorcombined--shouldcoordinate
regardingbothrequiredflowsandthenecessaryemergencypowersupport,toensurethat
minimalwastewaterflowsaremaintainedforsewagedisposal.
g. Power-conservativewastewatertreatmentoptions
Whereutilitiesuseaeratedsludgeoperationsforwastewatertreatment,thisapproach
typicallyconsumesmorethan75%ofwastewaterutilitypowerrequirements.Inaddition,
5InformationprovidedbytheU.S.ArmyCorpsofEngineers.
18
withoutpowerforaeration,withina48-72hourwindow,microbesforsewagedigestionwill
die,andre-establishmenttimesareaweekorlonger.Workingwiththeirrateregulators,
utilitiesshouldevaluateandpotentiallyimplementalternative,lesspower-intensive
wastewatertreatmentmethods,ifavailable.
h. Minimalsetofnecessarychemicals
Significantopportunitiesexistforwaterandwastewaterutilitiestodownsizetheirchemical
storageandresupplyrequirementsforBlackSkyoperations
Watersystemsmaintainsuppliesofchemicalsandother“consumable”suppliesforwater
treatmentthatareprimarilyusedtomeetregulatoryrequirementsfordisinfectantresidual
infinishedwaterandtoachievegoalsforcolor,odor,taste,andothercharacteristicsthat
aremostlyaesthetic(andnotenforcedbyregulation),ratherthannecessaryforpreventing
seriousillness.AchievingtheseaestheticgoalswillbecomelessurgentinBlackSkyoutages.
Instead,sustainingadequatequantitiesofwaterthatmeetbareminimum,health-related
requirements–orthatcanbetreatedbycustomerstoachievethatlevelofquality–willbe
ofparamountimportanceinsuchevents.
AswaterutilitiesdetermineBlackSkyservicelevelswithreducedqualityofwaterthey
deliver,theyshouldstringentlyprioritizethechemicalsthattheyneedtostoreandresupply.
Suchprioritizationwillneedtobedevelopedinagreementwithappropriateregulatory
entitiestoprovidevariancesfromchronichealthstandardswhileleavingthenecessary
acutehealthstandardsinplace.6SeeSection3.B.below.
i. Microgrids:Operatingwithinamicrogrid–asmallsectionoftheelectricgridthatcanbe
electricallyisolatedfromthelargerpowergrid–isanadvancedtechniquebeingpursuedby
ahandfulofwaterutilities,spurredbytherapidandextensiveprogressbeingmadewith
useofrenewables.
• Majorwastewatersystemshaveespeciallypromisingopportunitiestoestablish
microgridsforBlackSkyoperations.Anaerobicsludgedigestioncangenerate–via
capturedmethaneemissions–enoughpowertooperateawastewaterfacility,and
evengivebackexcesspower.However,fortheutilitytooperateasafunctional
microgrid,therequisiteelectricalinfrastructuremustalsobeinstalled.As
wastewaterutilitiesconsiderthedevelopmentofCombinedHeatandPower(CHP)
systems,utilityleaders,electedofficials,andotherresiliencestakeholdersshould
carefullyconsideropportunitiestoprovideformicrogridoperations,andbalance
theincrementalcostsofdoingfullCHPagainstthepotentialbenefitsforsustaining
wastewaterserviceinlongdurationblackouts.
• Overthelongerterm,microgrids(orsecurepowerenclaves)canprovideaholistic
meansofstrengtheningwaterandcross-sectorresilience.Whileelectricutilities
willneedtomakemanyoftheinvestmentsnecessarytocreatemicrogrids,water
andwastewatersystemsshouldhelpleadcross-sectordiscussionontheirdesignto
ensurethatminimalistwaterandwastewaterservicegoalscanbesustainedinBlack
Skyevents.
6SuchwasthecaseduringtheaftermathofHurricaneKatrina,wherethefocuswasonsafeguardingagainstacutemicrobialriskwhileobtainingavariancefromchronic70-yearlifetimeexposurestandardstoarsenicinwater.
19
2. EmergencyFuelTankage
Increasedonsiteemergencyfueltankage,orprovisionsforonsiteornearbystorageand
managementoftankertrucks,isneeded.Thisisafundamentalrequirementtoallowoperations
tocontinuedespiteinfrequentanduncertaindeliveriesofdieselfuel.Waterutilitiesshouldwork
withemergencymanagementagenciesandotherwaterutilities,todevelopregionalplans
(coordinatedwithfuelstorageanddeliverycompaniesforBlackSkyconditions)toensurethe
deliveryfrequenciesandvolumestheyanticipatewillberequiredinaBlackSkyscenarioaremet.
Asadefaultplan,wheresuchplansarenotyetavailable,sufficientfuelfor10daysofoperation
isrecommended(i.e.,approximatelyone9000gal.tankertruckcapacityforfullloadoperationof
a0.5megawatt(MW)generatorfor10days).UnderBlackSkyconditions,backuppower
generatorswillberunningunderfullload,whichwilldramaticallyincreasefuelrequirementsand
outstriptypicallevelsofon-sitefuelstorage.Note:Regulatoryrequirementswillinmostcases,
needtobeaddressedtoallowforadditionalfueltankage.Manyof thewatersectorutilities’BlackSkyresilienceinvestmentsandoperationalplanningwillrequireadditionalfundingthat
mustbeapprovedbyutilityboardsofdirectorsorgovernmentofficials.Otherinvestments,
includingthoseinwaterandwastewatertreatment,andinpowergeneratorsandfuelstorage,
mayrequireregulatorypolicydevelopmentandassociatedpre-orpost-outagewaiversofSafe
DrinkingWaterorCleanAirActstandardsandotherregulatorypolicychanges.Developingnew
approachestoaccountfortradeoffsbetweenBlackSkyresilienceandregulatoryobjectiveswill
beessentialforprogress.
3. Implementationapproachesandconsiderations:
a. Centralizedfuelstorage,oruseofpurchasedfueldeliverytrucks
Supportingalandscapewithmultiple,scatteredbackupgeneratorsandassociatedfuel
storagefacilitiespresentsmajorchallengesforexpandingfuelstorage,andforconducting
resupplyoperationsoveradisruptedtransportationnetwork.Anumberofwaterutilities
arecentralizingtheirstoragefacilitiestomeetthefuelneedsoflargenumbersof
generators,andarepurchasingorcontractingforfueldeliverytrucks.Contracting
arrangementsneedpre-coordinationwithemergencymanagementagencies,however,
sincetheymightcreateamajorcompetitionforresources,especiallyproblematicif
discoveredonceaBlackSkyeventtakesplace.
Theseoptionsshouldbeexaminedsector-wide,withtheaspirationalgoalofhavinganon-
site10-to30-dayfuelsupply,aswellasmaintainingarrangementswithmultiplevendorsfor
fuelresupply.Furthermore,utilitiesshouldtracetheirfuelsupplychainevenasfarbackas
therefineryorportandidentifythevulnerabilitiesthroughoutthesupplychain.
b. Fuelstoragepartnerships
FuelstorageandtransportationaremajorchallengesinmitigatingBlackSkyhazards.In
ordertoreducethenumberofstorageunitsandresupplytripsanddestinations,
partnerships(forlocaldeploymentofadequatefuel)withothercriticalusers/sectors
(emergencymanagement,hospitals,etc.)maypresentidealopportunitiesforfacilitysharing
andmutual-assistanceagreementsonfuelstorageandresupply.
20
c. Fuelsystemmaintenance
Acommonproblemfordiesel-rungeneratorsisthatdieselsludgewillbuildupovertime.
Fueltanksandlinesmustberoutinelycleanedtwiceperyear,andfuelfiltersmonitoredand
changedasneeded.Filtersmustbedraineddaily,especiallyincoldclimates,topreclude
waterbuildupandfreezing.
d. Alternativefuels
UnderemergencysituationssuchasthosefollowingaBlackSkyevent,itmaybenecessary
tousealternativefuelstooperateemergencypowergenerators.Oneapproachisto
acquiremulti-fuelbackuppowergenerators,buttheirpurchasecostishigher.Another
approachistouseaviation-gradefuel--whichwillpotentiallybemoreavailableinaBlack
Skyenvironment—butonlyafterconsultationwiththemanufacturer,distributororleasing
agent.TheuseofaviationfuelasanalternatefuelsourceisbeinginvestigatedbytheEIS
CouncilandthefindingswillbeincorporatedinasubsequentversionofthisPlaybook.
4. Facilitypreparationsforreducedtreatmentlevels:Waterandwastewatersystemsshouldbe
preconfiguredtoallowforswitchovertoBlackSkytreatmentlevels,designedtoassure
designatedtreatmentqualitywhileminimizingelectricityandconsumablesrequirements.Each
utilitywillhavetoestablishlevelsspecifictotheirsystem,conducttheassociatedplanningand
coordinationanddeveloptheconceptofoperationsforreducedlevels,andgaintheapprovalof
theappropriateregulatoryagencies,fortheseBlackSkyservicelevels.
5. Spares:Utilitiesmustplanforandmakeinvestmentsingeographically-stagedsparesfor
vulnerable/frequentlyreplacedcomponents,includingSCADAcontrollers,valves,and
geographicallydistributedstoresofbasictools.Utilitiesmustmaintainawarenessofspareparts
availabilityandsupplychainvulnerabilities.
B. WATERSYSTEM:RECOMMENDEDBLACKSKYOPERATIONALPLANNINGREQUIREMENTS
Operationalplansforoptimizeduseofemergencyfueltoextendoperationsbetweenrefueling,for
operationatreducedwatertreatmentqualitylevels,andotherchangesarerequiredtosupport
BlackSkyServiceLevels.Tobeeffective,theseplansshouldbefullycoordinatedwithsupporting
sectorsandexercisedfrequently.
1. OperationalPlansforTransitiontoBlackSkyServiceLevels:Utilitieswillneedtodevelopoperationalplanstoimplementeachoftheservicelevelchangesthatmayberequiredduring
BlackSkyemergencies.Theseshouldinclude:
a. Transitiontoreduceddrinkingwatertreatmentoperations,whereprocessesforenhanced
waterquality(clarity,odor)aresuspendedduringBlackSkyoperations.
b. Transitiontolimitedzonalservice,toincludeshuttingdownpressurezonestokeepservice
tohigher-priorityormoreeasilyserviced(lowerelevation)zones.
c. Transitionfromfull-powertoreduced-orintermittent-poweroperationsfortreatmentand
pumpingthroughoutthe(possiblyreduced)servicearea.
d. Transitionto“FireSuppressionOnly”operations.
21
e. Transitiontorecoverablesystemshutdown.Forthemostdiresituations,utilitiesshould
examinemethodstoshutthesystemdownwhilemaintainingpressurewithinthesystemto
allowformorerapidpost-eventrecoverybylimitingthedamagethatafulllossofpressure
wouldentail.Thisplanningshouldaddresspotentialpipebreaksduetolossofpressureor
totransientsproducedduringrestart.
2. OperationalplansforBlackSkyService:OncethetransitiontoBlackSkyoperationsismade,
thoseoperationsmayneedtostayineffectforamonthormore.StandingBlackSkyOperating
Procedureswillneedtobeadoptedasthe“newnormal”duringtheemergency.
3. OperationalPlansforRecoverytoNormalServiceLevels:Rapidsystemrecoverywillbe
enhancedifrecoveryplansfromeachofthediminishedBlackSkyservicelevelsbacktonormal
levelsareplannedinadvance,toensurethatthesystemsorcomponentsarenotinadvertently
damagedoncepowerisrestored.Closecoordinationwithsupportingelectricutilitiesisrequired,
inordertominimizedisruptivestarts/stopsthatmayoccurduringblackstart/crankingpath
operations.
4. Training:TrainingisrequiredtomakecriticalpersonnelawareofallBlackSkyoperationalplans,
toaddresseachoftheBlackSkyserviceleveloptions.Trainingneedsshouldaddressnon-typical
areassuchasuseofemergencypowerandcommunicationsmeans,aswellasmaintaining
reducedwaterqualitylevels.
5. CriticalPersonnelandFamilySupport:Utilitieswillneedtosupplementtheirowncritical
personnelandfamilysupport,toassuretheirlaborforcewillbeavailable.Criticalskillsand
proficiencylevels,aswellasfamilysupportrequirements,needtobespecificallyaddressedand
provisionsplannedforsecuringthemintheeventofanextendedemergency.
a. Identificationofkeypersonnelisessential,butforBlackSkyemergencies,itisimportantto
recognizethatsomekeypersonnelmaynotbeavailableastheymaybecaughtupinthe
emergencythemselves.Contingencyplansareneededtoensurethatoperationscanbe
maintainedwithoutthefullcomplementofkeypersonnel.
b. Utilitiesmayneedtoconsiderinternalrelocationofkeypersonnel.
c. Utilitiesmayneedtodevelop,coordinateandexerciseemergencytransportationplansthat
minimizethenumberofvehiclesandthatusepre-designatedemergencyroutesusedto
accesstheirplaceofduty.Carpoolingandpre-planningtheuseof4-wheeldrivevehiclesare
bestpractices.
d. Pre-eventcoordinationwiththeRedCrossorotherdisasterresponseandsupport
organizationisrecommendedtoensurethatcriticalpersonnelandtheirfamiliesaresafe
andsupportedduringanevent.
6. Emergencydatabase:Developmentandmaintenanceofapower-secureemergencydatabase
thatprovideskeyelementsofdataaboutthewaterandwastewaterassets,isvitaltoresponse
andrestorationdecisionsthatfollowaBlackSkyevent.
7. Emergencycommunications:Planningforuseofanappropriateemergencycommunication
systemforinternalandexternalcommunicationanddata,anduseofasynergisticemergency
situationalawarenessanddecisionsupporttoolwillbeessentialtoaccommodatebothinternal
operationalplans,andcoordinationwithexternalsectors.Asanexample,utilitiesmayreview
22
andutilize,asabasisforsupportingtheseplans,theavailableBlackSkyCoordinationandControl
System(BSX)architecture,beingdevelopedforthispurpose.7
8. Rigorousgeneratorserviceandmaintenance:BlackSkyemergencymaintenanceprocedures.
Generatorsaregenerallydesignedforshort-durationemergencyoperations,notlong-term24by
7operation.Planningmustconsidermaintenanceandfailurefactors,andactionstaken
accordingly.Generatorsmustbeexercisedunderfullloadforafewhours,oradayatatime,
everymonth.Utilitiesshouldmaintainamasterlistoftheavailabilitystatusofallbackup
generatorsforuseinBlackSkyoperationalplanning.
9. Critical(External)personnel:Utilitiesmustplanforobtainingpersonnelhavingcriticalsubject
matterexpertiseneededtorestoreormaintainoperations,suchasgeneratormaintenanceand
repair,electricalhookup,specializedplantoperators,etc.Oneexampleistheconceptof
CertifiedPowerRecoveryEngineeringTeams8(CPR)teams.Inthecaseofwatersystems,these
wouldbeemergencypowertechnicians,watersystemcivilengineeringsupportteamsand
cyber/industrialcontrolsystemexperts.
C. WASTEWATERSYSTEM:RECOMMENDEDBLACKSKYOPERATIONALPLANNINGREQUIREMENTS
WastewaterservicemaybeevenmoreimportantinachievingthekeyBlackSkygoalofreducing
massmigrationthandrinkingwaterservice–especiallyinurbanareas.Recommendedoperational
planningrequirementsforwastewaterutilitiesmirrorthoseforthedrinkingwaterutilitieslisted
above.Utilitieswillneedtodevelopoperationalplanstoimplementeachoftheservicelevel
changesthatmayberequiredduringBlackSkyemergencies.Additionalconsiderationsincludethe
following:
1. Transitiontolesspower-intensivewastewatertreatmentoperations,suchasfiltrationrather
thanaeratedsludge,wherefeasible.
2. Transitiontoareducedsetofwastewatertreatmentchemicals.
3. Conductalternativeoperationsthatensurewastewaterflowsaremaintained.
Internal Sector Requirements Matrix
7EISCouncil,BlackSkyCoordinationandControlSystem(BSX):http://eiscouncil.org/Protection/ItemDetails/62
8ForadditionalinformationontheCPREngineeringTeamconcept,see:
http://eiscouncil.org/Protection/ItemDetails/63
Phase Priority Requirement
Water
Response 4 OperationalPlansforTransitiontoBlackSkyServiceLevels
Response 1 OperationalplansforBlackSkyService
Response 8 OperationalPlansforRecoverytoNormalServiceLevels
Mitigation 3 TrainingandExercises
Response 5 CriticalPersonnelandFamilySupport
Mitigation 7 Emergencydatabase
Response,
Recovery
2 Emergencycommunications
Response, 9 Rigorousgeneratorserviceandmaintenance
23
External and Cross Sector Dependencies Overview WaterSectorutilitieswillbeabletomeettheirmissionrequirementonlyiftheyhaveassuredsupport
fromothersectors,tosupplythoseservicesandresourcesthatlieoutsideitsnormalcapabilities.
Definingthesector’srequestedexternalrequirement–andcoordinationwiththeappropriatesectors–
isacrucialpartofdevelopmentoftheBlackSkyProtocol.
A. Government–Non-
RegulatoryAuthoritiesandEmergencyManagers
1. NationalPlanningforEmergencyPower:Prioritizationofemployment,emergencyfuel
deliveries,replacementgeneratorsandgeneratortechsupportwillberequired,andappropriate
managementandcoordinationprocessesassociatedwithfederalandstateemergency
managementagenciesandothergovernmentstakeholderorganizationswillbeessential.TheEIS
Council’sNationalEmergencyPowerCouncil(NEPC)Initiative9providesanexampleofsuch
capability.
2. NationalPlanningforEmergencyUtilityConsumables:Areducedsuiteoftreatmentchemicals
willbeneededfordrinkingwaterandwastewatertreatmentatlevelsrequiredforminimal
health-basedquality.Foranextendedoutage,withnormaloperationofchemicalsupplyand
transportationseverelydisrupted,deliveryofsuchchemicalswilltakeplaceonlyifstateand
federalgovernmentagenciespreplananorganizedprocesstoassureeffectiveexecution.The
NationalEmergencyUtilityConsumablesCouncilInitiative10providesoneexampleofhowsucha
structuremightbeformulated.Fromthewaterutilityperspective,itwillbeimportanttoensure
thisrequirementisclearlyarticulatedasasectorexternalrequirement,tohelpgovernment
agenciesunderstandtherequirementthatsuchaprocesswillneedtoaddress.Key
considerationsinclude:
a. Typicalon-sitechlorinestorageis1–2weeksofneededsupply.Increasingon-sitestorage
however,posesmanyspaceandregulatorychallenges,whichneedtobeaddressedinBlack
Skyplanning.
9Emergencyfueldeliveries,replacementgeneratorsandgeneratortechsupportwillberequiredfromtheemergingNational
EmergencyPowerCouncilorequivalentorganization,onaschedulethatisnegotiatedbetweentherelevantsectors.See:
http://eiscouncil.org/Protection/ItemDetails/64.10Aminimalsetoftreatmentchemicalswillbeneeded,fromtheemergingNationalEmergencyUtilityConsumablesCouncilor
equivalentorganization,providingapre-negotiatedsetofchemicalsandquantities.See:
http://eiscouncil.org/Protection/ItemDetails/65.
Recovery
Mitigation,
Response,
Recovery
6 Critical(External)personnelsuchasgeneratormaintenancepersonneland
cyberexperts
Wastewater
Response 2 Transitiontolesspower-intensivewastewatertreatmentoperations
Response 3 Transitiontoareducedsetofwastewatertreatmentchemicals
Response 1 Conductalternativeoperationsthatensurewastewaterflowsare
maintained
24
b. Onsitechlorinegenerationgreatlyincreaseselectricpowerneeds,soitisnotagoodoption
forBlackSkyoperation.
c. Hypochloritebleachtypicallyrequires12timesthestorage/resupplyrequirementcompared
togaseouschlorineandmaythereforebeimpracticalwhenspaceislimited.
d. Onlyahandful(~5)facilitiesnowproducechlorineproductsintheU.S.Conductingregion
orindustry-widediscussionswiththesesuppliersisimperativeforBlackSkyplanning.
e. Forjointwater/wastewaterutilities,itmaybepossibletosacrificechlorineuseonthe
wastewatersidetoextendchlorinefordrinkingwatertreatment.Utilitiesneedto
understandandplanforthetradeoff,andcoordinatewiththeirwaterorwastewater
counterpart,ifthesefunctionsareperformedbyseparateutilities.
f. Otheressentialchemicalssuchascoagulantsmustbeconsideredinresupplyandstorage
planning.
g. CertifiedchemicaloperatorsandwaterqualityanalystsmaybeinshortsupplyinaBlackSky
emergency,andmustbeplannedfor.
3. CommunityDisasterPreparation:Mediaoutreach,communityforumsandotherformsof
advancecommunitypreparationforlongdurationoutageswillbeessentialtosuccessful
implementationofmanylifelineutilityBlackSkyplans.Forexample,governmentagencies,
utilitiesandmass-careNGOswillneedtoplayapivotalroleineducatingthepublicabout
preparingforhome-basedwaterdisinfection--byuseofchlorinebleachoriodinetablets,for
example--intheeventofsevere,longdurationoutages.
Similarly,governmentagencieswillneedtoprovideadvanceinformationtocommunitiesabout
boththecriticalityofreducingwaterconsumptioninsuchdisasters,andprovidebestpractices
fordoingso.TheEnvironmentalProtectionAgencyrecentlylaunchedtheCommunity-Based
WaterResiliency(CBWR)Tooltohelpfacilitatepartnershipsbetweenwaterutilitiesandthe
communitiestheyserve.11Finally,utilitieswillneedtocoordinatewithemergencymanagersand
otherresiliencestakeholdersregardingtheutility’srestorationplans,priorities,andtimelinesfor
limitedorrotatingservicewithinservicezones.
4. CommunityShelters:Preparationswillberequiredfromgovernmentagencies,incoordination
withtheNationalGuardandmasscareNGOs,tomovepeopletosheltersfromareas
predesignatedbyalocalwatercompanyasan“unserved”location,intheeventofaBlackSky
hazard.Governmentagencies,workingwithwaterandpowercompaniesandwithmasscare
NGOs,shouldpre-planshelterlocationsinareaswhereelectricityserviceandwaterservicewill
bemosteasilysustainedorrestoredfollowingaBlackSkyevent.
5. Limitedemergencywatersupplysupport–AlternativeHydro-Systems:Whilebottledwater
andotherformsofemergencywatersupplywillnotbeavailabletoreplacewaterservicefora
majorcity,governmentagenciesshouldplantoprovidelimitedemergencywatersupportfor
localuse,bothinsmallcommunitieswithoutothermeanstoobtainwater,andforlocalusein
unpowered“pockets”thatremainwithoutpowerandwater,astheoverallpowerrestoration
processproceedsintheaftermathofaBlackSkyevent.
11https://www.epa.gov/communitywaterresilience
25
Analternativedistributionsystem(hydro-systems,includingequipmentandoperations)couldbe
modeledafterIsrael’sconceptofoperationsforfurnishingpotablewaterduringwatercrises.
Theseemergencysystems,properlycoordinatedwithemergencymanagersandlocal
governmentofficials,wouldbedesigned,configured,operatedandmaintainedtoprovide
potablewaterunderNoWaterServiceorFireandSanitationServicesOnlyconditions.12
6. SecurityandTransportationSupport:Provisionswillneedtobemadeforfacilityandpersonnel
security,aswellasfortransportation,indisrupted,BlackSkyenvironments.Furthermore,if
securityisnotaddressed,unauthorizedtamperingwithhydrantsandstandpipescouldleadto
criticallosssupplyand/orpressure.Plansshouldincludecoordinationwithemergencymanagers
andtheNationalGuardtoassureaccessbycriticalrestorationpersonnelandtoprotectthe
distributionsystemfromtampering.
7. NationalSecurityConsiderations:Waterandwastewaterserviceisessentialfornational
security.Civiliancriticalinfrastructure–especiallyinfrastructurethatdirectlysupportsmilitary
installationsorcommandandcontrolelements–isapotentialtargetforanyadversary.Water
SectorutilityleadersshouldpartnerwithFederalandstateagenciestoaddressresilience
initiativeswithregardtonationalsecurity-relatedpriorities.Thiswillnotonlyenhanceresilience
ofthewaterutilityitself,butenhancedeterrenceontwolevels:
a. Activedeterrence--ournationalsecurityapparatuswillremainfunctional,ensuring
appropriatepoliticalandmilitaryresponse;and
b. Deterrencebydenial–reducingvulnerabilitytoattackandtheconsequencesthatanattack
wouldyield;adversarieswouldfindutilitiesalessdesirabletargetandbelesslikelytoattack
inthefirstplace.
B. Government–
RegulatoryBodies
RegulatoryPolicyAdaptations:Utilitieswillneedregulatorychangestorecognizetheneedsfor
BlackSkyresilienceinvestmentandoperations,inordertoprocurerequiredequipmentandbuild
appropriateplanstoimplementtheInternalRequirementsreflectedinthisBlackSkyPlaybook.
Thesewouldincludechangesinwaterpressure,waterquality,wastewatertreatment,airquality
(whenusingemergencybackupgenerators)andotherrelevantfeaturesofplanningforBlackSky
operation.Provisionsmayalsobeneededtoincreaserateschargedtocustomers,whensignificant
additionalinvestmentisrequiredtoimprovesystemresilience.Pointstobeconsideredarelisted
belowandexplainedindetailinAnnexB.
1. RegulatoryMechanisms:UtilitiesshouldlooktotheirAssociationstoidentifyandunderstand
regulatoryrequirementsthatmayneedtobechangedorthatrequirevariances,duetoBlackSky
BlackSkyoperations.
2. EmergencyStandards:Authorizationtomeetonlyacute–butnotchronic–waterquality
standardsduringBlackSkyemergencies.
12AbroaderstrategyforsuchactionisdefinedinPlanningforanEmergencyWaterSupply,issuedbyEPAand
AWWAin2011.
http://www.awwa.org/Portals/0/files/resources/water%20knowledge/rc%20emergency%20prep/Emergencywate
r.PDF
26
3. BlackSkyServiceLevelApproval:BlackSkyservicelevelswillrequireregulatorybodypolicydecisionsandcorrespondingruleadaptations,basedoncoordinatedreviewwiththeutilities
involved.Tofacilitatesuchcoordination,utilitiesarerecommendedtodevelopaBlackSky
CONOPSorPowerOutageAnnexthataddressesBlackSkyprotocols.
4. AirQualityStandards:Airqualityandotherregulatorystandardscreatechallengesforexpandedstorageoffuelandchemicals.Improvedairqualityisanimportantgoal.However,giventhe
absolutelyvitalrolethattheWaterSectorutilitywillplayinsavinglivesduringaBlackSkyevent,
sectorleadersshouldengagewithregulatorstoexplicitlyexaminethetradeoffsbetween
improvedsectorresilience(throughexpandedstorageandotherinfrastructureinvestments)and
environmentalregulations.AttheFederallevel,theEPAisboththeFederalsector-specific
agencyresponsiblefortheWaterSector,andistheFederalleadformakingandenforcing
environmentalregulations.BringingthosetwoportionsoftheAgencytoconductsuchatradeoff
analysisbetweensectorresilienceandemissionsgoalscouldprovideamodelfordoingsoat
stateandlocallevelsaswell.Limitedenvironmentalqualitywaiversshouldbedevelopedfor
emergencygeneratoroperation,torecognizetheneedsforongoingmaintenanceuse,aswellas
preplannedcontinuoususeinsevere,extendedpoweroutages
5. FuelStorage:Whileitisestimatedthat75%ofwaterandwastewaterfacilitieshaveinstalled
emergencypowergenerators,onsiteemergencydieselfuelstoragegenerallydoesnotexceed
suppliesadequatefor48to72hours.InaBlackSkyevent,whereelectricpoweroutagesare
expectedtolastweeksorlonger,backupfuelwillbedepletedquicklyandfuelresupplywillbe
extremelychallenging,anddeliveriesinfrequent.Regulatorybodiesshoulddeveloppoliciesthat
permitandencourageexpansionoftheseemergencyfuelreserves,eitheronsiteatwater
utilities,orstoredinaccessiblelocationswhereitcanalsobesharedwithotherlifelinesectors
anddisasterresponders.
6. ChemicalStorage:Atleastaspressingastheneedforadequateonsiteemergencyfuelstorage,
adequateavailabilityofchemicalsneededtotreatwaterandwastewaterwillbeessential.While
thesechemicalsareroutinelyresuppliedatthesefacilitiesundernormalconditions,availability
anddeliverywillbedrasticallyreducedinseverelydisrupted,BlackSkyscenarios.
TheU.S.EPAandresponsiblestatedepartments,incoordinationwithutilities,shouldexamine
optionsforexpandedonsitestorageofthesubsetoftreatmentchemicalsthatcorrespondto
designated,BlackSkytreatmentlevels,todevelopcorrespondingnewregulatorypolicyforsuch
chemicalstorage.
7. FuelandChemicalTransportation:Deliveryoffuelandwatertreatmentchemicalsisoftena
challengeforutilitiesandtheirregulators,duetoHazMatandothertransportationregulations.
Whilesuchregulationsarenecessarytolimitrisksandprotectpublicsafety,underBlackSky
conditions,thecontinuedfunctioningoflifelineinfrastructureswillbecomeacrucialrequirement
forsavingandsustaininglives.Unlesstreatmentchemicalsareaddressedasapriorityitemtobe
allowedin--particularlyinlargecities---ahealthandsafetycatastrophecouldresult.However,
advanceplanningforrequiredtransportationofthesechemicals–whetherbyutilitiesorby
governmentemergencymanagementagencies–cantakeplaceonlyifsuchtransportation
requirementsareembodiedinprearrangedregulatorypolicyadaptationsandrecognizedbythe
regulatoryagencyresponsible.
27
TheU.S.EPA,DepartmentofTransportation,andstateenvironmentaldepartmentsandother
regulatorybodiesshoulddeveloppoliciesfortransportationofessentialchemicalsunderBlack
Skyconditions,asanessentialframeworkforusebyothergovernmentagenciesandutilitiesin
theirplanningforsustainedwaterandwastewatersystemoperationinBlackSkyscenarios.
8. EmergencyGeneratorProcurement:Waterandwastewaterutilities,incoordinationwith
regulatorycommissions,shouldexaminetheirexistingbackuppowercapability.Forthose
utilitiesthatdonotcurrentlyhavesufficientemergencygenerationtoallowforsustained
operationatBlackSkyservicelevelsthataredesignatedforthatutility,theU.S.EPAshould
developpoliciesthatprovideforcostrecoverytoacquire,installandmaintaintheminimum
additionalgeneratorinventory.
Asanintermediatestep,infrastructureinvestmentinelectricalconnectionsandmanualtransfer
switcheswillallowforconnectionof“trucked-in”emergencygeneratorsduringseverehazards.
Thissolutionisnotideal,sinceavailabilityofsuchunitsisanticipatedtobeverylimited.
Nevertheless,itdoestakeanessentialsteptowardlateracquisitionandinstallationofdedicated
emergencygenerators.
9. Microgrids:Opportunitiesexistforsomewastewaterutilitiesthatuseanaerobicsludge
treatmentprocessestoself-power,reclaimingenergythroughtheuseofcombinedheatand
power(CHP)units,fueledbycapturedmethanefromthewastewatertreatmentprocess.Sucha
processisnotonlyenvironmentallyfriendlyandreducesenergycostsforthewastewaterutility
duringnormaloperations,butcanalsobeapowerfuldriverforsystemresilienceiftheadditional
investmentinelectricalinfrastructureismadetoallowtheutilitytofunctionasapowerislandor
microgrid.
C. MassCareNGOs
1. CriticalPersonnelandFamilySupport:Utilitieswillneedtosupplementtheirowncritical
personnelandprovidefamilysupport,toassuretheirlaborforcewillbeavailable.Masscare
NGOscanplayacrucialroleinthisarea,andwillrequiresuchinformationfromwaterutilitiesto
enablethemtoplanandtraintoprovidesuchsupport.
2. CommunityCoordination:MasscareNGOsshouldbeaskedtohelpcommunicatewiththe
public,tonotifythemofwatersafetyandusagelimitations,includingbothpre-eventandpost-
eventcoordination.NGOscanalsoplayanimportantroleinbothpreparingcommunitiesfor
waterconservationinsuchemergencies,andinreal-timecommunicationoftheurgencyofsuch
conservationduringamajoroutageemergency.
OneexampleisthatMassCareNGOscouldbetaskedwithdistributionofiodinetablets(pre-
eventpreferred),withinstructionalmaterialontheiruseforhomewaterpurification.
Finally,aswithemergencymanagers,utilitieswillneedtocoordinatewithmasscareNGOsand
otherresiliencestakeholdersontheutility’srestorationplans,priorities,andtimelinesforlimited
orrotatingservicewithinservicezones.
3. Shelters:Withwaterutilitiesbuildingplansforgeographicservicelimitations,NGOswillbean
importantpartofplanningtorelocatepeoplefrompredesignated,non-servedareas.Thisneed
shouldbeidentifiedbylocalutilitiesasanimportantexternalrequirementandcoordinatedwith
28
NGOs.Sheltersshouldbepreplanned,locatedinareaswhereelectricityserviceandwater
servicewillbemosteasilysustainedorrestoredinBlackSkyemergencies.Incollaborationwith
emergencymanagersandNGOs,utilitiesshoulddevelopplansformassevacuationstoshelters
havingassuredwaterandwastewaterservices.
D. Electricity
Subsector
1. Prioritizedpowerrestoration:Prioritizedpowerrestorationwillbeessential,tominimizethe
durationofoperationofemergencygeneratorsandrelatedequipment.TheWaterSector
utilitiesmustidentifyprioritizedrequirementsandcollaboratewithpowerproviderstoensure
theseprioritiesareunderstood,addressed,andsynchronized.Currently,mostpowerutilities
prioritizepowerrestorationona“customersserved”basis.Eventhoughwaterutilitiesareavery
largepowerconsumer,theyareoftenviewedbypowercompaniesas“justonecustomer(or
meter).”Furthermore,theyaretypicallylocatedoutsideofthemainpopulationcenters,and
oftenonanelectricalspurline,whichlowerstheirrestorationpriority.WaterSectorutilities
shouldcoordinatewithelectricpowerutilitiesandemergencymanagerstoensurethey
understandthatreturningpowertothewaterandwastewaterutilitiesmustbeapriority,as
waterservicetohomesandbusinesseswillallowthepopulationto“shelterinplace”untilpower
isrestored.
2. Earlypowerrestoration:Insomecases,prioritizednear-immediaterestoration(ifavailable)will
beneededforutilitieswhoseemergencygeneratorsfailorarenotoperable,andcannotfind
adequatespares.Waterandwastewatersystemsalsoprovidealargeloadthatcouldpotentially
beleveragedaspartofthesupportingelectricutility’sBlackStart/CrankingPathConceptof
Operations.Bothoftheseopportunitiesshouldbemutuallyplannedbytheelectricandwater
utilities.
External and Cross Sector Requirements Matrix Requirement
Area
Priority Requirement
Manpower 5 Maintainingallcriticalskillsneededtooperatethesystemat
sustainableminimallevels.ContinuallyimproveWARN.
Transportation 7 Assesstransportrequirementsforallsupportingsupplychains,
extendingbackthroughmainsupplierstotheirsuppliers.Factorinto
planningandmaintainsituationawarenessofprimaryandalternate
meansoftransportationsupportingcriticalsupplychains
EmergencyPower 6 Identifyandmaintainalistofrequiredbackuppowerunits,
includingreplacementsandmaintenancerequirements,provide
thoserequirementstoemergencyplannersandestablishthe
requiredconnectionpointsandoperatingprocedurestoreceive,
connect,operateandmaintainemergencypowerunits.(Consideran
EmergencyPowerPlanasanannextotheEmergencyResponsePlan
thatprovidesacomprehensiveplanofoperations)
Security 1 Providesecurityforfacilities(includingdistributionsystem)and
personnel.Whilepartofthedistributionsystem,hydrantsand
29
Requirement
Area
Priority Requirement
standpipesdeservespecialsecurityplanningtoavoidlossassociated
withunauthorizedaccess
Communications 2 Capabilitytocommunicatewithkeysubsystemstoreceivesituation
reportsandtoeffectcontrolofthesystem.Thisincludesmaintaining
SCADAcapabilities,butataminimum,oralcommunicationswithkey
personnel.
Information
Sharing
3 Capabilityforwaterandwastewaterutilitymanagerstoobtainfrom,
andprovideinformationtoWaterISACaboutpotentialthreatsand
riskstothenation’swaterinfrastructurefromallhazards.This
includesaccessthroughWaterISACtootherISACssuchasE-ISACand
FS-ISAC.
Communications
(Physical)
4 BlackSky-securecommunicationscapableofextendingtoallinternal
componentsandsubsystems.
Fuel 8 Resupplyoffuelnecessarytomaintainwater/wastewater
operationsatdesignatedBlackSkylevelofservice
Sector Specialized Resource Requirements Overview Waterandwastewatertreatmentrequiresthefollowingtypesofspecializedresourcestosustain
servicesfollowingaBlackSkyevent:
1. Skilledmanpower:treatmentplantoperators;distributionsystemrepairandmaintenance
personnel;cyber/ICSexperts;generatoroperatorsandmaintenancespecialists.
2. Disinfectantchemicals:continuingsupplytopreventtheoutbreakofillnessanddisease.
Additionalrequirementsexistforchemicalsusedtoachievelowerprioritytasteandodor
standards.
3. Fuel:foremergencypower,beyondthatstoredonsite.
4. BottledWater:asupplementalsupplyofwaterprovidedbyexternalpartiestomeettheneedsof
thosepeoplelosingtheiraccesstopotablewater.
Sector Commodity Specific List Matrix
Phase Commodity Estimated
Quantity
PotentialSource
Mitigation,Response,
Recovery
Skilledgeneratoroperators Variable WARN,Ex-
Military
Mitigation,Response,
Recovery
Skilledgeneratormaintenancepersonnel Variable WARN,Ex-
Military
All Skilledtreatmentplantandsystem
operators
Variable WARN
All DisinfectantChemicals Securean
initial30-
daysupply
WARN,Chemical
Plants,Food
Stores
30
Sector Black Sky Communications Overview Communicationsareveryimportanttotheoperationoflarge-scalewaterandwastewatersystems,
becausetheysupportpersontopersoncoordinationandtheincreasingadoptionofIndustrialControl
andSCADAsystemstomanagesystemoperations.Thesecommunicationsarenecessary tomonitorthe
systemandprovidereadingstotheSystemOperatorviasomeformofcommunication(BSXwouldbean
optionhere)hourlyinordertomaintainsustainableminimalistlevelsofservice.Communicationswith
WARNpartners,particularlyiftherehasbeenextensivepriorcollaboration,couldbeconsideredas
extendedinternalcommunications.
CommunicationswithEmergencyOperationsCentersandLawEnforcementwouldbenextinpriority
andcommunicationswithcomponentsofthesupplychainwouldfollow.Communicationswith
GovernmentauthoritieswouldgenerallybewithStates,toprovidestatusandtorequestintervention
neededtofacilitateoperations.Communicationwithpublicwillbevitalforongoingconservation
efforts,hometreatmentneeds,andpotentialrelocationrequirements.
Sector Communications Matrix
All Skilledtreatmentplantrepairpersonnel TBD WARN;local
trades
All CoagulantChemicals 30-day
supplyon
site
WARN,Chemical
Plants
All Fuels Securean
initial30-
daysupply
Contractswith
currentsupply
chain;alternate
fuels,when
necessary
Mitigation,Response BottledWater TBDby:
Pressure
Zoneshut
downand
where
quality
levelsare
lowered
FoodStores,
NGOs
Phase CommunicationsRequirement CoordinatedCrossSector
Element
Mitigation,
Response,
Recovery
WithPublic External
Mitigation,
Response,
BetweenOperatorsofthesub-systems(including
WARNpartners)
Internal
31
Sector Black Sky Assessment Tool (s) Overview EPA,withAmericanWaterWorksAssociationandDepartmentofHomelandSecurity,hasdevelopeda
numberofusefultoolsthatcanbeusedtoassessoverallresilienceandtheresilienceofcomponentsof
theWaterSector.
• RiskandResiliencyManagementofWaterandWastewaterSystems(2014)VSATVersion6.0
o http://water.epa.gov/infrastructure/water_security/techtools/vsat.cfm
o Vulnerabilityself-assessmenttoolbasedonANSI-AWWAJ100-10Standard(RAMCAP).
Thistoolisdesignedtoassistwaterandwastewaterutilitieswithdetermining
vulnerabilitiestoallhazardsandwithevaluatingpotentialimprovementstoenhance
theirsecurityandresilience.Itcanbeusedtoconductorupdateanall-hazardsrisk
assessment.(trainingvideosarealsoavailableatthissite).
• WaterSectorUtilityIncidentActionChecklists(2015)
o www.epa.gov/safewater
o ‘RipandRun’stylecheckliststofacilitateemergencypreparedness,responseand
recoveryactionsfortennaturaldisasterthatcansignificantlyimpactwaterand
wastewaterutilities:
§ DroughtEPA817-F-15-001
§ Earthquake:EPA817-F-15-002
§ ExtremeCold/WinterStorm:EPA817-F-15-003
§ ExtremeHeat:EPA817-f-15-004
§ Flooding:EPA817-f-15-005
§ Hurricane:EPA817-F-15-006
§ Tornado:EPA817-f-15-007
§ Tsunami:EPA817-f-15-008
§ VolcanicActivity:EPA817-F-15_009
• Wildfire:EPA817-F-15-010EmergencyWaterSupplyPlanningGuideforHospitalsandHealth
CareFacilities(2012)
Recovery
All BetweenElectricSubsectorandWaterSector External
Protection,
Mitigation,
Response,
Recovery
Sharethreatandhazardinformationacrosssectorsby
usingtheInformationSharingandAnalysisCenter
networkofSectorISACs
External
Protection,
Mitigation,
Response,
Recovery
MaintainICS/SCADAfunctions Internal
Mitigation,
Response,
Recovery
BetweenWaterSystemsandEmergencyOperations
Centers
External
Mitigation,
Response,
Recovery
BetweenWaterSystemsandLawEnforcement External
32
o www.cdc.gov/healthywater/pdf/emergency/emergency-water-supply-planning-
guide.pdf
o CentersforDiseaseControlandAmericanWaterWorksAssociationGuidelinestoassist
healthcarefacilitiesdevelopanEmergencyWaterSupplyPlan(EWSP)toenablethe
facilitytocontinueoperationswhenthedrinkingwatersupplyhasbeeninterruptedfor
anyreason.
• HazardMitigationforNaturalDisasters(AStarterGuideforWaterandWastewaterUtilities)
(2016)EPA810-B-16-001
o www.epa.gov/safewater
o Thisguideprovidespracticalsolutionstohelpwaterandwastewaterutilitiesmitigate
theeffectsofnaturaldisasters.
• AWWACybersecurityGuidanceTool(2014)
o www.awwa.org/cybersecurity
o Thistoolappliesonlytoprocesscontrolsystemsandnotbusinesssystems.Thoughnot
anassessmenttool,itusesanumberofquestionstocharacterizetheutility’sSCADA
system,andthenoffersanumberof‘controls’(suggestions)tohardentheprocess
controlsystem.Thistoolisconsistentwiththefederal‘CyberSecurityFramework’
issuedbytheNationalInstituteofStandardsandTechnologyin2014.
• WaterResearchFoundationBusinessContinuityPlanGuidanceDocument(2013)
o www.waterrf.org
o Thistoolassistsutilitiesinpreparingaplantoensurecontinuationofutilitybusiness
activitiesfollowingadisaster.Thedocumentdirectstheutilitytoidentifykey
parameterssuchas:missionessentialfunctions,criticalresources,andatimeframefor
reachingobjectives.
• SomeutilitiesarebeginningtoincorporatehydraulicmodelingandanalysisusingGeographic
InformationSystems(GIS)intoBlackSkyassessmentsandplanning
o Multipletools:hydraulicmodeling,GIS,simulationmodeling
EISCouncilworkswithU.S.EPA,hydraulicandGISmodelingtoolcompaniesandselectutilitiesto
developrecommendationsregardinghowtousehydraulicmodelingcombinedwithGIStoprovide
operatorsadeepunderstandingofwaterorwastewatersystembehaviorunderavarietyofBlackSky
scenarios,andtoenabledevelopmentof‘whatif’strategiesthatcouldbeusedtobuildadaptabilityinto
ResponseandRecoveryactivities.
Sector Black Sky Planning Requirements (On-going) AsnotedthroughoutthesectiononInternalRequirementsabove,extensiveplanningforBlackSky
operationsisveryimportanttomaintainingadequatewaterandwastewaterservicesvitaltopreventing
widespreadlossoflifeandthecontinuityofsociety.Planningmustaddresstherequirements
proprietarytoeachofthephases,andshouldcontinuetoberefinedinastructuredapproachthatseeks
thatleverages“everythingislearning”,aswellasmodelingandsimulation’tobuildandevolvea“body
ofknowledge”.Mostimportantly,whenperformedcollaborativelyacrosssectors,BlackSkyplanning
forcesthescopetobebroadened,generatesdiscoveryandservestoimprovecurrentoperations.
33
Inadditiontotherequirementsstatedabove,therearethreeotherareasthatneedtobeaddressed:
• CriticalSupportRequirements.BlackSkyeventswillprecipitateworkrequirementsfar
exceedingcapabilities,requiringresponseandrecoveryactionstobeprioritized.EPRO
HandbookIII,BlackSkyCoordinationandCommunication,advocatesforidentifying/designating
criticalfacilitiestoprovidefocusandprioritizationforresponseandrecoveryefforts.Listsof
designatedcriticalfacilitiesshouldformacommonoutcome-basedsetoftargetsusedforBlack
Skyplanning.
• ModelingandSimulation.Modelingandsimulationshouldbeusedextensivelyasanadjunctto
supportBlackSkyplanning.Eachsectorhasdevelopedintra-sectorcapabilitiestomodelits
systemsandsimulateoperations.IntheWaterSector,forexample,EPAhasdevelopedatool
calledWNTRthatutilitiescanusetomodelavarietyofscenariostodeterminearichsetof
vulnerabilitiesandtheirimpactonpressureandwater/wastewaterquality.Modelingand
simulationcanbeusedforexample,todetermineoptimaldispositionofalimitedsupplyof
emergencygenerators,workforce,andequipment.Itcanbeusedalsotoassessthe
effectivenessofplannedsequencingandresourcingofrestorativeactions.Mostimportantly,
whenmodelingandsimulationincorporatescross-sectoroperations,suchaswithGINOM,itcan
beusedtoassesstheimpactsofinterdependenciesthroughoutallphases.Finally,whenusedin
conjunctionwiththemetricsappropriatetoeachreportingactivity,modelandsimulation
providesaverypowerfulmeansofvalidatinginvestmentsinBlackSkyresilience.
• Water/WastewaterBlackStartProjectPlans.OneofthekeyelementsofcompleteBlackSky
planningisrecoveryplanning,thatis,autility’splanforrestoringservicestopre-event—or
otherwisespecified---levels.ThesizeandcomplexityofWater/WastewaterSectorfacilities
requireswell-plannedactivities,awell-consideredresourceallocationplanthatincorporates
contingentresourceavailability,andrisk-basedscheduling.Eachofthesecomponents—
activities,resources,schedule—arenotonlyinter-relatedbutalsomustbe“risk-adjusted”
meaningeachprojectplanmustincorporateprobabilitiestoprovidearangeofdurations,in
additiontomaximumlikelihoodofresourceavailabilityorduration.Theprobabilitiescanbe
usedinmodelingandsimulationtoolssuchasGINOMtoprovide“worst/bestcase”planning
parametersessentialtodecidingamongcoursesofaction,andtoestablishtheirsequencing.
Thesetoftheseprojectplansconstitutea“runningestimate”ofrecoverydurationswithinand
acrosssectors.
Sector Best Practices Matrix (On-going)
Areaof
Operations
Recommendation ExpectedImprovement
Waterand
Wastewater
andElectrical
DevelopcomprehensivePowerOutageIncident
AnnextoEmergencyOperationsPlans
Greaterawarenessand
understandingofwater/electric
nexuswillresultinimproved
resiliencetoBlackSkyhazards
Waterand
Wastewater
Developcomprehensiveemergencypowerplan Optimizescapabilitytoidentify,
planforandmaintain
34
Integrated and Shared Planning Actions (3.5/V4) MaintainingWaterSectorservicesfollowingaBlackSkyevent,evenatreducedlevels,requires
extensiveplanning,collaboratingandexercisingacrossallsectors.Integrationwithothersectors,
classifiedbytypeisasfollows:
andElectrical sustainableminimalistlevelsof
service
Water UsehydraulicmodelingandGIStosupport
resilienceplanningandinvestment
Improvedsituationawareness,
responseplanningandresilience
investments
Waterand
Wastewater
MaximizeWARNopportunities Enhancesresiliencebyadding
additionalcapabilitiesat
reducedlevelsofcost.Improves
thequalityofresilienceby
leveragingthe‘twoheadsare
betterthanone’principle
All Coordinateandmanageinterdependencies
amongsectorsintensively.
Improvedresilienceandgreater
efficiencyintheallocationof
resources
35 Planning and Coordination Actions M
atrix
Water
ElectricFederal
StateRegulatory
Telecommunications
NGO
Water
Support
• PriorityforRestorationofG
ridPow
er• TechnicalKnow
ledge• InclusioninPlanning,Training,Exercises
• CoordinatedBlackStartplanning
• SupportforMicrogridsand
SecureEnclaves• CoordinatedModeling
• ProvidedFederalEm
ergencyAssistance
• WaterInfrastructure
FinanceandIm
provementAct
(WIFIA)Funding
• Tools• M
odelingSupport• SetDesignStandardsforSustainedEm
ergencyOperations
• NERC-equivalent
PrimacyAgency
Support• Com
plianceGuidance
• LaboratoryServices• Dam
ageAssessm
ents• Technicalknow
ledge• RepresentationinEO
C• IncidentFieldSupport
• SupporttoWARN
• Financing(SRF)Em
ergencyManagem
entSupport• AccesstoSites• Credentialing• EvacuationPlans• ProtectionofAssets• State-levelCoordination
• InclusioninPlanning,Training,andExercises
• RequestingFederalAssistance
• SetDesignStandardsforEm
ergencyOperations
• RelieffromW
aterQualityStandards
(SDWA)
• RelieffromCleanAir
Act• Relieffrom
FuelStorageLim
itations• Relieffrom
highway
andInter-statetransportationrestrictions
• Plansforfacilitatingmovem
entofequipm
entandpersonnel
• • IncorporateW
aterSectorintoallEvacuationPlanning,TrainingandExercises
• Identifyevacuationlocations
Information
• Estimated
RestorationTime
• • Em
ergencyResponsePlans
• EOClocation
• PlannedEvacuationSites
• •
•
36
Sector Black Sky Resilience Considerations Overview (3.5/V4) TheWaterSectorhasidentifiedanumberofpotentialresilienceactions,activitiesandinvestmentsthatwouldstrengthenthesector’sresiliencetoBlackSkyhazardsandarelistedinthetablebelow.
Ingeneral,WaterSectorutilitiesaremunicipallyownedandratesareregulatedtokeepthemlow.ThissituationhinderstheirabilitytomakethetypesofinvestmentinresiliencethatwouldberequiredforBlackSkyhazards.
Resilience Initiatives Matrix Toolingandcapabilitiespreparedinadvance:Definethosecategoriesofadvanceplanning,investmentandeffortrequiredtomakeinfrastructurerestorationandpopulationsustainmentpossible,whenthenationfindsitselfthrustintoahighlydisrupted,BlackSkyscenario.
InitiativeTitle InitiativeDescription/Cost ExpectedOutcome
Water/EnergyNexus ParticipateinDOEinitiativetoanalyzetherelationshipsbetweenenergyandwateruseandconductresearchonwaterandenergysystems.`
DeepercollaborativerelationshipbetweenWaterSectorandElectricSubsector
EPRI ReviewandleverageEPRI’spublishedresearchonWaterSustainabilityandseektoestablishapartnershipbetweenEPRIandtheWaterSector.
DeeperrelationshipbetweenWaterSectorandElectricSubsector
HydraulicModeling/GeographicInformationSystem(GIS)
PromotehydraulicmodelingandGIStosupportBlackSkyplanning.Hydraulicmodelingcanassistautilityingainingafullrealizationofcapabilities,vulnerabilitiesandresponses,toavarietyof‘whatif’scenarios,whileGIScanprovideinformationonlanduse,population,criticalcustomers,shelterlocations,etc.
ImprovedBlackSkyplanningandSituationAwareness
EmergencyPowerAnnextoEmergencyResponsePlan
Conductanin-depthanalysisofemergencypowerrequirements;incorporatehydraulicmodelingtodeterminenumbersandplacementofemergencypowerunits.Assesssupportingrequirementssuchasfuel,maintenance,andunitreplacements.
Detailedexecutableplanforrestoringsustainableminimalistservicelevels,andbeyond.
Cyber-SecureWaterandWastewaterOperations
Developplansforacyber-basedattackonWaterandWastewatersystems,particularlyinconjunctionwithanaccompanyingattackontheElectricGrid
Reducedtimetorecover
37
Sector Black Sky Regulatory Impacts and Issues Overview (On-Going) BlackSkyregulatoryissuesaredescribedabove,inExternalandCrossSectorDependenciesOverview,PartB.,Government-RegulatoryBodies,andaresummarizedasfollows.Theyarefurtherdetailed,onanitembyitembasis,inAnnexB,RegulatoryIssuesDetailStatements.
Sector Regulatory Matrix Areaof
Operations
Issue RecommendedSolution/Resolution
WaterTreatment
Limitsonuseofspecificwatersources Identify,andcoordinatewithRegulatoryofficialstouse,specificsourcesofwaterduringBlackSkyresponseandrecovery
WaterTreatment
WaterQualityStandardsunderBlackSkyConditions
IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)
WastewaterTreatment
Relaxdischargestandardsforwastewatersystems
IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)
Waterand Responseandrecoverypersonnelshouldbe
SupplyChainResilienceAssessment(commodities)
Developadeepunderstandingofthecapabilitiesandvulnerabilitiesthesupplychainsthatsupportcontinued
Withanimprovedunderstandingofthevulnerabilities,threats,andconsequences,ownersandoperatorsofutilitiescancontinuetothoroughlyexamineandimplementrisk-basedapproachestoprotect,detect,respondto,andrecoverfromallhazardsbetter
FinancingResilienceInitiatives
Identifymeanstofinanceresiliencemeasuresintonewandexistingcapabilities.
Overcomecurrentobstaclesposedbyrequirementtouseratestopayforresilience
NIACWaterSectorResilienceMeasures
AdoptandsupporttheresiliencerecommendationscontainedintheNIACWaterSectorResilienceFinalReportandRecommendations,datedJune2016.
BlendsNIACandEISCouncileffortsandopenstheopportunitytobroadenBlackSkyconsiderationsinfutureNIACrecommendations.
38
Areaof
Operations
Issue RecommendedSolution/Resolution
Wastewater providedFirstResponderstatusEmergencyPower
LimitsPlacedonRunningEmergencyPower PartialExemptionfromCleanAirAct
WaterandWastewaterTreatment
Limitsonon-sitechemicalstorage IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)
EmergencyPower
LimitsPlacedonon-sitefuelstorage IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)
Sector Black Sky Essential Critical Infrastructure (MC) Overview (3.5/V4) Themostessentialelementsofwaterutilities’infrastructuretoachieveresiliencearedetailedbelow.TheseincludeitemsthatareuniqueormissionessentialtotheWaterSector.
Sector Critical Infrastructure Matrix (V3.5/V4) Element Function
Pumps Drawfromsource,convey,lift,maintainpressure:highenergyDistributionPipelines
Conveywater:notolowenergy
StorageTanks
Store:energytoliftbutnotoncestored
WaterTreatmentSettlingTanks
Clarificationbymixingandsettling;notolowenergy(mixingandsettledsolidremoval)
WaterOzonation
Useozonetokillmicro-organisms:highenergy
WaterFiltration:membrane
Improvequality;desalination:highenergy
WaterFiltration:mediafilter
Improvequality:notolowenergy
WaterDisinfection
Disinfectwater:lowenergy
WastewaterPrimary
PrimarySewageTreatmentcapabilities:lowtonoenergy
WastewaterSecondary
Secondarysewagetreatmentcapabilities,especiallybiosolidremoval:mediumtohighenergy
WastewaterTertiary
Sewagedischargetreatmentcapabilities:lowtomedium
39
Element Function
SludgeProcessing
Sludgeconveyanceanddisposal:highenergy
SCADA Controlsystemoperations:lowtomediumenergy
Sector Black Sky Specialized Skill Training Requirements Overview (V3.5/V4/V5) TobeBlackSkyresilient,theWaterSector,includingitswaterandwastewatersystemsshouldidentifymissioncriticalskills/positionsthatmustbetrainedtoaccomplishBlackSkyresilience,startingwithAssessmentandmovingthrough,Response,RestorationandRecovery.TheAmericanWaterWorksAssociationhasoutstandingtrainingprogramsforWaterandWastewateroperators,andconsiderationshouldbegiventoeitherincorporatespecialBlackSkyskillrequirementsintoexistingcoursesor2)startaseparatetrackforBlackSkyoperations.
AWWAhasdevelopedaresourcetypingcatalogforWARNthatcanservethedualpurposeofspecifyingtheskillsrequiredduringanemergencies.Resourcetypingisthecategorizationanddescriptionofresponseresourcesthatarecommonlyexchangedindisastersthroughmutualaidandassistanceagreements.Resourcetypingdefinitionscangiveutilitiestheinformationtheyneedtoensurethattheyrequestandreceivetheappropriateresourcesduringanincident.TheresourcetypingprotocolprovidedbytheNationalIncidentManagementSystemorNIMSdescribesresourcesusingtheparametersofcategory,kind,components,metrics,andtype.Utilityandsystemplannersshouldconsultthistypingcatalog,identifytheirspecializedskillrequirementsandincludethemintheirrespectiveBlackSkyemergencyandcontinuityofoperationsplans.http://www.awwa.org/Portals/0/files/resources/water%20knowledge/rc%20emergency%20prep/rc%20warn%20resources%20pdf/typingmanual.pdf
Sector Specialized Skill Training Requirements Matrix (3.5/V4/V5) Phase Position/Skill Training/CertificationRequirement
All EmergencyPlanner TBDMitigation HydraulicModeler TBDPrevention,ProtectionMitigation
VulnerabilityAssessment TBD
Response EmergencyPowerOperator TBDResponse EmergencyPowerMechanic TBDAll SCADATechnicalArchitect TBD
40
Annex A – Assessments (On-going) Sector Overall Resilience Assessment MostofthelargerwaterandwastewatersystemshaveconductedavulnerabilityassessmentusingtheVulnerabilitySelf-AssessmentTool(VSAT)6.0.VSATVersion6.0complieswiththeWaterSectorriskassessmentstandardandcanofferliabilityprotectionundertheDepartmentofHomelandSecurity’sSupportAnti-TerrorismbyFosteringEffectiveTechnologies(SAFETY)Actprogram.
TheEISCouncilisevaluatingthecurrenttooltodeterminethemodificationsrequiredforBlackSkyresilience.ItisunlikelythatmorethanafewsystemshavemadeaBlackSkyresilienceassessment,thoughLasVegasValleyWaterDistricthasbeguntoconductBlackSkyplanning.LessonslearnedfromtheLVVWDexperiencewillbeusedtorefineassessmentrequirementsinfutureupdatesofthisPlaybook.
2
Annex B – Regulatory Issues Detail Statements (On-Going) Issue Statement 1: Testing Emergency Generators
• Statement:CleanAirActrestrictsthetestingofemergencypowerunits.Reliefissoughttoallowpropertesting.
• DecisionAuthority:EnvironmentalProtectionAgency• RequiredDocumentation:obtain/compilemanufacturers’literature
regardingtesting• ResiliencyInvestmentstatement:generatorsmustbeperiodicallytested
underfullloadforextendeddurationtoassuretheywillproperlyfunctionduringemergencyoperations
• PlanRequirements:
• TrainingRequirements:
• LiabilityStatement/3rdPartyProtectionIssue:
• Explicitrequestedlegislativechanges/Insurance/Assurance/3rdParty
Indemnification:
Issue Statement 2: Reduced Water Quality Levels • Statement:SafeDrinkingWaterActestablisheswaterqualitystandards,
manyofwhicharedeterminedbyeffectsoflong-termuse.ReliefissoughtforBlackSkyconditions.
• DecisionAuthority:EnvironmentalProtectionAgency• RequiredDocumentation:listofvariances
• ResiliencyInvestmentstatement:reducingwaterqualitylevelsforBlackSkyoperationsmayextendthesustainabilityofwaterandwastewateroperations
• PlanRequirements:
• TrainingRequirements:• LiabilityStatement/3rdPartyProtectionIssue:
• Explicitrequestedlegislativechanges/Insurance/Assurance/3rdParty
Indemnification:
2
Annex C – Communications Requirements (V3/V4) Communications Requirement 1: Internal Coordination
• Internal/PlannedFormat/Path:Internalcommunications• External/PlannedFormat/Path:N/A• ExplicitModel
o Who:designatedsitesandmobileteamsoftheutilityo What:coordinationwithinternalresources
§ Statusofkeycomponents§ Damagetoassets§ Emergencypowergeneratorstatus—includingmaintenance§ Fuelstatus§ Personnelstatus§ Tanklevels§ Pressureatkeypoints§ Qualitymeasurements§ Firehydrantoperations§ Operationsorders§ Valveoperations
o When:asoccurs;followupuntilprovidedo Strategies(backup):walkie/talkieo Bandwidthrequirement(actualandnotional):HF/UHF/VHFo Format:voiceo Priority:High
Communications Requirement 2: External Support Requirements • Internal/PlannedFormat/Path:Internalcommunications• External/PlannedFormat/Path:BSX• ExplicitModel
o Who:designatedsitesandmobileteamso What:InterventionrequestedfromEmergencyOperationsCenter
§ Accesstosites§ LawEnforcement§ Generators§ TechnicalSupportbyType§ Fuel§ TreatmentChemicals§ BottledWater§ Other
o When:asoccurs;followupuntilprovidedo Strategies(backup):physicalpresenceo Bandwidthrequirement(actualandnotional):HF/UHF
3
o Format:Voiceo Priority:High
Communications Requirement 3: Status of Services • Internal/PlannedFormat/Path:SCADA• External/PlannedFormat/Path:BSX• ExplicitModel
o Who:SystemOperatoro What:SystemMapupdatedwithSCADA-reportedStatuso When:Initial,within1hourofBlackSkyincident;followup,twicedailyo Strategies(backup):CouriertoEmergencyOperationCentero Bandwidthrequirement(actualandnotional):Lowo Format:
§ NumberofPeoplewithoutwaterservice§ Pressurezones(designators)withoutservice(preregisteredwithEOC)§ Criticalfacilities(designators)withoutadequatefireflow/pressure§ Reductionorforecastedlossofserviceovernext24hours
o Priority:High
Communications Requirement 4: Damage to Components • Internal/PlannedFormat/Path:SCADA• External/PlannedFormat/Path:BSX• ExplicitModel
o Who:SystemOperatoro What:SCADA-reportedStatuso When:Initial,within1hourofBlackSkyincident;followup,twicedailyo Strategies(backup):Voice,followingphysicalinspectiono Bandwidthrequirement(actualandnotional):TBDo Format:
§ Assetsinoperable(selectfromprepositionedlistofsystem’sassets)§ Prioritizedlistofassetreplacements(requiredtooptimizelevelofservice):use
AWWAResourceTypeo Priority:High
Communications Requirement 5: Request for Personnel • Internal/PlannedFormat/Path:Supervisorreportsusinginternal
communications• External/PlannedFormat/Path:BSXtoWARN• ExplicitModel
o Who:UtilityManagero What:Requestforadditionalsupport,byResourceTypeo When:Initial,withinTBDhoursofBlackSkyincident;followup,twicedailyo Strategies(backup):Voice,followingphysicalinspectiono Bandwidthrequirement(actualandnotional):TBDo Format:
4
§ Prioritizedlistofassetreplacements(requiredtooptimizelevelofservice):useAWWAResourceType
o Priority:High
Communications Requirement 5: Request for Resupply of Treatment Chemicals • Internal/PlannedFormat/Path:Internal• External/PlannedFormat/Path:BSX• ExplicitModel
o Who:LogisticsManagero What:Treatmentresupplyrequirementso When:3daysandwheneversuppliesfallbelowreorderpoint(continuously
updatedbasedonexperiencegainedduringBlackSkyresponseandrecoveryo Strategies(backup):Preplanned‘push’supplyo Bandwidthrequirement(actualandnotional):TBDo Format:
§ Estimatedchemicalsupplybyquantity,projectedovera30-dayperiodo Priority:Medium
5
Annex D: Resilience Requirements by Layer Identifyrequirementsforeachphase,includingboth(A)GenericrequiredelementsforallBlackSkyhazards,and(B)Hazard-specificrequiredrequirements.Note,however,thatmanyrequirementswillexhibitheavyoverlapamongsectors,withmanyinfrastructuresectors,andtheirpartners,requiringthesameresiliencemeasures.
Area Phase BlackSkyGeneric BlackSkyHazardSpecific
Notes/Status
Emergency
CommunicationMitigate
Respond
Recover
• Planforcommunicationswithoutcellular
• Planfornon-gridpowerforcommunications
• Train/rehearseusingfrequenciesavailableduringBlackSky
• BSXcapabilities(EMPhardenedandprolongednon-gridpower)
• Planforcommunicationswithoutcellularandsatellite(EMP)
• StoreequipmentinFaradayboxes(EMP)
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
Cross-sector
situational
awareness,
coordinationand
decisionsupport
requirements
Mitigate
Respond
Recover
• Identifykeymetricsformonitoringinternalsectoroperations
• Identifykeymetricsformonitoringcross-sectoroperations
• Performall-hazard,multiple-scenariomodelingandsimulationtosupportBlackSkyplanning
• IncorporateGINOM-furnishedinformationintoallplanning
• EnsureBlack-Skysecure(EMP/GMDhardenedandlong-termpowersupply)toGINOMcapabilities
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
EmergencyFuel Mitigate
Respond
Recover
• Developfuelplanforextendedpoweroutagecontingencyo Coordinateplanwithsuppliers
o Identifypotentialcompetitionforresourceswithsuppliers
o Investinfueltankerstosupportlong-termfuelingoperations
• Maintainatleast7daysoffuelforfleetandemergencypower
• Planfordisruptedfuelandchemicaltransportationintheeventoffloodingorearthquake
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
OperationalFuel
(e.g.,naturalgas)
Requirements
Mitigate
Respond
Recover
• Plan/investinfuelingcapabilitiesforemergencyvehicles,usingonhandcapabilities
• Planfordisruptedfueltransportationduetofloodingorearthquake
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
Emergency
Generators(Black
Mitigate • Plan/investinsufficientemergencypowercapabilitytomaintain
• InvestinEMP/GMD-hardenedgenerators
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
2
Skyhazard-
hardened,designed
forlongduration
continuous
operation)
Respond
Recover
minimalservicelevels• Usehydraulicmodelingtoidentifyoptimaldeploymentofemergencypower
• RegisterrequirementsinEPFAT
• Provisionhookupsformobileandexternally-providedunits
• Investingeneratorsusedforlong-duration,continuousoperation
• Investinalternativeenergysources(wind,PVandco-generation)
• Planforspecializedmaintenancesupport
• Investinmulti-fuelcapability
• Securetheavailabilityofreplacementgenerators
Protectivemeasures:Hardware,software,operationalprocedures
Mitigate
Respond
Recover
• Protectelectricfacilitiesfromflooding
• Strengthendistributionsystemtominimizeearthquakedamage
• ProtectSCADAfromEMP• ProtectemergencygeneratorsandvehiclesfromEMP/GMD
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
Health/damage
assessment
diagnostics:
Mitigate
Respond
Recover
• Develop/exercisecomprehensivedamageassessmentCONOPS/CONPLAN
• IncorporateassetmanagementintoBlackSkyplanning
• Performcomprehensivemodelingwithmultiplescenarios(allhazards)toassesstherangeofpotentialdamagetosystem
• LeverageIoTtodesignacomprehensiveandintegratedsensorplanbasedonsituationawarenessgoals
• Plan/installhealthchecksforemergencypowerunits
• ProtectdiagnosticcapabilitiesfromEMP/GMD
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
Pre-deployed,automatedself-poweredremotereportingtocentralassessmentcontrollers
Mitigate
Respond
Recover
• Pumpstationpoweroutage
• Tank-levelindicator• Pressureindicatoratcriticalpointsindistributionsystem
• InvestinEMP/GMD-securesensing/communicationscapabilities
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
On-sitediagnostics Mitigate • Manual • N/A Meetingallrequirementswill
3
providingeasy1storderdamageassessmentforanonsite,deployedrestorationteam
Respond
Recover
water/wastewaterqualitytestingkits
haveMAJORimpactonmeetingsystem’smission
Diagnostictoolingdesignedformanualusebyadeployedrestorationteam
Mitigate
Respond
Recover
• Planforenhancedwater/wastewaterqualitytesting
• Planforwater/wastewaterqualitytestingcapability
• Providecapabilitytoreconfigurehydraulicmodelingtoolsinresponsetodamage
• SecureEMP/GMD/vulnerablediagnosticequipment
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
Restoration/
emergencytoolingMitigate
Respond
Recover
• Developblackstartplanforsystem–forseveraloutagescenarios
• Identifyspecializedsupportforrestoringsystemfollowingaprolongedpoweroutage
• Developresourcedprojectplanforsystemrestart
• IncorporatemodeledEMP/GMD/earthquakeandflooddamageforecastintorecoveryplans
•
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
BlackSkyExercise
andTraining
Requirementsand
Plans
Mitigate
Respond
Recover
• DevelopPowerOutageIncidentAnnex(POIA)CONOPSorCONPLANforconductingoperationsfollowingextensivegridoutage
• Plan/conductsectorPOIAexercise
• Plan/conductcross-sectorPOIAexercise
• IdentifyanddocumenttrainingrequirementsforPOIAoperations
• Incorporatehazard-causeddamage(inadditiontoeffectsofpoweroutage)intowater/wastewatersectorresponseandrecoveryplans
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
Materiel Mitigate
Respond
Recover
• Enhancedwater/wastewaterqualitytestingcapability
• Treatmentchemicalsandsupplies
• Suppliestomaintainworkforce
• ProtectelectronicequipmentfromEMP/GMD
• Protectelectricalequipmentfromflooding
• Investinhardeningmeasuresforearthquakeandflooding
MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission
4
1. Spares,withadvancefielddeploymentrequirements:Inthreecategories
a. Inexpensive,commonusehardwarethatwillbehardtofindinBlackSkyconditionsunlesspre-deployed(i.e.,“Forwantofanail…”)
b. Inexpensivetomoderatelyexpensivehazard-vulnerablehardwareelementsfordifferentBlackSkyhazardscenarios
c. Expensive,longleadhazard-vulnerablehardwareandassociatedinstallationtoolingfordifferentBlackSkyhazardscenarios
2. Personnelsupportrequirements:Whatteams,andhowmanyteams,willberequiredbyeachstakeholderorganization,withwhatteammakeupandwhatdeployment,foreachhazardtypeandforeachphase?
Note1.Thisassessmentwilldrive,forexample,therequirementsforprearranged,pre-certified,externalCertifiedPowerRecovery(CPR)EngineeringTeamstoaddsubstantiallytothecoreorganizationalteamsforthatsector.
Note2.Asupplementalrequirement:giventhenumber,makeupanddeploymentoftherequisiteteams,whatwillberequiredforassociatedfamilysupport?
Note3:Thesepersonnel/teamrequirementsshouldbeprovidedasafunction,atleastcrudely,ofthelevelofavailabilityofdiagnostics(in(g)i.->iii.Above).E.g.,ifthereareremotelypre-deployed,BlackSkyhardened,self-powered,embeddedandremotelyreportingdiagnosticsinallimportantpowergirdsubstationsandgeneratingsubstationsandwatersystemandgaspipelinekeyfacilities,vastlyfewerrestorationteamswouldneedtobedeployed,thosethataredeployedwouldbefarmoreeffective,andtheentiredamageassessmentphasewouldbefarquicker.
3. BlackSkyOperationalPlansandProcedures:Supplementingexistinghazardplansandproceduresforconventionalhazards
4. BlackSkyExerciseandTrainingRequirementsandPlans:Supplementingexistinghazardexerciseandtrainingrequirements