Risk Mitigation for Open NTP OpenNTP... · NTP clients synchronize their time with a local time...

Risk Mitigation for Open NTP

Copyright©2016,CyberGreen Sept2016

Agenda

1. Introduction2. AboutNTP3. MitigationrecommendationsforopenNTP4. Makingthecaseforimplementingmitigations

2 Copyright©2016,CyberGreen Sept2016

Introduction

WhencyberinfrastructureisinsecurethereisarisktotheglobalInternetcommunityNetworkTimeProtocol(NTP)isthestandardprotocolfortimesynchronizationfornetworkeddevicesNTPcanbefoundinnearlyeverynetworkenvironmentSynchronizedtimeiscriticaltologging,authentication,cryptographyandgeneralsystemadministrationNTPinfrastructureneedstobesecureandtrustworthy


About CyberGreen

• Globalnon-profitandcollaborativeorganizationfocusedonhelpingimprovethehealthofglobalCyberEcosystem

• WorkingtoprovidereliablemetricsandmitigationbestpracticeinformationtoCyberSecurityIncidentResponseTeams(CSIRTs),networkoperators,andpolicymakers

• Mission:helpCSIRTsandothersfocusremediationeffortsonthemostimportantriskso Helpunderstandwhereimprovementscanbemadeo Howwecanachieveamoresustainable,secure,and

resilientcyberecosystem


Copyright (c) 2016, CyberGreen

Thesematerialsaredistributedunderthefollowinglicense:Permissiontouse,copy,modify,and/ordistributethesematerialsforanypurposewithorwithoutfeeisherebygranted,providedthattheabovecopyrightnoticeandthispermissionnoticeappearinallcopies.THEMATERIALISPROVIDED"ASIS"ANDTHEAUTHORDISCLAIMSALLWARRANTIESWITHREGARDTOTHISMATERIALINCLUDINGALLIMPLIEDWARRANTIESOFMERCHANTABILITYANDFITNESS.INNOEVENTSHALLTHEAUTHORBELIABLEFORANYSPECIAL,DIRECT,INDIRECT,ORCONSEQUENTIALDAMAGESORANYDAMAGESWHATSOEVERRESULTINGFROMLOSSOFUSE,DATAORPROFITS,WHETHERINANACTIONOFCONTRACT,NEGLIGENCEOROTHERTORTIOUSACTION,ARISINGOUTOFORINCONNECTIONWITHTHEUSEORPERFORMANCEOFTHISMATERIAL.


About NTP


Network Time Protocol (NTP)

NetworkTimeProtocol(NTP)isstandardprotocolfortimesynchronizationfordevicesonanetwork,usedbyservers,mobiledevices,endpointsandnetworkingdevicesfromallvendorsThelatestdefinitionofNTPisversion4,asdescribedinRFC59051

1http://www.ietf.org/rfc/rfc5905.txt


Network Time Protocol (NTP)

NTPclientssynchronizetheirtimewithalocaltimeserver(liketheDomainControllerinWindowsenvironments),whichwillinturnsynchronize itsclockwithreliableNTPserversavailableontheInternetJusttogetthetime,veryfewtypesofmessagesareneeded• Additionalmessagesandmodesonlyneededfor

NTPserversthatneedtotalktoeachother


What is open NTP?

“OpenNTP”isaserverwhere• NPTisrunningonadeviceavailabletothepublic

Internet,and• NTPanswersMode6orMode7queries

o Thesequerieshavevulnerabilitiesthatcanbeexploitedbyattackers2


2https://community.rapid7.com/community/metasploit/blog/2014/08/25/r7-2014-12-more-amplification-vulnerabilities-in-

ntp-allow-even-more-drdos-attacks

How NTP works


Risks posed by open NTP

DevicesrunningopenNTPcanbeusedinreflectionattacks,atypeoftrafficamplificationattack• Denialofservice(DoS)– attackertriesmakeavictim’s

machineornetworkunavailabletoitsintendedusers• Amplification– whentheattackersendsasmallpacket

toaserverthatwillgeneratealargereplyInamplificationdistributeddenialofservice(DDoS)attacks,attackerssimultaneousabusemultipleamplifierssuchasNTPservers• Createshighly-distributedDoS attackconductedfroma

singlecommandandcontrolhost


Open NTP in reflection attacks

Attackertriestoexhaustthevictim'sbandwidthbyabusingthefactthatserversusingprotocolssuchasNTPallowspoofingofsenderIPaddressesReflectionattacksoftenexploitUserDatagramProtocol(UDP)traffic• UDPrespondstorequestswithoutanyvalidation

ofsenderidentity,i.e.IPaddress• UDPtrafficcanbespoofed(i.e.haveamisleading

apparentsourceIPaddress)andattackerisabletohidetrueidentity


NTP reflection amplification attack

ADDoSthatreliesonpublicallyaccessibleopenNTPserverstooverwhelmavictimsystemwithNTPresponsetraffic• Anattackerwithasingle1Gigabit/second(Gb/s)

connectioncantheoreticallygeneratemorethan200Gb/sofDDoStraffic3

Onlyscalableandeffectivemitigationistoreducenumberofserversthatcanbeusedbyattackers• Asof07/27/16,Shadowserverreported4,062,384

uniqueIPswithopenNTP;seehttps://ntpscan.shadowserver.org/stats/


3https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks

NTP amplification attack

AttackersgeneratealargenumberofUDPpacketsusingspoofedsourceIPaddressUDPpacketsaresenttoNTPserversonport123AttackersparticularlylikeNTPserversthatsupporttheMONLIST command4

• MONLIST commandreturnsalistwithlast600IPaddressesthatconnectedtotheNTPserver

• Actsasreconnaissancetoolforhackers:helpsbuildprofileoflocalnetwork


4AdiscussionofMONLIST canbefoundathttps://blog.qualys.com/securitylabs/2014/01/21/how-qualysguard-detects-

vulnerability-to-ntp-amplification-attacks


Real life attack using open NTP

Early2014reportofattackusingopenNTP5

• Generatedaround400Gbp/softrafficusing4,529NTPservers

• Eachserverreportedlysent87Mbp/softraffictothevictim

NTPamplificationattackscanresultinabandwidthamplificationfactorof556.96

5https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/

6http://www.christian-rossow.de/articles/Amplification_DDoS.php


Potential impacts from NTP attacks

Productivity• Serviceinterruptionorfailureofbusinessoperations

relyingonnetworkconnectivity,particularlyforseasonaloperations- e.g.onlineretailerswhereamajorityofsaleshappenbetweenThanksgivingandNewYears

• Timesensitiveoperations,e.g.collegeswithlimitedonlineregistrationperiodsoronlinewageringonupcomingsportingevents,etc.

Other potential NTP attack impacts

Brand• Lossofreputationwithcustomersandpartners• Becomingknownasa“DoSmagnet”inglobalcommunityTechnical• Networkserviceinterrupted• Isolationofvictimnetworkbynetworkprovidersfromthe

restofInternettomitigatecollateraldamagetoothercustomers

Financial• Lossofbusinessresultingfromserviceinterruption• CostofspecializedDDoSmitigationservices18 Copyright©2016,CyberGreen Sept2016


Indirect impacts from Open NTP attacks

YoumaybeimpactedifavictimorganizationsharesyourupstreamconnectivityOpenNTPdevicesonyournetworkmaybeusedtocontributetoanattackonanotherorganizationPotentialindirectimpactsinclude:Technical• Networkservicedegraded• Inboundoroutboundbandwidthmaybereduced• Networkprovidersmayisolateyournetwork(orat

leastyourinsecurerecursiveresolver)fromtherestofInternet


Other indirect impacts

Brand• Lossofreputationwithcustomersandpartnersduetoslow

orunreliablenetworkandsystemsFinancial• Unexpectednetworkusagecosts• Lossofbusinessresultingfromservicedegradation

Mitigate risks from open NTP



Mitigation options vary by environment

NotallmitigationbestpracticesareappropriateforallenvironmentsCyberGreenprovidesinformationrelevanttofourbasicenvironmentalprofilesLookfortheseiconstofindmitigationsforyourenvironment

1.

2.

3.

4.


Mitigate risks from open NTP

ThebestwaytomitigaterisksfromopenNTPmovingforwardistopurchaseanddeploydeviceswithminimalNTPconfigured,particularlyonoutsideinterfacesWorkwithyourinternalacquisitionandprocurementteams,orvendorsaboutotheroptions


Identify your open NTP risk

Evenifyoudon’tthinkyourdevicescurrentlyrunNTPacrosstheInternet,youshouldcheckyournetwork• ManydevicesmayberunningNTPwithoutyour

knowledge• NTPisoftenbuiltintoCustomerPremise

Equipment(CPE)gatewaysonnetworkequipmentsuchascablemodems,DSLrouters,“broadbandWiFi routers”,etc.


Find hosts running NTP

Thesimplestwayistouseaweb-basedprobe,suchastheoneathttp://openntpproject.orgTomanuallyidentifyNTPserverswithamplifiedresponsesenabled,runoneofthefollowingcommands:

ntpdc –n –c monlist 192.0.2.1

ntpdc –c sysinfo 192.0.2.1

ntpq –c readvar 192.0.2.1

Thecommandsonlyverifyifspecifiedfunctionsareenabled


Manually finding NTP hosts

Ifcommandwassuccessful,youwillseeastringofinformationlikethisfromtheIPyouqueried:

associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync, version="ntpd [email protected] Sun Oct 17 13:35:13 UTC 2010 (1)", processor="x86_64", system="Linux/3.2.0-0.bpo.4-

amd64", leap=00


Mitigation: Upgrade NTP

TheeasiestwaytomitigatetheriskistoupgradetoNTP-4.2.7p230(releasedin2011)orlater,whichremovestheMONLIST commandentirelyanddisablesMode7responsesbydefault• Protectsyournetworkfrominadvertentlybeing

usedinaDDoSattack• Protectsyournetworkfromunwanted

reconnaissance


Mitigation: Upgrade NTP

Ifyourenvironmentissofragilethatupgradingisnotanoption,modifytheNTPconffiletoaddthestatementdisable monitorandthenrestartyourNTPprocessesYoushouldalsoimplementanadditionalriskmitigation


Mitigation: Disable status queries or restrict access

NTPqueriesmayrevealinformationaboutthesystemrunningNTPthatyoudonotwantotherstoknow,suchastheoperatingsystemversionandntpdversionDisablingthesequeryfeaturesmayhelptoreducethelikelihoodofthisdataleakagetakingplace• Disablingthesequerieshasacost,asthesequery

capabilitiesalsoprovideusefulQ/Aanddebugginginformation


Mitigation: Restrict informational queries to authorized recipients

TodisableMONLIST functionalityonapublic-facingNTPserverthatcannotbeupdatedto4.2.7,addthefollowinglinestoyourntp.conf file:ForIPv4:restrict default kod nomodify notrap nopeernoquery

ForIPv6:restrict -6 default kod nomodify notrapnopeer noquery

Note:requiresarestartofthentpd servicetotakeeffect


Mitigation: Restrict access per network segment

Modifyyourntp.conf torestrictaccess:pernetworksegment(modifyline3tomatchyourLANsettings)*and*perhost(modifyline4):restrict default noquery

restrict localhost

restrict 192.168.0.0 netmask 255.255.0.0 nomodify notrap nopeer

restrict 192.168.1.27

Note:requiresarestartofthentpd servicetotakeeffect


Other NTP mitigations

ConsiderblockinglargeNTPpacketsatnetworkedge• Blockpackets234bytes– 482bytes(thesizeof

MONLIST replies)AdditionalguidelinesforsecuringtheNTPserviceondifferentplatformsandconfigurationsareavailablefromTeamCymru:http://www.team-cymru.org/secure-ntp-template.html


Mitigations for ASNs or ISPs

UsetrafficshapingonUDPservicerequests• Ensures repeated access toInternet resources isnotabusive

MonitorNTPinyournetworkforsignsofamplificationattacks(seehttps://www.us-cert.gov/ncas/alerts/TA14-017A)andgenerateabuseticketsforthesecustomers• Options:takeacustomer’smodemoffline,ornotifyviaphonecall

Notifyyourcustomersofissues,evenifyoucan’ttellthemhowtofixthem• TheymaynotbeintentionallyrunninganNTPserver - trafficmaybe

resultofmalfunctioninghomerouters thatCustomerCarehasnoideahowtoreconfigure


Spoofed Traffic Mitigation: Implement ingress filtering on networks

InternetEngineeringTaskForce(IETF)BestCurrentPractice(BCP)documentsConfigurationchangestosubstantiallyreducepotentialforsourceIPspoofedattacks,themostpopularDDoSattacktype• Howtofilternetworktrafficonnetworktoverify

thesourceaddressofapacket• Rejectpacketswithsourceaddressesthat

arenotreachableviatheactualpacket’spath


IETF BCPs recommended

AllnetworkoperatorsshouldperformnetworkingressfilteringasdescribedintheseBCPs:BCP-38NetworkIngressFiltering• DefeatingDenialofServiceAttackswhichemploy

IPSourceAddressSpoofing:https://tools.ietf.org/html/bcp38

BCP-84IngressFilteringforMultihomed Networks• https://tools.ietf.org/html/bcp84


More info on IETF BCPs

TestwhetheryournetworkcurrentlyfollowsBCP-38usingtoolsfromtheSpoofer Project:https://www.caida.org/projects/spoofer/

AdditionaldetailsabouthowtoimplementBCP-38:http://www.bcp38.info/index.php/Main_Page


Additional mitigations for ISPs

ISPsshouldensurethattheyhaveaDDoSdefensethatismulti-layered,anddesignedtodealwith:• Attacksthatcansaturatetheirconnectivity• “Lowandslow”sophisticatedapplicationlayer

attacks


Verify your fix

VerifyandmonitoryourinfrastructuretoensureitremainssecurebysubscribingtofreereportsfromShadowserver,availableathttps://www.shadowserver.org/wiki/pmwiki.php/Involve/GetReportsOnYourNetwork


Additional NTP resources

https://ntpscan.shadowserver.org/http://openntpproject.orghttp://www.us-cert.gov/ncas/alerts/TA14-017Ahttps://community.rapid7.com/community/metasploit/blog/2014/08/25/r7-2014-12-more-amplification-vulnerabilities-in-ntp-allow-even-more-drdos-attackshttp://www.acunetix.com/blog/articles/ntp-reflection-ddos-attacks

Making the case for implementing mitigations such as BCP 38



Making the case for mitigations

IHelpeveryoneunderstandthelevelofeffortneededtoimprovecyberhealthintheircommunityWhyshouldyouimplementthemitigationsinyourenvironment?1. ItistherightthingtodoasagoodInternet

neighbor2. YourorganizationmaybenexttobeattackedLet’sjointogetherandstopbadguysfromwinning!


Changing risk landscape

Increasedneedtodemonstrate“duecare”o Obtainingcyberinsuranceo Complyingwithriskframeworkstowinbusinesswith

local/nationalgovernmentsandlargecorporations

Ifwe(you!)don’tdoabetterjobofsecuringourowninfrastructureandreducingcyberrisk,governmentregulationmayforceadditionalmandatesand/orpenalties


Anticipated organizational benefits

Increasedproductivity• Fewerserviceinterruptionsandfailures

Improvednetworkperformance• Existingnetworkmore

reliableandresilient,withgreatercapacity

Improvedbrandreputation• Technicalreliabilityand

securityasellingpointtocustomers


More anticipated benefits

• Decreasedbudgetuncertaintyo FewerunanticipatedusagecostsforITo Budgetcanbeusedasplanned,e.g.- upgrading

technicalcapability/capacity,additionalpersonnel,etc.

• Systemadminsmayspendlesstimespenttryingtodealwithunexpectedproblems,whichinturnmayimprovetheirproductivityandreduceunexpectedovertime


What do you need to implement these mitigations?

Commandsandconfigurationdetailsformostimportantmitigationsarepublicallyavailable• Noadditionalsoftwaremustbepurchased• Implementingmitigationsdoesnotrequireanyspecial

knowledge,skills,orabilities

Note:AllmitigationsshouldbecarefullyreviewedinlightofyourspecificbusinessrequirementsandinfrastructureenvironmentbeforeproceedingAllorganizationalchangemanagementprocesses,includingtesting,shouldbefollowed


How long will mitigations take?

Systemadministratorsinsmallerorganizationsneedafewhourspernetworktoinvestigate,implementandverifyupgradeofNTP• Comparableeffortneededforothermitigations,suchas

disablingstatusqueriesandMONLIST functionality,andblockinglargeNTPpacketsatthenetworkedge

ISPsandlargeentitiescantakeadvantageofconfigurationmanagementsystemswithtaskexecution,suchasSaltandAnsible,toautomateadministrationofchanges


How long to implement BCP-38 network ingress filtering?

Smallbusinesses:fromafewminutestolessthananhour

Largerandmorecomplexorganizations:daystoweeks

Bonus:withnorealmaintenance,therecurringcostiseffectivelyzero!

Acknowledgement


CyberGreenwouldliketothanktheexpertswhomadethecreationofthisdocumentpossible:

Writtenby:- LaurinBuchanan,Applied Visions, Inc.– SecureDecisions Division

Contributed andReviewedby:- MattCarothers,CoxCommunications- Baiba Kaskina,CERT.LV- MotoKawasaki,JPCERT/CC- ArtManion,CERT/CC- Yoshinobu Matsuzaki, IIJ- JoeStSauver,Farsight Security- DavidWatson,ShadowServer Foundation

Disclaimer:CyberGreenbelievesthisguidanceandtheadvicefromourexpertsshouldbeofbenefittoanyonemitigatingariskconditions,butitisnotadvicespecifictoanyreaderornetwork.Ultimately,eachreaderisresponsibleforimplementinghisorherownnetwork remediationstrategyandweassumenoresponsibilityorliabilitytherefore.

Formoreinformationaboutriskmitigationbestpractices

pleasecontact:[email protected]


Risk Mitigation for Open NTP OpenNTP... · NTP clients synchronize their time with a local time...

Documents

Transcript of Risk Mitigation for Open NTP OpenNTP... · NTP clients synchronize their time with a local time...