th Quarterly Progress Report for CA NOO244-15-2-0005 ... · Python, with the Pandas, Statsmodels...
Transcript of th Quarterly Progress Report for CA NOO244-15-2-0005 ... · Python, with the Pandas, Statsmodels...
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
5thQuarterlyProgressReportforCANOO244-15-2-0005,IntraseasonalTropicalCycloneForecasting
I. Overview
TheprogressmadebyStatisticalSolutionsLLC(SSL)asoftheendofthefifthquarter(9/30/16)onresearchasdelineatedinthecooperativeagreementN00244-15-2-0005(hereafterreferredtoasCA)isevaluatedasonscheduleandonbudget.Attheendofthequarter,71.0%ofthetotalfundinghasbeenspentover71.0%ofthelifetimeoftheCA.FurtherdetailsonfundingandexpendituresareavailableinSSL’sInterimFinancialReport(SF-425).Forthesakeofbothtransparencyandforachievingthewidestpossibledisseminationofthepublicbenefitresultingfromthisresearch,thisprogressreportwillbemadepublicallyavailableandpostedonSSL’swebsiteat:http://www.statisticalsolutionsllc.com/recentongoing-research.html.Duetothepublicreleaseofthisform,thetotalgovernmentfundingofthecooperativeagreementisomitted,thoughallfinancialdetailswillbefoundintheSF-425,whichalreadyhasbeensenttoDr.TomMurphreeoftheNavalPostgraduateSchool(NPS),theGovernmentSponsor,andtheAdministrativeGrantsOffice.WhencomparingprogresswiththescheduleandmilestonesagreeduponintheCAandasadjustedindiscussionwithDr.TomMurphreeofNPS,progressmadetotheendofthequarterwouldindicatethattheprojectisonscheduleasbothNorthAtlantic(NA)andeasternNorthPacificENPhurricane/tropicalcyclone(TC)forecastingsystemmodelshavebeenconstructedandexperimentationwiththemodelsvs.realworldactivity(formations)isunderway.
II. NorthAtlanticHurricaneModelBuildingA. Background
TheworktobeperformedasdelineatedintheCAincludesthebuildingofastatistical-dynamicalmodelforthepurposeofforecastinghurricaneformationintheNA.AhighlevelviewofhowthesystemfunctionsisoutlinedinFig1.Large-scaleenvironmentalfactor(LSEF)andhurricaneformationdatafortheNAareusedtobuildastatisticalmodelrelating(bymeansoflogisticregression)formationtotheLSEFs.OncethemodelisbuiltitcanbeforcedwithforecastsoftheLSEFstointurngenerateaforecastofthelikelihoodofhurricaneformationatavarietyofdifferentleadtimes.Inordertodotheregression,30years(1985-2014)ofNALSEFandhurricaneformationdatawerecollected,archived,andgridded.TheLSEFsarethoseenvironmentalfactorsthoughttobenecessarybutnotsufficientforTC/hurricaneformationasproposedbyGray[1968,1975]andareessentiallylowshear,warmseasurfacetemperatures,positiveabsolutevorticity,upwardairflow,andhighhumidities).
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
Fig.1TheSSL-NPSstatistical-dynamicalhurricaneformationforecastingsystemfortheNorthAtlantic.LSEFdataisavailableormaybederivedfromtheClimatePredictionCenter’sClimateForecastSystemReanalysis(CFSR)dataset,availableat:https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2#CFS%20Reanalysis%20(CFSR)fordatathrough2010,andhttp://rda.ucar.edu/datasets/ds094.1/#descriptionfordatafrom2011throughcurrent.HurricaneformationdatausedinthemodelbuildingprocessisfromtheNationalHurricaneCenter’s(NHC)HURDAT2datasetavailableat:http://www.nhc.noaa.gov/data/hurdat/hurdat2-1851-2015-070616.txt.Thelocationanddate/timeofhurricaneformationistakenfromthefirstlineofdatafromthedatasetforeachlistedstorm.TheLSEFdatausedformodelbuilding,whichisavailableata0.5°,6-hourlyresolution,wasthenregriddedontoa1.0°,6-hourlygrid.NotethatwechosetouseashigharesolutionaspossibleconsistentwiththeuncertaintyofthetimingofactualcyclogenesisaswellasthespatialresolutionoftheClimateForecastSystemv.2(CFSv2)forcingdatathatisusedtoforcethestatisticalmodelforforecastingpurposes.
B. ModelRebuildingPython,withthePandas,StatsmodelsandPatsymodules,wasusedtoperformlogisticregressionontheLSEFandhurricaneformationdata.ThenecessarybutnotsufficientLSEFs,Gray[1968,1975],werefoundtobestatisticallysignificantataconfidencelevelofover0.999,thoughthehumiditytermwasdroppedduetomulticolinearity.However,followinginitialmodelbuilding(seeprevioustechnicalreportsforthisCA)itwasdeterminedthattheextendedleadforecastssufferedfrominconsistencyinthevaluesofprobabilitiescreatedbythestatisticalmodel,whenforcedwithCFSforecastsoftheLSEFs.Whiletheshortleadforecastsappearedbothskilledandconsistentinappearance,formandbehaviorwithwhathadbeenexpected(andwasobservedwithourENPshortleadforecasts),thelongleadforecaststendedtooverpredicthurricaneformationintheCaribbean/GulfofMexico/WesternNorthAtlantic,andunderpredictformationintheEasternNorth
1
Producesta%s%cal-dynamicalmodeloutput:ensemble-basedforecastsofhurricaneforma%onprobabili%es:1-dayleadforecaststo2monthleadmonthlyoutlooks
Sta%s%calmodelrelatestheLSEFstoTCforma%on
probability
Buildsta%s%calmodelbasedonrela%onshipsbetweenTCforma%onsandLSEFs(basedonNHCbesttracksandCFSR)
Forcesta%s%calmodelwithdynamical,ensemble-based,CFSv2LSEFforecasts
david.sta*s*cal.solu*[email protected]@nps.edu
SSL-NPSDynamical-Sta*s*calTCForma*onForecas*ngSystem
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
Atlantic.Moreover,thereweredifficulttointerpretpeaksofformationpotentialthatseemedbothrandomlyplacedinposition,andnotparticularlygroundedinphysicalplausibility.SSLandDr.Murphreespenttimegoingoverplots,comparingthemtootherbasinforecasts,anddiscussingthevariousbasinmodelsandhoweachperformed.Ultimately,itwasdecidedthatwewouldredotheENPstatisticalmodeltoitsmostbasicpredictors(theLSEFs)andnotincludehigherorderpolynomialandinteractionterms.Thiswasdonesothatwecouldevaluatetheeffectofreducingthenumberofpredictorstosomethingsimplerwhileexploringtheeffectofthereductiononwhatwasconsideredtobeaskilledstatisticalmodel.Dependingontheoutcomeofthemodelsimplification,wewouldthenre-approachmodelingtheNA.TheeffectoftheENPmodelsimplificationwasnegligible.Forecastsatallleadslookednearlyidentical,tothosecreatedusingthemorecomplexmodel,andtherewasnochangeinsystemskill.Emboldenedbytheapparentsuccess(thesimplifiedmodelwasonparwithskillincomparisonwiththemoredetailedmodel)withENPmodelsimplification,SSLcreatedanew,simplifiedstatisticalmodelfortheNA.Thegoalofthesimplificationwastoeliminatepotentialoverfitting,produceamodelthatmoreclearlyagreeswithGray’swork,Gray[1968,1975],andtocreateamodeleasiertoexperimentwithanddiagnose.Withthenew,simplifiedmodel,themagnitudeexcursionsappearedtobeeliminatedintheextendedleadforecasts,butthepreviouslymentionedover/underpredictionproblemwaslittleunchanged.
C. FurtherNAStatisticalModelExplorationAfterconsultationwithDr.Murphree,SSLproceededwithatwo-stepplan.Thefirstwastoexperimentwiththecreationandevaluationofmonthlyclimatologymasks.Thepurposeofthisstepwastoevaluate,onamonthlybasis,theagreement(orlackthereof)oftheNAhurricaneforecastingsystemproducedmonthlyoutlookswithwhatisobservedwithhistoricalNAhurricaneformations.Doingsowouldgiveustheopportunitytobettercomparewheretheforecastingsystemproducedresultsthatwereinroughagreementwithclimatology,andwheretheremightbesubstantialdepartures.ThecreationandusageofmaskshadlittleeffectonNAforecasts,againindicatingthattheproblemismoreoverprediction/underpredictionthanthemodelindicatingformationpotentialexistsinregionswhereitwouldbeunlikelytobetrue.Furthermore,themasksallowfordiscussionaboutthepotentialvalueofcreatingmorethanonestatisticalmodelfortheNA.Usingmorethanonemodel(oratleastparameterizingtheexistingmodel)wasanideathathadbeendiscussedbyDr.MurphreeandSSLimmediatelyaftercreationofthefirstgenerationNAstatisticalmodel.Figure2showsclimatological(mostrecent36yearsforwhichwehavedata)hurricaneformationlocationsfortheNAsuperimposedonourformation
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
mask(JuneandJuly).Notethatthetwomasksarenotaggressive.Otherthateliminatingafewonceinagenerationstorms,themasksroughlyencircletheclimatologicalformationlocations.Themasksclearlyshowtheexistenceoftwodistincthurricaneformationdevelopmentregions(onetothenorthwest,theothertothesoutheast,andwhilethemaskscannotindicatethatdifferentmodelsmustbeused,thedistinctlyseparateregionswouldimplythattheideashouldbeatleastinvestigated.
Fig.2ClimatologymasksforJune(Top)andJuly(bottom)fortheNorthAtlantic,with36yearsofactualTCformationsshown(fromtheNHCHURDAT2dataset).Whiletheremaainingmonthsofthehurricaneseasonarenotshown,Octoberistheonlymonthforwhichthereisnoobvioustworegionsourceofformations.Ultimately,basedonthemask/climatologyplots,coupledwiththemonthlyoutlooks(seeFigure3)producedbythecurrentstatisticalmodel,SSLandDrMurphreebothagreedthatthatthenextstepforNAforecastingshouldbethedevelopmentofatworegionmodel.Dr.MurphreeandSSLalsoagreedthatwehavelearnedmuchaboutNAhurricaneforecasting,thataskilledshortleadproductisinplace,butthatfurtherdevelopmentofanextendedleadNAhurricaneforecastingsystemmaynotbeaneffectiveuseoftimeincomparisontotwoothertasks(discussedinsectionIV.
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
WayForward),especiallyascurrentlywehavelessthanafullseasonofforecaststoworkwithtoevaluateandfinetuneskill.However,throughSeptemberof2016,ourprobabilityofdetectionforthecurrentmonthmonthlyoutlooksforthe2016hurricaneseasonwas83%(10outof12storms)wherethetwomissescamefromwhereonemightexpect,ourunderpredictivesoutheasternNAregion.
Fig.3ExtendedleadhurricaneformationpotentialforecastvalidforJuly2016,issued30June2016,withmaskapplied.Notethehighestprobabilitiesaretothenorthandwest,andthelowesttothesoutheast,despitetheformationoccurringinthesoutheastregion.
III. PreliminaryENPandNAForecastVerificationIncontrasttothechallengespresentedbytheNAwithrespecttoforecastingthepotentialforhurricaneformation,theENPhasbeenverystraightforward.Thesimplifiedmodel,asdiscussedaboveisinuseforallleads(shortandlong)andisusedasinputforSSL/NPS’weeklyrecommendationsregardingTC/hurricaneformationtotheCenterforEnvironmentalPrediction’sGlobalTropicalHazards/BenefitsTechnicalTeleconference(TheGTHBforecastmaybefoundhere:http://www.cpc.ncep.noaa.gov/products/precip/CWlink/ghazards/).Inadditiontothetechnicalteleconferences,whereSSLprovidesshortlead(48houroutlooks)andintermediatelead(week1andweek2outlooks)inputs,SSLmakes1and4dayleadforecasts,aswellascurrentmonthmonthlyoutlooks,onemonthleadmonthlyoutlooks,andtwomonthleadmonthlyoutlooksfortheWesternNorthPacificandtheENPavailableviaitswebpage(http://www.statisticalsolutionsllc.com/tropical-cyclonehurricane-formation-forecasts-2016.html).The1and4dayNAforecastsarealsomadeavailableinthewebsite.Allextendedleadforecastsarearchivedonthe
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
websitesothatusershaveanopportunitytojudgeforthemselvestheskilloftheforecasts.
A. ENPPreliminaryVerificationFigure4showsa1dayleadforecastissued25Aug2016,andvalid26Aug16.Oursynopticallyscaledshortleadforecastsaretypicallyhighlyskilled;somuchsothatapotentialfutureuseofourshortleadforecastscouldbeasasourceofunbiasedformationreanalysis.
Fig.41dayleadleadTCformationpotentialforecastvalidfor26Aug2016,issued25Aug2016,withverifyingTCindicated(dateandlocationasindicatedfromthefirstissuedTropicalCycloneFormationAlert).Notetheregionofelevatedprobabilitiesatroughly18°Nand115°W.Thisregionisactuallythelocationof13E,whichformed2daysearlieronthe24thofAugust.Beyondthehighskilloftheshortleadproductsatforecastingformation,theshortleadforecastsarealsohighlyskilledatforecastingfuturelocationsofTCs.InFigure4weseeata1dayleadtheforecastedpositionof13E,whichhadformedafewdayspriortotheindicatedformationof14E.MoreonanecdotalevidencethattheshortleadforecastsmayhaveuseforforecastingtrackswillbediscussedinSectionIII.BNAPreliminaryVerification.Figure5showsanextendedlead(currentmonthmonthlyoutlook)forecastfortheENP,issuedlateJuly,andvalidforAugust2016.InFigure5weseethat5TCsformed,with1miss(atabout130°W),1thatisjustoutsidethecontouredareabutstilljudgedasahit(weusea2.5°neighborhoodwhenscoringtoaccountforgrid
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
errors,CFSspatialerrorsandpositionalerrorsinthelocationoftheformation),and3solidhits.Theforecastshownistypicalfor2016,andasmentionedpreviouslymaybefoundwiththeother2016(and2015)archivedforecastsforuserevaluationofskill.Notshownarethelongerlead(1and2month)forecastsofformationpotential,buttheyareverysimilarinappearanceandskilltothecurrentmonthmonthlyoutlook.Whilenotarchivedonthewebsiteyet,thelongerleadforecastswillbemadeavailabletoosothatpotentialuserscangaintrustinasystemwhoseperformancetheycanjudgeforthemselves.
Fig.5CurrentmonthmonthlyoutlookofTCformationpotentialvalidforAugust2016,issued31July2016.Fromhttp://www.statisticalsolutionsllc.com/eastern-north-pacific-formation-forecasts.html
B. NAPreliminaryVerificationWehavediscussedinSectionII.CsomeoftheissuesregardingtheNAhurricaneformationpotentialforecastingsystem.However,sincetheissuesdidnotappeartoadverselyeffectshortleadforecastperformance,shortleadforecastsweremadeavailableontheSSLwebsite.Figure6showsthe1dayleadforecastvalidfor26Sep2016,thedaythathurricaneMatthew(14L)formedintheNA.Matthew’sformationlocationforthatdayisindicatedwiththepinkdot.
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
Fig.61dayleadofhurricaneformationpotentialvalidfor26Sep2016,issued25Sep2016.Thedisturbancecenteredon18°N,70°Wwasalsoofinterest,butenvironmentalconditionsthefollowingdaywerenotfavorabletoformation,andtheformationneveroccurred.Finally,Figure7(a-d)showsaseriesofshort(4day)leadforecastsfortheNAissuedandpostedonthewebsitedailybySSLonceitbecameapparentthatMatthewwouldimpacttheUS.Whiletheshortleadforecastswerenevercreatedtobeusedastrackforecasttools,Figure7correctlyindicatedthegeneraldirectionandspeedofMatthewatleadtimesusefultopeopleinitspath.
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
Fig.74dayleadforecastsofhurricaneformationpotentialvalidfor5-8Oct2016,andissued1-4Oct2016respectively.Consideringtheuncertaintyinforecastsavailableatthetimeata4daylead,thefourdayforecastswerequitereasonable.
C. SummarybyBasinSSLhaskeptrunningverificationfilesforbothbasinsthroughouttheTC/hurricaneseason.Theresultsarepreliminary,usingthefirstpositions/datesgivenintheirrespectivetropicalcycloneformationalert(TCFA),butthesepositions/datesfrequentlychangeoncethebesttracksfortheseasonareissued.Note,differencesbetweenpreliminarydataandbesttrackdatahistoricallydonotsignificantlyaltertheverificationresults,andwillendupimprovingresultsattimesaswell.TheENPresultsaregoodenoughthesomemodificationoftheminimumcontourusedmayneedtobeexperimentedwithovertheoff-seasonandoncethebesttrackscomeoutsothatprobabilityofdetect(POD)maybereducedinordertoreduceENPpercentageofcontouredarea(PCA).PCAisanSSLcreatedmeasureoftheamountofoceanareatheforecastsindicateasfavorabletoformationdividedbythatamountoftheoceancapableofsupportingTC/hurricaneformationasindicatedbyclimatology.ThusPCAwouldindicateforecastsystemdiscrimination,withthesmallerthePCAvaluetranslatingtosmallerareasoutofthebasinthatourforecastsindicateasformationsupportive.Figure8showsPODandPCAfortheNAandtheENP.Bothareskilled,evenwiththeissuesidentifiedfortheNA.Ideally,ourforecastswouldhaveroughlyan80%PODanda10%PCA.FortheNA,Figure8indicatesanacceptablePOD,butatoohighvalueofPCA(dominatedbythe
a)
c)
b)
d)
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
overpredictionissue).FortheENP,thereispotentialtoreducethePCAtoour10%target,byacceptingareducedPODinreturn.
Fig.8VerificationstatisticsforSSL’sNAandENPcurrentmonthmonthlyoutlooksforthe2016seasontodate.Thelongerleadforecasts(1and2monthleadmonthlyoutlooks)areverysimilarinskill.
IV. PublicBenefitKeyattributesofprojectsfundedbycooperativeagreementsarethattheresultsoftheresearcharetoprovideapublicbenefit,andthattheresultsaretobemadepublicallyavailablesothatthepublic,scientificcommunity,andothersmaytakeadvantageoftheresearchresults.Withoutdoubt,improvedunderstandingofTC/hurricaneformationandtheinfluenceoftheenvironmentonthoseformationsisofgreatbenefittothepublic.Tomakepublictheresearcheffort,andtheongoingresultsofthatresearch,onOctober14th,SSL,withDr.Murphreeasaco-author,presented“VisualizationandVerificationofExtendedLeadForecastsofHurricane/TropicalCycloneFormation”atthe2016Cincinnati-DaytonInformsTechnicalSymposium,outliningourTC/Hurricaneresearchtodate,ourresults,andthewayahead.TheabstractisavailableattheCincinnati-DaytonChapterINFORMSwebsitehttps://www.informs.org/Community/Cincinnati-Dayton-Chapter/Symposium.Additionally,allexperimentalforecastproducts,includingkeys,legends,andbackgroundmaterialsarepostedontheSSLwebsiteat:http://www.statisticalsolutionsllc.com/tropical-cyclonehurricane-formation-forecasts-2016.html.DuringearlyOctober,whileMatthewwasapproachingandimpactingtheUS,SSLupdateditsNApagedaily.Furthermore,SSL,onaweeklybasis,emailsparticipantsintheClimatePredictionCenter’sGlobalTropicalBenefits/Hazardstechnicalteleconferencewithtextualforecastinputandupdates
00.10.20.30.40.50.60.70.80.91
NA ENP
PreliminaryLongLeadVeriVicationbyBasin
POD
PCA
SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016
[email protected]://www.statisticalsolutionsllc.com/
theexperimentalforecastsonit’swebsite.SSLparticipatesintheteleconferenceandusestheinteractionwithotherscientistsasanopportunitytolearnmoreaboutformationforecasting,andaswellastheskillofourresearch.Finally,TC/hurricanerelatedresearchperformedtoenableSSL’sgreaterunderstandingofTCactivity(suchasTC/hurricaneformationclimatescience),thoughnotnecessarilyrequiredorfundedbytheCA,isalsomadeavailableontheSSLwebsite(http://www.statisticalsolutionsllc.com/tchurricane-climate-science.html).AmeasureofthepublicbenefitisthatSSL’spageviewsquadrupleeveryMonday(thedayofthetechnicalteleconference)to40pagesperday.Finally,forpurposesofbothtransparencyandpublicawareness,thisquarterlyprogressreport,pastandfuturequarterlyreports,andthefinaltechnicalreportforthiscooperativeagreementwillalsobemadeavailabletothepubliconSSL’swebsite.
V.WayForwardForthenextquarter,SSLwillfocusontwomainefforts.ThefirstistobegintheprocessofresearchingthestateoftheartforTC/hurricanetrackforecastingandoutlineoutanapproachtodevelopastatistical-dynamicalforecastingsystemfortracks.Thisdeviatesfromtheworkoriginallyincludedinthestatementofwork,butwasmutuallyagreeduponbetweenDr.MurphreeandSSLasthenexttasktoworkonfollowingarecommendationfromtheJointTyphoonWarningCenter.Thesecondeffortwillfocusonrobustverificationofthe2016forecasts(allbasins)toquantifytheexperimentalforecastingsystemperformance.Thisisakeycomponenttotheoverallforecastingresearcheffortasadvancesmadeaspartofourresearchwillbeignoredunlessitcanbeshownthatimprovementhavebeenmadetothestateofthescience.
VI.ReferencesGray,W.M.(1968),GlobalViewoftheOriginoftropicalDisturbancesandStorms,MonthlyWeatherReview,96,669-700.Gray,W.M.(1975),TropicalCycloneGenesisintheWesternNorthPacific,NavalAirSystemsCommand,WashingtonDC,20361.