FogNetworking:AnOverviewonResearchOpportunities

MungChiangArthurLeGrandDotyProfessorofElectricalEngineering

PrincetonUniversity

December2015A.IntroductionThepast15yearshaveseentheriseoftheCloud,alongwithrapidincreaseinInternetbackbonetrafficandmoresophisticatedcellularcorenetworks.Therearethreedifferenttypesof“Clouds:”(1)datacenter,(2)backboneIPnetworkand(3)cellularcorenetwork,responsibleforcomputation,storage,communicationandnetworkmanagement.NowthefunctionsofthesethreetypesofCloudsare“descending”tobeamongorneartheendusers,i.e.,totheedgeofnetworks,as“Fog.”WetakethefollowingasaworkingdefinitionofFogNetworks:“Itisanarchitecturethatusersoneoracollaborativemultitudeofend-userclientsornear-useredgedevicestocarryoutasubstantialamountofstorage,communicationandmanagement.”Architectureallocatesfunctionalities.EngineeringartifactsthatmayuseaFogarchitectureinclude5G,home/personalnetworking,andtheInternetofThings.TohighlightthecontrastbetweenCloudandFog,wecancompareFogarchitecturewiththecurrentstandardpracticealongthefollowingthreedimensions:

• Carryoutasubstantialamountofstorageatorneartheend-user(ratherthanstoredprimarilyinlarge-scaledatacenters).

• Carryoutasubstantialamountofcommunicationatorneartheend-user(ratherthanallroutedthroughthebackbonenetwork).

• Carryoutasubstantialamountofmanagement,includingnetworkmeasurement,controlandconfiguration,atorneartheend-user(ratherthancontrolledprimarilybegatewayssuchasthoseintheLTECore).

ItisnotabinarychoicebetweenCloudandFogeither:theyformamutuallybeneficial,inter-dependentcontinuum.Itisacontinuum:tothewearabledevices,amobilephonemaybeviewedastheCloud.Theyareinter-dependent,e.g.,coordinationamongdevicesinaFogmayrelyontheCloud.Theyarealsomutuallybeneficial:certainfunctionsarenaturallymoreadvantageoustocarryoutinFogwhileothersinCloud.TheinterfacebetweenCloudandFogisindeedakeyaspectofFogR&D.Fogarchitecturesmaybefullydistributed,mostlycentralizedorsomewherein-between.Theymayrelyonhardware,software,orcombinationofboth.Thecommon

denominatoristhattheydistributetheresourcesandservicesofcomputation,communication,control,andstorageclosertodevicesandsystemsatorneartheusers.Thereisalreadyalargeandincreasingrangeofsuchclientandedgedevicestoday:fromsmartphonestotabletsandfromset-topboxestosmallcellbasestations.Someofthemhavebecomedramaticallymorepowerfulincomputation,communication,storageandsensingcapabilitieswithinthepastseveralyears,whilestilllimitedinotherwayssuchasenergysupply.AsdifferentsegmentsofInternetofThings(IoT),InternetofEverything(IoE)orInternetofMe(IoM)starttotakeoff,e.g.,consumer,wearable,industrial,enterprise,automobile,healthcare,building,energy,etc.,therewillbeanevenmoreimpressivesurgeinthediversity,volume,andcapabilitiesofsuch“Fognodes.”Indeed,Fog’sfirstapplicationcontextswasconnectedcarsasproposedseveralyearsago[1].Information-transmittinglight-bulbs,computersonastick,andbutton-sizedRFtunersfurtherunderscoretheopportunitiesofFognodes.Concurrently,wirelessnetworksareincreasinglyusedlocally,e.g.,withinconnectedcars,smart-buildings,andpersonalbody-areanetworks;anddatageneratedlocallyisincreasinglyconsumedlocally.Whatcanacrowdofsuchdevicescollectivelyaccomplish,throughadense,distributedandunder-organizednetworkontheedge?Whatcantheyaccomplishtoenableultra-lowanddeterministiclatency,datamininginrealtimewithstreamingdata,andcyberphysicalnetwork’sactuationandcontrolfunctionswithinstringenttemporalconstraints?Ithasbecomebothfeasibleandinterestingtoaskthequestion:“Whatcanbedoneonthenetworkedge?”Forexample,whatistheset-topboxinyourlivingroomreplacesthedeepinspectionboxesinoperatornetwork?Andthedashboardinyourcarisyourprimarycachingdevice?WhatifyourphoneandphonesofotherscollectivelyactascontrollersimilartoanS-GWorPDN-GW?Whiletheanswerstoquestionssuchasthesemaynotbepositiveinallcases,ithasfinallybecomeworthwhileaskingthequestions.Wemaycontrasttheseclients,edgedevices,and“things”withthelarge,expensive,hard-to-innovate“boxes”intheCloud:S-GWandPDN-GWinLTEcore,largeserversandswitchesinsideadatacenter,andmetroandcoreroutersinwide-area-networkbackbone.Thetraditionalviewisthatedgeusesthecorenetworkanddatacenters.TheFogviewisthatedgeisthecorenetworkandadatacenter.Inthetensionbetweenthe“brick”versusthe“click,”thependulumisstartingtoswingbacktowardthe“brick,”wherephysicalinteractionswiththecyber-systemisbecomingonceagainimportant.

B.WhyFogandWhyNow?WhywouldwebeinterestedintheFogviewnow?Therearefourmainreasons:

1. Time:Realtimeprocessingandcyber-physicalsystemcontrol.Edgedataanalytics,aswellastheactionsitenablesthroughcontrolloops,oftenhave

stringenttimerequirementandcanonlybecarriedoutontheedge,“hereandnow.”ThisisparticularlyessentialforTactileInternet:thevisionofmillisecondreactiontimeonnetworksthatenablevirtual-reality-typeinterfacesbetweenhumansanddevices.

2. Cognition:AwarenessofClient-centricobjectives.Followingtheend-to-endprinciple,someoftheapplicationscanbebestenabledbyknowingtherequirementsontheclients.ThisisespeciallytruewhenprivacyandreliabilitycannotbetrustedintheCloud,orwhensecurityisenhancedbyshorteningtheextentoverwhichcommunicationiscarriedout.

3. Efficiency:Poolingoflocalresources.Therearetypicallyhundredsofgigabytessittingidleontablets,laptopsandset-topboxesinahouseholdeveryevening,oracrossatableinaconferenceroom,oramongthepassengersofapublictransitsystem.Similarly,idleprocessingpower,sensingabilityandwirelessconnectivitywithintheedgemaybepooledwithinaFognetwork.

4. Agility:Rapidinnovationandaffordablescaling.Itisusuallymuchfasterandcheapertoexperimentwithclientandedgedevices.Ratherthanwaitingforvendorsoflargeboxesinsidethenetworktoadoptaninnovation,intheFogworldasmallteammaytakeadvantagesofsmartphoneAPIandSDK,proliferationofmobileapps,andofferanetworkingservicethroughitsownAPI.

Therearealsotwomore“defensive”reasonsfortheriseoftheFog:Feasibilitytooperateonencryptedandmultipathtraffic.Amajortrendthesedaysisthatbythetimetrafficleavestheedgeandentersthebackbonenetwork,itisalreadyencryptedandpossiblytraversingmultiplepaths,makingitexpensiveifnotimpossibletooperateonsuchdata.InUnitedStates,implicationsofFCCTitleIIRuling.TheFCCvoteinFebruary2015toclassifyInternetservices,includingmobileservices,asa“utility”underTitleIIregulatorymandate,mayfurtherpushnetworkinnovationtotheedgeintheUS.Anewregulatoryenvironmentdoesnotmeannetworkscannotbeengineeredandmanagedanymore,butwemayneeddifferentvantagepointsofcontrol:notfrominsidethenetworkbutfromaroundtheendusers.Forexample,todaynetworkoperatorscanpickwhichlane(WiFi,Macro-cellular,Femtocell)auserdeviceshouldbein.Sincedifferentlaneshavedifferentspeedsanddifferentpaymentsystem/amount,suchpracticemaynotbeallowedanymoreintheUS.Instead,weneedtobetterdesignsystemwhereeachuserdevicemustchoosewhichlanetobeinforitself.ThechallengeresultingfromTitleIIregulationisa“hangingsword”thatchillsthedeploymentofnetworkinfrastructureinnovations,asrisk-Returnbalancenowtipstowards“keepthenetworkasis.”However,aslongasthegovernmentdoesnotprohibitend-userchoices,thenwecanrunnetworkingfromtheedge,throughclient/home-drivencontrol/configuration.

C.CaseStudies

Figure1:DataplaneandcontrolplaneofFognetworksenabledifferentapplicationsArchitecturalR&Dasksthequestionof“whodoeswhat,atwhattimescale,andhowtoputthemodulesbacktogether?”Asanarchitecture,Fognetworkingconsistsofbothdataplaneandcontrolplane,eachwitharapidlygrowingnumberofexamplesacrossprotocollayersfromthephysicallayertotheapplicationlayer:

• ExamplesofDataplaneofFog:o Poolingofclientsidlecomputing/storage/bandwidthresourcesandlocal

contento Contentcachingattheedgeandbandwidthmanagementathomeo Client-drivendistributedbeam-formingo Client-to-clientdirectcommunications(e.g.,FlashLinQ,LTEDirect,WiFi

Direct,AirDrop)o Cloudletsandmicrodata-centers

• ExamplesofControlplaneofFog:

o OvertheTop(OTT)contentmanagemento Fog-RAN:Fogdrivenradioaccessnetworko Client-basedHetNetscontrolo Client-controlledCloudstorageo Sessionmanagementandsignalingloadattheedgeo Crowd-sensinginferenceofnetworkstateso Edgeanalyticsandreal-timestream-mining

Edge-drivendataplane

Edge-drivenControl-plane

5GMobile

InternetofThings

Dataanaly<cs

Cyber-Physical

Data-planeofFoghasbeenmoreextensivelystudied,e.g.,[2].Inthefollowing,wehighlightafewparticularcasesthatillustratethepotentialandchallengesofFogcontrolplane,suchastheinference,control,configurationandmanagementofnetworks:Case1:Crowd-sensingLTEstates(incommercialdeployment).Throughacombinationofpassivemeasurement(e.g.,RSRQ),activeprobing(e.g.,packettrain),applicationthroughputcorrelationandhistoricaldatamining,acollectionofclientdevicesmaybeableto,inreal-timeandusefulaccuracy,inferthestatesofaneNBsuchasthenumberofResourceBlocksused[3].Case2:OTTnetworkprovisioningandcontentmanagement(incommercialdeployment).Thetraditionalapproachtoinnovatingnetworksistointroduceanotherboxinsidethenetwork,possiblyavirtualizedboxbutaboxnonetheless.Fognetworkingdirectlyleveragesthe“things”andphonesinstead,andremovesthedependenceonboxes-in-the-networkaltogether.WithSDKssittingbehindappsonclientdevices,throughtaskssuchasURLwrapping,contenttagging,locationtracking,behaviormonitoring,networkservicescanbeinnovatedmuchfaster.

Figure2:SDKsittinginsideclientscanenablenetworkinferenceandconfiguration

Case3:Client-basedHetNetscontrol(in3GPPstandards).Coexistenceofheterogeneousnetworks(e.g.,LTE,femto,WiFi)coexistenceisakeyfeatureincellularnetworkstoday.Ratherthanthroughnetworkoperatorcontrol,eachclientcanobserveitslocalconditionsandmakedecisiononwhichnetworktojoin.Throughrandomizationandhysteresis,suchlocalactionsmayemergegloballytoconvergetoadesirableconfiguration[4].

Figure3:Co-existenceofheterogeneousnetworksmaybemanagedinpartbyclients

Case4:Client-controlledCloudstorage(inbetatrial).Bydecouplingmassivecheapstorage(intheCloud)fromclientsidecontrolofprivacy(intheFog),wecanachievethebestofbothworlds.Forexample,byspreadingthebytes,inaclientshimlayer,ofagivenfileacrossmultipleCloudstorageproviders,itcanbeassuredthatprivacyofthedataismaintainedevenifencryptionkeyisleakedbyanygivenCloudprovider[5].

Figure4:ShredandSpread(CYRUSproject)storesinCloudbutcontrolsinFog

Case5:Real-timestreammining(inbetatrial).ConsidervirtualrealitytasksassociatedwithGoogleGlass.SomeoftheinformationretrievalandcomputationtasksmaybecarriedoutontheGlass(a“wearablething”),someontheassociatedphone(aclientdevice),someonthehomestorage(anedgedevice),andtherestintheCloud.An

architectureofsuccessiverefinementmayleverageallofthesedevicesatthesametime,withanintelligentdivisionoflaboracrossthem[6].Case6:BorrowingbandwidthfromneighborsinD4D(inbetatrial).Whenmultipledevicesbelongingtothesameperson,torelativesortoemployeesofthesamecompanyarenexttoeachother,onecanasktheotherstosharetheirLTE/WiFibandwidthbydownloadingotherpartsofthesamefileandtransmitting,viaWiFiDirect,clienttoclient[7].

Figure5:IdleresourcesinclientdevicescanbepooledinD4Dformoreefficientuse

Case7:Bandwidthmanagementathomegateway(inbetatrial).Byadaptingthehomeset-topbox/gateway,thelimitedbroadbandcapacityisallocatedamongcompetingusersandapplicationsessions,accordingtoeachsession’spriorityandindividualpreferences.Aprototypeonacommodityrouterdemonstratesascalable,economicalandaccuratecontrolofcapacityallocationontheedge[8].Case8:Distributedbeam-forming(inlabdemonstration).Fogcanalsohappeninthephysicallayer,forexample,byexploitingmulti-userMIMOtoimprovethroughputandreliabilitywhenaclientcancommunicatewithmultipleWiFiaccesspoints.Foruplink,wecanusemulti-userbeam-formingsothattheclientcansendmultipledatastreamstomultipleAPssimultaneously.Fordownlink,wecanuseinterferencenullingsothattheclientcandecodeparallelpacketsfrommultipleAPs.Thesecanbedoneentirelyontheclientside[9].Someoftheabovecasestudiesarecoretopicsinwhatmanypeopleimaginewouldpartiallydefine“5G:”HetNets/smallcell/densification,overthetopserviceprovisioning,cognitiveradioandcrowd-sensing.Othercasestudiespointtowardarchitectural

thinkingforIoTservices,questionsaboutownership,controlandvisibilityofpersonalareanetworks,suchas“shouldAppleWatchandthelikehavetheirowndataplan?”thatwillhelpdefinethebalanceofpowerbetween“AT&T”sand“Apple”softheworld.Ifthenetworkinoraroundtheendusershavealogicaltopologythatlookslikeastar,withafixedgateway(e.g.,iPhone),thevisibility,control,andvalue-addedbynetworkoperatorswillbedrasticallydifferentthaninthealternativescenariowherethegatewaysaredynamicallychosenortheThingscansometimeshavedirectcommunicationpathswithoutagateway.Formorereferencesfortheseexamplesandmoreexamples,pleaseseeaninitiallistofover100recentpublicationsoneightdifferenttopicsunderFogNetworkingathttp://fogresearch.orgD.OpenQuestionsandResearchChallengesAsistypicalofanyemergentareaofR&D,manyofthethemesinFogNetworkingarenotcompletelynew,andinsteadareevolvedversionofaccumulatedtransformationsinthepastdecadeortwo:

• Comparedtopeer-to-peer(P2P)networksinthemid-2000s,Fogisnotjustaboutcontentsharing(ordataplaneasawhole),butalsonetworkmeasurement,control&configuration,andservicedefinition.

• Comparedtomobileadhocnetwork(MANET)researchadecadeago,wehavemuchmorepowerfulanddiverseoff-the-shelfedgedevicesandapplicationsnow,togetherwiththestructure/hierarchythatcomeswithcellular/broadbandnetworks.

• Comparedtogenericedge-networkinginthepast,Fognetworkingprovidesanewlayerofmeaningtotheend-to-endprinciple:notonlydoedgedevicesoptimizeamongthemselves,buttheycollectivelymeasureandcontroltherestofthenetwork.

Alongwithtwoothernetworkarchitecturethemes:ICNandSDN,eachwithalongerhistory,Fogisrevisitingthefoundationofhowtothinkaboutandengineernetworks,i.e.,howtooptimizenetworkfunctions:whodoeswhatandhowtogluethembacktogether:

• Information-CentricNetworks:Redefinefunctions(tooperateondigitalobjectsratherthanjustbytes)

• Software-DefinedNetworks:Virtualizefunctions(throughcentralizedcontrolplane)

• FogNetworks:Relocatefunctions(tothenetworkedge)

WhileFognetworksdonothavetohaveanyvirtualizationortobeinformation-centric,onecouldalsoimagineaninformation-centric,software-definedFognetwork(sincethesethreebranchesarenotorthogonal).Asinanemergentareainitsinfantage,thereisnoshortageofchallengingquestionsinFognetworking,someofwhichcontinuefromearlierstudyofP2P,MANETandCloud,whileothersaredrivenbyaconfluenceofrecentdevelopmentsinnetworkengineering,devicesystemsanduserexperience:

• Cloud-Foginterface:Thefundamentalquestionofarchitectureis“whodoeswhat,atwhattimescale,andhowtoputthembacktogether?”CloudwillremainusefulasFogarises.ThequestioniswhattasksgotoFog(e.g.,thosethatrequirereal-timeprocessing,enduserobjectivesorlow-costleverageofidleresources)andwhatgotoCloud(e.g.,massivestorage,heavy-dutycomputation,orwide-areaconnectivity),andwhatwillbetheFog-CloudandFog-Foginterfaces:thespecificationofinformationpassage,fromitsfrequencytogranularity,acrosstheseinterfaces.

• Interactionswithclient/thinghardwareandOS:Oncetheactionsaretakenon

theclientsorthings,theinterfacewiththeiroperatingsystemsandhardwarebecomeessential.MorethanjustusingD4Dforpoolingidleedgeresources,thereisalsothepossibilityofspecializedprotocolstackjustfornetworkingwithinanedge.

• Trustworthinessandsecurity:WhileFogmayhelpenhancesecurityinsome

cases,itmaypresentnewsecuritychallengesinothercases.Giventhatitisofteneasiertohackintoclientsoftware,perhapssecurityathardwarelevelonclientdevices.Atthesametime,becauseoftheproximitytoendusersandlocalityontheedge,nodesinFognetworkscanoftenactasthefirstnodeofaccesscontrolandencryption,providecontextualintegrityandisolation,andenablethecontrolofaggregatingprivacy-sensitivedatabeforeitleavestheedge.

• Incentivizationofclientparticipation:Sometimesitisnottoomanyun-

trustworthyclientsthatcreateconcernbuttoofewclientswillingtoparticipate.Marketsystemsandincentivemechanismswillbecomeuseful.

• Convergenceandconsistencyarisingoutoflocalinteractions:Typicalconcernsof

distributedcontrol,divergence/oscillationandinconsistencyofglobalstates,becomemoreacuteinamassive,under-organized,possiblymobilecrowdwithdiversecapabilitiesandvirtualizedpoolofresourcessharedunpredictably.Usecasesinedgeanalyticsandstreamminingprovideadditionalchallengesonthisrecurrentchallengeindistributedsystems.

• Ingeneral,thetradeoffbetweendistributedandcentralizedarchitectures,betweenwhatstaysonlocalandwhatgoesonglobal,andbetweencarefulplanningandresiliencethroughredundancy.Onthistopic,weneedtobesensitivetotheopportunitywheremanydifferent,ordynamic,logicaltopologiesmayarisefromthesameunderlyingphysicalconfigurationofaFognetwork.

Toaddresstheabovechallenges,weneedboth

• Fundamentalresearch,acrossnetworking,devicehardware/OS,pricing,HCIanddatascienceasshowninFigure6,and

• Industry-academiainteractions,asexemplifiedintheOpenFogConsortium,aglobal,non-profitconsortiumlaunchedinNovember2015withfoundingmembersfromARM,Cisco,Dell,Intel,MicrosoftandPrincetonUniversity.

Figure6:Arangeoffieldsisneededforaninter-disciplinarystudyofFognetworksIndeed,FogNetworkingisstartingtoshapethefutureofthebalanceofpoweranddistributionofdrivinginnovationacrosstheentireindustryfoodchain,includingthefollowing:

• Enduserexperienceprovider(e.g.,GE,Toyota…)• Networkoperators(e.g.,AT&T,Verizon,Comcast…)• Networkequipmentvendors(e.g.,Cisco,Nokia,Ericsson,Huawei…)• Cloudserviceproviders(e.g.,VMWare,Amazon…)• Systemintegrators(e.g.,IBM,HP…)• Edgedevicemanufacturers(e.g.,Linksys…)• Client/IoTdevicemanufacturers(e.g.,Dell,Microsoft,Apple,Google…)• Chipsuppliers(e.g.,Intel,ARM,Qualcomm,Broadcom…)

2015isaninterestingyeartostartsystematicallyexploringwhatFogmightlooklikeandthedifferencesitwillmakeintheworldofnetworkingandcomputinginthenext15years.Acknowledgements:Theauthorisgratefulfortheinspiringconversationswithmanycolleaguesinindustryandacademia,especiallyFlavioBonomi,RussellHsing,BharathBalasubramanian,AakankshaChowdhery,YanSchvartzshnaider,SangtaeHa,JunshanZhang,RajSavoor,JohnSmee,ChonggangWangandrepresentativesofARM,Cisco,Dell,IntelandMicrosoftinOpenFogConsortium.

Reference:(morereferencescanbefoundathttp://Fogresearch.org)[1]F.Bonomi,R.Milito,P.NatarajanandJ.Zhu,“Fogcomputing:AplatformforInternetofThingsandAnalytics,”inBigDataandIoT,Springer2014.[2]M.Satyanarayanan,P.Bahl,R.Caceres,andN.Davies,“ThecaseforVM-basedCloudletsinmobilecomputing,”IEEETransactionsonPervasiveComputing2009.[3]A.Chakraborty,V.Navda,V.N.PadmanabhanandR.Ramjee,“CoordinatingcellularbackgroundtransfersusingLoadSense,”ProceedingsofACMMobicom2013.[4]E.Aryafar,A.Keshavarz-Haddard,M.WangandM.Chiang,“RATselectiongamesinHetNets,”ProceedingsofIEEEINFOCOM2013.[5]J.Chong,C.Joe-Wong,S.HaandM.Chiang,“CYRUS:Towardclient-definedCloudstorage,”ProceedingsofEuroSys2015.[6]L.CanzianandM.vanderSchaar,“Realtimestreammining:Onlineknowledgeextractionusingclassifiernetworks,”IEEENetworksSpecialIssueonNetworkingforBigData2014.[7]Z.Zhang,J.ZhangandL.Ying,“Multimediastreamingincooperativemobilesocialnetworks,”Preprint.[8]F.M.F.Wong,S.Ha,C.Joe-Wong,Z.LiuandM.Chiang,“Mindyourownbandwidth:Adaptivetrafficmanagementonnetworkedge,”ProceedingsofIEEEIWQoS2015.[9]Y.Du,E.Aryafar,J.CampandM.Chiang,“iBeam:Intelligentclient-sidemulti-userbeamforminginwirelessnetworks,”ProceedingsofIEEEINFOCOM2014.