Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream...
Transcript of Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream...
![Page 1: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/1.jpg)
ManySolutions,OneGoal.
SplunkAppforStreamDavidShpritz,ApluraLLC.BaltimoreAreaUserGroup
3/21/2016
![Page 2: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/2.jpg)
ManySolutions,OneGoal.
Agenda
• WhatisSplunkAppforStream?• WhyuseSteam?• WheretouseStream?• DeployingStream• Questions
![Page 3: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/3.jpg)
ManySolutions,OneGoal.
WhatIsSplunkAppforStream?
![Page 4: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/4.jpg)
ManySolutions,OneGoal.
Somehistory
• SplunkacquiresCloudmeter,December2013• RenamedSplunkAppforStream• ReleasedwithSplunk6.0(August,2014)• Nowatversion6.4.3(January,2016)
![Page 5: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/5.jpg)
ManySolutions,OneGoal.
PurposeofStream
• Rapiddeployment• Rapidconfiguration• Capturewiredata• Interpretwiredata• Summarize/filter/aggregate• Index• KindoflikeBro,butmoreSplunky,andGUI
![Page 6: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/6.jpg)
ManySolutions,OneGoal.
Sowhatcanwecapture?
• Well,wearen’treallycapturingandindexingpackets• Forwarderscapturepackets,analyzetheprotocols• Whatprotocols(alot):• TCP/UDP• Applicationprotocols(HTTP,databases,email,filesharing,chat)• About30differentprotocolscurrently• http://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/Whattypeofdatadoesthisappcollect
![Page 7: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/7.jpg)
ManySolutions,OneGoal.
WhytouseSplunkStream
![Page 8: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/8.jpg)
ManySolutions,OneGoal.
Nologs
• Noownership• Novisibility• Noforwarders(asendpoints)• Nologgingoptions
![Page 9: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/9.jpg)
ManySolutions,OneGoal.
Poorlogs
• Loggingishighoverhead• Logsmakenosense• Keyeventsarenotlogged
![Page 10: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/10.jpg)
ManySolutions,OneGoal.
Cloud
• Manycloudservicesdon’tofferlogsonthings• Nochokepoints
![Page 11: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/11.jpg)
ManySolutions,OneGoal.
VS.BroIDS
• LowerCPUusage• LowerRAMusage• MoreOSsupport(Linux,Windows,OSX)But• Hightrafficrequiresnetworkpacketbrokers(Gigamon,Ixia,etc.)• Can’twriteyourworkinterpreters• NoSnortrules
![Page 12: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/12.jpg)
ManySolutions,OneGoal.
Otherfeatures
• Filtering• Aggregation• EphemeralStreams(shortterm)• SSLdecrypt• Centralizedmanagement• IntegrationwithES• StartastreamafterNotableevent• Protocolanalysisdashboards
![Page 13: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/13.jpg)
ManySolutions,OneGoal.
DataEstimation
• “WhatifIturnthison?”• Tellsyouhowmuchdatayouwouldbeindexing
![Page 14: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/14.jpg)
ManySolutions,OneGoal.
Granularcontrolofthedata
• Notjustwhichsystems,butalsowhatdata,whichfields
![Page 15: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/15.jpg)
ManySolutions,OneGoal.
GlobalFilters
• Filteroutnoisefromtheenterprise• Thingslikevulnerabilityscanners
![Page 16: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/16.jpg)
ManySolutions,OneGoal.
DistributedForwarderManagement• Setupgroupsforcapture• Usesregexforgroupsonthe“ForwarderID”• ForwarderIDisconfigurableviaXMLconfig file• Yes,it’sanotherSplunkdeployment/controlmechanism
![Page 17: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/17.jpg)
ManySolutions,OneGoal.
WheretouseSplunkStream
![Page 18: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/18.jpg)
ManySolutions,OneGoal.
DedicatedStreamForwarders
• SenddataoffofaswitchSpanorTap• ToolslikeGigamon,Ixia,Etc.• Youneedtheseforreallybigpipestospreadthelove
• Purposebuilt• HigherCPUandRAM• Betternetworkcards
• AlsoagoodoptionisyouwanttoperformSSLdecrypt• Notethatifyoudothisyouwillwanttochangesomeofyourkernelsettings(buffersizes)• Makesuretomonitoryourforwardersforthruput warnings!
![Page 19: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/19.jpg)
ManySolutions,OneGoal.
![Page 20: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/20.jpg)
ManySolutions,OneGoal.
DeploytotheEndpoints
• Deploydirectlytothesystemsyouwanttomonitor• Goodforapplicationdebugging• NiceoptionforSplunkES• CanbedonefromDeploymentServer• Granularcontrolovergroups• Couldmeanalotof“handon”
![Page 21: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/21.jpg)
ManySolutions,OneGoal.
![Page 22: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/22.jpg)
ManySolutions,OneGoal.
DeployingSplunkStream
![Page 23: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/23.jpg)
ManySolutions,OneGoal.
![Page 24: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/24.jpg)
ManySolutions,OneGoal.
Twoparts
• TheSplunkAppforStream• Dashboardsforanalyticsonprotocols• Administrativepanelsforconfiguration• StreamEstimate(reallycool,morelater)• GoesonSearchHead/Controller
• SplunkStreamAdd-on• Binaries• Index-timeoperations(linebreaking,timestamping)• GoesonIndexersandForwarders(UForHF)
![Page 25: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/25.jpg)
ManySolutions,OneGoal.
InstalltheSplunkAppforStream
• Canco-locatewithES• Canco-locatewithDMC• Insmaller(lessthan100forwarders)don’tusewiththeDS• Possibleexhaustedconnections(DSandStreampollseparately)
• InstallsjustlikeanyotherSplunkapp
![Page 26: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/26.jpg)
ManySolutions,OneGoal.
HarvesttheAddOn• Installstoafewplaces• $SPLUNK_HOME/etc/apps/Splunk_TA_stream• $SPLUNK_HOME/etc/apps/splunk_app_stream/install/Splunk_TA_stream• $SPLUNK_HOME/etc/deployment-apps/Splunk_TA_stream• Willcreatethelocalinputs.conf withtheappserverlocation
*SkipthisisyourSHisyourDS
![Page 27: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/27.jpg)
ManySolutions,OneGoal.
Makesureyourforwarderscantalkback
• YourforwarderswillneedtobeabletotalktotheSHwithsplunk_app_stream installed• TheportisthesameastheGUIforyourSH
![Page 28: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/28.jpg)
ManySolutions,OneGoal.
Configureyourforwarders
• Don’thavetoberootonLinux• Usetheincludedsetuid.sh script
• MustbelocaladminorlocalsystemonWindows• OnUFsyoushouldmonitoryourthruput limits
![Page 29: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/29.jpg)
ManySolutions,OneGoal.
Inputs.conf
• Rememberthattheinputs.conf islayerable• JustlikeotherSplunkconfigs• Doesn’thavetobeintheSplunk_TA_stream• OntheDSyoucandeploytwoapps,onewiththeinputtopointbacktothesplunk_app_stream• ThenalsodeploytheSplunk_TA_stream
![Page 30: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/30.jpg)
ManySolutions,OneGoal.
Configureyourstreams• Thedefaultsmaysendmorefieldsthanyouneed• Cantellforwarderswhichpartsofthedatayouwant• Youcanhavedifferentconfigs fordifferentgroups!
![Page 31: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/31.jpg)
ManySolutions,OneGoal.
Configureyourforwardergroups
• Usesgoodol’regex• LetsyousayaheadoftimeifEphemeralStreamsshouldbeallowed
![Page 32: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/32.jpg)
ManySolutions,OneGoal.
GotchawithGroups• JustregexontheStreamforwarderID(notIP,hostname)• ThisisconfiguredinanXMLfile• Messy• The“defaultgroup”forwardergroupforallunmatchedhostswillgatherALLTHETHINGS
![Page 33: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/33.jpg)
ManySolutions,OneGoal.
Waitfordatatoflowin
• That’sprettymuchit!• Docsmakeitlookalotharder
![Page 34: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/34.jpg)
ManySolutions,OneGoal.
Questions?
![Page 35: Splunk App for Stream - Aplura · PDF fileMany Solutions, One Goal. Splunk App for Stream David Shpritz, Aplura LLC. Baltimore Area User Group 3/21/2016](https://reader034.fdocuments.in/reader034/viewer/2022050807/5a90a6fd7f8b9a7f398e1f94/html5/thumbnails/35.jpg)
ManySolutions,OneGoal.
Credits• ThankstotheBaltimoreAreaSplunkUserGroup• CoverSlide:UpperSwallowFallsinOakland,MD,ChrisFlees,http://fineartamerica.com/profiles/chris-flees.html?tab=artwork&page=7
• Slide3:PotomacRiverinMaryland,TerryJ.Adams,http://www.fhwa.dot.gov/byways/byways/60807/photos
• Slide7:Timanus MillontheJonesFallsinBaltimore,“MonumentCity”,http://www.panoramio.com/photo/57148558
• Slide8:“MissingHomeworkLog”by“RedBeetleRB”.https://www.teacherspayteachers.com/Product/Missing-Homework-Log-4112• Slide9:Rotton log,NationalWildlifeFoundation,https://www.nwf.org/kids/family-fun/outdoor-activities/investigate-a-rotten-log.aspx
• Slide10:TheSimpsons,http://i.imgur.com/91sn32Q.jpg?fb
• Slide11:BroNetworkSecurityMonitor,https://www.bro.org/
• Slide17:IanAdamsPhotography,http://ianadamsphotography.com/news/galleries/bridges/• Slides19and21:SplunkConf 2015,“SplunkAppforStreamDeploymentsintheRealWorld:EnhanceOperationalIntelligenceAcrossApplication
Delivery,ITOps,SecurityandMore”,http://conf.splunk.com/session/2015/conf2015_SUdovicic_CChing_MDickey_Splunk_SplunkEntWhatsNew_StreamDeploymentsInTheReal.pdf
• Slide22:GunpowderFallsinBaltimoreCounty,MD,http://hdrcreme.com/photos/1818-gunpowder-falls• Slide23:SplunkDocs,http://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/DeploymentArchitecture
• Slide34:YoughioghenyRiveratFriendsville,MDbyJoeDawson,https://www.flickr.com/photos/jmd41280/5066756138