FreeSWITCH Monitoring

FreeSWITCHMonitoringClueCon,August2016

MoisésSilva<[email protected]>Manager,So?wareEngineering

We’reHiring

•  LinuxdevelopersC/C++orPython

•  Anywhereintheworld,paidrelocaJontoTorontoorfullJmeremoteopportuniJes

•  Funandrelaxedworkenvironment

2

SangomaTechnologies-©2016

Agenda

•  MonitoringBasics

•  MetricsandLogs

•  AlerJng

•  TradiJonalSoluJons•  EmergingSoluJons

•  A?erword(FreeSWITCHOxidized)

3


MonitoringBasics

•  Monitoringisaboutknowingwhatishappeningonyoursystems.Thegoodandthebad

•  MonitoringhelpsyouwithpredicJon/forecasJng.PlanyourfuturegrowthandanJcipateproblems

•  Monitoringprovidesyouwiththedatayouneedtotroubleshootproblemsfaster

4


TypesofMonitoring

•  ApplicaJonMetricMonitoring•  NumbersaboutapplicaJonusage&performance•  Youcanthenusethosenumberstodefinealerts

•  ServiceStatusMonitoring•  Simplebinarychecks.It’stheserviceupornot

5


MonitoringModels

•  PushvsPullsystems•  Push:Graphite,StatsD,CollectD,SNMP•  Pull:Prometheus,SNMP

•  DetermineswhoiniJatesthemetrictransfer

•  Pushsystemsaremoredynamicandtendtorequirelessmaintenance

•  Pullsystemsrequireasourceoftruthandnodediscovery

•  PullinghoweverallowsyoutomoreeffecJvelydetectifanodeorapplicaJonisdown

6


PushMonitoring

7


MonitoringServer

Node1

Node2

Node3

MetricsandCheckResults

PullMonitoring

8


MonitoringServer

Node1

Node2

Node3

MetricandCheckRequests

Metrics

•  Timeseriesdata.SequenceofnumericaldatapointslistedinJmeorder,usuallysampledatregularintervals.

•  DifferentgranulariJesofdatastoredandaggregatedoverJmeonfixed-sizestorage

•  Timeseriesanalysisoverthedataresultsinfancygraphs

•  Exampledatabases:RRDTool,Graphite(Whisper),InfluxDB,Prometheus

9


FreeSWITCHMetrics

•  AcJveCalls

•  TotalCalls

•  FailedCalls

•  RegistraJons

•  FailedRegistraJons

10


FreeSWITCHMetrics

•  ASR(Answer/SeizureRaJo)

•  NER(NetworkEffecJvenessRaJo)

•  ACD(AverageCallDuraJon)

•  PDD(PostDialDelay)

11


Logs

•  Logscanbeasourceofmetricswhenmetricsandservicechecksarenotenough

•  Youmaywanttosendalertsbasedonlogpagernsandrepeatedlogerrors

•  ELKStack•  ElasJcSearch•  Logstash•  Kabana

12


AlerMng

•  Email,Chat(Slack/HipChat),SMS,etc

•  Alertbasedonexpectedmetricsandcheckfailures

•  ProcuretousepercenJlesinsteadofaverages

•  SeveralalternaJves:Prometheus,Alerta,Flapjack

13


AlertFaMgue

14


•  Thishappenswhenalertsaretriggeredtooo?enandcausesthereceivertobedesensiJzed

AlertFaMgue

•  Youknowyouhaveitwhenyoustartignoringalerts

•  Ignoringalertsleadstomissingrealproblemsortakingtoolongtorespondtothem

15


AlertFaMgue

•  WarningsmightbewellintendedandinisolaJonworkwell,however,whentheyaddup,theymaycausemoreharmthangood

•  Adjustyourthresholds.BecauJouswithwarningvscriJcalseverityandadjustyournoJficaJonsaccordingly

•  Killorfixalertsthatarefiringtooo?en.YouneedawaytoeasilymutetheminthemeanJme

16


•  They’reallaboutgejngthejobdone.Andtheydo:

•  Butatsomepointyougogaaskyourselfifthere’sabegerway...

TradiMonalSoluMons

17


SNMP

18


SNMP

•  MIBs/OIDsarecumbersometouse(tosaytheleast)•  OIDsavailableinFreeSWITCHgiveyounumberofsessions,acJvecalls,sessionspersecondetc.

•  Noteasytocollectcustommetrics

•  VerylimitedinformaJonexposedbymod_snmpandoverallseemskindofabandoned

•  Themonitoring/managementinterfaceinFreeSWITCHneedssomework

19


CACTI&RRDtool

•  WebfrontendfortheRRDtoolJme-seriesdatabase

•  CollectmetricsfrommulJplesources(e.gsnmp)

•  Noonecaneverrememberhowtoaddaserver(convolutedprocess)

20


CACTI&RRDtool

21


•  Youendupwithuglygraphslikethis:

It’sbegerthannothingthough….

MonitoringSucks

22


•  Somuchthataround2011a“MonitoringSucks”communitywassettofixit:

•  hgps://github.com/monitoringsucks

•  hgps://github.com/monitoringsucks/blog-posts

•  hgps://vimeo.com/monitorama

NewSoluMons

23


•  Becausenewisalwaysbeger…

Sensu

•  Monitoringthatdoesn’tsuck(theirmogo)

•  Youcanreusenagios/zabbixchecks.SensuwasdesignedasareplacementforagingNagiosinstallaJons

•  NoJme-seriesdatabaseincluded,youhaveseveralopJonsavailable(e.gGraphite,InfluxDB)

24


Sensu

•  Pull/Pushsystemwithanagent

•  Pub/Submodelwithpluggabletransports(RabbitMQ,Redis,AmazonSQS)

•  Nodessubscribetogroupsofchecks.Thechecksarescheduledbytheserverorthenode.

•  Bothserviceandmetriccheckssupported

•  CheckresultscanbesentasynchronouslybyexternaljobsviarawTCP+JSON(supportinagithubPRforHTTP)

25


Sensu

•  HTTPAPItoreadaccessnodes,checks,silencealarms,etc

•  NiceandsimpledashboardUI

•  Dynamicinfrastructure.Serversaddthemselvestothemonitoringsystem.

•  ComposablejsonconfiguraJonforeasyautomaJon(e.gAnsible,SaltStack,Chefetc)

26


Sensu

27


Source:sensuapp.org

Graphite

•  Time-seriesstoragesystem

•  WhisperdatabaseformatdesignedasanimprovementoverRRDtooltomanageoutofJmedatapoints

•  Carbon-cacheisadaemonlisteningformetricsonaTCPsocket(typicallyusedintandemwithStatsD)

•  Metricsarestoredinfixed-sizefiles

•  Comeswitha(quiteoutdated)UIformetricgraphs.Thisismosto?enusedasanhgpendpointbyothergraphingtoolslikeGrafana

28


Graphite

•  EveryJmeseriesisidenJfiedbyahierarchyofdot-separatedidenJfiers(e.gstats.freeswitch.total_calls)

•  Thetextformattosendstatstographiteisdeadsimple:<metricname><value><Jmestamp>•  stats.freeswitch.total_calls501470843323

•  Nofurtherdimensionscanbeencoded.Thetypicalwork-aroundistoencodetheminthemetricname:•  stats.freeswitch.profile.internal.total_calls20

1470843323

29


Graphite

30


Source:hgp://www.aosabook.org/en/graphite.html

Prometheus

•  NewJme-seriesmonitoringsystemwithbuilt-inalerJng

•  IncludesitsownJme-seriesdatabaseformat

•  MulJ-dimensionaldatamodel(fancytermforkey/valuepairslabelsagachedtotheJme-seriesmetrics)

•  HTTP-basedpullmodel(PrometheusscrapsnodesformetricsviaHTTP)

•  PushgatewayssupportedforapplicaJonswithoutanaJveHTTPendpoint

31


Prometheus

•  ClientlibrariesavailableformanylanguagessoyoucaninstrumentyourapplicaJons

•  Flexiblequerylanguage(thinkSQLforforJme-series)

•  Commandlinequeryingtool

•  Built-inalerJng

•  CanbefederatedforaggregaJonofdatainmulJpledatacenters

32


Prometheus

•  GostandaloneapplicaJon(nodependencies)

•  Allmetricsstored,nolossofgranularity

•  Similarformattographite,withtheaddeddimensions:•  freeswitch_total_calls{profile=internal}20

1470843323

33


Prometheus

34


GraphingwithGrafana

•  PowerfuldashboardbuilderapplicaJon

•  Supportformanybackenddatabases(Prometheus,Graphite)

•  AutomatecreaJonofdashboardswiththeirHTTPAPIandJSONdashboarddefiniJonformat

•  RichgraphingcustomizaJons,styling,accesscontrol,etc

35


GraphingwithGrafana

36


GraphingwithGrafana

37


mod_prometheus

•  NaJvePrometheusFreeSWITCHintegraJonmodulewrigeninRust

•  Earlydevelopment(startedjustafewdaysago)butalreadyprovidesusefulmetricsoutofthebox,readytobescrappedbyPrometheus

•  MetricslikeCalls,Failedcalls,RegistraJons,CPS,ASR,etc

•  YoucanthenusePrometheusAlertmanagertocreatealerttriggerrulesbasedonthosemetrics

•  hgps://github.com/moises-silva/mod_prometheus

38


mod_prometheus

•  InstallingrequiresRustandCargo(theRustpackage/buildtoolandFreeSWITCHmaster)

#gitclonehgps://github.com/moises-silva/mod_prometheus#cdmod_prometheus#cargobuild#cptarget/debug/libmod_prometheus.so/usr/local/freeswitch/mod(Donotagempttorenamethemoduletomod_prometheus,itwon’twork)

39


mod_prometheus

•  Thenjustloadit

40


mod_prometheus

•  Testwithcurl

41


mod_prometheus

•  Usecustomcountersandgaugesfromthedialplan

42


mod_prometheus

•  NextPrometheusscrapwillgetthosemetrics

43


FinalThoughts

•  ApplicaJoninstrumentaJonmustbepartofapplicaJondevelopment

•  Monitoringtoolsofallkindsareavailable.Noexcusesforlackofmonitoring

•  Monitorallthethings!

44


AUerword

•  BringingRusttoFreeSWITCH.Themod_prometheusmoduleisthefirstRustmodulecomingintoFreeSWITCH

•  RustisanewsystemsprogramminglanguagewithinthesamespeedrangeasC/C++butwithmemorysafetyguaranteesusinganownershipsystemandmovesemanJcsbydefault

•  RustbindingsareavailablebythefreeswitchrsprojectcreatedbyMichaelGiagnocavo:hgps://gitlab.com/wiresight/freeswitchrs/

•  RustmemorysafetycouldmakeforaninteresJngaddiJontothestabilityofFreeSWITCH.Ifyou’rewriJngmodulespleaseconsiderusingRust

45


QUESTIONS

ContactUs

•  SangomaTechnologies100RenfrewDrive,Suite100Markham,OntarioL3R9R6Canada

•  Websitehgp://www.sangoma.com/

•  Telephone+19054741990x2(forSales)

•  [email protected]


47

THANKYOU

FreeSWITCH Monitoring

Software

Transcript of FreeSWITCH Monitoring