Debugging@Scale
KumarSrinivasamurthyBing&CortanaEngineering
Looksfamiliar?
The(dual)natureofdebugging
Whenauto-mitigationfails…
DebuggingisHardhttps://xkcd.com/1722
Whomostofusatare- Explorers
WhoweshouldBe?- ER
TraditionalToolsoftheDebuggingTrade
DEBUGGER LOGSMEMORYPROFILER
Traditionaltoolsdon’tscaleJ
Conceptswecanuse…
MAKEERRORMESSAGESUSEFULFORUSERS*AND*ENGINEERS
Acommonerrormessage
Abetterapproach
USEDISTRIBUTEDTRACINGASTHENEW‘DEBUGGER’
-
APPLYTRANSFORMSAND‘MACHINELEARNING’TOMAKELOGSUSEFUL
https://xkcd.com/1838
…FromOur‘PileOfLinearAlgebra’
• Log-relevantTokenization• FeatureExtraction• Instance-BasedLearning• KNN:k-NearestNeighbor
• LinearClassifiers• Clustering• k-Means
DesignforReproducibility
Reproducibility
• Logplayback• Forkingtraffic• Singlebox/Mocking• SnapshotDebugger• Experimentation
INVESTINDEEPOPTICSWITHNRTMETRICS
REALUSERMEASUREMENTS(RUM)NEARREALTIMEMETRICS
NavigationTimingAPI
Ø Expandtobusinessmetrics
DEVELOPANOMALYDETECTIONASAPLATFORMCAPABILITY
• Detectanomaliesacrossmetrics• Enableoutlierdetectiononlogs• Highdatadimensionalityallowsbetterdetection
PRIVACY*HAS*TOBEAFIRST-CLASSCITIZENFORDEBUGGING
BeMindfulOf:
• GDPR– VerypertinentforEU• Logsanitization• LogRetentionandOndemandpurge• Exposureto3rd PartyAnalysisSystemsandHosting• TheLatentPowerofAggregatedLogs
OLDTRICKSSTILLWORK,BUTCANBETRICKY
Stillinuse…
• DebugModeforsubsetinProduction• Deployingprivatebuildsonasubset• Loggingontomachineinstances• Attachingdebugger• Others?
CLOSINGTHOUGHTS
• NRTTelemetryandDistributedTracing– KeyElements• Atscale,machine-learningisn’tjustabuzzword– itactuallydelivers• Bemindfulofprivacyconcerns.Yourbusinessisatstake• DebugginginfunJ
Questions?
• [email protected]• @00kumars
Onlyhalfofprogrammingiscoding.Theother90%isdebugging./r/ProgrammerHumor
Top Related