Debugging @ Scale - USENIX...CLOSING THOUGHTS • NRT Telemetry and Distributed Tracing –Key...
31
Debugging @ Scale Kumar Srinivasamurthy Bing & Cortana Engineering
Transcript of Debugging @ Scale - USENIX...CLOSING THOUGHTS • NRT Telemetry and Distributed Tracing –Key...
-
Debugging@Scale
KumarSrinivasamurthyBing&CortanaEngineering
-
Looksfamiliar?
-
The(dual)natureofdebugging
-
Whenauto-mitigationfails…
-
DebuggingisHardhttps://xkcd.com/1722
-
Whomostofusatare- Explorers
-
WhoweshouldBe?- ER
-
TraditionalToolsoftheDebuggingTrade
DEBUGGER LOGSMEMORYPROFILER
-
Traditionaltoolsdon’tscaleJ
-
Conceptswecanuse…
-
MAKEERRORMESSAGESUSEFULFORUSERS*AND*ENGINEERS
-
Acommonerrormessage
-
Abetterapproach
-
USEDISTRIBUTEDTRACINGASTHENEW‘DEBUGGER’
-
-
-
APPLYTRANSFORMSAND‘MACHINELEARNING’TOMAKELOGSUSEFUL
-
https://xkcd.com/1838
-
…FromOur‘PileOfLinearAlgebra’
• Log-relevantTokenization• FeatureExtraction• Instance-BasedLearning• KNN:k-NearestNeighbor
• LinearClassifiers• Clustering• k-Means
-
DesignforReproducibility
-
Reproducibility
• Logplayback• Forkingtraffic• Singlebox/Mocking• SnapshotDebugger• Experimentation
-
INVESTINDEEPOPTICSWITHNRTMETRICS
-
REALUSERMEASUREMENTS(RUM)NEARREALTIMEMETRICS
NavigationTimingAPI
Ø Expandtobusinessmetrics
-
DEVELOPANOMALYDETECTIONASAPLATFORMCAPABILITY
-
• Detectanomaliesacrossmetrics• Enableoutlierdetectiononlogs• Highdatadimensionalityallowsbetterdetection
-
PRIVACY*HAS*TOBEAFIRST-CLASSCITIZENFORDEBUGGING
-
BeMindfulOf:
• GDPR– VerypertinentforEU• Logsanitization• LogRetentionandOndemandpurge• Exposureto3rd PartyAnalysisSystemsandHosting• TheLatentPowerofAggregatedLogs
-
OLDTRICKSSTILLWORK,BUTCANBETRICKY
-
Stillinuse…
• DebugModeforsubsetinProduction• Deployingprivatebuildsonasubset• Loggingontomachineinstances• Attachingdebugger• Others?
-
CLOSINGTHOUGHTS
• NRTTelemetryandDistributedTracing– KeyElements• Atscale,machine-learningisn’tjustabuzzword– itactuallydelivers• Bemindfulofprivacyconcerns.Yourbusinessisatstake• DebugginginfunJ
-
Questions?
• [email protected]• @00kumars
Onlyhalfofprogrammingiscoding.Theother90%isdebugging./r/ProgrammerHumor
