Debugging: Love It, Hate It or Reverse It?
-
Upload
undo -
Category
Technology
-
view
58 -
download
1
Transcript of Debugging: Love It, Hate It or Reverse It?
Debugging:LoveIt,HateItOrReverseIt?
Debugging:LoveIt,HateItOrReverseIt?.
JulianSmith,co-founderandCTO,[email protected]://undo.io/
OverviewTesting.Debugging:
Debuggingwithgdb.Strace.Valgrind.Recordingexecution.
(Linux-specific.)
Testinghaschanged:
Resultingin:
Testing.
Continuousintegration.Test-drivendevelopment.Cloudtesting.
1,000softestsperhour.Manyintermittenttestfailures.Verydifficulttofixthemall.
Testing.Securitybreaches.Productionoutages.Unhappyusers.
Fixingtestfailuresishard.
Testing.
Recreatecomplexsetups:Multi-application.Networking.Multi-machine.
Re-runflakeytestsmanytimestoreproducefailureRecompile/linkwithchangeswheninvestigating.
Changesbehaviour.Slow.Requiresadevelopermachine.
Fixingtestfailuresisslow.
Testing
Reproducingslowfailuresis…slow.Reproducingintermittentfailuresisalsoslow.
Requiresrepeatedlyrunningatestmanytimesinordertocatchthefailure.
Criticalbugs:
Canoccuroneinathousandruns.Eachruncantakehours.
Toolstofixtestfailures
Testing.
Debuggers.Logging.Systemlogging.Memorycheckers.Recordingexecution.
GDB
Debugging.
Betterthanyoumayremember.Ctrl-XCtrl-Ashowssourcecodewithinterminalwindow.GDB-7haspythonextension.
Scripteddebugging,e.g.toreproduceintermittentfailures.
GDBscripting.
Debugging.
repeat_until_non_zero_exit.py'''Repeatedlyrundebuggeeuntilitfails.'''importgdb
while1:print'-'*40gdb.execute('run')e=gdb.parse_and_eval('$_exitcode')print('$_exitcodeis:%s'%e)ife!=0:break
(gdb)sourcerepeat_until_non_zero_exit.py
GUIsforgdbaregettingbetter:
Debugging.
Eclipse.CLion.QtCreator.KDbg.Emacs.
Logging.Cansometimesworkwell.Needtocontrolwhattolog.
Defineareasoffunctionalityandassigndifferentdebuglevels.E.g.parser,lexer,network.Moredetailed:memoryallocator,socket,serialiser.
Wecandefinedebuglevelsfordifferentcategoriestomatchthebugweareinvestigating.
Thiscangetcomplicated.
logcategory_t*io_category=...;logcategory_t*serialisation_category=...;...logf(io_category,"haveread%zibytesfromsocketfd=%i",n,fd);...logf(serialisation_category,"serialised%pto%zibytes",foo,actualsize);...
Problemswithloggingcategories.
Logging.
Howmanycategories-howdetailedshouldwego?
Dependsonthebugweareinvestigating.Mayneedtorecompilewithnewcategories.
Whatcategorydoweuseforcodethatwritesserialiseddatatoafile-io_categoryorserialisation_category?
Useprogrammestructureforcategories.
Logging.
Wealreadyhaveareasoffunctionality:
Sourcecodedirectories.Sourcefiles.Functions.
Wecanusetheseasimplicitcategories:
Noneedtodefineourowncategories.Wegetdifferentlevelsofcategoriesforfree.Wegetnestedcategoriesforfree.
Controllingverbosityprogrammatically:
Logging.
debug_add("network/socket",NULL,1);//Extraverboseforalldiagnosicsinnetwork/socket*.*.
debug_add("network/",NULL,1);debug_add("network/socket",NULL,1);//Extraverboseforalldiagnosticsinnetwork/*.*.//Evenmoreverboseinnetwork/socket*.*.
debug_add("heap/alloc.c","",1);debug_add("network/socket.c",Send,2);debug_add("parser/","",-1);//Verboseforheapoperations.//Veryverboseforalldiagnosticsinnetwork/socket.c:Send().//Lessverboseinparser/.
Controlverbositywithenvironmentalvariables:
Example:
Logging.
QA-friendly.Noneedtorecompile/link/build.Activateloggingindifferentpartsoftheprogrammedependingonthebugwhichisbeinginvestigated.
DEBUG_LEVELS=heap/alloc.c=1,parser/=-1,network/socket.c:Send=2myprog...
Strace.Linux/unix-specific.
Getadetailedlogofallsyscalls.
>stracedateexecve("/bin/date",["date"],[/*34vars*/])=0brk(0)=0xd50000access("/etc/ld.so.nohwcap",F_OK)=-1ENOENT(Nosuchfileordirectory)mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f7602059000access("/etc/ld.so.preload",R_OK)=-1ENOENT(Nosuchfileordirectory)open("/etc/ld.so.cache",O_RDONLY|O_CLOEXEC)=3</etc/ld.so.cache>fstat(3</etc/ld.so.cache>,{st_mode=S_IFREG|0644,st_size=144491,...})=0mmap(NULL,144491,PROT_READ,MAP_PRIVATE,3</etc/ld.so.cache>,0)=0x7f7602035000close(3</etc/ld.so.cache>)=0access("/etc/ld.so.nohwcap",F_OK)=-1ENOENT(Nosuchfileordirectory)open("/lib/x86_64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC)=3</lib/x86_64-linux-gnu/libc-2.19.so>read(3</lib/x86_64-linux-gnu/libc-2.19.so>,"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"...,832)=832fstat(3</lib/x86_64-linux-gnu/libc-2.19.so>,{st_mode=S_IFREG|0755,st_size=1738176,...})=0mmap(NULL,3844640,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_DENYWRITE,3</lib/x86_64-linux-gnu/libc-2.19.so>,0)=0x7f7601a90000mprotect(0x7f7601c32000,2093056,PROT_NONE)=0mmap(0x7f7601e31000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE,3</lib/x86_64-linux-gnu/libc-2.19.so>,0x1a1000)=0x7f7601e31000mmap(0x7f7601e37000,14880,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,-1,0)=0x7f7601e37000close(3</lib/x86_64-linux-gnu/libc-2.19.so>)=0mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f7602034000mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f7602033000mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f7602032000arch_prctl(ARCH_SET_FS,0x7f7602033700)=0mprotect(0x7f7601e31000,16384,PROT_READ)=0mprotect(0x60e000,4096,PROT_READ)=0mprotect(0x7f760205b000,4096,PROT_READ)=0munmap(0x7f7602035000,144491)=0brk(0)=0xd50000brk(0xd71000)=0xd71000open("/usr/lib/locale/locale-archive",O_RDONLY|O_CLOEXEC)=3</usr/lib/locale/locale-archive>fstat(3</usr/lib/locale/locale-archive>,{st_mode=S_IFREG|0644,st_size=1607760,...})=0mmap(NULL,1607760,PROT_READ,MAP_PRIVATE,3</usr/lib/locale/locale-archive>,0)=0x7f7601ea9000close(3</usr/lib/locale/locale-archive>)=0open("/etc/localtime",O_RDONLY|O_CLOEXEC)=3</etc/localtime>fstat(3</etc/localtime>,{st_mode=S_IFREG|0644,st_size=3661,...})=0fstat(3</etc/localtime>,{st_mode=S_IFREG|0644,st_size=3661,...})=0mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f7602058000read(3</etc/localtime>,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\7\0\0\0\0"...,4096)=3661lseek(3</etc/localtime>,-2338,SEEK_CUR)=1323read(3</etc/localtime>,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0"...,4096)=2338close(3</etc/localtime>)=0munmap(0x7f7602058000,4096)=0fstat(1</dev/pts/50>,{st_mode=S_IFCHR|0620,st_rdev=makedev(136,50),...})=0mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f7602058000write(1</dev/pts/50>,"Mon26Sep12:27:50BST2016\n",29Mon26Sep12:27:50BST2016)=29close(1</dev/pts/50>)=0munmap(0x7f7602058000,4096)=0close(2</dev/pts/50>)=0exit_group(0)=?+++exitedwith0+++
Subsetofsyscalls-fileoperations:
>strace-y-etrace=filedateexecve("/bin/date",["date"],[/*34vars*/])=0access("/etc/ld.so.nohwcap",F_OK)=-1ENOENT(Nosuchfileordirectory)access("/etc/ld.so.preload",R_OK)=-1ENOENT(Nosuchfileordirectory)open("/etc/ld.so.cache",O_RDONLY|O_CLOEXEC)=3</etc/ld.so.cache>access("/etc/ld.so.nohwcap",F_OK)=-1ENOENT(Nosuchfileordirectory)open("/lib/x86_64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC)=3</lib/x86_64-linux-gnu/libc-2.19.so>open("/usr/lib/locale/locale-archive",O_RDONLY|O_CLOEXEC)=3</usr/lib/locale/locale-archive>open("/etc/localtime",O_RDONLY|O_CLOEXEC)=3</etc/localtime>Mon26Sep12:29:01BST2016+++exitedwith0+++
Subsetofsyscalls-memoryoperations:
>strace-y-etrace=memorydatebrk(0)=0x25b8000mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f14cc871000mmap(NULL,144491,PROT_READ,MAP_PRIVATE,3</etc/ld.so.cache>,0)=0x7f14cc84d000mmap(NULL,3844640,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_DENYWRITE,3</lib/x86_64-linux-gnu/libc-2.19.so>,0)=
0x7f14cc2a8000mprotect(0x7f14cc44a000,2093056,PROT_NONE)=0mmap(0x7f14cc649000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE,3</lib/x86_64-linux-gnu/libc-2.19.so>,0x1a1000)=0x7f14cc649000mmap(0x7f14cc64f000,14880,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,-1,0)=0x7f14cc64f000mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f14cc84c000mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f14cc84b000mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f14cc84a000mprotect(0x7f14cc649000,16384,PROT_READ)=0mprotect(0x60e000,4096,PROT_READ)=0mprotect(0x7f14cc873000,4096,PROT_READ)=0munmap(0x7f14cc84d000,144491)=0brk(0)=0x25b8000brk(0x25d9000)=0x25d9000mmap(NULL,1607760,PROT_READ,MAP_PRIVATE,3</usr/lib/locale/locale-archive>,0)=0x7f14cc6c1000mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f14cc870000munmap(0x7f14cc870000,4096)=0mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)=0x7f14cc870000Mon26Sep12:29:40BST2016munmap(0x7f14cc870000,4096)=0+++exitedwith0+++
Summary:
Strace.
Notperfect-onlyworksonsyscalllevel.Butstillusefulforlow-levelinvestigations.Norecompilationrequired.
Overview:
Valgrind.
Linux,OSX,Solaris,Android.Verydetailedcheckingofexecution.Free.SimilartoPurifyetc.
Memorychecking:
Threadchecking.
Other:
Valgrind.
Illegalmemoryaccesses:Overrun/underrunheapblocks.Overrunstack.Use-after-free.
Doublefree.Memoryleaks.
Inconsistentlockorderings.Dataraces(e.g.missingmutex).
CPUcachebehaviour.Heapprofiler.
Highlyrecommended!
Valgrind.
Newdebuggingtechnologyinlastfewyears.
Recordingexecution.
Linux:UndoLiveRecorder.RR.
Windows:Intellitrace(partialrecordingonly).TimeMachineFor.Net(partialrecordingonly).
Java:Chronon.Undo(soon).
LiveRecorder.Alibrary,forlinkingintoanapplication.Allowstheapplicationtocontroltherecordingofitsownexecution.ProvidesasimpleCAPItostart/save/stoprecording.APIisdefinedinundolr.hheaderfileandimplementedinlibundolrlibrary.
LiveRecorder.LiveRecorderrecordings:
ArestandardUndoRecordingfiles.Containeverythingneedtoreplayexecution:
Non-deterministicevents(inputstoprogram).Initialstate(initialmemoryandregisters).
Alsocontaininformationneededforsymbolicdebugging:Completeexecutableand.sofiles.Debuginfofiles.
Allowsdebuggingevenwhenlibrariesand/ordebuginformationisnotavailablelocally(e.g.loadandreplayonadifferentLinuxdistribution).LoadedintoUndoDBaswithSave-Load:
undodb-gdb--undodb-load<filename>
(undodb-gdb)undodb-load<filename>
Fullreversibledebugging.
LiveRecorder.LibraryAPI(undolr.h):intundolr_recording_start(undolr_error_t*o_error);
intundolr_recording_stop(void);
intundolr_recording_save(constchar*filename);
intundolr_recording_stop_and_save(constchar*filename);
intundolr_save_on_termination(constchar*filename);
intundolr_save_on_termination_cancel(void);
intundolr_event_log_size_get(long*o_bytes);
intundolr_event_log_size_set(longbytes);
intundolr_include_symbol_files(intinclude);
LiveRecorder.UseLiveRecorderininternaltesting:
Investigatetestfailureseasilyusingreversibledebugging.Avoidproblemswithdifferingenvironments.Noneedtoreproducecomplexmulti-machinesetups.Canbeusedindifferentways:
Disabledbydefault,butre-runfailingtestswithLiveRecorderactivated.Enabledbydefault,buttellLiveRecordertosaverecordingonlyiftestfails.
Havemultipledevelopersworkonthesametestfailure.
LiveRecorder.UseLiveRecorderincustomerreleases:
Nooverheadifnotused.Youandyourcustomercontrolwhen/ifrecordingisenabled.CustomerhascontroloverpruningtherecordingtoprotecttheirIP.Debuganexactcopyofacustomerfailure,withouthavingtocreateatest-case.Havemultipledevelopersworkonthesamecustomerbug.
LiveRecorder:Demo.
Questions?
LiveRecorder.
EOF.
http://undo.io/