Introduction to segmentation fault handling

Intro to Segmentation Intro to Segmentation Fault Handling in LinuxFault Handling in LinuxBy Khanh Ngo-DuyBy Khanh [email protected]@elarion.com

mailto:[email protected]

SeminarSeminar

What is Segmentation Fault (Segfault)What is Segmentation Fault (Segfault)

Examples and ScreenshotsExamples and Screenshots

Tips to get Segfault informationTips to get Segfault information

Segfault Debugging TechniquesSegfault Debugging Techniques

What is Segmentation What is Segmentation Fault?Fault?

Segmentation Fault (Segfault)Segmentation Fault (Segfault) or or access access violationviolation is a particular error condition that can is a particular error condition that can occur during the operation of computer occur during the operation of computer softwaresoftware

A Segfault occurs when a program attempts to A Segfault occurs when a program attempts to access a memory location that is not allowed access a memory location that is not allowed to access, or attempts to access a memory to access, or attempts to access a memory location in a way that is not allowedlocation in a way that is not allowed

Write to a read-only locationWrite to a read-only locationTo overwrite part of the operating system or protected memory To overwrite part of the operating system or protected memory locationslocationsAccess to invalid memory location. e.g : memorry address NULL, -1 . . .Access to invalid memory location. e.g : memorry address NULL, -1 . . .etc . . . etc . . .

Examples and Examples and Screenshots Screenshots (1 of 3)(1 of 3)

Write to Read-Only memory address


Write to Invalidmemory address(NULL = 0x00)


Stack overflow

Tips to get Segfault Tips to get Segfault information information (1 of 7)(1 of 7)

Generally, when Segfault occurs, very less Generally, when Segfault occurs, very less information is provided (see previous slides)information is provided (see previous slides)• → → Very hard to debugVery hard to debug


Use Use dmesgdmesg to show information that saved by to show information that saved by the Kernel when any application crashesthe Kernel when any application crashes

Last Segfault information


How to read How to read dmesgdmesg outputs ? outputs ?

S eg fault[19960]: s eg fault a t 7fffff7feff8 ip 400480 s p 7fffff7ff000 error 6 in S eg fault[400000+1000]

Application name

Reasonit crashed

Address caused fault

InstructionPointer address

Additionalerror code

StackPointer address

Other values ???I DON'T KNOW...


Add Add -g-g when compiling the source code. The when compiling the source code. The compiler will add debugging symbols into the compiler will add debugging symbols into the binarybinary

Will provide more useful information when debugging with gdbWill provide more useful information when debugging with gdb

Compiled binary size will be largers (debugging symbols are added)Compiled binary size will be largers (debugging symbols are added)

Application runs slower, takes more RAMApplication runs slower, takes more RAM

Maybe, some other drawbacksMaybe, some other drawbacks


Add Add -g-g when compiling the source code. The when compiling the source code. The compiler will add debugging symbols into the compiler will add debugging symbols into the binarybinary

Will provide more useful information when debugging with Will provide more useful information when debugging with gdbgdb

Without Without -g-g, , gccgcc still adds some minimal debugging information still adds some minimal debugging information

Compiled binary size will be largers (debugging symbols are added)Compiled binary size will be largers (debugging symbols are added)

Application runs slower, takes more RAMApplication runs slower, takes more RAM

Maybe, some other drawbacksMaybe, some other drawbacks


Use Use nmnm to view the symbols in the binary file to view the symbols in the binary fileAddress of symbol, symbol type, symbol name can be listedAddress of symbol, symbol type, symbol name can be listedGive us chance to know the Segfault occurred with what symbolGive us chance to know the Segfault occurred with what symbol

$man nm$man nm for more information for more informationon the usageon the usage

Tips to get Segfault Tips to get Segfault informationinformation (7 of 7) (7 of 7)

Use Use ldd ldd to view the shared library dependencies to view the shared library dependenciesShow shared library name, starting address of libraryShow shared library name, starting address of library

We know Segfault occurred in our application or in shared libraryWe know Segfault occurred in our application or in shared library$man ldd $man ldd for more information how to usefor more information how to use ldd ldd

Segfault Debugging Segfault Debugging TechniquesTechniques

Using Using gdb gdb – The GNU Debugger– The GNU Debugger

Core dump fileCore dump file and and gdbgdb

objdumpobjdump

Segfault Debugging Segfault Debugging TechniquesTechniquesUsing gdb – The GNU DebuggerUsing gdb – The GNU Debugger (1 of 6) (1 of 6)

gdb gdb supports:supports:Starting programs , attaching to running programs or debugging Starting programs , attaching to running programs or debugging crashed programscrashed programs

Debugging locally or remotely (via Debugging locally or remotely (via gdbservergdbserver))

Setting breakpoints and watchpointsSetting breakpoints and watchpoints

Examining variables, registers and call stackExamining variables, registers and call stack

Changing data and calling functionsChanging data and calling functions

Automating debug tasksAutomating debug tasks

Multi threaded programsMulti threaded programs


In order to effectively debug program, add In order to effectively debug program, add -g-g when compiling with when compiling with gccgccLoad a program into gdb:Load a program into gdb:

$gdb$gdb program program

Once you are in gdb, you can run the programOnce you are in gdb, you can run the program(gdb)(gdb)runrun [parameters to program][parameters to program]

To stop program, press To stop program, press Ctrl+CCtrl+C To quit gdb, execute command To quit gdb, execute command qq


← Step 1: Load the program into gdb

← Step 2: execute the program

← Step 3: Quit from gdb

← gdb detects Segfaultbut very less info (-g is not add when compiling)


← add -g when compile← Step 1: Load

← Step 3: Quit from gdb

← gdb detects Segfault, showsthe lines which caused SegfaultLine 6, in main(), file: Segfault.c

← Step 2: Run


Is this Is this usefuluseful and and easyeasy ? ? YES!!!YES!!! But But why?why?Because of Because of -g-g → we can see the → we can see the file namefile name, , function name function name andand line number line number

Because of the source code is available → can see the exactly line of codeBecause of the source code is available → can see the exactly line of code

If there is no source code → we can see the filename, function name, line If there is no source code → we can see the filename, function name, line number but NOT contents of the line cause Segfault ← No problem, still number but NOT contents of the line cause Segfault ← No problem, still GOOD! :-)GOOD! :-)

Because this situation is simple, sometime you can NOT use this Because this situation is simple, sometime you can NOT use this technique! See next...technique! See next...


This technique can This technique can ON LYON LY be used when: be used when:You know for sure, Segfault will occurYou know for sure, Segfault will occurOnly when testing. When in production time, you can NOT → gdb causes Only when testing. When in production time, you can NOT → gdb causes many many side effectsside effects: slow down the running, running is not stable etc …: slow down the running, running is not stable etc …

Even when testing, if application is so Even when testing, if application is so Big Big oror Complicated Complicated (many (many threads, many resources)threads, many resources) → → gdb can not handlegdb can not handleTo be able to debug when your application is in production mode and not To be able to debug when your application is in production mode and not able to reduce the Segfault ? See the next techniques → ....able to reduce the Segfault ? See the next techniques → ....

Segfault Debugging Segfault Debugging TechniquesTechniquescore dump file and gdb core dump file and gdb (1 of 5)(1 of 5)

A core dump consists of the recorded state of A core dump consists of the recorded state of the working memory of a computer program at the working memory of a computer program at a specific time, generally when the program a specific time, generally when the program has terminated abnormally (crashed)has terminated abnormally (crashed)

Core dump file might contain: processor registers, which may include the Core dump file might contain: processor registers, which may include the program counter and stack pointer, memory management information, and program counter and stack pointer, memory management information, and other processor and operating system flags and informationother processor and operating system flags and informationCore dumps are disabled by default on some Linux distributionsCore dumps are disabled by default on some Linux distributionsTo force the core dump generation, you can using command lineTo force the core dump generation, you can using command line

$ulimit $ulimit -c <limit size of core file>-c <limit size of core file>To force the core dump generation, you can also insert code to your To force the core dump generation, you can also insert code to your application to request generating the core dump when it crashesapplication to request generating the core dump when it crashesTo disable the core dump just set To disable the core dump just set <limit size of core file><limit size of core file> to to 00


← Enable core dump, limit to 1024 MB, just once

← core file is generated when app crashes

← It is here!


Once you have core dump, what to do ?Once you have core dump, what to do ?→ → Just load it into gdb and seeJust load it into gdb and see

$gdb$gdb <application name> <core file name> <application name> <core file name>


← Load the application and core file

← gdb reads core file andshows the results as if theApplication has just run andcrashed, actually core dumpjust shows the actual HISTORY


Is this better than the previous technique ?Is this better than the previous technique ?Yes, because I could not reproduce the Yes, because I could not reproduce the Segfault, however, core file shows meSegfault, however, core file shows meThough it is good, still some disadvantagesThough it is good, still some disadvantages

Core file may grow very large if your application uses much memory, so Core file may grow very large if your application uses much memory, so sometime you simply can not use this methodsometime you simply can not use this methodIn case of complicated application, there In case of complicated application, there might be some side effectsmight be some side effects when when forcing core dump → your application might run unstableforcing core dump → your application might run unstable

What I read till now, just What I read till now, just s ide effec tss ide effec ts .. Is there any Is there any else? I don't want to risk the production system! → else? I don't want to risk the production system! → YES. There is, see the last techniques ..YES. There is, see the last techniques ....

Segfault Debugging Segfault Debugging TechniquesTechniquesobjdump objdump (1 of 9)(1 of 9)

AdvantagesAdvantagesNo need to add No need to add -g-g → do not affect the memory and reduce the size of binary → do not affect the memory and reduce the size of binary filefileNo need to generating core dump → no side effects, do not take disk spaceNo need to generating core dump → no side effects, do not take disk spaceActually, you do not need to do anything, what will come will come, and you Actually, you do not need to do anything, what will come will come, and you will solve it!will solve it!

Disadvantages ?Disadvantages ?You need a little knowledge about You need a little knowledge about assemblyassembly language :-) → don't be scare, still language :-) → don't be scare, still easy!easy!If adding optimization flag to If adding optimization flag to gccgcc ( (-O-O, , -O2-O2, , -O3-O3) it will be a little hard you to ) it will be a little hard you to read assembly code laterread assembly code later


First of all, what you need is the output of dmesg First of all, what you need is the output of dmesg (very first slide told you).(very first slide told you).

Note the Note the “Address caused fault”“Address caused fault” and and “Instruction pointer address”“Instruction pointer address”Use the tool named Use the tool named objdumpobjdump to generate information from your application to generate information from your applicationOutput of objdump should be redirected to a file, we need this file later!Output of objdump should be redirected to a file, we need this file later!

$objdump$objdump -DCl <application name> > <output file> -DCl <application name> > <output file>


OK, so my fault address is OK, so my fault address is 0x40058c0x40058c and and instruction pointer is instruction pointer is 0x40048c0x40048cmyDumpmyDump contains the assembly code of my app contains the assembly code of my appNow I will see at what line of code, my app Now I will see at what line of code, my app crashed → just find where is crashed → just find where is 0x40048c0x40048c in in myDumpmyDump

$grep$grep -n -A 100 -B 100 “40048c” ./myDump -n -A 100 -B 100 “40048c” ./myDumpWhat it does it just find the line having 40048c in ./myDump, and also show What it does it just find the line having 40048c in ./myDump, and also show 100 more lines after the found line, and 100 lines before the found line. You 100 more lines after the found line, and 100 lines before the found line. You can customize the grep command as you want ;)can customize the grep command as you want ;)


← Step 1: Find instruction pointer address

← Step 2: Found, This caused segfault

← Step 3: Look above to see the codethat caused segfault in what function?Here it is in main()


Now you know the code that cause Segfault in Now you know the code that cause Segfault in assemblyassembly..What to do is open your source code (in C, C++ What to do is open your source code (in C, C++ …) to see the appropriate line of code …) to see the appropriate line of code corresponding to that Assembly code, you will corresponding to that Assembly code, you will figure out what caused Segfault :-)figure out what caused Segfault :-)


0x48 = 'H'


You are done now! Bravo !!!You are done now! Bravo !!!Just with “Just with “Instruction PointerInstruction Pointer””, you know where , you know where caused Segfault exactlycaused Segfault exactlyHow about the “How about the “Address caused faultAddress caused fault” ” ((0x40058c0x40058c), we have not used it, haven't we?), we have not used it, haven't we?

No we don't. BUT, till now I can say the line caused Segfault is No we don't. BUT, till now I can say the line caused Segfault is *s = 'H ';* s = 'H ';

And, the address of variable And, the address of variable ss at that time is 0x40058c at that time is 0x40058cMeaningless to know this?Meaningless to know this? NO! There is sometime you will need it to know NO! There is sometime you will need it to know the root cause, see the next slidethe root cause, see the next slide


Sometimes, the “Sometimes, the “Address caused faultAddress caused fault” tell you ” tell you the root cause. See the following example, we the root cause. See the following example, we can say that, value of can say that, value of ss is is N U LLN U LL

Thanks for watchingThanks for watchingIf you see it useful → clap your hands :-)If you see it useful → clap your hands :-)

Introduction to segmentation fault handling

Technology

Transcript of Introduction to segmentation fault handling