July 2, 2015 Sam Siewert

Verify Code Modules, Configuration and Debugging

Basic Code Level Verification Tool Primer for Linux

http://www.stsc.hill.af.mil/resources/tech_docs/gsam4.html

Integration and Test Fundamental Linux Tools Used for Unit Testing and Integration and Test of Modules

Sam Siewert 2

GitHub and GitHub Desktop Primer Use CMVC from Now On – Install GitHub Desktop on Windows or Mac – Create GitHub Account - https://github.com/ – Code is Public for FREE Use – Alternative is bitbucket (https://bitbucket.org/ ) – Clone from siewertserau or PRClab or Desktop

git clone https://github.com/siewertserau/Examples-RAID-Unit-Test.git You need my permission to push changes back, but anyone can pull

Sam Siewert 3

GitHub New Project Create Your Own Student Project Start GitHub Desktop, Download Code from Web, Drag Folder over to Create new repository Publish for Use by Others [on your team, public] Push to GitHub so Others can Clone on Your Team [Pull]

Sam Siewert 4

#1

#2 #3

Use on University/Corporate Machines Git Clients Can Securely Connect to the Hub with SSH or HTTPS See Cloning Path on GitHub Desktop / Web – View on GitHub – Use SSH or HTTPS to Clone [SSH is Preferred]

Sam Siewert 5

Grab clone path here

Clone to Desktop here

Get Archive here

Use CVMC (Git) for All Module Work

Clone/Update, Debug, Update, Test, Commit/Push to Base … Repeat

Sam Siewert

6

Git Example – Use SSH Tunnel Set up your SSH key on PRClab and copy it to your key-ring on GitHub (described here too - https://help.github.com/articles/generating-ssh-keys/ ): – ssh-keygen -t rsa -C siewerts@erau.edu [follow prompts as described on

web page above] – ssh-agent –s [start ssh agent for your current login – do this on each login] – cat ~/.ssh/id_rsa.pub [copy the output that should be something like “ssh-

rsa ****************** siewerts@erau.edu”, where ****’s are my encrypted key, so your’s will be unique and different

– got to github.com/siewertserau [replace with your github username] – Go to “gear” configuration tool and add your SSH key, here:

Sam Siewert 7

note that I have erased portions of my keys for security

Git Example – SSH Cloning Now that you have your SSH key generated on PRClab on your GitHub keyring, you can clone with SSH instead of the HTTPS clone – Now, for me, I just copied from the web page

and then issued the clone command on PRClab “git clone git@github.com:siewertserau/Examples-Imageproc-Unit-Test.git” and I’m prompted for my SSH passphrase now, which is great, and if I enter it correctly, I will get an SSH clone sandbox.

– It should look like this:

Sam Siewert 8

Use SSH PATH

Git Example – Sandbox Work Now edit your files, add files, build, test and when your happy, do “git add *” to add any new files you copied into your sandbox, and do a pull to make sure you are current. Now do “git commit –a” to commit the change set to your clone sandbox and add a meaningful comment and after do “git push”, which should look like:

Sam Siewert 9

Git Example – Browse Change History Use Browse code and Side-by-side “Split” view to Compare Old and New Versions of a file Browsing Code Change History is One of the MOST Powerful Features of CMVC

Sam Siewert 10

Binary File Example Make changes, build, test and then commit and push Now Run to Generate Chunk files

Sam Siewert 11

Git RAID Example - Browse To See What I Changed, Browse First, 4 New “+” Lines for Includes for Simple file I/O Next, 2 New Lines to Declare Simple file Descriptors

Sam Siewert 12

Git RAID Example – Test Case Update New Block of Binary Output for External Verification Can “diff” and use “od –t x1” on files to Examine

Sam Siewert 13

Line 129 to 158 is New Line 124 and 160 Sync Up Again

Basic Binary File Dump and Compare Chunk1.bin, …, Chunk4.bin are Data Files in 4+1 RAID-5 Set ChunkXOR.bin is the XOR Parity File in the RAID-5 4+1 Chunk4_Rebuilt.bin is the Recovered Data in Test Case #0 Identical Chunks, so XOR=0 Sam Siewert 14

System Test of Integrated Modules

Profiling, Tracing and Workloads for I&T

Sam Siewert

15

Profiling Basics Gprof, Available with GCC Since Beginning of Time Still Very Useful Today Read the Manual, Then Try Quick Start Following Example in These Notes https://sourceware.org/binutils/docs/gprof/ Read numerous Tutorials - https://en.wikipedia.org/wiki/Gprof

Sam Siewert 16

Performance Tests Profiling [Classic Tools] – Gprof – Open souce tool [similar to Gcov, but for Profiling] – Vtune – Commercial Tool from Intel – Logic Analyzer, Agilent SPA (Statistical Performance Analysis)

Tracing – E.g. Timestamps output to syslog

Statistics – top, htop – iostat – memstat

Workloads – Iometer – stress

Sam Siewert 17

Performance - Sysprof What is Using CPU on my System Rather than Profile of an Application – Sub-System [Service]

Sam Siewert 18

Gprof Simple –pg compile opiton Run, gprof on gmon.out to get analysis

Sam Siewert 19

%make cc -O3 -Wall -pg -msse3 -malign-double -g -c raidtest.c raidtest.c: In function 'main': raidtest.c:99: warning: format '%d' expects type 'int', but argument 2 has type 'long unsigned int' raidtest.c:68: warning: unused variable 'aveRate' raidtest.c:68: warning: unused variable 'totalRate' raidtest.c:66: warning: unused variable 'rc' raidtest.c:212: warning: control reaches end of non-void function cc -O3 -Wall -pg -msse3 -malign-double -g -c raidlib.c cc -O3 -Wall -pg -msse3 -malign-double -g -o raidtest raidtest.o raidlib.o %./raidtest Will default to 1000 iterations Architecture validation: sizeof(unsigned long long)=8 RAID Operations Performance Test Test Done in 453 microsecs for 1000 iterations 2207505.518764 RAID ops computed per second %ls Makefile gmon.out raidlib.h raidlib64.c raidtest raidtest.o Makefile64 raidlib.c raidlib.o raidlib64.h raidtest.c raidtest64 %gprof raidtest gmon.out > raidtest_analysis.txt

Gprof Analysis 1 million iterations of RAID test XOR and Rebuild

Sam Siewert 20

Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls ns/call ns/call name 82.13 1.54 1.54 main 15.47 1.83 0.29 2000001 145.38 145.38 xorLBA 2.67 1.88 0.05 2000001 25.07 25.07 rebuildLBA % the percentage of the total running time of the time program used by this function. cumulative a running sum of the number of seconds accounted seconds for by this function and those listed above it. self the number of seconds accounted for by this seconds function alone. … calls the number of times this function was invoked, if this function is profiled, else blank. self the average number of milliseconds spent in this ms/call function per call, … total the average number of milliseconds spent in this ms/call function and its descendents per call, … name the name of the function. …

RAID Operations Performance Test Test Done in 206417 microsecs for 1000000 iterations 4844562.221135 RAID ops computed per second

Call Graph Profile from Gprof

Sam Siewert 21

Call graph (explanation follows) granularity: each sample hit covers 2 byte(s) for 0.53% of 1.88 seconds index % time self children called name <spontaneous> [1] 100.0 1.54 0.34 main [1] 0.29 0.00 2000001/2000001 xorLBA [2] 0.05 0.00 2000001/2000001 rebuildLBA [3] ----------------------------------------------- 0.29 0.00 2000001/2000001 main [1] [2] 15.4 0.29 0.00 2000001 xorLBA [2] ----------------------------------------------- 0.05 0.00 2000001/2000001 main [1] [3] 2.7 0.05 0.00 2000001 rebuildLBA [3] ----------------------------------------------- This table describes the call tree of the program, and was sorted by the total amount of time spent in each function and its children… % time This is the percentage of the `total' time that was spent in this function and its children… self This is the total amount of time spent in this function. children This is the total amount of time propagated into this function by its children. called This is the number of times the function was called…

Tracing and Code Coverage Basics What Code Has Been Executed? - Coverage What Events Triggered Code Execution (Function Call) and at What Time? - Trace Read the Manual, Then Try Quick Start Following Example in These Notes for Coverage https://gcc.gnu.org/onlinedocs/gcc/Gcov.html Read numerous Tutorials - https://en.wikipedia.org/wiki/Gcov

Sam Siewert 22

Summary of Gcov Commands

Sam Siewert 23

%make clean rm -f *.o *.d *.exe sclogic *.gcov *.gcno *.gcda *.info %ls Makefile SC-Logic-2-LCOV-results SC-Logic-2-LCOV-results.zip sclogic.c << ALWAYS MAKE CLEAN FIRST>> %make cc -Wall -O0 -fprofile-arcs -ftest-coverage -g sclogic.c -o sclogic << DO NEW BUILD ON YOUR TEST MACHINE AND NOT PROFILE-ARCS and TEST-COVERAGE GCOV INSTRUMENTATION DIRECTIVES >> %./sclogic function_A function_B do function_C do function_D function_A do function_D … do function_C function_A do function_D function_A function_B do function_D <<RUN OF CODE TO BE TESTED WITH SOME PRINTF DEBUG OUTPUT HERE>>

%gcov sclogic.c File 'sclogic.c' Lines executed:100.00% of 29 sclogic.c:creating 'sclogic.c.gcov' <<RUN POST RUN COVERAGE ANALYSIS ON SOURCE FOR LAST RUN OF INSTRUMENTED CODE>> %gcov sclogic File 'sclogic.c' Lines executed:100.00% of 29 sclogic.c:creating 'sclogic.c.gcov' <<RUN POST RUN COVERAGE ANNOTATED TEXT GENERATION>> %cat sclogic.c.gcov -: 0:Source:sclogic.c -: 0:Graph:sclogic.gcno -: 0:Data:sclogic.gcda -: 0:Runs:1 -: 0:Programs:1 -: 1:#include <stdio.h> … 1: 41:int main(void) -: 42:{ … -: 52: // Test Case #2, Test use in logic 11: 53: for(testIdx=0; testIdx < 10; testIdx++) -: 54: { 10: 55: if((rc=(function_A() && function_B()))) 2: 56: function_C(); -: 57: else 8: 58: function_D(); -: 59: -: 60: } -: 61: 1: 62: return(1); -: 63:}

Summary of Lcov Commands

Sam Siewert 24

%lcov -t 'SC LOGIC 2 REPORT' -o sclogic.info -c -d . Capturing coverage data from . Found gcov version: 4.4.7 geninfo: WARNING: invalid characters removed from testname! Scanning . for .gcda files ... Found 1 data files in . Processing sclogic.gcda Finished .info-file creation <<RUN LCOV TOOL ON GCOV RESULTS>> %genhtml -o result2 sclogic.info Reading data file sclogic.info Found 1 entries. Found common filename prefix "/home/facstaff/siewerts/se420/src" Writing .css and .png files. Generating output. Processing file MCDC2/sclogic.c Writing directory view page. Overall coverage rate: lines......: 100.0% (29 of 29 lines) functions..: 100.0% (5 of 5 functions) % <<GENERATE THE WEB PAGES FROM LCOV RESULTS FOR BROWSING>>

What Does Gcov Show? Impact of Short-Circuit Logic in C http://mercury.pr.erau.edu/~siewerts/se420/code/MCDC2/SC-Logic-2-LCOV-results/MCDC2/sclogic.c.gcov.html

Sam Siewert 25

Every Other Driver If-call goes Into B, But B must return “1” after A does too

Function_C called 2x in 10 iterations, due to toggle + short-circuit logic

Tracing Tools for Linux LTTng – http://lttng.org/ Systemtap – https://sourceware.org/systemtap/ Ftrace – http://elinux.org/Ftrace Kernelshark – http://rostedt.homelinux.com/kernelshark/, https://lwn.net/Articles/425777/ Wireshark - https://www.wireshark.org/ (network) Syslog - http://linux.die.net/man/5/syslog.conf, http://linux.die.net/man/2/syslog

Sam Siewert 26

Kernelshark Trace VxWorks System Viewer