Post on 11-Jan-2016
description
1 / Platform Development Process
In-house limited
Automatically Measuring Code Coverage in Distributed EnvironmentLi Fei PengPlatform QA Part, NSC2009- 08-10
2 / Platform Development Process
In-house limited
Content
• Introduction• Tools for code coverage, lcov and
bullseye• Executing command remotely, jsch• A tool for automatically measuring code
coverage• Troubles in this work• Improvement
3 / Platform Development Process
In-house limited
Introduction
• A complicated application which runs in a distributed environment, CUBRID
• Code coverage need to be measured automatically
• Set up environment, update and compile source code
• Run test cases• Process code coverage result and upload to QA
website• Linux (support SSH by default)
4 / Platform Development Process
In-house limited
……
Introduction
SVN Server
Machines Running TC
QA Website
Control Machine
5 / Platform Development Process
In-house limited
Tools for code coverage
• Lcov– Line coverage– Function coverage– Free – Compile: -fprofile-arcs -ftest-coverage
• Bullseye– Function coverage– Condition/Decision coverage– Proprietary– User command cov01 to enable/disable coverage
build
6 / Platform Development Process
In-house limited
Tools for code coverage - lcov
• a graphical front-end for gcov
• collect gcov data for multiple source files
• create HTML pages containing the source code annotated with coverage information
• add overview pages for easy navigation within the file structure
7 / Platform Development Process
In-house limited
Tools for code coverage - bullseye
• Function coverage gives you a quick overview and condition/decision coverage gives you high precision
• Works with everything you can write in C++ and C, including system-level and kernel mode
• Short learning curve • Include or exclude any portion
of your project code • Run-time source code included,
for custom environments • Simple licensing. • Merge results from distributed
testing • Integration with Microsoft Visual
Studio
8 / Platform Development Process
In-house limited
Jsch
• JSch is a pure Java implementation of SSH2.• JSch allows you to connect to an sshd server
and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs.
• Ant provides sshexec and scp tasks by using Jsch
• We use two functions: – Remote exec: ls, lcov, cubrid server start– File transfer: scp, sftp
9 / Platform Development Process
In-house limited
Jsch
• We use two functions: – Remote exec:
• lcov• cubrid server start• cov01• make install• …
– File transfer: • scp• sftp
10 / Platform Development Process
In-house limited
A tool for automatically measuring code coverage
• Configure environment in each machines– Install lcov, bullseye, subversion– Set up environment variables, add the bin directory of lcov and
bullseye to PATH– Checkout the source code, save password
• Runing process– Update source code– Set up environment, compile and Install in local machine– Compress the source code and upload the source code to other
machines– Set up enviroment, uncompress, compile and install in each remote
machine– Run test cases– For lcov, collect code coverage data– Copy the code coverage data from other machines to control machine– Generate the visual result of code coverage in control machine– Upload the result to QA website
11 / Platform Development Process
In-house limited
A tool for automatically measuring code coverage
• SSHClient.java– Encapsulate the exec and file transfer
functions in this class– SSHClient.exec(String host, String user,
String password, String command)– SSHClient.scpFrom(String host, String user,
String password, string rfile, String lfile)
12 / Platform Development Process
In-house limited
A tool for automatically measuring code coverage
• CovUtil.java– Main()– Update and install:
• updateAndInstallCubrid()– updateAndInstallCubridLocal– updateAndInstallCubridRemote
– Code coverage• collectCCAndUpload()
– collect bullseye data or lcov data– analyze code coverage data(genhtml, no analyze for
bullseye)– Upload the result to QA website
13 / Platform Development Process
In-house limited
A tool for automatically measuring code coverage
• UsageUsage: java
com.nhncorp.cubridqa.codecoverage.CovUtil <mode> <properties file>
mode: install, sa, cc
0 20 * * 5 java -jar ~/cubrid_cov/cubrid_cov.jar install ~/cubrid_cov/cov.properties0 20 * * 6 java -jar ~/cubrid_cov/cubrid_cov.jar sa ~/cubrid_cov/cov.properties0 23 * * 6 java -jar ~/cubrid_cov/cubrid_cov.jar cc ~/cubrid_cov/cov.properties
java -jar ~/cubrid_cov/cubrid_cov.jar <mode> <properties file>
14 / Platform Development Process
In-house limited
A tool for automatically measuring code coverage
• Configuration file– The production installation directory– The directories of related tools– QA website– Remote machines– Other parameters:
• removeCubrid• Bullseys or lcov• Exec locally or remotely• Etc.
15 / Platform Development Process
In-house limited
A tool for automatically measuring code coverage
• cubridBuildDir=/home/xdbms/build/RB-8.2.0• cubridHome=/home/xdbms/CUBRID• lcovHome=/home/xdbms/opt/lcov-1.7• svnPath=/home/xdbms/opt/subversion/bin/svn• removeCubrid=true• debug=true• enable64bit=false• uploadHost=59.108.93.74• uploadUser=qahome• uploadPassword=password• uploadPath=/home/qahome/tomcat-6.0.18/webapps/qaresult/qaresultfile/cubrid/code coverage• sqlDir=/home/xdbms/workspace/qa_repository/scenario/sql• execLocal=true• execRemote=true• svnUpdate=true• bullseyeHome=/home/xdbms/opt/bullseye• enableBullseye=true• #remote.1=59.108.93.81,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/
subversion/bin/svn• #remote.2=59.108.93.67,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/
subversion/bin/svn• #remote.3=59.108.93.68,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/
subversion/bin/svn• #remote.4=59.108.93.69,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/
subversion/bin/svn• #remote.5=59.108.93.70,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/
subversion/bin/svn• #remote.6=59.108.93.71,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/
subversion/bin/svn• remote.7=127.0.0.1,xdbms,password,/home/xdbms/build/RB-8.2.0,/home/xdbms/CUBRID,/home/xdbms/opt/lcov-1.7,/home/xdbms/opt/
subversion/bin/svn
16 / Platform Development Process
In-house limited
Troubles in this work
• Environement variables: execute . /etc/profile . ~/.bash_profile before executing each command
• Bullseye can not enabled with lcov• In order to successfully merge two coverage
files, the source files used to build them must have the same timestamps
• There is some problems in utility lcov of v1.7: v1.6 is ok
• An error will be thrown when using genhtml: modifying the source code of genhtml
17 / Platform Development Process
In-house limited
Improvement
• Analyzing the data generated by bullseye
• Output the visual result and upload them to QA website
• Code reorganization
18 / Platform Development Process
In-house limited