Automated Testing Environment by Bugzilla, Testopia and Jenkins
-
Upload
walkerchang -
Category
Software
-
view
21 -
download
3
description
Transcript of Automated Testing Environment by Bugzilla, Testopia and Jenkins
Automated Testing Environment by Bugzilla, Testopia and Jenkins
Chin Pin Chang04/07/2014
1
2
Outline
• Bugzilla• Testopia• Jenkins • Automated Testing Environment (Hypervisor
Testing as an example)– Automated Testing Process– Example
• References
3
Bugzilla Introduction
• Bugzilla is a Defect Tracking System and it is server software designed to help you manage software development
4
Bugzilla Features
• Excellent security to protect confidentiality• Optimized database structure for increased
performance and scalability• Integrated email capabilities• Comprehensive permissions system• Editable user profiles and comprehensive email
preferences• Proven under fire as Mozilla's bug tracking
system
5
Bugzilla Requirements
• Hardware– CPU : 3 GHz– RAM : 4 GB or more recommended– Hard Disk : 50 GB of free space is large enough
• Operating System : Including Windows, Linux, and Mac OS X
• Software– Perl : 5.12– Database Server : Bugzilla supports MySQL 5.5,
PostgreSQL 8.3, Oracle and SQLite– Web Server : Apache 2.2
6
MySQL Installation
Step 1. Update
Step 2. Install MySQL
Step 3. Set the password
7
MySQL Installation (cont.)
Step 4. Login MySQL– user– password
8
MySQL Configuration
Step 5. Create an user for Bugzilla
Step 6. List users
9
MySQL Configuration (cont.)
Step 7. Create database for Bugzilla
10
Apache Configuration
Step 1. Install Apache2
Step 2. Modify Apache2 configuration file
11
Bugzilla Installation
Step 1. Download Bugzilla : http://www.bugzilla.org/
Step 2. Decompress the file to /var/www/
Step 3. Modify folder name and switch to /var/www/bugzilla
12
Bugzilla Installation (cont.)
Step 4. Execute checksetup.pl
13
Bugzilla Installation (cont.)
Step 5. Install module
14
Bugzilla Installation (cont.)
Step 6. Re-run checksetup.pl
Step 7. Need to modify localconfig
15
Bugzilla Installation (cont.)
Step 8. Modify Bugzilla configuration
Step 9. Modify permission
16
Bugzilla Installation (cont.)
Step 10. Re-run checksetup.pl
Step 11. Set up user information
17
Bugzilla Installation (cont.)
Step 12. Modify Apache configuration and restart Apache2
18
Bugzilla Installation (cont.)
Step 13. Installation complete
19
Testopia Introduction
• Testopia is a test case management extension for Bugzilla
• Testopia Requirements– Bugzilla– MySQL or PostgreSQL– Mozilla compatible browser– Additional Perl Modules
• Testopia Architecture– Test Plans– Test Cases– Test Runs
20
Relationship between Bugzilla and Testopia
• Since Testopia is an extension to Bugzilla, it relies on many of the same objects that Bugzilla uses to track bugs
• Testopia is comprised of several objects that are interdependent, so they make managing the testing procedure possible
21
Relationship between Bugzilla and Testopia (cont.)
• Component– A Bugzilla component – An attribute of a product
22
Relationship between Bugzilla and Testopia (cont.)
• Product– The product name belong to Bugzilla
23
• Milestone– A Bugzilla object– Testopia builds are associated with milestones
Relationship between Bugzilla and Testopia (cont.)
24
• Test Plan– At the top of the Testopia hierarchy are Test Plans. Before you can do anything else in
Testopia, you need a Test Plan – Test plan will serve as the storage point for all related Test Cases and Test Runs and it will
act as the Dashboard for your testing
Relationship between Bugzilla and Testopia (cont.)
25
Relationship between Bugzilla and Testopia (cont.)
• Test Case– Test Cases are the heart of all testing – Test Cases spell out what steps to take while running a test and what outcomes to expect – Test Cases are associated with one or more Test Plans and with zero or more Test Runs
26
Relationship between Bugzilla and Testopia (cont.)
• Test Run– Once you have defined a set of Test Cases, you are ready to run through those tests in a
Test Run – Each run is associated with a single Test Plan and can consist of any number of Test Cases
from that plan
27
Relationship between Bugzilla and Testopia (cont.)
• Build– Software development is usually an iterative affair. In Testopia, each iteration is called a build– In software testing, a string denoting the compiled results of a period of development– Builds are often associated with milestones of the project. This is reflected in the build's
relationship to Bugzilla's target milestones object
28
Relationship between Bugzilla and Testopia (cont.)
• Case Run– A Test Case-Run is the record of how a particular Test Case fared in a particular run for a given
Build in a given Environment
29
Relationship between Bugzilla and Testopia (cont.)
• Category– A property of a product that is used to classify Test Cases
30
Relationship between Bugzilla and Testopia (cont.)
• Environment– A list of the surrounding conditions that a test run is performed in
31
Testing Process of Testopia
The testing process in Testopia is as follows : Step 1. Create a Product and multiple ComponentsStep 2. Create a Test PlanStep 3. Create Test CasesStep 4. Create a Build
32
Testing Process of Testopia (cont.)
The testing process in Testopia is as follows : Step 5. Create EnvironmentStep 6. Create a Test RunStep 7. Execute the Test RunStep 8. Produce a test report on Dashboard
33
Testopia Installation
Step 1. Download Testopia– http://www.mozilla.org/projects/testopia/
Step 2. Decompress the file to Bugzilla folderStep 3. Run checksetup.pl and install module
34
Access to Testopia
Step 4. Re-run checksetup.pl
Step 5. Installation complete
35
Jenkins Introduction
• Jenkins is an open source continuous integration tool written in Java
• Jenkins provides a web-based user interface
36
Jenkins Features
• Easy installation• Easy configuration• Change set support• E-mail integration• Test reporting• Plugin support
37
Jenkins Installation
Step 2. Access to Jenkins– http://localhost:8080
Step 1. Install Jenkins
38
Install Testopia plugin on Jenkins
39
Integrate Jenkins with Testopia
• This plug-in integrates Jenkins with Testopia and generates reports on automated test execution
• With this plug-in you can manage your tests in Testopia, schedule and control in Jenkins
• It reads TAP test report formats, used to update Testopia test case's executions
40
Integrate Jenkins with Testopia(cont.)
• Testopia configuration section
Auto Test Process
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)Shell scripts (expect)
Shell scripts (expect)
Step 1. Retrieve Test Case from Testopia
41
1.
Step 2. Invoke corresponding script to initialize the test environment
2. Auto testing environment for ITRI ARM hypervisor :
Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest VM
Guest VM
Guest VM
Test Target
Step 3. Initialize the test environmentStep 4. Launch necessary Guest VMs
42
3.
4.
Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest VM
Guest VM
Guest VM
Test Target
Step 5. Execute testing tasks
43
5.
Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest VM
Guest VM
Guest VM
Test Target
Step 6. Generate and return test result (by TAP, Test Anything Protocol)
44
6.
Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)Shell scripts (expect)
Shell scripts (expect)
Step 7. Return test result to Jenkins
45
7.
Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)Shell scripts (expect)
Shell scripts (expect)
Step 8. Write corresponding result to Test Case in Testopia based on TAP
46
8.
47
Auto Test Example
Step 1. Add user on Bugzilla to execute auto testing
48
The process to create a Test Plan
• To create a new Test Plan in Testopia – Step 1. Click the "New Plan" link in the Bugzilla footer– Step 2. Enter a name for this Test Plan– Step 3. Select a product from the product list– Step 4. Select a type for this plan – Step 5. Select a product version– Step 6. Type or paste your plan document into the Plan Document Editor– Step 7. Click the "Submit" button
49
Step 2. Create a new Test Plan
• Plan Name – Test Plan name
50
Step 2. Create a new Test Plan (cont.)
• Product – Bugzilla product name
51
Step 2. Create a new Test Plan (cont.)
• Product Type– The testing type for this Test Plan
52
Step 2. Create a new Test Plan (cont.)
• Product Version– The Bugzilla product version
53
Step 2. Create a new Test Plan (cont.)
• Plan Document– This is document that spells out the type of testing and testing methods used
for this plan
54
The process to create a Test Run
• To create a new Test Run in Testopia – Step 1. Click "Create a New Test Run" link on your Test Plan or the New Run link in the footer– Step 2. Select which Test Cases to include – Step 3. Enter a Summary for this Test Run– Step 4. Select a build from the Build list or type the name for a new one – Step 5. Select an environment from the Environment list – Step 6. Click the "Submit" button
55
Step 3. Create a new Test Run
• Product Version– The Bugzilla product version
56
Step 3. Create a new Test Run (cont.)
• Run Manager– The user in charge of this Test Run
57
Step 3. Create a new Test Run (cont.)
• Target Completion Rate– The expect completion rate of target
58
Step 3. Create a new Test Run (cont.)
• Build– The default product build
59
Step 3. Create a new Test Run (cont.)
• Environment– The environment in use for this Test Run
60
Step 3. Create a new Test Run (cont.)
• Target Pass Rate– The expect pass rate of target
61
Step 3. Create a new Test Run (cont.)
• Summary– A short description of this Test Run
62
Step 3. Create a new Test Run (cont.)
• Notes– This field is a place to make notes and observations about this run
63
The process to create a Test Case
• To create a new Test Case in Testopia – Step 1. Click "Create a New Test Case" on the plan page or
New Case from the footer– Step 2. Enter a short description of your Test Case in the Summary field– Step 3. Select a Category– Step 4. Add a default tester – Step 5. List the steps for testing in the Action field – Step 6. Provide the expected outcomes in the Expected Results field– Step 7. Click the "Submit" button
64
Step 4. Create a new Test Case
• Summary– A short description of the Test Case
65
Step 4. Create a new Test Case (cont.)
• Default Tester– The person is assigned to this Test Case
66
Step 4. Create a new Test Case (cont.)
• Alias– A unique string that uses to identify test result
67
Step 4. Create a new Test Case (cont.)
• Priority – The level of testing
68
Step 4. Create a new Test Case (cont.)
• Category– The product category
69
Step 4. Create a new Test Case (cont.)
• Estimated Time– The estimated time that this Test Case should take to complete
70
Step 4. Create a new Test Case (cont.)
• Bugs– You can attach bugs to your Test Cases
71
Step 4. Create a new Test Case (cont.)
• Blocks– If this Test Case prevents others from being run their ID numbers will appear
here
72
Step 4. Create a new Test Case (cont.)
• Status : Test Case status – PROPOSED : This test case is a work in progress and has not been reviewed for accuracy. It is therefore not ready to be included in test runs– CONFIRMED : This test case has passed review and is ready to be included in future test runs. Only test cases with this status can be included in new test runs– DISABLED : This test case is no longer applicable to current testing
73
Step 4. Create a new Test Case (cont.)
• Add Tags– A user defined string used to classify Test Cases
74
Step 4. Create a new Test Case (cont.)
• Requirements– The requirement number or URL to a document containing the requirement
this Test Case is designed to test
75
Step 4. Create a new Test Case (cont.)
• Automated – Automatic Test Cases are run by a script
76
Step 4. Create a new Test Case (cont.)
• Scripts – The name of the script that runs this Test Case
77
Step 4. Create a new Test Case (cont.)
• Arguments – The specific arguments sent to the script
78
Step 4. Create a new Test Case (cont.)
• Add to Run– Entering a run ID here will include this Test Case in a Test Run if it is not
already included
79
Step 4. Create a new Test Case (cont.)
• Depends On – If this Test Case requires other Test Cases be run before this one, their case
numbers should appear here
80
Step 4. Create a new Test Case (cont.)
• Action – This field lists the steps of testing
81
Step 4. Create a new Test Case (cont.)
• Expected Results– This details the expected outcomes of a test
82
Process to Launch Test Script
Configure
1. Clean Environment2. Check Image3. Launch Test Script
Start Test Script1. 3.
2.
1. Login to Host2. Start VM3. Ping Host4. Turn off VM5. Return Result
Testopia
Contents : 1. Parameters2. Functions
4.
1 . Retrieve Test Script
83
Set up a new project
Step 5. Create a project on Jenkins to invoke Test Script
84
Step 6. Set Testopia configuration section
• Testopia Version– The version name of Testopia
85
Step 6. Set Testopia configuration section (cont.)
• Test Run ID– The Test Run ID of Testopia
86
Step 6. Set Testopia configuration section (cont.)
• Single Test Build Steps– The command to execute by single step
87
Step 6. Set Testopia configuration section (cont.)
• Iterative Test Build Steps– The parameters of Testopia are used to execute by test cases
88
Step 7. Set Result configuration section
• Test Anything Protocol (TAP)– TAP is a simple text-based interface between testing
modules in a test harness– It allows individual tests to communicate test results
to the testing harness
89
TAP File Example
• The path of the file
• Success contents
• Fail contents
90
Launch The Project
Step 8. Start this project on Jenkins
91
Jenkins Console
Step 9. Monitor the shell scripts
92
Jenkins Result
93
Return Result to Testopia
95
References
• http://www.bugzilla.org/• https://developer.mozilla.org/en-US/docs/Mozilla/
Bugzilla/Testopia#Requirements• http://blog.codylab.com/testcase-management-
using-testopia/• https://wiki.jenkins-ci.org/display/JENKINS/
Testopia+Plugin• http://testanything.org/• https://wiki.jenkins-ci.org/display/JENKINS/
Meet+Jenkins