Top100summit christina
-
Upload
christina-geng -
Category
Documents
-
view
140 -
download
1
Transcript of Top100summit christina
Automation Test Framework Continue Improvement
Welcome PresentationPresented by Christina Geng
AGENDA
❶ Case Introduction❷ Testing Strategy Evolution❸ Case ROI Analysis❹ Success factor❺ Revelation
Title :7 Times Evolution of Test Automation Framework under
JS Product for Efficiency Improvement
Target :
❶ Build a robust test automation framework
❷ Manage test cases efficiently
❸ Reduce test case running time and maintenance cost
❹ Shared framework under scrum development
CASE INTRODUCTION
WHO AND WHAT?
Data Analysis Visualization
• reporting, online analytical processing • analytics• data mining• process mining• event processing• business performance management• benchmarking• text mining• predictive analytics
WHO AND WHAT?
Data Analysis Visualization
• reporting, online analytical processing • analytics• data mining• process mining• event processing• business performance management• benchmarking• text mining• predictive analytics
Programming language: Javascript/Java
Provide service: The Visualization Service is a software module which provides various type of visualization objects that can be
used in different applications on different platforms (Web, Mobile and Desktop).
Visualization types: • A basic visual component like chart (Bar, Column Pie, Line, etc ), Geo Map, Gauge, Table or Grid.
• An advanced visual component like (Tree map, Candlestick, Tagcloud) • Above visual component with interactions and animations
• Combination of multiple visual components, it could be composed by same type of visual components or different type of component.
AGENDA
❶ Case Introduction❷ Testing Strategy Evolution❸ Case ROI Analysis❹ Success factor❺ Revelation
We have it!
Selenium
Data D
riven
Jenkins
Test Link
Full Automation
Developer environm
ent
What’s m
ore…2011.6 2011.12 2012.032012.06 2012.09 2012.10 2012.12
VERSION-1
Test page
SilkTest
Pie Bar Line BackgroundTitle Legend Layout Tooltip Theme ValueAxis AnimationPie y y y y y yBar y y y y y y y y yLine y y y y y y y y y
Scheduler: ASTECTC Management: ExcelTC: test pageTest tool: SilkTest
VERSION-1
Test page
SilkTest
+ -HTMLTest
page
1. First automation framework setup2. Overview functional test case
coverage3. Reduce testing overlap for shared
component
writing test case
maintenance
the relation between TC and record
resource
We have it!
Selenium
Data D
riven
Jenkins
Test Link
Full Automation
Developer environm
ent
What’s m
ore…2011.6 2011.12 2012.032012.06 2012.09 2012.10 2012.12
VERSION-2
DIV
DIVDIV
DIVTest page
Selenium
Scheduler: ASTECTC Management: Excel/XML (test manager)TC: test pageTest tool: Selenium Test Driver
Other tools• Auto generate test plan from Excel to xml• Test Manager
DIV
DIVDIV
DIVTest page
VERSION-2
+ -1. Reduce test pages2. Reduce resource running cost3. Test steps turns to be keyword driven in xml4. Easy case migration5. Write test case in .xls, auto generate .xml
Selenium
DIV
DIVDIV
DIVTest page
test driver
test case sync
case unit
test data redundancy
We have it!
Selenium
Data D
riven
Jenkins
Test Link
Full Automation
Developer environm
ent
What’s m
ore…2011.6 2011.12 2012.032012.06 2012.09 2012.10 2012.12
VERSION-3
Wrapper API
Unified Test Driver
Abstract Test Data
Discard Test Page
And How?
VERSION-3
Wrapper API
backgroudcolor
visiblelegendposition
verticalOffsethorizontalOffset
visiblelabelFont
labelColorformatString
layoutpaddingtooltip
enablecustomization
textmainValue
subValuetitle
main subSelectability
modedeSelectale
defaultSelectedItemsxAxisMemberyAxisMember
barbarMarkerSize
measureMember
initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
updateChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
testUtil.generateChart(viz/line; TestData.a1a1m1m1)
testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])
VERSION-3
Abstract Test Data
publicpublic
privateprivate
testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])
mySQL
testUtil.generateChart(viz/line; TestData.a1a1m1m1; legendVisibleTrue)
VERSION-3
Discard Test Page
<testsetcase id="test.workflow.layout.line.showTitle" enabledtype="enabled"/><testsetcase id="test.workflow.layout.line.hideLegend" enabledtype="enabled"/><testsetcase id="test.workflow.layout.line.hideAxis" enabledtype="enabled"/><testsetcase id="test.workflow.layout.line.showAxisTitle" enabledtype="enabled"/>
titleVisibleTrue = [['title.visible',true]];titleVisibleFalse = [['title.visible',false]];titleNormal = [['title.visible',true],['title.text',"Set customer title"]];titleLong = [['title.visible',true],['title.text',testData_Text.Long]];titleAlignmentCenter = [['title.visible',true],['title.alignment',"center"]];titleAlignmentLeft = [['title.visible',true],['title.alignment',"left"]];titleAlignmentRight = [['title.visible',true],['title.alignment',"right"]];legendVisibleTrue = [['legend.visible',true]];legendVisibleFalse = [['legend.visible',false]];legendHierarchicalTrue = [['legend.visible',true], ['legend.isHierarchical', true]];legendPositionLeft = [['legend.visible',true], ['legend.position',"left"]];legendPositionRight = [['legend.visible',true], ['legend.position',"right"]];legendPositionTop = [['legend.visible',true], ['legend.position',"top"]];legendPositionBottom = [['legend.visible',true], ['legend.position',"bottom"]];
VERSION-3
Unified Test Driver
selenium1
selenium2{StartBrowser}IE, Chrome, Firefox (selenium2)
RemoteWebDriver webDriver = null;
webDriver = new FirefoxDriver();
Robot()
ImageComparer
VERSION-3
Unified Test Driver
selenium1
selenium2
{StartBrowser}Safari (selenium1)
SeleniumServer seleniumServer = new SeleniumServer();
seleniumServer.boot();
selenium = new DefaultSelenium("localhost", 4444, "*safariproxy "+ safariLocation,urlRoot);
CommandExecutor executor = new SeleneseCommandExecutor(selenium);
DesiredCapabilities dc = new DesiredCapabilities();
webDriver = new RemoteWebDriver(executor, dc);Robot()
ImageComparer
VERSION-3
Unified Test Driver
selenium1
selenium2 {TakeSnapshot}Point p = this.getElementOffset(targetElement);
Dimension d = this.getElementSize(targetElement);
BufferedImage bi = new Robot().createScreenCapture(p.x, p.y, d.width, d.height);
Robot()
ImageComparer
VERSION-3
Unified Test Driver
selenium1
selenium2
ImageComparer
{FileCompare}cmd.exe ImageComparer.exe –file filepath1 filepath2 – threshhold 100
Robot()
We have it!
Selenium
Data D
riven
Headless
Test Link
Full Automation
Developer environm
ent
What’s m
ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
VERSION-4
Tooltip Selection
Gesture
CSSButton
Link
Dropdown
Checkbox
Table Text
Scroll
Hover
Animation
Theme
Upload
Download
Tooltip Selection
Gesture
VERSION-4
Selection
VERSION-4
Gesture
VERSION-4
var eventObj = {type : originalEvent.type,timeStamp : originalEvent.timeStamp,pageX : originalEvent.pageX,pageY : originalEvent.pageY
};eventRecord.push(eventObj);
var handler = function(event){var clickEvt = document.createEvent('MouseEvents');clickEvt.initMouseEvent(event.type, event.pageX,
event.pageY…);d3.select('.main .plot')[0]
[0].dispatchEvent(clickEvt)};testAPI_lasso_heatmap = [
{"type":"mouseover","timeStamp":1348739963751,"pageX":126,"pageY":94},{"type":"mouseover","timeStamp":1348739963751,"pageX":485,"pageY":94},{"type":"mousemove","timeStamp":1348739963766,"pageX":485,"pageY":208},{"type":"mousemove","timeStamp":1348739963773,"pageX":126,"pageY":208}
]
VERSION-4
Record
Replay
VERSION-4
start to checkdiff image folderdiff image location
p4 folder
replace BM directly
We have it!
Selenium
Data D
riven
Headless
Test Link, Jenkins
Full Automation
Developer environm
ent
What’s m
ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
VERSION-5
How to arrange test case?
VERSION-5
Test Link
TC (.xls/.xml)
Executable TP
import
export
Result logsave
VERSION-5
Continue Integration
1. Trigger
CI Server Test Executor
Build Machine Code repository
Manual Auto
TC mngmt
Test Case Pool
Test Driver
Selenium2 Java/C++
Web Server
2.TP/Platform/build info
resources
4. get assets
libs
4. get test plan3. Build info 5. result write back
5. result log
.xml.xls
VERSION-5
Where we are.
1. Trigger
CI Server Test Executor
Build Machine Code repository
Manual Auto
TC mngmt
Test Case Pool
Test Driver
Selenium2 Java/C++
Web Server
2.TP/Platform/build info
resources
4. get assets
libs
4. get test plan3. Build info 5. result write back
5. result log
.xml.xls
VERSION-5
Test Link
JenkinsJava Tool
PerforceWhere we are.
We have it!
Selenium
Data D
riven
Headless
Test Link, Jenkins
Full Automation
Developer environm
ent
What’s m
ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
Engineering
VERSION-6
http://casperjs.org/
VERSION-6
Paint
Layout Display
http://phantomjs.orgPhantomJS (Headless Webkit)
VERSION-6
Test driver inner logic
CasperJS IE/Safari/FF/Chrome
configBrowser?
Open browser
RunJS (test step)
….
Take Snapshot
Result validation
Save result log
Read JS listTP
PhantomJS:1. open webkit2. runJS3. Take snapshot4. close
not completenot complete
Reporting
And that’s OK?
We have it!
Selenium
Data D
riven
Headless
Test Link, Jenkins
Full Automation
Developer environm
ent
What’s m
ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
trigger
CI Server Test Executor
Build MachineP4 server
TC Management
Test Case Pool
Test Driver
Selenium2 Java/C++
get resources
sync up
.java
per check-in
local
Test Executor
result log
VERSION-7
Casper/PhantomJS
trigger
CI Server Test Executor
Build MachineP4 server
TC Management
Test Case Pool
Test Driver
Selenium2 Java/C++
get resources
sync up
.java
per check-in
local
Test Executor
result log
VERSION-7
Casper/PhantomJS
Eclipse Plug-in
PerforceJenkins
1. Set up environment
2. Test case execution
3. Test result reporting
4. Maintenance
CASE ROI ANALYSIS
1) Test machine2) Assets (util, BM…) 3) TP4) Libs
1) Browser compatibility2) Run in headless
1) Email sending2) Sonar integrated with CI
1) BM update2) TC management
Functional Test: BAT (nightly), FA
1000+ cases
4.5h47min
SUCCESS FACTOR ❶ ❷❸❹
Focus
SUCCESS FACTOR ❶ ❷❸❹
Team work
SUCCESS FACTOR ❶ ❷❸❹
Continue Improvement
SUCCESS FACTOR ❶ ❷❸❹
Learn 360°
OUR REVELATION
不被传统的测试方法束缚;从产品用到的技术找到突破;持续改进的思想;不断尝试的理念
CONTRIBUTORXiaolu, Ye
Christina, GengHelen, JiangChen, ZhouPeipei, Jiang
Coco, DaiGrace, ZhaoMinghao, LuKent, Wang Izzie, Wang
Phil, Xu
SUPPORTED FROMDaoyu, Wei
Li, ZhaoJimmy, YangAlex, FengAlex, Su Jiwei, Li ... …
THE ENDNO