3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis...

22
Eff ti f Eff ective use of Automated GUI Interactions on System Verification LeRoy Dunn & Michael Mullins 2 March 2016 Copyright © 2014 Raytheon Company. All rights reserved. Customer Success Is Our Mission is a registered trademark of Raytheon Company.

Transcript of 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis...

Page 1: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Eff ti fEffective use of Automated GUI Interactions on

System Verification

LeRoy Dunn & y

Michael Mullins

2 March 2016Copyright © 2014 Raytheon Company. All rights reserved.

Customer Success Is Our Mission is a registered trademark of Raytheon Company.

Page 2: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Overview

This presentation seeks to provide an insight to the use of automation of data entry and interaction via a Graphical User Interface (GUI) to provide the input required to conduct integration and verification efforts of a large system.

2 March 2016 2

Page 3: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Topics

Why to automate and expected benefits

Ho to incorporate a tomationHow to incorporate automation

Practical lessons learned from application

Summary

2 March 2016 3

Page 4: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Congressional and Customer Desire

Annually addressed in House Reports for D.O.D. budgetsg

Navy NAVSEA Instruction 4215.1 authored in 2010 to provide guidance for implementation for2010 to provide guidance for implementation for Automated Test and Re-Test (ATRT)

From the House of Representatives Report for National Defense Authorization Act for FY2016

The committee believes that the valuable lessons from such activities as ATRT should be more widely leveraged across the Navy and the rest of the Department of Defense

2 March 2016 4

rest of the Department of Defense.

Page 5: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Return on investment

Return on investment–Direct cost savings for test execution–Direct cost savings for test execution Automating an existing test procedure

– First Procedure – investment resulted in a break even point of 8.6 test executions after automation

– Subsequent Procedures – investment resulted in a break even point of 6.1 test executions after automation

Automating a test without developing a test procedure– Automation resulted in 30% increase in development cost and 80%

reduction in execution costreduction in execution cost

– Break even point realized in 4 test executions

–Numerous indirect savingsg Vary based on test environment, customer relations, etc.

2 March 2016 5

Page 6: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Technical Benefits (1 of 3)

Increased test run count provides broader coverage of test objectivescoverage of test objectives–Execute automated script hundreds of times in a

Monte Carlo approachMonte Carlo approach

–Allows for wider range of test inputs

Provides better opportunity to witness intermittent–Provides better opportunity to witness intermittent failures Tool may even help provide characteristics that support debugTool may even help provide characteristics that support debug

of the failure

2 March 2016 6

Page 7: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Technical Benefits (2 of 3)

Broader exercise of operator entered values for data entry pointsfor data entry points–Random selection of enumerations rather than specific

entries dictated by test procedure for manualentries dictated by test procedure for manual execution

– Increased breadth of invalid entries to move away fromIncreased breadth of invalid entries to move away from testing boundary conditions

Manual Test Points

0 99 255Special Characters za

Manual Test Points

2 March 2016 7

Automated Test Points

Page 8: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Technical Benefits (3 of 3)

Immediate determination of Pass/Fail evaluations rather than post-execution data analysisp y

Screen captures of Pass/Fail evaluationsCreates evidence of findings for what was typically a–Creates evidence of findings for what was typically a Demonstration test procedure

–Consistent capture points–Consistent capture points

Verifying exact measurement that are difficult to conduct in real timeconduct in real-time–Stringent timing requirements

C l i i d d–Color mappings to requirements or standards

2 March 2016 8

Page 9: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Why to automate and expected benefits

Ho to incorporate a tomationHow to incorporate automation

Practical lessons learned from application

Summary

2 March 2016 9

Page 10: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Evaluating for Implementation

Review your Contract and/or Statement of Work– Incentives/Contractor Performance Assessment ReportIncentives/Contractor Performance Assessment Report

–May require updates to Master Test Plan and/or Test Procedures which require customer approvalq pp

Dozens of options based on desired approachp pp–More than 30 tools have developed in this area and

some are already obsolete

–Wide range of characteristics drive tool selection

–Understand both your corporate tool providers and the y p ptools available to your customer

2 March 2016 10

Page 11: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Gaining Customer Acceptance

Determine if tool selected requires customer certification for use in program qualificationp g q–Plan for collaboration as automated testing is likely to

be new to the customer as well as the contractor

–Understand limitations of the selected tool

Issue concerning Method of Verification (MOV)–Many tests of an HMI or heavily utilizing an HMI areMany tests of an HMI or heavily utilizing an HMI are

Demonstration, but automated testing replaces the human witness with a computer witness (i.e., a tool)

–May have ramifications for an existing specification2 March 2016 11

Page 12: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Incorporation into Formal Test Events

Customer acceptance/reluctance will vary greatly

One approach that has gained concurrenceOne approach that has gained concurrence –For converted tests, verify identical execution of an

automated test script and paper procedure during theautomated test script and paper procedure during the same witnessed test event Identical outcomes allows for future automated execution only y

–For automated only tests, allow the customer to observe the execution of the automated test procedure Demonstrate against specification or approved use case

–Address use of automation during Test Readiness Review and gain the board’s approval to proceed

2 March 2016 12

Page 13: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Why to automate and expected benefits

Ho to incorporate a tomationHow to incorporate automation

Practical lessons learned from application

Summary

2 March 2016 13

Page 14: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Execution Challenges (1 of 2)

Test automation infrastructure–Hardware upgradesHardware upgrades

–Lab support

–Scripting software and supportScripting software and support

–Documentation

Personnel training–Personnel training

–Security Plans

2 March 2016 14

Page 15: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Execution Challenges (2 of 2)Test maintenance cost

–Evolving System Under Test (SUT)g y ( )–Evolving test tool GUIs–Evolving lab environment

Starting the implementation of test automation either too early or too late in the development process –Too early: “spinning wheels” during initial development

i d d ROIperiods reduces ROI–Too late: Not enough test execution runs to recoup

investment losses (reduced ROI)investment losses (reduced ROI)

2 March 2016 15

Page 16: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Scripting Tradeoffs

Go path testing vs failure and error handling–Challenge of handling test step failure and errorChallenge of handling test step failure and error

exception sprawl

–Determine the appropriate level of test robustness and pp preliability for each automation case

Test step3 Test step4 Test step5 Test step6pass pass pass pass

Log step3 failure

Execute errorException3_1

Terminatetest and logtest results

Log step4

Execute errorException4_1

Execute errorException5_2

Execute errorException4_2

Execute error

Execute errorException5_1

err err

err

err

err

errfail

fail

cont contcont

RandomizationHuman in the loop introduces inherent variability vs

Log step4 failure

Log step5 critical failure

Execute errorException5_3fail

term

term

–Human in the loop introduces inherent variability vs scripting introduces repeatability and repetitiveness

2 March 2016 16

Page 17: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Graphics Comparison and OCR Tuning

For graphics comparison or Optical Character Recognition

Click (ImageName: “CloseWindow”, SearchType: Tolerant, HotSpot: (45 15) WaitFor: 10 Tolerance:p g

(OCR), start with default settings, then depending on

(45, 15), WaitFor: 10, Tolerance: 10, Discrepancy: 5, Pulsing: false, SearchRectangle: (180, 250, 240, 300))g , p g

test performance tune searches as needed for the specific GUI and test goals ReadText ((300, 200, 350, 220),

DPI: 80, Contrast: true, ContrastColor: black

Tuning can be done for individual searches or

ContrastColor: black, ContrastTolerance: 50, ValidCharacters: “A0123456789”, IgnoreSpaces: true)

globally (all searches)2 March 2016 17*eggPlant code snippets. For eggPlant documentation see http://docs.testplant.com/

Page 18: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Execution Limitations

Human-In-The-Loop (HITL) limitations–Non-GUI system controlsNon GUI system controls

Safety limitationsRadar ranges–Radar ranges

–High power equipment

S i li i i!

Security limitations–Screen locks, Password control, VNC servers

–Prevention of disclosure of classified capabilities

Consult SMEs as needed 0100010010010101001010110101011010101010001010001010 10101010100

–Test Engineers, Lab Manager, Security, etc.2 March 2016 18

010001010 10101010100010101000 00100100

Page 19: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Why to automate and expected benefits

Ho to incorporate a tomationHow to incorporate automation

Practical lessons learned from application

Summary

2 March 2016 19

Page 20: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Summary

Infrastructure– HW, SW, support, training

Customer Satisfaction

Value , , pp , gMaintenance

– SUT, GUIs, lab environmentD l t

– Direct long-term cost savings

– Indirect efficiencies Development

– Scripting, tuning Limitations

Improved Technical Approach– Thoroughness Limitations

– HITL, Safety, Cybersecurity– Robustness

– Expanded test capabilities

ROI– Frequency

2 March 2016 20

Consider ALL benefits, costs and drawbacks in the ROI analysis

Page 21: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Go Forward Plan

Start early–Use ROI analysis to drive plans

ROI analysis

Use ROI analysis to drive plans

–Get customer buy-in on approach

Start conservative

Buy In

Implementation

Execution

Start conservative–Better off with realistic ROI estimate

S ll

p

Start small–Focus on a small set (1-3) of near term test cases

Start easy–Automate the “low hanging fruit”

Reevaluate2 March 2016 21

Page 22: 3.2.16 430pm Effective use of Automated GUI …...–Use ROI analysis to driveplans ROI analysis analysis to drive –Get customer buy-in on approach Startconservative Buy In Implementation

Biographies Mike Mullins is a Senior Principal Systems Engineer at Raytheon Company,

Integrated Defense Systems. He has 20+ years of experience in real time communications, networking systems, and track fusion, including algorithmcommunications, networking systems, and track fusion, including algorithm development and IV&V. In his current assignment, Mike is the Integrated Product Team Lead for a Combat System Element. Mike has a Bachelors Degree in Computer Science from Florida State University.eg ee Co pute Sc e ce o o da State U e s ty

LeRoy Dunn Jr. is a Principal Systems Engineer at Raytheon Company, Integrated Defense System. He has 10+ years of experience in mechanical design of test equipment and radar electronics as well as 10+ years ofdesign of test equipment and radar electronics, as well as 10+ years of experience in systems requirements development, architecture, IV&V and data analysis. In his current assignment, LeRoy is a data analyst for a Combat System Element. LeRoy has a Bachelor’s Degree in Mechanical EngineeringSystem Element. LeRoy has a Bachelor s Degree in Mechanical Engineering from University of Massachusetts at Amherst, a Master’s Degree in Mechanical Engineering from Northeastern University, and a Master’s Degree in Computer Science from Worcester Polytechnic Institute.Degree in Computer Science from Worcester Polytechnic Institute.

2 March 2016 22