testAutomation

Automated Software Testing using Open Source Testing ToolsBy Elfriede Dustin

http://www.idtus.com

AgendaWho is IDTState of Software TestingWhat is Automated Test and Re-test (ATRT) Approach to ATRTAdvantage of open sourceATRT and open source Our IDE and solutionATRT Challenges


Who is IDT?IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutionsDeliver turn key automated test suiteAutomated test strategy and identification of highest payoff areas to apply automation for your projectSelection of best automation tools for your projectUsing your existing test cases or develop test cases for youTest results documented in requirements traceability matrixTraining and pilot project implementation with companies


Who is IDT?IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutionsDeliver turn key automated test suiteAutomated test strategy and identification of highest payoff areas to apply automation for your projectSelection of best automation tools for your projectUsing your existing test cases or develop test cases for youTest results documented in requirements traceability matrixTraining and pilot project implementation with companies

We are currently hiring Java DevelopersPlus wed like to hire 2 or 3 grad students to work for us part time ( i.e. 20 hours week ) must be US citizen


State of Software Testing - Why ATRT? ?


State of Software Testing - Why ATRT? Utilize Automated Testing Strategies and Technology to Improve Productivity and Quality ManualAutomatedReduction in Test Days IDT Provides Automated Software Testing SolutionsIncreased Test Coverage Manual vs. AutomatedSpiral / Release 1Spiral / Release 2Spiral / Release 3Spiral / Release 4Size & Complexity of SW BaselineTest DaysTest DaysTest DaysTest Days50% or More of Overall Development Cost is Typically Spent on Testing


What is ATRT?Our definition of ATRT is:

Application and implementation of software technology throughout the entire Software Testing Life Cycle (STL) and QA lifecycle; with the goal to improve STL efficiencies and effectiveness


ATRT Spans the Software DevelopmentLife Cycle Visual ModelingUse casesDesignBuildAssembleRound-Trip EngineeringTest


ATRT Spans the Software DevelopmentLife Cycle Requirements


ATRT Spans the Software DevelopmentLife Cycle Automated Test ToolsAutomated Software TestingVisual ModelingExecutionManagementDevelopmentAutomated Testing Tools


ATRT Spans the Software DevelopmentLife Cycle Integrated Suite of ToolsAdditionally:MiddlewareInfrastructureDefect TrackingConfiguration ManagementMemory Leak DetectorsPerformance Testing ToolsDocumentation ToolsothersVisual ModelingRequirements Management andProcess Automation

Infrastructure


Types of Software Test Life-cycle Support Tools

Which tools are you familiar with?


Types of Automated Test Tools


Types of Tools (Contd)

Programming Phase Memory Leak and Runtime Error DetectionDetects runtime errors and memory leaksSource Code TestingVerifies maintainability, portability, complexity, and standards complianceStatic and Dynamic AnalyzersDepict quality and structure of code

Life-CyclePhaseType of ToolTool DescriptionAnalysis and Design PhaseDatabase DesignDevelops second generation enterprise client-server systemsStructure Charts, Flowcharts, and Sequence DiagramsManage processesTest Procedure GeneratorsGenerate test procedures from requirements, design, or data and object modelsSyntax Checkers/ DebuggersPerform syntax checking and have debugging capability; usually available with built-in programming language compiler

Metrics ToolsCode (Test) Coverage Analyzers or Code InstrumentorsIdentify untested code and support dynamic testingUsability MeasurementsProvide usability testing as conducted in usability labs


Types of Tools (Contd)


Deployment Approach Adapted ATLMCollect/Create: Requirements Test Cases Test Procedures Expected Results Interface Specifications System/Component Configuration DescriptionIdentify or Develop & Pilot Test Manager Test Scripts Data Comparators Performance Test Tools GUI Record/Playback Storing/Retrieving Results Populate Test Manager with Test Cases and Requirements

Develop Test Scripts from Test Procedures

Test Results: Pass/Fail by Test Case Pass/Fail Summary Test Time Required


Why Open Source?


Why Open Source?Advantage of Open Source:no licensing issues; i.e. license cost; maintenance; etc.easily modifiableadaptablelightweightflexibility - not tied to one vendor


ATRTs IDE - Eclipse Eclipse this site lists all the plugins http://www.eclipse-plugins.info/eclipse/plugins.jsp?category=SCM&pager.offset=0&firstItem=1 Eclipse/TPTP Open Source - Developed by IBMEclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions:Requirements Management http://sourceforge.net/projects/osrmt/ Configuration Management - SubversionTesting Tools (http://www.nabble.com/Eclipse-TPTP---Testing-Tools-f2262.html)Profiling: default Java applications, but can be extended to other apps


ATRTs IDE - Eclipse Eclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions (cont):Static Analysis: default for C++ and Java, but can be extended to other appsApplication monitoring and log analysisBuild tool antXml input and output - using an open source Java package http://jakarta.apache.org/ecs/ , which is a Java library that has an API for constructing and outputting XML.Test Manager STAF/STAX


Current Test Tool Implementation Example and ChallengesVendor-provided (Capture/Playback) Tool:

Automated test tools mimic actions of the test engineer.During testing, the engineer uses the keyboard and mouse to perform some type of test or action.Testing tool captures all keystrokes and subsequent results, which are baselined in an automated test script.During test playback, scripts compare latest outputs with previous baseline.Testing tools have built-in, reusable test functions.Most test tools provide for non-intrusive testing; i.e., they interact with the application-under-test as if the test tool was not involved.


Current Testing Tools Challenges (cont)

In-House Software Development required:

Capture/Playback tools generate hard-coded values; test scripts are not reusable, nor do they implement software development best practices right out of the box; scripts need to be modified.Capture/Playback tools dont necessarily provide all testing features required; code enhancements are often required to meet testing needs.Capture/Playback tools are not necessarily compatible with system engineering environment, and software testing scripts need to be developed in-house.Developed testing scripts are not cross-vendor compatible, even if the same scripting language is used


Linux Capture/Playback Tools IDT Findings

Replay Xcessoryhttp://www.scl.com/products/ics/motif/testing/replay/ www.ics.comQF-Testhttp://www.qfs.de/en/qftestJUI/index.htmlFroglogichttp://www.froglogic.com/pg?id=Google&category=squishjavaRedstone SoftwareEggplantKD Executorhttp://www.kdab.net/?page=products&sub=kdexecutorJameleonhttp://sourceforge.net/projects/jameleon/ShUnit2 for shell scripts testingGNU Linux Desktop Projecthttp://ldtp.freedesktop.org/wiki/HomeCantanahttp://www.ipl.com/pdf/p0003.uk.pdfcrontabPart of Unixcrontab, cruisecontrol, www.adminschoice.com/docs/crontab.htmDogtailhttp://people.redhat.com/zcerza/dogtail/about.html http://www.cyberciti.biz/tips/linux-automated-gui-testing-with-python-based-dogtail-tool.htmlExpecthttp://expect.nist.gov/


Automated Test and Re-Test (ATRT)ATRT includes a common framework comprised of re-usable software servicesOpen Source Software Test Automation Framework (STAF)

Provides automated test procedures, test inputs, and test results for SW componentsSuitable for use by 3rd parties not associated with development of the component


STAX is an automation systemWith STAX you can automate and monitor your entire test environmentSystem/product setupTestcase distributionTestcase executionTestcase results collection and analysisSTAX consists ofA programming language designed for automationPrograms written in this language are called jobsAn execution engine (similar to an interpreter for other programming/scripting languages) which manages the execution and runtime behavior of the jobA GUI application, called the STAX Monitor, which provides a dynamically updated view of your executing jobsA tool (STAXDoc) that is used to generate documentation for your STAX jobs What is STAX?


STAF can run on: * Windows 95 * Windows 98 (and 98 SE) * Windows Millenium Edition * Windows NT Version 4.0 * Windows 2000 * Windows XP * Windows Server 2003 (IA32, IA64, AMD64) * Windows Vista (IA32, AMD64) * Linux (Intel32/64, AMD64, PPC32/64) * Linux on zSeries (31-bit, 64-bit) * AIX Version 4.3.3.0 or higher (32-bit, 64-bit) * Solaris (Sparc) 2.6 and higher * Solaris AMD Opteron 64-bit * HP-UX 11.00 and higher (PA-RISC, IA64 32-bit and 64-bit) * OS/400 V5R2 or higher * z/OS UNIX V1.4 and higher


STAF Internal Services

DIAG Provides diagnostics services Internal ("DIAG")DELAY Provides a means to sleep a specified amount of time Internal ("DELAY")ECHO Echos back a supplied message Internal ("ECHO")FILE SYSTEM Allows you to get and copy files across the network Internal ("FS")HANDLE Provides information about existing STAF handles Internal ("HANDLE")HELP Provides Help on STAF error codes Internal ("HELP")MISC Handles miscellaneous commands such as displaying the version of STAFthat is currently running Internal ("MISC")PING Provides a simple is-alive message Internal ("PING")PROCESS Allows you to start, stop, and query processes Internal ("PROCESS")QUEUE Provides a network-enabled IPC mechanism for STAF Programs Internal ("QUEUE")SEMAPHORE Provides network-enabled named event and mutex semaphores Internal ("SEM")SERVICE Allows you to list services available on a machine and to examine theRequests that have been submitted on a machineInternal ("SERVICE")SHUTDOWN Provides a means to shutdown STAF and register for shutdownNotifications Internal ("SHUTDOWN")TRACE Provides tracing information for STAF services Internal ("TRACE")TRUST Interfaces with STAF's security Internal ("TRUST")VARIABLEProvides a method for maintaining configuration and runtime data(variables) Internal ("VAR")


STAF External ServicesThe executable code for external STAF services resides outside of STAFProc, for example in a Java jar file, a C++ DLL file, or a Rexx script file.

CRON Calls into STAF services at a specified time interval External (Java)EMAIL Allows you to send email messages External (Java)EVENT Provides a publish/subscribe notification system External (Java)EVENTMANAGER Allows you to call STAF services when a specified Event occurs External (Java)HTTP Allows you to make HTTP requests which can be grouped together in a session External (Java)LOG Provides a full-featured logging facility External (C++)MONITOR Allows a testcase to publish its current running execution status for others to read External (C++)RESOURCE POOL Allows you to manage exclusive access to pools of elements, e.g. VM UserIDs or Software Licenses External (C++)STAX Provides an XML-based execution engine External (Java)ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives External (C++)


STAFSTAF provides the infrastructure on which STAX builds. The full power of STAF and its services is exposed for use within STAX jobs.XMLThe STAX programming language is based on XML. This provides built-in structure to your jobs, as well as providing a set of existing tools for use in constructing your jobs, such as XML (aware) editors and XSLT.PythonThe STAX programming language builds on Python to provide a rich and accessible data modelSTAX's Python integration also allows you to access the wealth of existing Python librariesJavaThe STAX programming language allows you to access existing Java classes/libraries, providing another source for reuseTechnologies used in STAX


Automation TasksExecutionSynchronizationMonitoringResource ManagementAutomation CompletionTestcase Output AnalysisResults NotificationTestcase CleanupTestcase ExecutionSystem SetupAutomation Startup


End-to-End Automation with STAF and STAXEvent, EventManager, CronFS, ProcessProcess, Monitor, Log, Variable, Queue, ResPoolLog, Process Process Email, HTTP STAX

Job

Automation CompletionResults NotificationTestcase CleanupTestcase Output AnalysisTestcase ExecutionSystem SetupAutomation Startup


Key ATRT Architecture Requirements Support applications running on multiple computers Support applications developed in different languages Support applications running on different types of OSs Support applications which have GUI and those which do not (for example Interface testing) Support applications which use different types of network protocols such as TCP/IP, DDS, etc Support integration of multiple commercial testing tools from different vendors ( allowing as new or better products emerge in the market they can be utilized ) Support testing w/o having to install ATRT on the same computers as the application under test and be able to be distributed across computers


Display ApplicationDisplay ApplicationAnalysis & Display Application Simulation of Application Processing Messages Simulation of Application Processing MessagesSystem Under TestTest ManagerRTMBugzillaATRTEggplantR/PVNC RobotR/PSTAF/STAXPrototype Proof of ConceptWindowsBasedLinux Based


Able to successfully execute ATRT for 48 hours without operator intervention

During the tests:

VNCRobot - Approximately 25,000 operator key strokes were replayed STAF/STAX - More than 1,000,000 messages processed and verified by ATRT ATRT Prototyping Effort


Additional RequirementPristine test environment required automated testing tool cannot be installed on Application-Under-Test PC

Eggplant allows for remote capture/playback via VNCserver and KVM switch VNCRobot


Automated Software TestingSoftware testing paradoxWhat is it?


Effective Automated Software TestingCurrent State: Developing software in order to test softwareAutomate the test support development: automated IDL to code generation; test data generation (using FireEye from NIST); compare utilities; log file analyzers;Our Automated Software Testing Goal: Minimize Human interactionPossibly get on par with Hardware Automated Testing?


Hardware Testing: current stateMany automated test generators for manufacturing/ logic devices, circuit testing, etc., available i.e. SEI/ATG, uMaster, etc.Automated testing without human interaction is the norm for hardware testing


Hardware vs. SoftwareATHow is hardware testing different?Software Challenge:Changing requirementsGUI interfacesCode access/availabilityDefects uncovered earlier in the software development lifecycle are cheaper to fixSolution flexibility required problem dependent


Proposed solutionsFocus on Reusable ComponentsTest Data / Expected ResultsInterfaces MiddlewareAssuming code and models are available:Model based testing; automated test case generators based on modelsEclipse effective ATRT IDEProbes see Aprobe Code instrumentationBuilding self-testable componentsUsing standard language MOF to Text to generate automate test casesGUI Testing tool ideasNew tools - otherWorking with NIST to produce additional tools


GUI TestingJemmy - java library to test UI

http://jemmy.netbeans.org/

http://abbot.sourceforge.net/doc/overview.shtml

Perl modules

run via VNCServer?


Questions?Email [email protected] with any follow-up questions


IDT We are hiring.Please come see me now if interested or sendemail [email protected] with any follow-up

***********

testAutomation

Documents

Transcript of testAutomation