© 2006 by IBM Corporation; made available under the EPL v1.0 | September 6, 2006 Achieving...

© 2006 by IBM Corporation; made available under the EPL v1.0 | September 6, 2006

Achieving Continuous Integration with the Eclipse Test and Performance Tools Platform

Using TPTP Automatable Services to continually test and report on the health of your system

Achieving Continuous Integration with the Eclipse Test and Performance Tools Platform | © 2006 by IBM Corp.; made available under the EPL v1.0

Agenda Part 1

Introductions (10 mins)

What’s on this USB Key / CD (5 mins)

TPTP Overview (The Big Picture) (30 mins)

TPTP Test Project Overview and Details (15 mins)

Agile Software Development and Tooling (15 mins)

TPTP Automatable Services Framework (45 mins)

Part 2 Assembling a Test Automation Framework (75 mins)

Building Automatable Services (135 mins)

TPTP Resources (5 mins)

Conclusion (10 mins)


Introductions

Instructors

Scott E. SchneiderIBM Rational Software

Eclipse TPTP Project Committer

[email protected]

Joe ToomeyIBM Rational Software

Eclipse TPTP Project Committer

[email protected]


Introductions (continued)

Attendees

Please introduce yourself: Name and company or university affiliation

Level of experience with Eclipse development and the TPTP Testing Tools Project (previously Hyades)

Your goals for this tutorial


Agenda Part 1












What’s on this USB Key / CD

Eclipse, TPTP and dependencies



TPTP Agent Controller



Code template files for the hands-on work

Other ancillary xml files (for the hands-on work)

Informa open source library for authoring RSS feeds (for the hands-on work)


Agenda Part 1












TPTP Project Overview

Test and Performance Tools Platform (TPTP) project is an Eclipse top-level project.

TPTP project was formed in August 2004 by promoting and expanding Hyades project. The former Hyades project was reorganized into four new projects.

Hyades project was formed in December 2002 under Eclipse Tools project.

For more on the project visit http://www.eclipse.org/tptp


TPTP Project Overview (continued)

Mission is “to build a generic, extensible, standards-based tool platform upon which software developers can create specialized, differentiated, and interoperable offerings for world class test and performance tools.”

Being developed by three Eclipse Strategic Developer members and a total of eight Eclipse member organizations.


TPTP Project - Participants

In alphabetical order:

Computer Associates

Compuware

FOKUS

IBM

Intel

OC Systems

SAP

Scapa Technologies


TPTP Project - Composition TPTP is composed of 4 projects:

PlatformTestTrace and ProfilingMonitoring

Each project is divided into technology domains:

UIAnalysisEMF ModelCommunication/Collection/ControlExecutionAgents/EventsDoc

Each technology domain divided into components, each with an assigned owner and committers.


TPTP Project - ArchitecturePresentation System

Test

Trace

EMF Data Models

Log

StatisticalXMI

Real TimeExport

Runtim

eM

onitor / log

Trace Analysis

And Profiling

Test Creation

and Execution

Artifact

Managem

ent

Eclipse TPTP UI

Standard Widgets andCore Plug-ins

Reference PerspectivesAnd Workflow

Target System

Data C

ollection

ApplicationD

ata Collection

Interface

InjectionCorrelation

ExecutionEnvironment

Log Collection

Trace Collection

System PerformanceMonitor

JVMPI Monitor

Data Loader

DistributedData Collection

Framework

Agent

Control Interface

Agent

Control Interface

DistributedControl

Framework

Testability Interface

Test Engine

Eclipse Platform

JVMTI Monitor

TBD


TPTP Platform Project: The Foundation1) Reference UIs and perspectives

Basic metaphors for interacting with target systems and resources - includes both remote and local systems.

UI Frameworks and common navigators, viewers, editors, and wizards. It provides many extension points used by TPTP tools and vendor tools.

2) Standard data models, and assets repository EMF-based information model implementations for test,

trace, log and statistical data.

Framework for running rule based queries against data model instances and some simple queries. This is the primary mechanism that ties UI with data models

3) Common data collection and execution framework Execution environment that supports deployment,

launch, and control of test cases and applications

Data collection and control frameworks and agents.

Communication service which is used by the distributed data collection and control frameworks.

Eclipse Reference UIs

Execution Framework

XMI Assets


TPTP Tracing and Profiling Tools Project Provides frameworks for tracing and

profiling tools by extending the TPTP platform.

Includes profiling tools for both single-system and distributed JavaTM applications.

Provides correlation service A JVMPI monitoring agent that collects

trace and profile data.Collects and analyzes heap and stack information

Anticipate JVMTI-based monitoring agent.

Anticipate additional language and protocol support.

A generic tool kit for probe insertion - can instrument byte code of Java applications.


TPTP Tracing and Profiling Tools Project (cont)Profiling and Logging Perspective

TPTP profiling tools provide comprehensive information on the performance of a Java application

JVM performance

Object allocations and references

Garbage collection

Object methods performance

Object->Object interactions

Thread interactions

And more…


TPTP Tracing and Profiling Tools Project (cont) Profiling Monitor View

Monitor contains processes

from hosts that are being

monitored

Hosts being

monitored in a

profiling session

Java processes

associated with a host

Profiling agents


TPTP Tracing and Profiling Tools Project (cont) Interactions View

The Interactions Views present execution flow of an application according to the notation defined by UML2 Trace Interactions.

Host Interactions Process Interactions Thread Interactions Agent Interactions Class Interactions Object Interactions


TPTP Monitoring Tools Project Provides frameworks for building monitoring tools

by extending the TPTP platform.

Includes tools for monitoring application servers (JBoss, Jonas, and Websphere) and system performance.

Collects, analyzes, aggregates, and visualizes data captured in the log and statistical models.

A typical example is collection of CPU or memory utilization and viewing, aggregation, and analysis of that data.

Supports Common Base Event (CBE), provides services for mapping of custom log formats to CBE, and regular expression based log filtering.

Correlates data across multiple instances of log and statistical models; also across instances of trace and test history models

Enables symptom and pattern analysis.


TPTP Testing Tools Project

Provides testing tools framework by extending the TPTP platform.

Includes testing tools such as JUnit based component testing tool, Web application performance testing tool, and a manual testing tool.

Common nomenclature and metaphors facilitate integration of disparate test types and environments.

Deployment and execution of tests on remote and distributed systems.


Agenda Part 1












TPTP Testing Tools Tasks and Concepts Extensible platform for test creation,

deployment, execution and results analysis.

Test PerspectiveTest Navigator and Test UI framework

ToolsJUnit

JUnit Plugin

URL

Manual

Automated Eclipse GUI Datapooling

Editor for creation and API for runtime access

Automatable Execution frameworkLocal or distributed execution of tests

Results analysisExecution History viewer

BIRT based reporting system


Data Models (Test Model)

The TPTP Test Model encompasses:

UML 2 Testing Profile Implementation

Behavioral model

Execution Histories



UML 2 Testing Profile Implementation

TPTP was part of the finalization process (as Hyades) in the OMG’s acceptance of the U2TP submission. We have implemented our reference implementation based on the MOF independent meta model.



Behavioral model

The behavioral portion of the Test Model is an implementation of UML2 Interactions. We have provided a façade on top of this behavioral model for several reasons:

To provide a more semantically familiar programming model (loops and decisions, not combined interaction fragments with operators.)

To facilitate interoperability among tests from different test tool vendors (multiple ways of representing the same transaction)

To (potentially) allow TPTP based products to access behavior that is not modeled (via another implementation of the same façade interface.)



Execution Histories (Test Logs) Stores the results of test executions

References to test, trace, deployment info

Verdicts Test Log messages and console out

from test Provides an extensibility mechanism for

custom typed attributes


Test Specific Extension Points

Test Creation Wizard Extension point for defining wizard to create specific types of tests

Test Editor Extension point provided for associating editor with specific type of test

Code Generator Extension point for registering a custom code generator for a type of test

Publish Reports Extension point for custom report types with custom content


Execution Environment

The TPTP Testing Tools project extends the capabilities provided by the platform to allow distributed execution, control and data collection of tests.

Extenders of the TPTP Testing Tools project can enable distributed execution, control and data collection of their own test types in two ways

By leveraging and extending the components provided by the Testing Tools project (less code to write, less control over behavior)

By extending the Platform Project (more code to write, more control over behavior)


Agenda Part 1












Agile Software Development Background I Agile software development is an

abstract conceptual framework that concrete agile software development methodologies realize (such as Extreme Programming or Scrum)

Spawned from a momentum in the late nineties towards methodologies prescribing frequent software releases, increased programmer to customer collaboration and less up-front documentation proliferation


Agile Software Development Background II The Manifesto for Agile Software

Development specifies the agile software development belief system

Additionally defined principles further detail and reinforce the manifesto, in particular*:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

Working software is the primary measure of progress

http://www.agilealliance.org/

*Complete Manifesto and agile principles hosted athttp://www.agilemanifesto.org/

More valuable to agile methods


Agile Software Development Background III Approaches to software

development can be informally graphed with respect to predictability (or inversely with respect to adaptability)

Approaches shown with increasing adaptability as farther left

More Adaptive More Predictive

What if the requirements drift after the design is concretized?

What if 25% of the staff’s programmers leave to another company?

What if a new version of a dependent piece of software is released?


Prevalent Themes Across Agile Methods ITest-Driven Development [TDD] (integral to agile methods)

Construct automated tests validating the feature functional requirements (and other non-functional requirements if appropriate) – should fail at first

Code implementations to ensure automated tests succeed (a single unit test or a suite of unit tests)

Refactor code subsequently executing automated tests again until clean minty feeling (code is not delivered until tests pass)

Continuous Integration [CI] (integral to agile methods)

Design a build process and configure build machinery to enable frequent, clean and direct from source automated integration builds (including automation of tests)

Deliver code frequently to minimize integration pains and ease source control merge complexity – builds are triggered by delivery or scheduled time

Publish successful (and therefore good) builds and give interested parties a warm fuzzy feeling


Prevalent Themes Across Agile Methods IITest-Driven Development [TDD] (integral to agile methods)

First, design and implement (or record) tests using TPTP (JUnit, JUnit Plug-In, Manual, URL or UI test types as appropriate)

Next, design and implement feature using Eclipse (and appropriate modeling tools if desired)

Lastly, execute tests using TPTP, interpret test results, if successful refactor (and test again) and deliver to source control via Eclipse team support frequently

Continuous Integration [CI] (integral to agile methods)

Create automated build machinery around Apache Ant and/or with Eclipse PDE build (or a continuous build framework such as CruiseControl)

Leverage the TPTP automatable services framework and the published services performing: test execution, test result interpretation and test result reporting

Use the command line interface, supplied ant tasks or another client adapter (add your own) to command TPTP from the build environment


Contrast of TDD and CI Process Concepts Examining test-driven

development process concepts alongside continuous integration process concepts leads to some observations

TDD and CI share the common goal of seeking test success and the common activities of frequent building, frequent testing and frequent integration TDD and CI seek a steady-state of assured quality (assertion is that the software is in a good state before integration) TDD and CI each contain repetitive steps in their respective processes TDD is a process local to the developer and CI is a process in the large (where the sources out of TDD are piped into the CI process input)

• TDD/CI is scalable where a build server might deliver to a downstream build server (as if the upstream build server assumed the role of developer in TDD)• Each arrow from upstream to downstream represents a delivery and an assertion that the software is in a good state (all tests pass)

• CIx is continuous integration layer x• TDDy is test-driven development layer h

CI1

CI0

TDD0


Compartmentalized Development Process Pattern Carefully abstracting out commonalities in TDD and CI (and parameterizing the process) a

software process pattern occurrence appears in each (pattern in the design pattern sense but applying to the software process domain instead of the software design domain)

Each process instance is compartmentalized and therefore greatly minimizes the possibility of negatively affecting the other process instances (assured quality before publishing)

ExecuteTests

InterpretResults

AcquireAssets

MutateAssets PublishAssets

RealizeTests

InitiateBuild

not typically feeding the same process instance

TDD and CI specify operations for the various boxed named operations (the arguments to the pattern parameters)

Compartment

Input

Output


Agenda Part 1












Automatable Services Framework Motives I

Desire for bi-directional communication with Eclipse applications (as the server) that may or may not be running already, where the client is either internal or external to Eclipse and where the interaction is headful or headless

Desire for heterogeneous client environment support including command line interface, scripting environments, Eclipse-external Java applications and non-Java languages

Desire for an open-source framework that abstracts the complexities

Can you execute a test for me?

CLI (Client)

Can you publish my test results?

Jython (Client)

Did the test have a pass verdict?

Ant Script (Client)

Huh?

Eclipse(Server)


Generalize the problem from concrete problem instances and then design a framework that solves the general solution through concrete solution instances

Command a particular parameterized remote operation within the context of an Eclipse instance

Interact with the operation in a method compatible with the client’s environment (script-friendly data types)

Automatable Services Framework Motives II

SOA (service-oriented architecture)

Automation Paradigm (flexible typing model as well as reflection)


An automatable service is a service that exhibits automation-like characteristics (and in particular one that conforms to the TPTP automatable services framework) – it can be commanded in a programmatic way with only interface and semantics clearly specified by the service (the service provider is dynamically plugged-in at runtime)

A service is a particular behavior/ability of the system that takes inputs and produces outputs (the actual implementation and location of the service is not precisely known to the caller but the process of how to request the service is known)

For example, two concrete implementations of a service might be installed within a given Eclipse/TPTP instance and the one that best meets the caller’s or system's criteria would be used (separates interface/contract from implementation), a service is defined via Eclipse extension points and implemented via creating extensions implementing these extension points

Automation is the ability to send requests into a running instance of Eclipse/TPTP in a programmatic fashion to command a certain operation to be performed – this is usually done in headless mode where a UI is not visible and interacted with

Automatable Services Framework Concepts I


Given this definition of an automatable service (and the guts of the frameworks that enables ease of creating and using these automatable services) it is possible to take program functionality that is typically limited to running inside of an isolated Eclipse instance and expose it publicly to consumers outside of the Eclipse environment (by wrapping it with TPTP automatable service abstractions)

If the program features are implemented using a service-oriented architecture from the start, then it might be a matter of simple service policies to define which services are public TPTP services and which are further restricted to a tighter access policy (not the case currently)

Automatable Services Framework Concepts II

Can you execute a test for me?

CLI (Client)

Can you publish my test results?

Jython (Client)

Did the test have a pass verdict?

Ant Script (Client)

Your request is my

command!

Eclipse(Server)


Automated Services Framework Architecture I

Automatable Services Framework

Architecture Explosion


Automated Services Framework Architecture II

ComponentsOutsideEclipse

ComponentsInsideEclipse




Automated Services Framework Architecture III


Components Outside Eclipse Components Inside Eclipse

Layer I Consumer

Layer II Marshaller

Layer III Broker

Layer IV Provider



Automated Services Framework Architecture IV


Layer I Consumer

Layer II Marshaller

Layer III Broker

Layer IV Provider


Components Outside Eclipse Components Inside Eclipse

Ant Script(Ant task

client adapter)

Shell Script(CLI client adapter)

Other such as Jython

(using client adapters)

Java Code (Java client

adapter)

AutomationClient

AutomationServer

Test Execution

Service

Test Results Report

PublicationService

Test Results Interpretation

Service

Others, not restricted to Test or TPTP

(format source code)


Interaction Points Touched by end-user or commanding process

Touches marshaller layer passing in service request (service identity, directive and properties)

Extensibility Points Additional client adapters can be written to expose automatable services to unique environments (API)

Concrete Extensions Java, Ant and Shell-CLI client adapters

Automated Services Framework Consumer Layer

Excerpt of Ant script example packaged with TPTP, uses the Ant automatable services framework client adapter (exposed via custom TPTP ant tasks)

(External Clients and Client Adapters)


Automated Services Framework Marshaller Layer

Interaction Points Touched by the consumer layer, receives service request and complementary data

Touches broker layer (the automation server) directly or indirectly based on marshaller strategy

Extensibility Points Additional marshaller strategies are expected as usage evolves (internal)

Concrete Extensions Out-of-process and in-process strategy to start

Layer II Marshaller

Layer III Broker

Layer II Marshaller

Layer III Broker

In-process marshaller strategy used for intra-Eclipse service request and invocation, no additional Eclipse is launched

Out-of-process marshaller strategy used for cases where an Eclipse instance is launched to satisfy service request

Eclipse JVM

Standalone JVM

Eclipse JVM

(Automation Client)Launch Eclipse (send request)


Automated Services Framework Broker Layer

Interaction Points Touched by marshaller layer receiving service request and associated data

Touches provider layer conjuring up appropriate service provider per request and passes request data onwards

Extensibility Points Additional service provider selection strategies (internal)

Concrete Extensions First service provider wins strategy

(Automation Server)

Automation server implements Eclipse’s standard runtime.applications extension point serving as a broker for the service provider population, also notice the execution.service extension point is implemented by the “HelloWorldService” – a simple service sample shipped with TPTP


Automated Services Framework Provider Layer

Interaction Points Touched by the broker layer passes in directive and properties (to the concrete service provider in this layer)

Touches broker layer indirectly with return state

Extensibility Points Additional service and service providers can be published (API)

Concrete Extensions Test execution service, test result interpretation service and test result publication service

Service provider hierarchy, from the most abstract to the most concrete classes – shows the test execution service (a concrete extension) as the leaf class

(Automatable Services)


The automated services framework was contributed to Hyades 3.3 and TPTP 4.0 at the same time (TPTP 4.1 adds increased quality and minor refactoring)

The framework spans three logical source folders within one plug-in and is built as two JARs (tptp-automation-client.jar and tptp-automation-server.jar)

TPTP 4.2 will offer two new automatable services (test result interpretation and test result publication) complementing the test execution service

End-users can add their own custom automatable services

Additional client adapters are easily and quickly written, especially in environments that can delegate to the Java client adapter (such as the Ant task client adapter)

Automated Services Framework Logistics

Automatable Service Framework


Agenda Part 1












Assembling a Test Automation Framework

Authoring TPTP JUnit (15 mins)

Hands-on Demo



Authoring TPTP AutoUI (30 mins)

Hands-on Demo



Using Automatable Services (30 mins)

Hands-on Demo


Agenda Part 1












Building Automatable Services

RSSService (75 mins)

Hands-on Demo



Creating a de.nava.informa plugin from the open source library

(See Eclipse Rich Client Platform, chapter 10.1 and chapter 20.)


Building Automatable Services: RSSService

Create a new Plugin Project org.eclipseworld.tptp.execution

.rssservice

No UI contributions

Open MANIFEST.MF and Add dependencies on de.nava.informa, org.apache.ant & org.eclipse.hyades.execution,



Import provided code templates

Window -> Preferences -> Java -> Editor -> Templates

c:\eclipseworld2006\etc\ RSSService_templates.xml



Open extensions tab and create service extension ID = RSSService

New service

Click class

Put in created package, class name RSSService



Quick-fix to correct serialization warning Add member variables for each service parameter

Insert code template rss1_define_member_variables Add initialize() call to execute method Add body of initialize method to read service parameters

Insert code template rss2_add_initialize_body



Add call to RSSWriter.append to body of execute() method Insert code template rss3_call_rsswriter

Quickfix to create RSSWriter writer class Quickfix to RSSWriter.append() method RSSService – return this; Add sysout to RSSWriter



Create RSSTest with main() Populate the body of the RSSTest main method

rss4_main_rsstest

import org.eclipse.hyades.automation.client.adapters.java.AutomationClientAdapter;



Test Export deployable plugins and fragments

Directory: c:\eclipseworld2006\eclipse

Run RSSTest (should see service was called)



Fill in the body of the RSSWriter.append method rss5_rsswriter_append_body

Organize imports – (util.date, basic channel, net.URL) Implement the RSSWriter.createChannel method

rss6_create_channel_method Implement the RSSWriter.appendChannel method

rss7_create_appendChannel Organize Imports (iterator & collection)



Test Export deployable plugins and features

Directory: c:\eclipseworld2006\eclipse

Run RSSTest (should see RSS feed written)



Ant Adapter Make new source folder src_ant

Add new source folder to build.properties (in . target and new ant-rss.jar)



Make new class Package org.eclipseworld.tptp.execution.ant.rssservice

Class RSSClientAutomationAdapter

Superclass org.eclipse.hyades.automation.client.adapters.ant.AutomationClientAdapter.Automation

Constructors from superclass



Override/Implement methods: execute & init Define service parameters

rss8_define_member_variables Add parameters as properties & set command to execute

rss9_add_properties Generate setters for attributes overwrite body of init() method

rss10_implement_init_method



Drag antlib.xml and build.xml from c:\eclipseworld2006\etc to org.eclipseworld.tptp.execution.ant.rssservice

Export deployable plugins and features Directory: c:\eclipseworld2006\eclipse



Add plugins to ant-home classpath (in Window -> Preferences -> Ant -> Runtime

Our new RSSService plugin ant-tptp from plugin

org.eclipse.hyades.execution



Run the build.xml file Run As -> Ant Build

To publish to a webserver, simply change the feedfile parameter in the build.xml file to point to the desired location

C:\Program Files\Apache Group\Apache2\htdocs\myrss.xml



FSSService (building and debugging) (60 mins)

Hands-on Demo


Agenda Part 1












TPTP Resources

Getting TPTP source and tests with CVS


TPTP Resources

Where can I see if this is a known issue, file a defect, etc.

https://bugs.eclipse.org/bugs


TPTP Resources

Where can I find documentation, information on mailing lists, newsgroups, release plans, etc?

http://www.eclipse.org/tptp


TPTP Resources

Other TPTP based testing tools:

IBM Rational Performance Tester IBM Rational Functional Tester IBM Rational Manual Tester

Scapa Test and Performance Platform


Agenda Part 1












Conclusion

TPTP Provides powerful testing tools

You can use the TPTP Automatable Services Framework to incorporate automated tests into your build cycle to achieve continuous integration

By using TPTP in your CI cycle, you gain More powerful test types

Better / more extensible reporting

Distributed test execution (easy platform coverage)


Conclusion

What we have shown you in the tutorial is a small fraction of what TPTP offers. If you want to learn more, please attend some of the many other TPTP talks being offered.


Legal Notice

Java and all Java-based trademarks are trademarks of SunMicrosystems, Inc. in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

© 2006 by IBM Corporation; made available under the EPL v1.0 | September 6, 2006 Achieving...

Documents

Transcript of © 2006 by IBM Corporation; made available under the EPL v1.0 | September 6, 2006 Achieving...