TTCN, Past, Present and Future

O. Monkewich

Chairman, ITU-T Joint Coordination Activity on Conformance and

Interoperability Testing (JCA-CIT)

Overview

Past - some pre-history, before TTCN-3 Present - new ITU initiative Future - how can TTCN-3 and TTCN-3

experts help

In the Beginning OSI

Implementations from different suppliers to interconnect October 10-13, 1983, Ottawa, Canada,

ISO/TC97/SC16/WG1 Dave Rayner of NPL, UK, appointed Rapporteur of OSI

Conformance Testing Methodology and Framework First meeting achievements

Static Conformance – visual review Dynamic Conformance – in an instance of communication Will need a test notation

Test specifiers to be allowed their own notation The standard would provide some examples

Test methods, Verdicts, Multi-layer, Single-layer, Embedded testing

February 13-15, 1984 GMD Darmstadt, Germany ISO/TC97/SC16/WG1 Conf. Group First discuss of requirements for test notation

ASPs, PDUs, Test Methods Verdicts

Must consider single-layer testing embedded single-layer testing multi-layer testing

Neutral test notation or an FDT? November 12-16, 1984, Ft. Lauderdale

Florida, ISO/TC97/SC16/WG1 Conformance Anthony Wiles, Uppsala University, Sweden

Uppsala University developing tools for testing Conformance Group developing a neutral test

notation but is also looking at FDTs Plan to meet with the ISO FDT experts

Prepare testing examples for comparison with other potential notations

Tables, Trees and FDTs February 3-13, 1985, Paris, France, ISO/TC97/SC16

France - a table per test case UK proposed behaviour trees – textual, branching with indentations Both adopted – birth of TTCN Met with FDT groups - LOTOS, Estelle

Continue with TTCN but keep looking at FDTs

July 7-12, 1985, Manchester, UK, ISO/TC97/SC16 Conformance Rapporteur Group Upper Testers, Test Coordination Procedures Requirements on test notation:

Verdict assignment: Pass, Fail, Inconclusive Valid, Invalid and Inopportune PDUs timing, addressing sending, receiving PDUs Compare observed vs. required outcome

The “Bows-and-Arrows” notation October 18-30, 1985, Paris, France, ISO/TC97/SC6

Conformance test suites 3 X.25 test suites in ad hoc time sequence diagrams

DTE-DCE physical link – computer-modem DTE-DCE link access to network – computer-modem DTE-DTE packets over network – computer-computer

Canada argues for using TTCN, labels the US notation “bows-and-arrows” notation

Canada, please translate to TTCN or … Initiated effort to develop the University of Ottawa/Nortel

“Workbench” tool for TTCN

Stabilize TTCN for X.25, MHS April 21-29, 1986, ISO/TC97/SC21 jointly with

CCITT Q.47/VII, San Diego (La Jolla), California Should ASN.1 be added for PDU description Canada wants TTCN syntax to expressed in BNF TTCN is informal - sufficient to define it by example CCITT (now ITU-T) will develop a test suite for MHS, they

prefer TTCN A version of TTCN was stabilized (San Diego version)

TTCN still informal, but, can we describe it in BNF

September 8-26, 1986, Egham, UK, ISO/TC97/SC21 Canada argues need BNF for machine readability Keep San Diego (La Jolla ) version of TTCN for X.25 and MHS LOTOS gaining interest Criteria for test notation identified

Notations not evaluated by February 1987 will not be considered TTCN is still considered informal: it is for human readability,

abstract test suite specifiers, not for test realizers PIXIT introduced (tongue-in-cheek acronym) but was adopted Test suite work not for the conformance group, shifted to protocol

specifier groups

X.25 Test Suites will Allow Other Test Notations (in an Annex)

October 13-24, 1986, Tokyo, Japan ISO/JTC1/SC6 Conformance testing Definition of any new test notation will be added

to the test suite annex Canada promises X.25 Packet Level test suite in

TTCN to the next meeting

BNF for TTCN arrives February 3-12, 1987, Nice, France,

ISO/JCT1/SC21/WG1 Conformance Rapporteur Canada tabled BNF description of TTCN, TTCN.MP to come

April 27-May 1, 1987, Washington, DC, TTCN meeting Construct to permit TTCN trees to execute in parallel introduced

February 28-March 9, 1988, McLean, Virginia, USA, ISO/JCT1/SC21/WG1 TTCN should be separate part of DP9646 rather than an annex

of DP9646 Part 2

TTCN.MP May 23, 1988, Teddington, NPL, UK, TTCN Editing meeting July 14-22, 1988, Stockholm, Sweden, SO/JCT1/SC21/WG1

Conformance editing meeting (hosted by Telia Research/Teletest) Teddington/Stockholm TTCN becomes Part 3 of DP9646

(DP9646-3) TTCN made self-consistent, integrated with BNF, machine

readable TTCN.MP Could not agree on Operational Semantics

November 28-December 8, 1988, Sydney, Australia ISO/IEC JTC1/SC21/WG1 Conformance Group DP9646 Part 3 did not make it to DIS Operational Semantics substantially changed Stable “kernel” of TTCN could go to DIS after the next Stockholm

editing meeting, July 18-29, 1989.

TTCN extensions

February 7-17, 1991, Phoenix, Arizona, ISO/JCT1/SC21/WG1 Conf. Group TTCN extensions

Rational for concurrency, TCs, PTCs and MTCs Coordination Messages, Coordination Points, Test

Configurations Starting and stopping Test Components CREATE and

TERMINATE May 20-31, 1991, Arles, France, ISO/JCT1/SC21

Editing TTCN Extensions DEFAULT ACTIVATE, DEACTIVATE, RETURN,

OTHERWISE Handling background activity (cannot be forced or ignored)

TTCN-1 was considered completed

TTCN-1 approved as ISO/IEC International Standard

Ottawa, February 18-19, 1992, ISO/IEC/JTC1/SC21/WP1 TTCN-1 (TTCN edition 1) approved as ISO/IEC International

Standards and published on 12 November 1992 Anthony Wiles hands over TTCN Editorship to Os

Monkewich Further Extensions as well as the ongoing Extensions

Concurrency: Draft Amendments One and Two to TTCN-1:

RTP North Carolina, 19-27 October, 1992, ISO/IEC/JTC1/SC21/WP1 TTCN Editing Meeting Extensions and Further Extensions

TC, PTC, MTC Test Suite Operations and control agreed CREATE(List of PTCs) passed by value RETURN for default trees only ?DONE added TERMINATE deleted

The Last TTCN-2 Meeting

BSI, Chiswick, London, UK, September, 1994, ISO/IEC/JTC1/SC21/WP1 TTCN Editing Meeting Cleanup, over 200 comments, defects reports

on Extensions and Further Extensions Modular TTCN – Southampton, July 1994 Remaining work done by e-mail

TTCN Edition 2 (TTCN-2)

TTCN-1 vs. TTCN-2 TTCN-1 Non-concurrent:

Maximum of two PCOs Single service provider Behaviour is expressed successive branch

indentations only TTCN-2 Concurrent

Unlimited number of PCOs Multiple service providers Behaviour expressed by parallel execution of

behaviour trees Modules for reuse of ATSs Uses 54 different tables

Years after the beginning Started in 1983 TTCN-1 completed and published in 1992 TTCN-2 completed and published in 1998 ISO/IEC JTC1 stops work on OSI

Defect Report processing more difficult ETSI corrected defects to produce TTCN-2++

ITU-T SG10 and SG7 (now SG17) Languages, SDL, MSCs, ASN.1 and TTCN Adopted TTCN-2++ in 2002 as Rec. X.292 2001- 2011, SG17 approved and continues to updated 11

TTCN-3 Recommendations based on ETSI standards

Since 2002, ITU-T has approved11 TTCN-3 Recommendations

Z.161: TTCN-3 core language Z.162:TTCN-3 tabular presentation format (TFT) Z.163: TTCN-3 graphical presentation format (GFT) Z.164: TTCN-3 operational semantics Z.165: TTCN-3 runtime interface (TRI) Z.166: TTCN-3 control interface (TCI) Z.167: TTCN-3 mapping from ASN.1 Z.168: TTCN-3 mapping from CORBA DL Z.169: TTCN-3 mapping from XML data definition Z.170: TTCN-3 documentation comment specification Z.171: TTCN-3 and the use of C/C++

New Horizons ITU-T WTSA-8 Resolution 76 approved

TSB to carry out specific tasks with the view to introducing the use of ITU Mark Program

ITU Mark Program permit manufacturers and service providers to add a

visible ITU mark to their product if it conforms to ITU-T Recommendations

Assist the Developing Countries Capacity building Establishing of regional test laboratories

Interoperability – the ultimate objective Interoperability does not follow from

conformance Today’s ITU-T Recommendations not good

enough for global interoperability No explicit conformance or interoperability

requirements Interoperability is achieved by means of on-the-fly

adjustments Adjustments not made publicly available Interoperability exclusive

Developing Countries want both Conformance and Interoperability

Role of TTCN-3 Conformance testing is required to

Qualify for the ITU Mark Be entered into the ITU Database

Testing quality Must support the criteria for formal testing Test laboratory accreditation Criteria for Global harmonization and recognition Product certification

Voluntary – wide acceptance in the marketplace Regulatory – meet requirements of national law

Can anything go wrong?

ITU Recommendations for the modern times Packaging of conformance requirements

specifications from a standard implemented in different boxes no declaration of which requirements are in what box boxes may communicate over proprietary interfaces no standardized interfaces or access points for testing

Boxes and network components from multiple vendors Streamed video utilizing a variety of "hidden" codecs Operators and service providers downloading their software

onto user devices Plug-ins downloaded for use with the implementation Hidden proprietary actions as providers make changes to the

user system

CIT Section for ITU Recommendations

Proposal to include requirements for Reporting any factors needed for achieving

conformance and interoperability Defining the essential elements for end-to-end

communication Information for network element management for

multi-vendor mix of network component JCA-CIT draft CIT Section and Guide

Use Guide to complete the CIT Section Include Section in each Recommendation

Joint Coordination Activity-Conformance and Interoperability Testing (JCA-CIT)

Membership TSB staff and Study Group chairmen and vice-chairmen Representatives appointed by the Study Groups Experts form A.4/A.5 SDOs such as ETSI

Meet 3 to 4 times per year Combination of in body, via Webinar or GoToMeeting

Focus Recommendation ingredients for testing Assistance in the implementation Resolution 76

Roadmap for implementation ITU Mark, ITU Database, capacity building, test laboratories

URL: http://www.itu.int/en/ITU-T/jca/cit/Pages/default.aspx

New opportunities for experts Training in Developing Countries

Conformance, compliance and interoperability Production of Requests For Proposal (RFPs)

procurement of products Establishing new test laboratories Test laboratory accreditation Product certification On-site, hands-on training

New test laboratories Establishing regional test laboratories

Setting up a test laboratory from start Defining the requirements for the lab Mission statement Business plan

laboratory operation cost recovery Funding land purchase

Specification of the equipment needed for its operation

Scope of what will be tested Plans for accreditation

Contribution to standards Conformance and interoperability

Improving Recommendations for conformance and interoperability

Development and standardization of test suites The testing process

Recognition criteria Assertion of compliance Credentials for admission to be listed by ITU

Conclusions

15 years to develop early concepts used in TTCN-3 10 years of TTCN-3 New ITU and Regional initiatives creating new

opportunities for TTCN-3 and its experts Testing experts are needed to

Develop and improve standards to allow testing Help Developing countries gain technical capabilities Help build new test laboratories in regions

Wide recognition of test results Accreditation and Certification ITU Database population ITU-Mark management