World Class Standards Review of TC MTS (Methods for Testing and Specification) Dieter Hogrefe ©...

World Class Standards

Review ofTC MTS

(Methods for Testing and Specification)

Dieter Hogrefe© ETSI 2009. All rights reserved

Board#75, 3-4 November 2009


B65(08)xx 2

Description

MTS is responsible for the identification and definition of advanced specification and testing methods, which take advantage of innovative techniques to improve the efficiency and economics of both the standard description and associated conformance and interoperability testing processes

Methodologies are established, in co-operation with the CTI and relevant Technical Committees, for future use in the development of standards, and of testing specifications, including interoperability issues.


B65(08)xx 3

Terms of Reference

Standardization of testing languages and methodology development Standardization of TTCN-3 Interoperability testing Methodology

IP Testing Voip SIP IPv6 generic test suites

Standards Engineering Standards Development Process Making Better Standards


B65(08)xx 4

Structure

Plenary level TTCN-3 Interoperability methodology Standards Engineering

WG IPT Voip, SIP, IPv6


B65(08)xx 5

Technical overview

TTCN-3 - Extended standard series New language concepts e.g. module interfaces Updated and additional language mappings: ASN.1, IDL, XSD New documentation tags New extension packages: advanced parameterization, advanced

types, static configurations

Development of TTCN-3 test suites in many application domains VoIP testing, ETSI IPv6 testing, ETSI LTE testing, ETSI/3GPP Automotive software testing, AUTOSAR Mobile enabling services, OMA Mobile communication, TETRA


B65(08)xx 6

Technical overview

Establishment of a TTCN-3 syllabus and certificate Cooperation of ETSI, GTB and iSQI Two accredited TTCN-3 training providers First exams Over 20 certificates issued

Improvement of TTCN-3 tool interoperability Development of a TTCN-3 reference test suite based on a snapshot of

3GPP LTE benchmark tests 1st TTCN-3 Plugtest test with commercial tools The Plugtest has brought tool makers closer together No grey areas identified in TTCN-3 standards based on the

compilation and execution of this LTE benchmark test suite


B65(08)xx 7

1st ETSI TTCN-3 Tool Plugtest 2009


B65(08)xx 8

Technical overview

Automated interoperability testing (IOT) Developed generic methodology for automated IOT that integrates

conformance checks with “pure” IOT• Extends ETSI’s general approach to interoperability testing• Independent of technology to be tested• Independent of test language, i.e., TTCN-3• Defines generic IOT test architecture

Applied concepts in development of highly reusable, library based TTCN-3 framework for automated IOT

Specified 50 tests for IMS core network NNI IOT (TC INT) Developed first TCI based ETSI SIP/SDP codec Developed first TRI based ETSI IOT adapter

• Includes upper tester and PCAP capture reader adapter

Used TTCN-3 test system in TC INT’s 3rd IMS Plugtest (Oct 09)


B65(08)xx 9

Technical overviewMeans of Interoperability Testing

EUT 1

InterfaceMonitor

Test Coordinator

Configuration interface

Application Support Nodes

Monitoring interface

SUT

Test Oracle

EUT 2

EUT N

Interconnecting Network

Stimulating interface

EquipmentUser


B65(08)xx 10

STFs

STF 370 Automated Interoperability Testing Funded by EC and ETSI

• Objective is to extend existing ETSI interoperability testing concepts for the purpose of automation and in context of distributed systems

Main stakeholders• ETSI TC Methods for Testing and Specification (owner of ETSI WIs)• ETSI TC INT (IMS Network Testing)• ETSI TC GRID• B2B community (mainly around HL7)• TETRA Association• WiMax Forum (NWIIOT)

Further signalled interest• IPv6 community• ITS community• Testing labs


B65(08)xx 11

STFs

STF 380 TTCN-3 extension and maintenance continuation Resolution of open CRs Cooperation with STF 160 on LTE testing Extensions for real-time and performance

• Real-time concepts and semantics • Timing constraints• Support for simulated time• Support for performance tests and benchmarks

Reference test suite (voluntary, based on inputs from tool vendors)


B65(08)xx 12

Major activities/deliverables

Standardization of testing languages and methodology development TTCN-3, Edition 4 Automated IOT

IP testing IPv6 generic test suites

Standards Engineering MBS WEBSITE addition of Security pages to MBS in support of OCG SEC


B65(08)xx 13

Present problems

the usual ones, i.e. workload, funding, travel restrictions of members, etc.


B65(08)xx 14

Successes/failures

TTCN-3 Success

• Fast growing user community• Increase of application domains• Strong international conference series T3UC• Growing number of TTCN-3 tools• International TTCN-3 syllabus and certificate

Shortcomings• Limited number of freely available/open source tools• Learning not well supportedconsider to establish TTCN-3 OS tools

2009, Bangalore, India2009, Sophia-Antipolis2008, Madrid, Spain * 2007, Beijing, China 2007, Stockholm, Sweden 2006, Berlin, Germany2005, Sophia-Antipolis2004, Sophia-Antipolis


B65(08)xx 15

Successes/failures

MBS Success

• Cooperation with OCG-SEC to produce Security Standards process guidelines (TR 102 790) and security extensions to the Making Better Standards website

Shortcomings• Problem to provide the necessary methodological and validation

documentation to support the ETSI Secretariat in their implementation of the MTS/OCG-IOP recommendations to The Board on improving the Standards Engineering Process.


B65(08)xx 16

Successes/failures

Automated interoperability testing (IOT) Successes

• Automated IOT concepts successfully validated in IMS Plugtest• Even event participants strongly valued conformance results• Feedback on standards collected

• Demonstrated that TTCN-3 is independent of a specific compiler by working in all project phases with at least two commercial TTCN-3 tools

• Open source contributions to TTCN-3 community , i.e., TCI SIP/SDP codecs and TRI IOP test adapter

• Effort savings of more than 50% for IOP trace analysis in IMS Plugtest due to TTCN-3 based IMS IOT tool!

Shortcomings• Due to lack of IMS client availability no validation of TTCN-3 framework

part dedicated to driving interoperability tests• Due to funding shortage not all IMS IOT conformance checks could be

automated, e.g., XML SIP message body & DNS message checks


B65(08)xx 17

How we operate !

Meetings Semi-annual plenaries Ad hoc meetings

Electronic working Mailing lists Mantis for CR handling


B65(08)xx 18

Different to other TBs because…

Horizontal issues Testing language TTCN-3 for use in other bodies and CTI Background material for use in other bodies and CTI

Research oriented Responsible for the identification and definition of advanced

specification and testing methods Methodologies are established, in co-operation with the relevant

Technical Committees, for future use in the development of standards, and of testing specifications, including interoperability issues


B65(08)xx 19

Globalization aspects

ITU-T TTCN-3 became Recommendation Z.16x Close collaboration with SG 17 on standardization methodology

TTCN-3 User Conferences Asia (in the meantime 10000s of users, esp. China and India) Europe (same) North America coming

TTCN3 usage all over the world is a big success


B65(08)xx 20

Where next ?

TTCN-3 extensions Real time testing Performance testing

Test Abstractions High Level Test descriptions

Test Automation Model based?

World Class Standards Review of TC MTS (Methods for Testing and Specification) Dieter Hogrefe ©...

Documents

Transcript of World Class Standards Review of TC MTS (Methods for Testing and Specification) Dieter Hogrefe ©...