STANDARDS, TESTING AND VALIDATION · 2017-12-06 · Achieving Interoperability – Best Practices...

Partners:

STANDARDS, TESTING AND VALIDATIONPresented by Sebastian MüllerProject Manager ETSI Centre for Testing and Interoperability © All rights reserved

ETSI Symposium onAchieving Interoperability – Best Practices in Standardization

• ETSI – A member‐driven Standards Organization (SDO) setting globally‐applicable standards in ICT since 1988

• Whether it is from:• LTE to Multi‐access Edge Computing; or• Digital Broadcasting to Network Functions Virtualisation; or• Emergency Telecommunications to the Internet of Things …

• … ETSI strives to produce high‐quality, relevant, timely, testable and interoperable standards

• Our members have the same ambitions for their products based on ETSI standards

• This presentation is about how together we achieve these goals• But first, a few words about what we do …

ABOUT THIS PRESENTATION

2


• ETSI – A member‐driven Standards Organization (SDO) setting globally‐applicable standards in ICT since 1988

GLOBAL STANDARDS COLLABORATION

(Canada)

(USA)

(USA)

(International)

(Japan)

(Korea)

(Japan)

(China)

(USA)(India)

3


VARIOUS TECHNICAL GROUPS26 Technical Committees (TC)• ATTM, BRAN, BROADCAST, CABLE, CYBER, DECT, EE, ERM, ESI, HF,

INT, ITS, LI, MSG, MTS, NTECH, PLT, RRS, RT, SAFETY, SCP, SES, SmartBAN, SmartM2M, STQ, TCCE.

4 Special groups• EP eHealth, SC EMTEL, SC SAGE, SC USER. 2 ETSI Partnership projects• 3GPP, oneM2M.14 Industry Specification Groups (ISG)• CCM, CIM, ECI, ENI, IP6, ISI, MBC, MEC, mWT, NFV, NGP, OEU,

QKD, SMT.1 Open Source Group (OSG)• OSM

4


ETSI DELIVERABLES AND APPROVALEN – European Standard: Normative• Approval level: NSO (minimum 90 days)EN – Harmonized Standard: Normative• Approval level: NSO + European Commission review (minimum 120 + 60 days)ES – ETSI Standard: Normative• Approval level: ETSI Membership (90 days)EG – ETSI Guide: Informative (guidance)• Approval level: ETSI Membership (Immediate)TS – ETSI Technical Specification: Normative• Approval level: relevant Technical Committee (Immediate)TR – ETSI Technical Report: Informative (informational)• Approval level: relevant Technical Committee (Immediate)GS – ETSI Technical Specification: Normative• Approval level: relevant Industry Specification Group (Immediate)GR – ETSI technical Report: Informative• Approval level: relevant Industry Specification Group (Immediate)

Harmonized Standards are ENs with a special status. They are produced in response to an EC mandate. They provide the technical detail necessary to achieve the ‘essential requirements’ of an EC Directive. Harmonised betweed CEN, CENELEC and ETSI. They are thus key enablers of the European Single Market.

5


STANDARDS ‘ECOSYSTEM’ (much SIMPLIFIED!)

6

ITU

ETSI

… to deliver standards that enable ...• Testable• Interoperable• Reliable• High‐quality ...... products

IEEE

IETF

Fora

Working with Vendors …

Working with Operators, Service Providers and Users …

Partnerships3GPP

OneM2M

Working with Partners …


OUR ROLE IN THE STANDARDS MAKING PROCESS

Validation and

ReviewDrafting Editorial

Check

Approval and

Publication

Work Item

Feedback

Feedback

Feedback

Published

Standard

Implementationand use

Maintenance and evolution resulting in a new Work Item for a revision of the standard

CTI is active here ... ... to influence here

7


OUR APPROACH TO ENSURE HIGH‐QUALITY, INTEROPERABLE STANDARDS

Implementations mature from prototypes to INTEROPERABLE commercial products

Market Awareness and

Pre‐Standardization:

Proofs of Concepts

Testing and Validation

Methodologies test techniques, frameworks,

repositories and tools

INTEROPERABLE standards evolve in parallel with product development

Standardized Test SpecificationsConformance, Interoperability etc.

PLUGTESTS EventsInteroperability, pre‐conformance, coexistence etc.

Product Testing (not done by ETSI)

Certification (not done by ETSI)

8


• Trial implementations built around well‐defined scenarios• demonstrate viability of standards solutions• provide feedback to guide the development of the standards• help build awareness and confidence in the technology• help develop diverse and open ecosystems

• Usually small groups (2‐3 vendors + 1‐2 operators)• exhibits at specific events• demonstrations in laboratories• experiences from (restricted) real deployment situations

• Successfully used by• ISG NFV (over 40 PoCs running) http://www.etsi.org/technologies‐clusters/technologies/nfv/nfv‐poc

• ISG MEC (4 PoCs running ‐more on the way)• TC NTECH/AFI (PoCs programme just initiated)

PROOFS OF CONCEPT (PoC)

9


INTEROPERABILITY TESTING

Tests end‐to‐end functionality between equipment or systems

11


WHY ETSI INTEROPERABILITY EVENTS?

11

• Foster interoperability

• Gather feedback• Validate and improve standards

• Debug and validate implementations (products)

• Free and open to anyone who has an implementation to test


• Over 200 events since 1999• 10 ‐ 15 events per year• Events on‐site and/or remote• Over 60 different technologies• http://www.etsi.org/about/what‐we‐do/plugtests

ETSI INTEROPERABILITY EVENTS LOOK LIKE THIS

12


• Hackathons• Recent initiative (for ETSI)• Combining Plugtests events with hackathon activity• Usually dedicated to addressing a specific technical ‘problem(s)’• Quick resolution of bugs found in the standard or implementations during the test event

• Specfests/Hackfests• Mostly oriented to APIs• Developer guides• Hands‐on tutorials• Bringing developers and standardizers together

HACKATHONS AND SPECFESTS/HACKFESTS

15


CONFORMANCE TESTING

Tests equipment for compliance to requirements in a base standard

16


DEVELOPMENT OF CONFORMANCE TEST SPECS

WHAT to test HOW to test(high‐level)

Test‐relevant behaviourand data

HOW to test(detailed)

Complete behaviourComplete data

Verdict assignment

Pre‐Conditions

Cleanup

Test Purposes

Compiled and Validated Executable Code

(binary)

Full protocol and system knowledge

Run‐time support

Test Descriptions Test Cases Executable Test Cases

Prose or TPLAN

Proseor TDLExtracted

Requirements

TTCN‐3(Source)Base Standard(s)

Certification

17


ALIGNING WITH THE STANDARDS DEVELOPMENT PROCESS

R0 R1

Validation Testing

Test Descriptions

Test Purposes

Plugtest#1

Test Cases

R2

R1 Products

R1Test Spec

Plugtest#2 Plugtest#3

19


SUMMARY (OR WHY WE ARE DOING THIS)

PoCsAwareness (external)Feasibility, relevance, timeliness of standardsProposed standardization topics… ...interoperable

solutions

PlugtestsCommunity building

(standards makers and implementors together)Validated standards

Tested (debugged) productsFeedback to testers and standardizers

Interoperable products……more likely to conform

Conformance TestingInternal proprietary testing

RegulationCertification

Conformant products……more likely to interoperate

Hackathons and SpecfestsInvestigate specific (technical) issues

EducationTaking the standards to

developers


• Air Traffic Management VoIP (with EUROCAE) events• Intelligent Transport Systems (ITS) events and conformance test specifications and frameworks

• Next Generation Emergency Communications (NG112) events• eCall events (with partner ERTICO)• Mission Critical Push‐to‐Talk (MCPTT) event (with partner TCCA)• oneM2M events (with partner TTA), conformance test specifications, user guides and hackathons

• Network Functions Virtualization (NFV)• Interop, API testing, Open Source project ...

• 3GPP UE certification tests

SOME ‘SUCCESS STORIES’

18


• Capitalising on synergy between Standardizers – Vendors –Users to deliver standards that are• Interoperable, timely, testable, high‐quality, relevant etc.

• Pre‐standardization activities are becoming an integral part of the standardization process

• Validated standards are key to interoperable products• Testing and certification is crucial in key technologies

• 3GPP LTE, ITS, oneM2M, DECT, TETRA ...• Interop, conformance and indirectly performance, robustness, reliability ...

• Need to encompass Open Source activities and approaches...

TAKEAWAYS

20


Sebastian Müller on behalf of

Anthony WilesDirector

Centre for Testing and Interoperability European Telecommunications Standards Institute

[email protected]

Thank you!

STANDARDS, TESTING AND VALIDATION · 2017-12-06 · Achieving Interoperability – Best Practices...

Documents

Transcript of STANDARDS, TESTING AND VALIDATION · 2017-12-06 · Achieving Interoperability – Best Practices...