Carrier Ethernet World Congress 2009 Multi-Vendor ......Nokia Siemens Networks FlexiPacket Microwave...

GlobalUbiquitous

Manageable

Carrier Ethernet WorldCongress 2009

Multi-Vendor Interoperability EventWhite Paper

Berlin, September 21–25, 2009

2

Carrier Ethernet World Congress 2009 Multi-Vendor Interoperability Test

EDITOR’S NOTEWhere do Carrier Ethernetimplementations stand today?After six months of prepa-ration and a two-week hotstaging with 60 engineersfrom 24 vendors, here is thelatest news for the CarrierEthernet World Congress ‘09:Despite the global economiccrisis, the majority of CarrierEthernet vendors (listed on

the next page) continue to develop new productswith consideration for multi-vendor interoperability.Our tests show that the technology becomes broader— opening up new markets such as in E-NNI andLTE backhaul. It becomes also deeper — enablingadvanced end-to-end, multi-vendor faultmanagement. And it certainly gets more mature —for example, Synchronous Ethernet interoperability.On the other hand, more deployments in more areasalso results in growing challenges for the technologyand continuous work for the standards bodies: Thereis plenty of work to standardize MPLS-TP; E-NNIPhase 1 is in its final stages ofstandardization in the MEF but notquite fully implemented yet, andgood quality synchronization withIEEE-1588:2008 is possible butcannot be taken for granted.“Global Interconnect”, the latestMEF project, is a topic of highvisibility these days. Serviceproviders have been connectingtheir Carrier Ethernet networks fora while — we believe the futurewill belong to scalable and resilientinterconnects, successfully testedthis time with MPLS and Ethernet.ENNI failure recovery times wereon the long side - understandablegiven all the protocol translationsfrom one network to the other, butwe hope for improvements.Our Ethernet OAM tests advancedfurther for both Link and ServiceOAM. The vendors accepted and mastered thechallenge. In the end most issues were configura-tional: Vendors did not agree how to carry CFMacross the network (S-VLAN tagged, untagged,with/without MPLS label) — clarification bystandards bodies seems to be required.On the transport side, MPLS-TP pre-standard,sometimes even pre-IETF-draft testing was interesting.There are still a lot of strategic questions to bedecided but technical progress is moving fast.Furthermore, it was a surprise to see PBB-TE back onmultiple vendors’ agenda. Finally, we tested nativeEthernet ring resiliency (ERPS / G.8032).Please review the detailed results below and visit thelive demos in Berlin or the virtual booth after the event!

INTRODUCTIONFrom a technical perspective, Carrier Ethernet is arich, broad topic which potentially covers and takesadvantage of many different technologies in itsimplementation. In our efforts to craft the technicalscope of our interoperability events in order to givethe industry a focused message, we decided to testthree major topics. Each topic is correlated to afocus area of the Metro Ethernet Forum (MEF) - thedefining body for Carrier Ethernet. Our choiceswere also influenced by service providers’ need toquickly develop new applications and services togenerate new revenues. As such, we focused on thefollowing areas:

• Global Interconnect - the MEF coined term forCarrier Ethernet Phase 3 - the worldwidedeployment of Carrier Ethernet services. Therealization of Global Interconnect will requiresuch tools as External Network to NetworkInterface (ENNI) which is nearing completion ofthe first phase specification. Aware that much isto be defined in this phase, we take a look at thevarious solutions and technologies that are thecurrent state of the art.

• Mobile Backhaul - With theMEF 22 technical specifi-cation completed, carriers areready and waiting to takeadvantage the PacketSwitched Network (PSN) forbackhauling their mobiletraffic. The tests conductedfocus on packet basedsynchronization mechanismsand legacy mobile transport.

• Managed Ethernet Services -Both from our discussions withservice providers and ourfeedback from a survey givento providers at last year’sCEWC, the message wasclear: providers are most inter-ested in seeing interoperablesolutions for managingEthernet services. Weextended our Ethernet OAMtests (as a response to

providers’ feedback) and added an extensive setof tests for management solutions.

Last, but not least, we present a detailed view intothe current state of transport mechanisms necessaryto provide Carrier Ethernet services to customers,including tests for transport technology features fromMPLS, MPLS-TP, and PBB-TE.Within each of these areas, in a tightly scheduledtwo week hot staging at our Berlin lab, over 60engineers configured network equipment spanningfifteen racks in order to provide the results we detailin the following report. We hope you enjoy theread.

Carsten RossenhövelManaging Director

TABLE OF CONTENTSParticipants and Devices.............. 3Network Design ......................... 4Mobile Backhaul ........................ 4Synchronization ......................... 4GSM and UMTS Transport........... 6Global Interconnect .................... 8Managed Ethernet Services ....... 12Performance Monitoring ............ 12Ethernet in the First Mile (EFM) ... 12Continuity Fault Management..... 13Carrier Ethernet Transport.......... 15References ............................... 18Acronyms ................................ 19

Participants and Devices

3

PARTICIPANTS AND DEVICES

INTEROPERABILITY TEST RESULTSThe description and results of the technical areastested are documented in their own respectivesections. Each section includes the technicalbackground of the technology, the test procedureused, the successful results, and in some cases someadditional information. The document generallyfollows the structure of the test plan. Also includedfor some tests is a logical diagram which depicts theresults and in some cases the test setup. Somesuccessful tests which involved several vendors wereincorporated to the demonstration network,described in the Network Design section below.It is important to note that the multi-vendor combina-tions documented detail the successful resultscompleted within the 9 business days available forthe hot staging (two weeks minus 1/2 day for setup,1/2 day for tear-down). Given the time constraint, afull mesh of vendor combinations for some of thetechnologies tested is impossible to test. If the readerdoes not find a combination which includes hisvendors of choice we encourage him/her to contactthe vendor and make sure that the test combinationis performed at our next event.Testers. Throughout the test event the generation oftraffic, as well as the measurement of out of servicetimes, was required; both for debugging and in theverification process for many of our tests. We would

Vendor Participating Devices

Agilent Technologies N2X

Albis Technologies ULAF+ MCU-CESULAF+ Acceed

Alcatel-Lucent 1850 TSS-3201850 TSS-160

Calnex Solutions Paragon

Ceragon Networks FibeAir IP-10

Ciena CN 5305CN 3960CN 3940CN 3920

Cisco Systems ME 3400EG-12CSME 3400EG-2CSCatalyst 3750ME76067604ASR 1002Active NetworkAbstraction (ANA)a

ECI Telecom SR9705BG9305

Ericsson DemonstratorSM480

Ethos Networks E-80Domain ManagementSystem (DMS)a

Harris StratexNetworks

Eclipse Packet Node

Huawei Technologies NE40E-X8CX600-X3PTN910PTN950PTN1900PTN3900

Ixia IxNetwork

JDSU QT-600HST-3000MTS-8000MTS-6000AEtherNIDSmartClass E1

MRV Communications OS910MOS9124-410GOS904-DSLOS904-MBH

NEC Corporation Pasolink NEO iPPasolink NEO HPCX2600

Nokia SiemensNetworks

FlexiPacket MicrowaveFlexi WCDMA BTS

RAD DataCommunications

ACE-3220ACE-3105ETX-204AETX-202IPmux-24

SIAE Microelettronica ALplus2

Siklu Communication EtherHaul-250

SpirentCommunications

xGEMSpirent TestCenter

Symmetricom TimeProvider 5000TimeProvider 500SSU 2000TimeWatchCesium

Tejas Networks TJ2050TJ2030

Telco Systems - aBATM Company

T-Metro-XGT-Metro-200T5C-XGT5C-48TT-Marc-380T-Marc-280T-Marc-254P

a. Management system

Vendor Participating Devices

4


like to thank Agilent Technologies, Ixia, SpirentCommunications, and JDSU not only for providingthe test equipment, but also for the time and effortspent helping debugging and performing tests.Terminology. For the purposes of readability (andconsistency with our previous white papers) we willuse the term “tested” to describe interoperability testsbetween equipment from multiple vendors andperformed according to the test plan. The term“demonstrated” will be used both when a test wasperformed with equipment from the same vendor,and also when functionality was used or demon-strated and not thoroughly tested.

NETWORK DESIGNBased on the interoperability test results, EANTCand the participating vendors constructed a singlemulti-vendor network topology. The network includedall participating devices and a number of CarrierEthernet demonstrations. The showcase networkaimed to mirror modern converged networksincluding mobile radio access elements, IP/MPLSrouters, and both aggregation and access switches.The resulting services spanned equipment frommultiple vendors as well as different technologies.The network showed end-to-end Mobile Backhauland business services demonstrating the ability ofthe underlying technologies and devices not only tosuccessfully provide these services, but also tomanage them. The detailed results are described inthe Mobile Backhaul and Managed EthernetServices sections below.The network consisted of two distinct providerdomains which were connected using results fromthe Global Interconnect tests enabling end-to-endnetwork services and applications. Each of theprovider domains was created from the conjunctionof two transport technologies - one with IP/MPLSand MPLS-TP technologies, and one with IP/MPLSand PBB-TE technologies. The detailed transporttechnology and administrative domain intercon-nection test results are described in the CarrierEthernet Transport and Global Interconnect sections.

MOBILE BACKHAULIn January 2008 we performed our first MobileBackhaul interoperability event. We quickly realizedthat we were a too early for the industry and that notenough Mobile Backhaul solutions existed forinteroperability testing. This is clearly no longer thecase. We tested deployment and test solutions from20 of the 24 vendors attending the hot staging fromthis time around - from solutions to support legacyGSM services all the way to forward looking imple-mentations of Long Term Evolution (LTE).Mobile backhaul readiness is measured by two yardsticks: The ability to provide transport between basestations and their controllers and the solution’s abilityto provide clock synchronization to the base stations.

SYNCHRONIZATION IN THEPACKET SWITCHED NETWORKOne of the biggest challenges facing carriersplanning to use packet based networks for theirmobile backhaul is synchronisation of their mobilebase stations. Traditionally, synchronization inmobile networks is provided by the synchronousphysical layer (Layer 1) of the network, however, thissynchronisation capability is not inherent to Ethernetas it is with TDM/SDH based technologies.Therefore, with the move to Ethernet based transport,carriers must solve this problem independently fromthe transport network.Depending on the mobile technology used on the airinterface, and base station type, the mobilebackhaul plan must consider different aspects andlevels of synchronization. It is nearly impossible todefine a single set of synchronisation requirementsfor all mobile backhaul networks. In our event wemeasured the clock quality of multi-vendor systemstypically consisting of a master and slave clockdevice and provided for each a statement regardingwhich clock quality level these systems achieved.Whether this clocking quality is sufficient or not liesin the decision of each mobile operator.Several solutions, from a variety of Standard Devel-opment Organizations (SDOs), currently exist toprovide synchronization over packet basednetworks. In our test event we evaluated threemechanisms which can be used to synchronize thepacket based network: adaptive clocking, IEEE1588-2008 protocol, and Synchronous Ethernet - atechnology that changes the asynchronous nature ofthe Ethernet PHY to be synchronous.For all tests in which we verified the clock accuracywe measured time interval error (TIE) of the slave’sclock output (E1 or Synchronous Ethernet interfacetype) using a jitter and wander analyzer. The TIEwas measured against a high precision externalatomic clock whose quality was better than thequality of a Primary Reference Clock (PRC) asspecified by ITU-T G.811. We then converted the TIEmeasurements to Maximum Time Interval Error(MTIE) and Time DEViation (TDEV) curves andcompared to the synchronization accuracy require-ments expressed as MTIE and TDEV masks definedby ITU-T.We have seen an increase in participation in thesetest areas and a significantly higher number ofimplementations. This facilitated extensive testingwhich expanded on the procedures used in previoustest events.

IEEE 1588-2008 (PTPv2)In order to verify the synchronization functions ofPrecision Time Protocol version 2 (PTPv2) defined inIEEE 1588-2008, we measured the time needed forthe slaves to synchronize with their master clocks.Following this we measured synchronization qualityon the slaves’ clock output interface by an externaljitter and wander analyzer. Wander was measured

Synchronization in the Packet Switched Network

5

for at least 1 hour after initial synchronization of theslave clock to the master.During all tests we used the Calnex Paragon wasconnected between the IEEE 1588-2008 clockmaster and slave devices. The Paragon emulated aPacket Switched Network (PSN). The impairmentgenerator was configured with a network profileaccording to ITU-T G.8261 Test Case 12.The following devices served as PTP masters:Huawei CX600-X3, MRV OS904-MBH, Symmet-ricom TimeProvider 5000, and Telco Systems T-Marc-254P. The Huawei CX600-X3, HuaweiPTN910, RAD ACE-3220 and Telco Systems T-Marc-254P successfully tested PTP slave functionality. Allsuccessful test combinations passed the ITU-T G.823wander traffic mask, and are found in Figure 1. Forwander measurements we used the SymmetricomTimeWatch and JDSU wander analyzer.

Figure 1: Precision Time Protocol version 2

In this event we housed more PTPv2 implementationsin our lab than in any of our previous test events.Much time was spent figuring out the commonsupported configuration settings and fixing configu-ration issues. For all test combinations we tested 1-step clock, and used UDP/IP unicast encapsulation.The configured SYNC messages rate was between16 and 128 per second.Our results show that the industry is in a progressingstage regarding IEEE 1588-2008 implementations.For our next events we expect to further testadvanced IEEE 1588-2008 features like 2-stepclock, phase and time synchronization, peer delay,

delay/response protocol mechanisms, boundaryand transparent clocks, and other IEEE 1588-2008encapsulation types.

Synchronous Ethernet and ESMCSynchronous Ethernet (SyncE) provides mechanismsfor high-precision frequency distribution in Ethernetnetworks similar to functionality provided by SDH.Unlike traditional Ethernet, where the receiving nodeonly performs synchronization on a per-packetbasis, internal clocks of the SyncE enabled nodesare constantly synchronized.A node operating in synchronous mode recovers theclock frequency from an external high precisionclock signal which is received either from anEthernet link or from a clock link such as BuildingIntegrated Timing Supply (BITS). As a result of thisoperation the node “locks” its internal clock to theexternal clock signal. This node can thensynchronize Ethernet peers in the transmit directionwith its internal clock allowing the directly attachedSyncE nodes to synchronize their clocks to thenode’s clock. With this mechanism, synchronizationcan be provided throughout an Ethernet network.One of our test goals was to verify the ability of anEthernet device operating in synchronous mode tosynchronize its clock frequency with another SyncEnode. We verified the synchronization quality bymeasuring the jitter and wander of the node’s clockoutput signal. The wander analyzer was connectedto the same external clock source as the masterSyncE node in order to measure the difference. Theresults were compared against the synchronizationaccuracy requirements defined by the ITU-T.Tests were successfully completed between thedevices listed below. The first device in the pairindicates the master, and the second the slave role.All combinations passed at least the ITU-T G.813SEC Option 1 MTIE and TDEV masks. For wandermeasurements we used the Calnex Paragon andJDSU MTS-8000 devices: RAD ETX-204A andAlcatel-Lucent 1850 TSS-320, SIAE MicroelettronicaALplus2 and Alcatel-Lucent 1850 TSS-320, EricssonDemonstrator and Alcatel-Lucent 1850 TSS-320,Alcatel-Lucent 1850 TSS-320 and Albis ULAF+Acceed, RAD ETX-204A and Albis ULAF+ Acceed,SIAE Microelettronica ALplus2 and Albis ULAF+Acceed, Huawei PTN3900 and Cisco 7604, ECIBG9305 and Albis ULAF+ Acceed, ECI BG9305and Cisco 7604, ECI BG9305 and RAD ETX-204A,Albis ULAF+ Acceed and RAD ETX-204A, NokiaSiemens Networks FlexiPacket Microwave and AlbisULAF+ Acceed, Nokia Siemens Networks Flexi-Packet Microwave and MRV OS904-MBH, ECIBG9305 and Huawei PTN3900, SIAEMicroelettronica ALplus2 and MRV OS904-MBH,Ericsson Demonstrator and Nokia SiemensNetworks FlexiPacket Microwave, SIAEMicroelettronica ALplus2 and Nokia SiemensNetworks FlexiPacket Microwave, Alcatel-Lucent1850 TSS-320 and Albis ULAF+ Acceed, SIAEMicroelettronica ALplus2 and Albis ULAF+ Acceed,

Master SlaveCalnexParagon

Calnex Paragon

Transport service Provider Network

Access network

HuaweiCX600-X3

HuaweiPTN910

E1/2.048Mhz clock link

TDM link

RADACE-3220

SymmetricomTimeWatch



SymmetricomTimeProvider 5000

HuaweiCX600-X3

TelcoT-Marc-254P


TelcoT-Marc 254P

RADACE-3220

CalnexParagon

CalnexParagon

RADACE-3220


Calnex Paragon

MRVOS904-MBH

JDSUWander Analyser

JDSUWander Analyser

6


Ericsson Demonstrator and MRV OS904-MBH,Huawei PTN910 and MRV OS904-MBH.

Figure 2: ESMC

In the case of multiple clock sources the networkelements should be able to select the best one, andin failure situations to switch from the failed clocksource to the working source. This would allow theService Providers to build resiliency into the clockinginfrastructure. This mechanism is called Synchroni-zation Status Messaging (SSM) and its usage forSyncE is defined in G.8264. The origin of the SSMlies in TDM and SDH.This brings us to the second goal of the SyncE tests -verification of the SSM mechanism. During thesetests we first verified the establishment of an EthernetSynchronization Messaging Channel (ESMC)between the SyncE nodes and then analyzed thatthe nodes properly generated and interpreted theESMC status messages. ECI BG9305, EricssonDemonstrator, Huawei PTN910, MRV OS904-MBH,Nokia Siemens Networks FlexiPacket Microwave,and RAD ETX-204A all successfully passed this test.For ESMC packet analysis we used Agilent N2X andCalnex Paragon. In addition, we used Agilent N2Xfor ESMC message generation for some tests. Allsuccessful test combinations are shown in Figure 2.Similar to PTPv2 tests, we housed more SyncE imple-mentations in this event than any previous event. The

results showed a high level of interoperability forsynchronization between the tested equipment. Wealso tested Ethernet Synchronization MessagingChannel (ESMC) for the first time - we hope to teststill more implementations in the future.

GSM AND UMTS TRANSPORTLegacy networks such as GSM (commonly referredto as 2G) and UMTS (referred to as 3G) are notgoing away. Carriers tend to add equipmentbelonging to the next generation radio technology toa cell site, but rarely do we see legacy technologiesdecommissioned. As part of the move to packetbased backhaul carriers are looking to offload theirTDM and ATM base station to controller links topacket networks. This sets a new requirement onmobile backhaul networks: support native and trans-parent TDM and ATM connectivity between basestation and controller.Both the MEF and the IETF have developedstandards to address these requirements before therequirement for mobile backhaul arose. However,with the new drive from mobile carriers to decom-mission their TDM and ATM networks and toconverge all services into one packet base network,these standards are experiencing renewed interests.We verified the interoperability of the devicesproviding base station to controller connectivity inestablishing and maintaining emulated circuits.

TDM and ATM EmulationMost of today’s Mobile Backhaul deployments arebased on TDM technology (e.g. GSM) or ATMtechnology (e.g. UMTS). On the migration path ofthe existing Mobile Backhauls to Carrier Ethernet,the legacy TDM and ATM equipment can beconnected to the new Carrier Ethernet MobileBackhaul by using translation technologies allowingto transport TDM and ATM data over CarrierEthernet. During our hot staging we verified ATMtransport over MPLS implementations according tothe RFC 4717 and TDM structure agnostic andstructure aware transport in the IETF version (RFC4553 and RFC 5086) and MEF version (MEF 8).Additionally to the ability of transport data over aPacket Switched Network, we also verified thequality of adaptive clocking recovery from thePacket Switched Network implemented at thedevices. All devices were required to configure astructure aware network service that transport thePCM30 or PCM31 E1 slots 9, 20, 21, and 22.The following devices were able to successfullytransport TDM data over a structure aware networkservice (CESoPSN): Cisco 7606, Ericsson Demon-strator, Huawei PTN910, Huawei CX600-X3, MRVOS910M, NEC Pasolink NEO TE, and RAD ACE-3220. The Albis ULAF+ MCU-CES, Cisco 7606, ECIBG9305, Ericsson Demonstrator, Huawei CX600-X3, Huawei PTN910, MRV OS910M, MRVOS9124-410G, NEC Pasolink NEO TE, Telco T-Metro-200, RAD ACE-3220, RAD IPmux-24 success-

Master Slave Slave(PRC) (SEC) (SSU)

Agilent N2X

Ethernet tap

SyncE network

Clock link

AgilentN2X

AgilentN2X

HuaweiPTN910

RADETX-204A

EricssonDemonstrator


ECIBG9305

Calnex Paragon


MRVOS904-MBH

Calnex Paragon

NSNFlexiPacketMicrowave

Agilent NX2

NSNFlexiPacketMicrowave


ECIBG9305

Clock Source

GSM and UMTS Transport

7

fully tested TDM structure agnostic CES (SAToP). Alltests performed with Ericsson Demonstrator deviceswere done for STM 1 and STM 4 interfaces towardsthe circuit network. Also Telco Systems used in onetest STM 1 interface. The E1 data structure wasencapsulated into STM. For TDM traffic generationand analysis we used the following JDSU testerdevices: MTS-8000, MTS-6000A, andSmartClass E1.We tested Ethernet CES (MEF8) as well as MPLSCES (RFC 4553 and RFC 5086) encapsulation. Thefollowing devices tested TDM Circuit Emulation withEthernet encapsulation: Albis ULAF+ MCU-CES, ECIBG9305, Huawei CX600-X3, Huawei PTN910,Ericsson Demonstrator, MRV OS910M, MRVOS9124-410G, NEC Pasolink NEO TE, RAD IPmux-24, Telco T-Metro-200.The following devices tested TDM Circuit Emulationwith MPLS encapsulation: Cisco 7606, EricsssonDemonstrator, Huawei CX600-X3, RAD ACE-3220,and NEC Pasolink NEO TE.For both structure agnostic and structure aware TDMCircuit Emulation tests the slave devices wereconfigured to perform adaptive clock recovery fromthe Packet Switched Network. We used CalnexParagon and Spirent xGEM impairment devices toemulate a real network between master and slavedevices according to G.8261 Test Case 1. Theimpairment was introduced from the beginning ofthe test as the slave clock was in Free Running state.As soon as clock changed in synchronous state wemeasured time interval error from the reference clockfor at least one hour, using the JDSU jitter/wandertesters.The following device combinations passed the ITU-TG.823 traffic MTIE and TDEV masks, the first ofwhich indicates master role, the second a slave:RAD IPmux-24 and Albis ULAF+ MCU-CES, AlbisULAF+ MCU-CES and Telco Systems T-Metro-200,Telco Systems T-Metro-200 and Albis ULAF+ MCU-CES, Telco Systems T-Metro-200 and ECI BG9305.These pairs performed TDM Structure Agnostic CES.The Cisco 7606, Huawei CX600-X3, HuaweiNE40E-X8, Huawei PTN910, NEC CX2600, andNEC Pasolink NEO TE tested successfully ATMtransport over MPLS. All tests were performed forATM VPI n-to-1 encapsulation, with n set to 1. Forthe test between Cisco 7606 and Huawei CX600-X3 and the demonstration between Huawei NE40E-X8 and Huawei PTN910 we configured cell concat-enation of 10 cells per PDU and concatenation time-out of 10 ms. In all other tests we tested without cellconcatenation, which means the devices encapsu-lated a single ATM cell per PSN PDU.Our tests show that TDM structure agnostic and ATMtransport over MPLS have achieved a mature imple-mentation status - we did not observe any interoper-ability issues. In regards to CESoPSN testing, weobserved that not all implementations support bothPCM30 and PCM31 E1 and also some implementa-tions could not configure different E1 slots (9, 20,21, and 22) as required by our tests. Also in mostcases the implementations either did not fully support

E1 alarm propagation but rather implemented it in adifferent way. For this reason we see a reason tocontinue testing CESoPSN interoperability in furtherevents. We also observed increased success foradaptive clocking tests.

Figure 3: TDM and ATM Emulation

Microwave TransportWe have seen a great deal of development in theproducts offered by Microwave vendors since westarted testing these devices at our interoperabilityevents. As the Microwave transport market becamemore and more commoditized vendors started imple-menting more and more intelligence in their devicesmoving away from pure transport into advancedswitching functionality with Microwave uplinks.The tests focusing on Microwave devices demon-strated these advanced capabilities - from reactingto degraded Microwave link conditions to propa-gation link state information to the remote end. Theformer test, more specifically referred to as AdaptiveCoding and Modulation (ACM), involved trans-mitting two classes of Ethernet traffic each distin-guished by the Priority Bits in the customer VLANtag. An attenuator was then used to degrade thesignal, at which point the system was expected toautomatically lower the Quadrature AmplitudeModulation (QAM). Since lowering the QAM alsoreduces the bandwidth of the air interface, thesystem was then expected to only drop traffic of alower priority. This test was successfully conductedwith the Ceragon FibeAir IP-10, Harris StratexEclipse Packet Node, Huawei PTN 950 and 910,NEC Pasolink NEO HP, Nokia Siemens NetworksFlexiPacket Microwave and the SIAEMicroelettronica ALplus2. Siklu performed a similartest with their EtherHaul-250 device where thechange in modulation was manually configured asthe attenuation was increased.

TDM CESoPSN emulated service

TDM SAToP service

ATM emulated service

MRVHuaweiPTN910

HuaweiCX600-X3

Albis ULAF+MCU-CES

MRVOS9124-410G

RADIPmux-24

TelcoT-Metro-200

ECIBG9305

HuaweiNE40E-X8NEC

Pasolink NEO TE

RADACE-3220

Cisco7606

NECCX2600

OS910M


8


In order for fault detection to work even over themicrowave air interface, some systems have imple-mented the capability to propagate the loss of signalon the air interface to the Ethernet interfaces, as wellas the propagation of one Ethernet link going downto the associated Ethernet link on the opposite sideof the air interface. Both functionalities were demon-strated by the Ceragon FibeAir IP-10, Harris StratexEclipse Packet Node, NEC Pasolink NEO HP, NokiaSiemens Networks FlexiPacket Microwave and theSIAE Microelettronica ALplus2.Some vendors demonstrated additional features.Ceragon, for example, showed a resilient ring of sixFibeAir IP-10 switches. Failure tests were run wherethe Ethernet link was disrupted as well as the radiolink. In all cases the out of service times ranged from12 to 50 milliseconds. Harris Stratex demonstrated aGigabit Ethernet service aggregated over multiplemicrowave paths. To demonstrate that jumbo framescan be transmitted over microwave links, HarrisStratex, NEC, SIAE Microelettronica, and Siklu allpassed traffic consisting of 7600 byte frames.The following microwave systems demonstrated thetransport of Precision Time Protocol version 2 (IEEE1588-2008): Ceragon FibeAir IP-10, Harris StratexEclipse Packet Node, NEC Pasolink NEO HP, NokiaSiemens Networks FlexiPacket Microwave, SIAEMicroelettronica ALplus2 and Siklu EtherHaul-250.The master PTPv2 devices were the SymmetricomTimeProvider 5000 and the Telco Systems T-Marc-254P, with the Symmetricom TimeProvider 500 andRAD ACE-3220 as slaves. The demonstration wasprepared for the Carrier Ethernet World Congress inBerlin, and was run over a VPLS service provided bythe network. We observed the wander with thepreviously mentioned wander analyzers after thedemonstration was setup and recorded that it did notexceed the expected masks.

GLOBAL INTERCONNECTHow are service providers going to offer complete,end-to-end Ethernet services that span multipleadministrative domains? Global interconnect aims toanswer this question and to expand Carrier Ethernetservices from one provider to the next. The endresults of such service is true global-encompassingCarrier Ethernet services.The Global Interconnect term stands also for all inter-connections of a network that consist of multipleadministrative domains. A single interconnectionbetween two administrative domains is called anENNI (External Network Network Interface). Devicesimplementing ENNI must map network services andtheir attributes between the administrative domainand ENNI, in particular:

• Service delimiting mapping (e.g. MPLSpseudowire labels into 802.1ad S-Tags),

• Service monitoring (e.g. between Ethernet OAMon ENNI and Pseudowire OAM)

• Service attributes (e.g. Availability, QoS)We verified service delimiting mapping between

MPLS pseudowire (PW) intra-domain segments toENNI PW segment or ENNI S-Tag, and we wereable to create resilient multi-vendor ENNI scenarios,increasing the availability of the network services.The results of these tests are described below.

MPLS and Ethernet InterconnectAn end-to-end Carrier Ethernet network servicepassing multiple provider domains is realized viasegmented network service, a segment per providerdomain and a third at the ENNI. In case of ProviderBridging based ENNI, the 802.1ad S-Tags are usedfor service delimiting and treated as virtual AccessCircuits (AC) at the devices implementing ENNI. Theswitching between the intra-domain PW segmentsand ACs at ENNI is called “Pseudowire Switchingusing AC” and described in IETF draft draft-ietf-pwe3-segmented-pw.

Figure 4: MPLS and Ethernet Interconnect

With MPLS based ENNI, the devices implementingENNI maintain two PW segments per networkservice. One PW segment is traversing the ENNI,and the other towards the native domain. The ENNIdevices implement switching between the intra-domain and ENNI PW segments. This mode ofoperation is called “PW Control Plane Switching” inIETF draft draft-ietf-pwe3-segmented-pw.

HuaweiCX600-X3

Cisco7606

HuaweiCX600-X3

Cisco7604

VPWS segment

Provider networkMPLS based ENNI

802.1ad based ENNI

HuaweiCX600-X3

HuaweiNE40E-X8

EricssonSM480

HuaweiNE40E-X8

HuaweiCX600-X3

Cisco7606

EricssonSM480

HuaweiCX600-X3

Cisco7606

HuaweiCX600-X3

Cisco7604

HuaweiNE40E-X8

EricssonSM480

HuaweiCX600-X3

Cisco7606

ECISR9705

CienaCN 5305

Cisco7604

ECISR9705

HuaweiCX600-X3

CiscoASR 1002

Cisco7604

ECISR9705

HuaweiCX600-X3

Cisco7604

ECISR9705

EricssonSM480

ENNI area

CienaCN 5305

CienaCN 5305

CienaCN 5305

HuaweiPTN950

Telco SystemsT-Metro-200

Global Interconnect

9

We tested MPLS based Global Interconnect on thefollowing devices: Cisco 7606, Ericsson SM480,Huawei CX600-X3, and Huawei NE40E-X8. Thedevices performed MPLS PW switching betweenprovider and ENNI PW segments. eBGP with MPLSlabel extension was used on ENNI in order to signalthe MPLS transport LSP ENNI segment.We tested Ethernet based Global Interconnect onthe following devices: Cisco 7606, ECI SR9705,Ericsson SM480, Huawei CX600-X3, and HuaweiNE40E-X8 - which switched between a providersegment PW and an S-VLAN tag on the ENNI.The following devices served as UNI Provider Edgedevices: Ciena 5305, Cisco 7604, Cisco ASR1002, Huawei CX600-X3, Huawei NE40E-X8, andTelco Systems T-Metro-200. These devices estab-lished inter-provider VPWS over the Ethernet andMPLS based ENNIs mentioned above.The tests were not conducted as quickly as onemight think, requiring manual configuration andextensive troubleshooting of Ethernet based ENNIs.The vendor implementations differed regarding thehandling of the PW VC types (“Raw” vs “Tagged”mode) and VLAN tag manipulation. For each devicecombination we discovered a configuration thatworked, determining where the S-VLAN tag had tobe pushed and popped (at UNI or at ENNI), andwhich type of PW to use (Ethernet or VLAN type).Figure 4 shows the successfully tested combinations.

ENNI Failure ScenariosThe forwarding of various OAM notifications andfailure triggers between the segments of a networkservice is often required in order to synchronize thestatus of the network service across all domains anddevices. We configured two inter-domain point-to-point network services between two access devices -one service served as primary, and other as backup.We expected the simulated ENNI failure to besignaled across the service from UNI to UNI.There are different triggers and status notificationfunctions that can be used for such a scenario.During our testing two multi-vendor scenarios weresuccessfully tested as shown in Figure 5.In one scenario the primary service at the ENNI wasrealized by Huawei CX600-X3 and ECI SR9705with a backup through Ericsson SM480 and Cisco7606. CFM sessions monitored the liveliness of theprimary service between the Cisco 7604 and ECISR9705 in one provider domain, and between theCisco 7604 and Cisco ME3400EG-2CS in theaccess area. Pseudowire protection was configuredin the second provider domain on the Ciena CN5305. The ENNI failure was triggered by asimulated power failure on the ECI SR9705. Uponthe Loss of Signal (LOS) towards the ECI device theHuawei CX600-X3 signaled the PW down via anLDP Withdrawal Message towards the Ciena CN5305, causing the Ciena CN 5305 to switch to thebackup PW. In parallel, the Cisco 7604 received aContinuity Check (CC) error for the CFM sessiontowards the ECI SR9705 and reacted by trans-

mitting a CFM Alarm Indication Signal (AIS),defined in ITU-T Y.1731, towards the CiscoME3400EG-2CS which then switched over to thebackup service based on the CFM AIS.In the second scenario the failure was triggered by aLOS between the Cisco 7606 and Huawei CX600-X3. The Cisco 7606 reacted by transmitting aPseudowire Status Notification (AC down) to theHuawei CX600-X3 at the UNI, and the HuaweiCX600-X3 at the ENNI reacted to the LOS bysending an LDP Withdraw Message to the CienaCN 5305. After receiving of the Pseudowire StatusNotification the Huawei CX600-X3 device at theUNI shut down the local PW Access Circuit (AC).Upon LOS on the primary AC, the RAD ETX-204Aswitched over to the backup AC. Agilent N2X wasused in both scenarios as an Access Deviceemulator connected to the Ciena CN 5305.

Figure 5: ENNI Failure Scenarios

In both scenarios we measured failover times below400 milliseconds (ms). The switch-over from thebackup to the primary service upon recovery of theprimary service was performed manually. Thoseworking in transport networks might find failuretimes greater than 50 ms to be high, however, theseout of service times were expected given the numberof different time-outs and protocol translations. Ourprevious events showed failover mechanisms withina single technology to converge under 50 ms acrossdifferent implementations, however here the value intesting such multi-domain scenarios was underlined.

RADETX-204A

CiscoME3400EG-2CS

HuaweiCX600-X3

EricssonSM480

Cisco7606

HuaweiNE40E-X8

HuaweiCX600-X3

Cisco7604

HuaweiCX600-X3

EricssonSM480

ECISR9705

Cisco7606

Cisco7604

HuaweiCX600-X3

Provider networkPrimary service

Backup service ENNI area

Access network

Agilent N2X Agilent N2X

Ethernet link

CienaCN 5305

CienaCN 5305


Cisco ANA

Albis TechnologiesULAF+ Acceed

HuaweiPTN1900

SIAE ALplus2

RADETX-204A

RADETX-202

Ixia IxNetwork

JDSUHST-3000

JDSUMTS-6000A

JDSUSmartClass E1

JDSUEtherNID

Ixia IxNetwork

RADIPmax-24

HuaweiPTN910/950

HuaweiPTN1900

Telco SystemsT-Marc-280

Agilent N2X

RADETX-204A

MRV9124-410G


JDSUSmartClass E1

RADACE-3220

Harris StratexEclipse Packet Node

ECI BG9305 Siklu EtherHaul-250

CeragFibeAir I

NEC Pasolink NeoTelco SystemsT5C-XG

Harris StratexPacket Node


HuaweiNE40E-X8

Cisco7606

Cisco Catalyst3750ME

Telco SystemsT-Metro-XG

HuaweiCX600-X3

HuaweiCX600-X

NEC CX26

EricssonSM480

ECI SR970

Alcatel-Lucent1850 TSS-320


HuaweiPTN3900




Albis TechnolULAF+ MCU

Calnex Paragon


Cisco 7604

CienaCN 3920

Spirent xGEN

NEC CX2600

NEC Pasolink Neo HP

MRVOS904-DSL

MRVOS904-DSL

MRVOS904-MBH

CiscoME3400EG-2CS

CeragonFibeAir IP-10

HuaweiPTN910/PTN950

Spirent TestCenter

Ixia IxNetwork

NEC PasolinkNeo HP

Telco SystemsT-Marc-254P

SIAE ALplus2

Provider A

ECI SR9705

Siklu EtherHaul-250

Physical Network Topology

Network Areas

Connection Types

Tester

Aggregation Device

Radio Access Network

TDM link

ATM link

10 Gigabit Ethernet

Radio Link

Device TypesMetro/CoreNetwork Device

Microwave Device

Access Device

Mobile Base Station

Access network

PBB-TE network

MPLS-TP network

Provider network

Global interconnect

Management System

JDSUQT-600

Gigabit Ethernet

Fast Ethernet

1588 Clock Master/Slave

SHDSL/ADSL/VDSL link

MPLS network

OTN link

SDH link

RADACE-3220

Nokia Siemens NetworksNC Demonstrator

Access Device

Ixia IxNetwork

CiscoME3400EG-12CS

SymmetricomSSU 2000

Ixia IxNetwork

TejasTJ2050

CienaCN 3960


RADETX-202

RADACE-3220

Agilent N2X

Tejas2030

00

JDSUNetComplete OSS

JDSUEtherNID

EthosE-80

Telco SystemsT-Marc-254P

Nokia Siemens NetworksFlexiPacket Microwave

Nokia Siemens NetworksFlexi WCDMA BTS


gonIP-10


MRVOS9124-410G

Telco SystemsT5C-XG

Agilent N2X

o iPECI BG9305

iX3

2600

n0

HuaweiNE40E-X8

705

CiscoASR 1002

EthosE-80

TejasTJ2050

TejasTJ2050

CienaCN 5305

TejasTJ2050

TejasTJ2050

RADACE-3105

logiesU-CES



Calnex Paragon

MRVOS910M

MRVOS910M

CienaCN 3940


Harris StratexEclipse Packet Node

Calnex ParagonSpirent TestCenter

Spirent TestCenter

Provider B

JDSUMTS-8000

Controller

12


MANAGED ETHERNET SERVICESAs Carrier Ethernet standards are becoming moremature, service providers are feeling more and morecomfortable deploying Ethernet Services. To anextent this is thanks to the evolution from EthernetServices – an Ethernet based service similar toleased lines in which the customer receives Ethernetnetwork access, to Managed Ethernet Services – ahigh margin service in which the provider is respon-sible for the health of the service (usually regulatedthrough strict SLAs) and has a demarcation device atthe customer site.We have been testing Ethernet Services at EANTC’sinteroperability events since 2005. Now that manyof the standards that compromise operations, admin-istration and maintenance (OAM) as well as provi-sioning are more mature we incorporated all aspectsinto one test area.

Performance MonitoringEthernet service OAM is specified in two comple-menting standards – IEEE 802.1ag and ITU-T recom-mendation Y.1731. The ITU-T recommendationspecifies the performance monitoring function,including an array of performance monitoringmessages. Of these messages, we focused on delaymeasurement and reply messages (DMMs andDMRs). In addition, the Loopback Messages (LBM)and Loopback Replies (LBR) specified in 802.1agwere used to demonstration frame delay and delayvariation measurement based strictly on the IEEEstandard.All tests were performed using either the CalnexSolutions Paragon or Spirent xGEM impairmentequipment. First, messages were exchangedbetween the DUTs without any impairment to showbaseline interoperability. Then, 20 ms delay and 5ms delay variation impairments were generatedusing the impairment devices. The DUTs wereexpected to recognize and report this increase indelay, and we expected the reports to match.The following devices successfully participated intests for both two-way frame delay, and two-wayframe delay variation: Ciena CN 5305, Ciena CN3960, Cisco ME 3400EG-12CS, IXIA IxNetwork,JDSU EtherNID, MRV OS904-DSL, NEC CX2600,NEC Pasolink NEO TE, RAD ETX-204A, TelcoSystems T-Marc-380 and Telco Systems T-Marc-280.Additionally, the Tejas TJ2050 device tested two-way frame delay only.During the tests we found that some devicessupported more precise delay measurements byusing two optional time stamps in the DMMs andDMRs to exclude the processing of delay, reflectingonly the transmission delay. This allows a provider tomonitor the delay induced by the network, omittingdelay from OAM processing by the endpoints of theEthernet OAM. These tests revealed successful resultswithout issues.In addition the Cisco ME 3400EG-12CS demon-strated performance monitoring using LBM and LBR

messages towards the Telco Systems T-Marc-280.The ME 3400EG-12CS transmitted LBMs to measureround trip time from which the frame delay anddelay variation were calculated. Between bothdevices the Spirent xGEM was used as animpairment generator. The demonstration showedan accurate measurement, which included OAMprocessing by the far end nodes.

Figure 6: Performance Monitoring Y.1731

Ethernet in the First Mile (EFM)Ethernet in the first mile (EFM) is specified in the IEEEstandard 802.3ah-2004. The standard defines aloopback mode which intends to assist carriers inperforming fault localization as well as testing linkperformance. Additionally, different types of eventsare specified in the standard, which can be used forthe immediate reporting of local status to a remotedevice. Operators can use these indications arrivingfrom an Access Device to localize failures anddetermine the reasons for an inactive customerconnection.We verified the ability of the Access Devices andProvider Edge to generate and properly interpret thelink events that signal the number of frame errors,and the number of errored frame seconds within aspecified time period. Each device was tested usingthree thresholds configured as following: 10 erroredframes in 10 seconds, 2 errored frame periods persmallest window size supported by the implemen-tation, and 5 errored frame seconds in one minute.First, a traffic generator sent IMIX traffic, filling thewindow of the peered device. Then, an impairmentdevice introduced CRC errors into the traffic fortwenty seconds at one CRC error per second,adding errors to the frames. The device receivingtraffic was then expected to send at least oneErrored Frame Event, one Errored Frame PeriodEvent and one Errored Frame Second Event.Four vendors successfully tested these link events asshown in the diagram, using the following devices:Alcatel-Lucent 1850 TSS-320, Ciena CN 3920,Ciena CN 3940, Cisco ME 3400EG-12CS, andIxia IxNetwork. In tests of Ixia IxNetwork, IxNetworkperformed impairment and EFM emulation at the

DMMs exchanged in both directions

NECPASOLINK

RADETX-204A

MRVOS904-DSL

TejasTJ 2050

NECCX2600

IxiaIxNetwork

Telco SystemsT-Marc-380Ciena

CN 5305 JDSUEtherNID

JDSUEtherNID

CalnexParagon

SpirentxGEM

NEO TE

CalnexParagon

Managed Ethernet Services

13

same time. Otherwise, impairment was provided bythe Calnex Paragon or Spirent xGEM as shown inthe diagram. Additionally, the Telco Systems T-Marc-380 successfully tested Errored Frame Events withvendor pairs shown in the diagram.We also verified that each device was able to bringits peer into loopback mode. The device in loopbackmode was expected to discard all incoming framesdestined towards its peer and loop back all framescoming from the peer. On receipt of these loopbackframes the peer was expected to drop them.The following devices successfully tested loopbackmode: Agilent N2X, Alcatel-Lucent 1850 TSS-320,Ceragon FibeAir IP-10, Ixia IxNetwork, MRVOS904-DSL and Telco Systems T-Marc-380.

Figure 7: EFM on the UNI

In some cases we also verified the signalling ofcritical link events such as Dying Gasp when theAccess Device was disconnected from power. DyingGasp is a message defined in the EFM standardwhich is used by a device when it is being shutdown to signal the event to its peer gracefully.Four vendors also successfully tested the signaling ofcritical link events: Alcatel-Lucent 1850 TSS-320,Ciena CN 3920, Ciena CN 3940, Cisco ME3400EG-12CS, Ixia IxNetwork and Telco Systems T-Marc-380. Two such devices: CN 3920 and T-Marc-380 successfully generated Dying Gaspmessages and the Alcatel-Lucent 1850 TSS-320,

Cisco ME 3400EG-12CS and Telco Systems T-Marc-380 successfully received the messages. Ixia’sIxNetwork was used for emulation and reception ofDying Gasp, Link Fault and Critical Event.In addition the Ceragon Fibe-Air IP-10 demonstratedEFM link discovery with the following devices: AlbisTechnologies ULAF+ Acceed, Cisco ME 3400EG-12CS, RAD ETX 204A, Ixia IxNetwork and MRVOS910M.

Hierarchical Continuity FaultManagementThe IEEE 802.1ag Connectivity Fault Management(CFM) standard defines end-to-end Ethernet basedOAM mechanisms. Its major use for serviceproviders and enterprises is to verify connectivityacross different domains managed by independententities, such as the Customer, the Service Providerand the Operator(s). The MIPs (Maintenance Associ-ation Intermediate Points) present in a certainMaintenance Domain can be used to perform thelinktrace procedure which is similar to the Traceroutetool known from IP.We configured four E-Lines in the network usingCFM capable devices to enable the testing. To verifythe linktrace function we instantiated CFM Mainte-nance Associations (MAs) across the E-line servicesusing three Maintenance Domains (MD): CustomerMD, Service Provider MD and Operator MD. TheMA at the Customer MD Level was built betweentwo Down-MEPs, which were residing at each UNI-C. A Down-MEP is a MEP communicating through aphysical port, where as an Up-MEP communicatesthrough the internal switch function of the device.The Service Provider MD Level was configuredbetween two Up-MEPs at the two UNI-N, trans-mitting LTMs towards the provider network. Finally,the Operator MD Level was built between theprovider edges (two providers were simulated,correlating with our Physical Topology in thecenterfold of this document). MIPs were thenconfigured on devices which supported MIP function-ality, and had a MEP configured at a lower MDLevel.In each MD Level of each scenario LinktraceMessages (LTMs) were transmitted by every MEP ineach direction both to the pair MEP at that MDLevel, and also to all MIPs configured at that MDLevel.Thirteen vendors participated the test, building sixmulti-vendor scenarios, and a total of 18 differentdevices all together: Agilent N2X, Alcatel-Lucent1850 TSS-320, Alcatel-Lucent 1850 TSS-160,Ceragon FibeAir IP-10, Ciena CN 5305, CiscoCatalyst 3750ME, Cisco 7604, Cisco ME 3400EG-12CS, ECI SR9705, Ixia IxNetwork, JDSU EtherNID,RAD ETX-204A, Ericsson SM480, SpirentTestCenter, Tejas TJ2050 and Telco Systems T5C-XG, Telco Systems T-Metro 200 and Telco Systems T-Marc-280. All MEP to MEP LTMs possible in the sixscenarios were successfully returned. AdditionallyMRV OS904-DSL was able to response LTMs sent by

Link eventsLoopbackCritical link events

SpirentxGEM



SpirentxGEM


SpirentxGEM

CalnexParagon

IxiaIxNetwork

IxiaIxNetwork

AgilentN2X

T-Marc-380Telco Systems

T-Marc-380Telco Systems

CN 3940Ciena

CN 3920Ciena


MRVOS904-DSL

ME 3400EG-12CSCisco

CiscoME 3400EG-12CS

IxNetworkIxia


IxNetworkIxia

IxNetworkIxia

14


Agilent N2X.During the testing we encountered severalconstraints due to conflicting configuration require-ments and support of the DUTs. In one case a devicecould not create a MEP and MIP using the sameVLAN-ID on the same port. Another device would notrespond to LTMs which were 802.1ad tagged. Onedevice required VLAN tagged (802.1Q or802.1ad) CFM frames in order to respond. Finally,one device in the provider domain did not supportMIP functionality at all. Due to these constraints,some devices shown in the diagram did notconfigure a MIP where shown, and we thereforerefer you to the following text.The following MIP devices properly responded toLTMs destined to the pair MEP: Alcatel-Lucent 1850TSS-320, Alcatel-Lucent 1850 TSS-160, Ceragon

FibeAir IP-10, Cisco Catalyst 3750ME, Ciena CN3505, Cisco 7604, Ericsson SM480, Tejas TJ2050,Telco Systems T5C-XG, Telco Systems T-Marc-280,and T-Metro 200.The following MIP devices properly responded toLTMs destined to them as a MIP: Ceragon FibeAir IP-10, Cisco Catalyst 3750ME, Cisco 7604, EricssonSM480, Tejas TJ2050, Telco Systems T5C-XG, andTelco Systems T-Marc-280.After our achieved success, and many discussions ofconfiguration requirements from different devices,we are confident that all parties left with a moreglobal understanding of CFM. Additionally we lookforward to the specifications from the differentstandards bodies aligning in regards to CFM andhow it is realized over different encapsulations andtransport technologies.

E-LMI and IP/MPLS InterworkingThe Ethernet Local Management Interface (E-LMI)specification defines a protocol and procedures thatprovide the means for Ethernet Virtual Connection(EVC) status notification of the UNI-N to the accessdevice implementing UNI-C. In particular, the E-LMIprotocol includes the following procedures:

• Notification to the UNI-C of an added EVC• Notification to the UNI-C a deleted EVC• Notification to the UNI-C of the availability state

of an EVC (Active, Not Active, Partially Active)

• Communication of UNI and EVC attributes to theUNI-C.

We were able to test also the interworking of MPLSLDP protocol status notification messages with E-LMI.Huawei NE40E-X8 and Cisco 7604 configured anMPLS Virtual Private Wire Service (VPWS). TheCisco 7604 then configured a UNI with the CiscoCatalyst 3750ME. As soon as Huawei NE40E-X8signalled VPWS Attachment Circuit (AC) status“down”, the Cisco 7604 mapped this notification toan asynchronous E-LMI EVC report message towardsthe Cisco Catalyst 3750ME signaling an EVC "NotActive" status. Upon reception of this E-LMI event,

Customer

Service Provider

Operator A Operator B

UNI SpirentTestCenter

UNIIxiaIxNetwork

Site A Site B

UNI

AgilentN2X

UNI

CeragonFibeAir

AgilentN2X

AgilentN2X

TelcoSystems

UNI JDSUEtherNID

RAD

UNIAgilent

N2X UNISpirentTestCenter

E-NNI



MD Level

MD Level

Maintenance AssociationUp-MEP

Down-MEP

MIP802.1ad (0x88A8)Q-in-Q (0x8100/0x8100)

UNI UNICiscoME 3400

Cisco7604

CienaCN 5305

TejasTJ2050

OperatorMD Level

ECISR9705

EricssonSM480

CienaCN 5305

ECISR9705

802.1Q (0x8100)PseudowireE-Line

UNIETX-204A

TelcoSystems

Figure 8: Connectivity Fault Management IEEE 802.1ag

IP-10Ceragon

FibeAirIP-10

UNI

CiscoCatalyst

3750ME

CiscoCatalyst

3750ME

T-Metro 200

T-Marc 280

TelcoSystemsT5C-XG

TelcoSystemsT5C-XG

UNI EG-12CS

IxiaIxNetwork

UNIUNI

CeragonFibeAirIP-10

CeragonFibeAir

IP-10

Carrier Ethernet Transport

15

the Cisco Catalyst 3750ME successfully shut downthe EVC and did not forward any traffic to it. Oncethe Huawei NE40E-X8 signaled VPWS status “up”via an LDP notification message, the Cisco 7604mapped this notification to asynchronous E-LMI EVCreport message and sent it to the Cisco Catalyst3750ME signalling the EVC "Active" status, causingthe device to enable the EVC and forward traffic toit. Finally when we administratively deleted the EVCon the Cisco 7604, the EVC deletion was success-fully signalled via E-LMI to Cisco Catalyst 3750MEand Cisco Catalyst 3750ME deleted the EVC andstopped forwarding data to it.

Monitoring and ProvisioningWhile multi-vendor networks are more commontoday than they were five years ago, multi-vendorprovisioning systems are still a bit rare. Never-theless, they are of great use to providers.Management systems which display the networktopology or the health of certain links and protocolsare much more interoperable thanks to the standard-ization of network protocols, SNMP, and protocolManagement Information Bases (MIB). Several of theparticipating vendors showed their managementsystems, some of which were able to display multi-vendor information.Using their Domain Management System (DMS),Ethos Networks successfully provisioned PBB-TEtrunks across the Ethos E-80 and Tejas TJ2030devices and services across their respective accessdevices - Telco Systems T-Marc-254P and TelcoSystems T-Marc-380.

Figure 9: PBB-TE Provisioning on Ethos DMSCisco ANA management system demonstratedinteroperability across Ceragon, Cisco, Ciena,Ericsson, SIAE Microelettronica, and Telco Systemsequipment, automatically displaying inventory andtopology between the various devices. Using eitherthe built in interface or by cross launching othervendors’ web interfaces, the ANA was able toinitiate CFM linktrace and loopback messagesduring CFM LTM testing. The graphical interface, asshown in Figure 10, aided in visualizing the CFMtesting.

Figure 10: CFM Test Topology on Cisco ANA

CARRIER ETHERNET TRANSPORTEthernet transport embodies the basis of a CarrierEthernet service - the ability of a network to determin-istically transport Ethernet frames throughout aservice provider network. Answering to the transportcall were the following technologies: MPLS, MPLS-TP, and PBB-TE.

Figure 11: MPLS, MPLS-TP, PBB-TE Interop

* Dashed line - No tunnel label used, leaving one (1) labelin MPLS stack.

It is still important to ensure that Carrier Ethernet

MPLS network2 Static MPLS Labels

PBB-TE network

HuaweiNE40E-X8

HuaweiCX600-X3

HuaweiNE40E-X8

HuaweiCX600-X3

HuaweiPTN910

HuaweiPTN3900

CiscoASR 1002

ECISR9705

CiscoASR 1002

ECISR9705

EricssonSM480

TejasTJ2050

CienaCN 5305

MPLS-TP networkPBB-TE TrunkMPLS Pseudowire

802.1ad

MPLS-TP Pseudowire*



Static MPLS LSP (over MPLS-TP PW)


ECISR9705

EricssonSM480

{


{Alcatel-Lucent1850 TSS-320

{Alcatel-Lucent1850 TSS-320

{

HuaweiNE40E-X8

HuaweiCX600-X3

HuaweiCX600-X3

HuaweiNE40E-X8


{

EthosE-80

{

TejasTJ2050

{

16


services may be realized not only by choosing oneof these technologies, but also by a provider who isinterested in leveraging multiple. In Figure 11 youcan observe the different ways in which the MPLS-TPand PBB-TE devices were configured to interoperatewith MPLS equipment.In the PBB-TE network both Ethos and Tejasequipment terminated PBB-TE trunk into 802.1adinterfaces towards the Cisco ASR 1002 whichpassed the payload into a pseudowire. Ciena on theother hand, with an implementation for both PBB-TEand MPLS, was able to stitch PBB-TE trunks to MPLSpseudowires, passing the payload from one to theother.A triple segment pseudowire model was used in theMPLS-TP network. Both Ericsson and Huaweiequipment handed off to the MPLS network bymeans of statically configured Label Switched Path(LSP) and Pseudowire (PW) labels. In these cases theMPLS enabled Huawei NE40E-X8 performed thestitching from the statically configured pseudowiresegment to the signaled pseudowire segment.Alcatel-Lucent also configured the above three-segment approach using MPLS-TP pseudowireswithout an LSP label in the MPLS-TP part of thenetwork. Additionally, such single-label pseudowireswere stitched to statically configured LSPs towardsan MPLS router. The MPLS router swapped the outerstatic label without needing to peer beneath into thepseudowire label. The MPLS-TP nodes on either endof the MPLS router were able to pass OAM over thisconnection.

MPLS VPLS - BGP Auto-DiscoveryWith IP/MPLS Provider Edge Equipment (PEs)growing in functionality, the configuration workrequired is increasing as well. To help reduce config-uration efforts when activating new customerconnections, some standardization has been madein the IETF’s L2VPN working group. The idea is totake advantage of the peering already establishedwith BGP which many PEs will also have configuredand leveraging those BGP sessions when setting upVPLS connections. The feature, Auto Discovery (AD),was defined in the VPLS specification which usesBGP for signaling (RFC 4761), but an IETF draft hasalso specified the use of this feature in LDP signaledVPLS scenarios (RFC 4762) - draft-ietf-l2vpn-signaling, which is currently in RFC Editor Queue,last edited in 2006).The Cisco 7606, Cisco 7604, and ECI SR9705successfully discovered each other as VPLS PEsthrough BGP, where LDP was used to signal thepseudowires. Using BGP signaled pseudowires, theHuawei CX600-X3, Huawei NE40E-X8, and IxiaIxNetwork emulating a VPLS PE also successfullydiscovered each other as a VPLS PE. Followingsuccessful discovery, both scenarios were verified bypassing Ethernet traffic.

Figure 12: MPLS BGP AD

MPLS Transport Profile UnfoldingCurrently the IETF and ITU-T are working together ina ground breaking liaison to define an MPLSTransport Profile (MPLS-TP). The joint effort hasalready passed their first RFC document, RFC 5586,which specifies an IP-less mechanism for identifyingMPLS and pseudowire (PW) maintenance andmanagement packets. The working group strives todefine the framework and requirements for MPLS-TP,standardizing the technology to fulfill the needs ofcurrent transmission networks without changing theforwarding rules of MPLS.

MPLS-TP ChannelsThe latest framework draft proposes multipleservices: Virtual Private Wire Service (VPWS),Virtual Private LAN Service (VPLS), Virtual PrivateMulticast Service (VPMS), and IP-only LAN-likeService (IPLS) that can be provided using the PWclient. The following devices successfully forwardedpoint-to-point Ethernet traffic over MPLS-TP channels:Alcatel-Lucent 1850 TSS-160, Alcatel-Lucent 1850TSS-320, Ericsson Demonstrator, and the HuaweiPTN 3900. The channels were verified to use theencapsulation defined by RFC 5586 for theirrespective OAM mechanisms. The RFC generalizesthe applicability of the Associated Channel Header(ACH) defined in RFC 4385, enabling therealization of a Generic Associated Channel Header(G-ACh) under which to run different OAM mecha-nisms. The Generic Associated Channel Label (GAL)is used to recognize the presence of the ACH.

Proposed OAM and ProtectionSolutions (individual drafts)OAM. Recent discussion in the IETF MPLS workinggroup has been focused on creating a consensusaround a standard OAM solution for MPLS-TP.Different solutions have been proposed in individualauthor drafts such as draft-fulignoli-mpls-tp-bfd-cv-proactive-and-rdi and draft-bhh-mpls-tp-oam-y1731,however the working group has not yet acceptedthese documents as IETF working group drafts. (Thestatus is easy to distinguish; a working group draft

MPLS network

BGP AD with RFC 4761 Signaling

HuaweiCX600-X3

HuaweiNE40E-X8

BGP AD with RFC 4762 Signaling

Ixia IxNetwork

Cisco7606

Cisco7604

ECISR9705

Carrier Ethernet Transport

17

carries “draft-ietf” in its name.)Therefore, the solutions tested and demonstratedhere are based on individual author drafts not yetaccepted into the MPLS-TP framework by theworking group — some of them may never beaccepted, as they are contradictory to some extent.Implementations based on these drafts are all propri-etary today; some of them may become blessed byan RFC in the future, some not.The purpose of our effort is to evaluate the currentstatus of the industry and to examine progress madesince our last test in February. The two OAMproposals tested have received substantial vendorand service provider backing, so we were able tostage multi-vendor interoperability tests.The Alcatel-Lucent 1850 TSS-160, Alcatel-Lucent1850 TSS-320, and Huawei PTN 3900 successfullytested an interoperable OAM solution based ondraft-bhh-mpls-tp-oam-y1731 which proposes modifi-cations to Ethernet OAM tools as defined in ITU-TY.1731 for use with MPLS-TP OAM. Ericsson demon-strated an OAM solution consisting of the IETF-defined Bidirectional Fault Detection (BFD) protocolfound in draft-ietf-bfd-mpls with proposed enhance-ments based on draft-fulignoli-mpls-tp-bfd-cv-proactive-and-rdi and draft-boutros-mpls-tp-cc-cv. Asof IETF 75 at the end of July 2009 it was planned toconverge these two enhancement drafts into draft-asm-mpls-tp-bfd-cc-cv, and submit the convergeddraft for adoption as an IETF working group draft.The enhanced BFD protocol was used to check thecontinuity (liveliness) of the MPLS-TP LSP establishedbetween the Ericsson devices. Both OAM solutionswere verified for the use of GAL and G-ACh.Protection. The Alcatel-Lucent 1850 TSS-320 and1850 TSS-160 both with the Huawei PTN 3900tested an interoperable protection solution, refer-encing the individual draft draft-weingarten-mpls-tp-linear-protection. Ericsson also demonstrated thisprotection between Ericsson equipment. The draftspecifies the mechanisms needed to coordinate theprotection switching on top of the previouslymentioned OAM solutions. Out of service timesamongst the link failure tests ranged from 7 to 24milliseconds upon link failure and from less than 1up to 3 milliseconds upon link recovery. All threevendors also demonstrated such switching viamanual commands as opposed to an automaticaction as a reaction to a link failure, which is alsospecified in the author draft. The Alcatel-Lucent1850 TSS-320 Huawei PTN 3900 additionallydemonstrated the ability to switch to a backup pathwhen the far end OAM source was consideredinvalid based on a mismatch of OAM configura-tions.

Provider Backbone Bridging TrafficEngineering (PBB-TE)We have been testing PBB-TE here at our interopevents since before the standard received its officialIEEE name (back then it was called ProviderBackbone Transport (PBT)). Now a completed

standard (IEEE 802.1Qay), three vendors haveparticipated in this year’s interoperability testingwith solutions.The devices involved in PBB-TE testing included theCiena CN 5305, Ciena CN 3960, Ciena 3940,Ethos E-80, Tejas TJ2050, Tejas TJ2030, and IxiaIxNetwork where Ixia emulated a PBB-TE node trans-mitting PBB-TE encapsulated frames. All equipmentwas involved in the establishment of multi-vendorPBB-TE trunks in order to forward Ethernet frames.CFM CCMs as specified by IEEE 802.1ag wereused by all devices to monitor the liveliness of atrunk. The standard Ethertypes were also used by allvendors for Ethernet frame encapsulation as well asCFM encapsulation. Additionally, all vendors partic-ipated in tests where connectivity was establishedwith the MPLS network, as seen in Figure 11.Protection. Equipment from all three vendorssuccessfully tested PBB-TE protection across primaryand backup tunnels consisting of multiple vendors.For each test, a primary and backup PBB-TE trunkwas configured for a particular service. A link notphysically connected to both end nodes was thendisconnected, and each end node was expected torecognize the failure through a loss of three CCMs.A test was considered a success when the out ofservice time was under 50 milliseconds, which wasthe case for all tests. Failover times observedspanned from 15 to 42.5 milliseconds, andrecovery tests were hitless.

Ethernet Ring Protection SwitchingCarrier Ethernet access networks are oftenconstructed in ring topologies in order to provideresilient path protection while using less cablingresources than other network topologies. The ITU-Thas defined Ethernet Ring Protection Switching (ERPS- ITU-T G.8032) in order to provide resiliencymechanisms for such topologies using the definedAutomatic Protection Switching (APS) protocol.We tested a multi-vendor six node ring consisting ofthe NEC Pasolink NEO TE, RAD ETX-202, and TejasTJ2050. When the tests were first configured therewere inconsistent expectations of how the Mainte-nance Entity group Level (MEL) values were set,however this issue was quickly resolved throughconfiguration.We performed several link failure test runs whereboth NEC and Tejas functioned as the RPL owner.We measured out of service times between 14 and26 milliseconds in case of link failure and 3 to 14milliseconds for recovery.In addition, we successfully tested a protectedEthernet ring between NEC and Tejas using an NECPasolink microwave link. A failure of the airinterface was recognized by NEC equipment whenreception of Continuity Check Message (CCM)frames ceased. This then triggered the ERPS fail-overprocedure. It took 20 milliseconds for the ring toconverge after the air interface failed and 3 milli-seconds to switch back.

18


Figure 13: ERPS

Tejas also demonstrated an open ring topologyplanned for standardization proposal. The ring,consisting of four Provider Backbone Bridging (PBB)nodes, is loosely based on ITU-T G.8032 ERPShowever removing the requirement of a closed ring.The open ring created a dual homed connection to aVPLS service in the MPLS network via the HuaweiNE40E-X8 and ECI SR9705. Failure times rangedfrom 4 to 8.6 milliseconds, and recovery timespeaked at 4 milliseconds.

AcknowledgementsEANTC would like to acknowledge and thank KariAsheim and Terje Krogdahl from nLogic, Oslo, aswell as Stephen Murphey and Thomas Peterson fromthe University of New Hampshire InterOperabilityLab (UNH-IOL) for their excellent support during thehot staging.Thanks to three T5C-48T switches provided by TelcoSystems, management connectivity was extendedthroughout the 15 racks filled by the hot stagingequipment in the lab and the work spaces provided.Editors. This document has been edited by JambiGanbar, Sergej Kälberer, Jonathan Morin, CarstenRossenhövel, Thomas Sladek, and Xiao Tai Yu.

REFERENCES“Ethernet Services Attributes Phase 2”, MEF 10.1“Ethernet Service Definitions”, MEF 6“Implementation Agreement for the Emulation ofPDH Circuits over Metro Ethernet Networks”, MEF 8“Ethernet Local Management Interface”, MEF 16“User Network Interface (UNI) Type 2 Implemen-tation Agreement”, MEF 20“External Network Network Interface (ENNI) - Phase1”, Draft 7.1“Provider Bridges”, IEEE 802.1ad“Provider Backbone Bridges”, IEEE 802.1ah“Provider Backbone Bridges Traffic Engineering”,IEEE 802.1Qay

“Media Access Control (MAC) Bridges”, IEEE802.1D“Media Access Control (MAC) Bridges Amendment2:Rapid Reconfiguration”, IEEE 802.1W“Virtual Bridged Local Area Networks”, IEEE802.1Q“Link Aggregation Control Protocol”, IEEE 802.3ad“Carrier Sense Multiple Access with CollisionDetection (CSMA/CD) Access Method and PhysicalLayer Specifications. Amendment: Media AccessControl Parameters, Physical Layers, andManagement”, IEEE Std 802.3ah“Virtual Bridged Local Area Networks - Amendment5: Connectivity Fault Management”, IEEE 802.1ag“OAM Functions and Mechanisms for EthernetBased Networks”, ITU-T Y.1731“Ethernet ring protection switching”, ITU-T G.8032“Segmented Pseudo Wire”, draft-ietf-pwe3-segmented-pw“Provisioning, Autodiscovery, and Signaling inL2VPNs“, draft-ietf-l2vpn-signaling“Virtual Private LAN Service (VPLS) Using BGP forAuto-Discovery and Signaling”, RFC 4761“Virtual Private LAN Service (VPLS) Using LabelDistribution Protocol (LDP) Signaling", RFC 4762“Encapsulation Methods for Transport ofAsynchronous Transfer Mode (ATM) over MPLSNetworks", RFC 4717“Structure-Agnostic Time Division Multiplexing (TDM)over Packet (SAToP)", RFC 4553“Structure-Aware Time Division Multiplexed (TDM)Circuit Emulation Service over Packet SwitchedNetwork (CESoPSN)”, RFC 5086“The Control of Jitter and Wander within DigitalNetworks which are Based on the 2048 kbit/sHierarchy”, ITU-T G.823“IEEE Standard for a Precision Clock Synchroni-zation Protocol for Networked Measurement andControl Systems”, IEEE 1588-2008“Timing and synchronization aspects in packetnetworks”, ITU-T G.8261/Y.1361“Timing characteristics of synchronous Ethernetequipment slave clock (EEC)”, ITU-T G.8262/Y.1362“Distribution of timing through packet networks”,ITU-T G.8264/Y.1364“Timing characteristics of primary reference clocks”,ITU-T G.811“Timing requirements of slave clocks suitable for useas node clocks in synchronization networks”, ITU-TG.812“Timing characteristics of SDH equipment slaveclocks (SEC)”, ITU-T G.813“MPLS-TP Linear Protection", draft-weingarten-mpls-tp-linear-protection“Pseudowire Setup and Maintenance Using theLabel Distribution Protocol (LDP)”, RFC 4447“BFD For MPLS LSPs", draft-ietf-bfd-mpls“MPLS-TP BFD for Proactive CC-CV and RDI", draft-fulignoli-mpls-tp-bfd-cv-proactive-and-rdi“Connection Verification and Continuity Check forMPLS Transport Profile Label Switched Path", draft-boutros-mpls-tp-cc-cv“MPLS-TP OAM based on Y.1731”, draft-bhh-mpls-tp-oam-y1731

TejasTJ2050 TejasTJ2050

NEC Pasolink NEO iP

Radio link

ProtectedRing NodeEthernet Ring

NECCX2600

RAD

RAD

TejasTJ2050

NECPasolink NEO TE

ETX-202

ETX-202Tejas

TJ2050

NECPasolink NEO TE

Ethernet link

Acronyms

19

ACRONYMSTerm Definition

AC Access Circuit

ACH Associated Channel Header

AIS Alarm Indication Signal

APS Automatic Protection Switching

ATM Asynchronous Transfer Mode

BITS Building Integrated Timing Supply

BFD Bidirectional Fault Detection

BGP Border Gateway Protocol

CCM Continuity Check Message

CE Customer Edge

CE-VLAN

Customer Edge Virtual LAN

CES Circuit Emulation Service

CFM Connectivity Fault Management

DMM Delay Measurement Message

DMR Delay Measurement Reply

DSL Digital Subscriber Line

E-LAN A multipoint-to-multipoint Ethernetservice. A LAN extended over a widearea

E-Line Point-to-Point Ethernet Service similarto a leased line ATM PVC or FrameRelay DLCI

E-LMI Ethernet Local Management Interface

ENNI External Network Network Interface

EFM Ethernet in the First Mile

ERPS Ethernet Ring Protection Switching

EVC Ethernet Virtual Connection

ESMC Ethernet Synchronization MessagingChannel

G-ACh Generic Associated Channel Header

GAL Generic Associated Channel Label

GSM Global System for Mobile

IEEE Institute of Electrical and ElectronicsEngineers

IETF Internet Engineering Task Force

ITU-T International TelecommunicationUnion Telecommunication Standard-ization Sector

LDP Label Distribution Protocol

LMM Loss Measurement Message

LOS Loss Of Signal

LSP Label Switched Path

LTE Long-Term Evolution

MAC Media Access Control

MBMS Multimedia Broadcast MulticastServices, integral part of LTE

MEG Maintenance Entity Group

MEL Maintenance Entity group Level

MEP Maintenance Association End Point

MEF Metro Ethernet Forum

MIP Maintenance Association IntermediatePoint

MIMO Multiple-Input and Multiple-Output

MPLS Multi-Protocol Label Switching

MPLS-TP MPLS Transport Profile

MTIE Maximum Time Interval Error

OAM Operations, Administration andMaintenance

OPEX OPerating EXpenditure

OSPF Open Shortest Path First

PBB Provider Backbone Bridge

PBB-TE Provider Backbone Bridge TrafficEngineering

PCM30 Pulse-Code-Modulation, 30 of 32 E1slots used for data

PCM31 Pulse-Code-Modulation, 31 of 32 E1slots used for data

PE Provider Edge

PDU Protocol Data Unit

PHY PHYsical layer

ppb parts-per-billion

ppm parts-per-million

PSN Packet Switched Network

PTP Precision Time Protocol

PW PseudoWire

QAM Quadrature Amplitude Modulation

RDI Remote Defect Indication

RFC Request For Comments

RSVP-TE Resource reSerVation Protocol TrafficEngineering

RTP Real Time Protocol

SEC SDH Equipment slave Clocks

SDH Synchronous Digital Hierarchy

S-Tag Service Tag

SAToP Structure-Agnostic Time Division Multi-plexing (TDM) over Packet

SLA Service Level Agreement

SyncE Synchronous Ethernet

TDEV Time DEViation

TDM Time Division Multiplexing

TIE Time Interval Error

UMTS Universal Mobile Telecommunica-tions System

UNI User Network Interface

VLAN Virtual Local Area Network

VPLS Virtual Private LAN Service

VPN Virtual Private Network

VPWS Virtual Private Wire Service

Term Definition

EANTC AGEuropean Advanced Networking Test Center

IIR Telecoms

Einsteinufer 1710587 Berlin, GermanyTel: +49 30 3180595-0Fax: +49 30 [email protected]

29 Bressenden PlaceLondon SW1E 5DR, UKTel: +44 20 7017 7483Fax: +44 20 7017 [email protected]

This report is copyright © 2009 EANTC AG. While every reasonable effort has been made to ensure accuracy andcompleteness of this publication, the authors assume no responsibility for the use of any information containedherein.All brand names and logos mentioned here are registered trademarks of their respective companies in the UnitedStates and other countries.v1.2

Editor’s NoteIntroductionParticipants and DevicesInteroperability Test ResultsNetwork DesignMobile BackhaulSynchronization in the Packet Switched NetworkIEEE 1588-2008 (PTPv2)Synchronous Ethernet and ESMC

GSM and UMTS TransportTDM and ATM EmulationMicrowave Transport

Global InterconnectMPLS and Ethernet InterconnectENNI Failure Scenarios

Managed Ethernet ServicesPerformance MonitoringEthernet in the First Mile (EFM)Hierarchical Continuity Fault ManagementE-LMI and IP/MPLS InterworkingMonitoring and Provisioning

Carrier Ethernet TransportMPLS VPLS - BGP Auto-DiscoveryMPLS Transport Profile UnfoldingMPLS-TP ChannelsProposed OAM and Protection Solutions (individual drafts)Provider Backbone Bridging Traffic Engineering (PBB-TE)Ethernet Ring Protection SwitchingAcknowledgements

ReferencesAcronyms

Carrier Ethernet World Congress 2009 Multi-Vendor ......Nokia Siemens Networks FlexiPacket Microwave...

Documents

Transcript of Carrier Ethernet World Congress 2009 Multi-Vendor ......Nokia Siemens Networks FlexiPacket Microwave...