Anella Científica: Anella Científica:
A virtualized research networkA virtualized research network
Maria Isabel Gandía CarriedoCommunications Department, CESCA
Terena Networking ConferenceUniversidad de Málaga, 11/06/2009
AgendaAgenda
Anella Científica• Introduction• Functionalities• Evolution
The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment
Examples of use
AgendaAgenda
Anella Científica• Introduction• Functionalities• Evolution
The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment
Examples of use
About CESCA and Anella Científica About CESCA and Anella Científica
CATNIX created in 1999
Commercial Internet
Public consortium Created in 1991 Formed by:
• Generalitat de Catalunya
• Fundació Catalana per a la Recerca i la Innovació
• 9 Catalan universities
• Consejo Superior de Investigaciones Científicas
Anella Científica created in 1993
About CESCA and Anella CientíficaAbout CESCA and Anella Científica
Anella Científica is the high-speed communications network that connects the universities and research centres in Catalonia
(“Scientific Ring”)
CESCA, as the manager of the Regional Research and Education Network (RREN) in Catalonia and as a Local Internet Registry (LIR) has:• Addresses for the connected institutions:
– IPv4: 84.88.0.0/15– IPv6: 2001:40B0::/32
• An Autonomous System (AS):– AS13041
About CESCA and Anella CientíficaAbout CESCA and Anella Científica
Even for enterprise researchers not only RedIRIS members
CESCA and Anella Científica provide services...CESCA and Anella Científica provide services...
For all the universities and research community not only Ethernet
For all Catalan universities and researchers, not only in Barcelona area not everyone has fibre
For special projects and regular traffic special circuits, dedicated fibres, QoS
(Unicast || Multicast) && (IPv4 || IPv6)
The Anella Membership PolicyThe Anella Membership Policy
A
A.1Public and Private
Universities
A.2Public Research
Organizations& LSF
A.3 Research Centres
and Institutes
A.4Hospital Research
Departments
B
B.1Manager of R+D+i
Program with Public Funding
B.2Institutions with
Relevant Digital Contents for the Scientific and Technical Community
B.3Entities Participating
in R+D+i Projects
B.4Entities of Special
Interest
C
C.1Technological and
Scientific Parks
C.2Other Hospital Departments
Own connection
CAR
Guttmann
2-02
UB
UAB
UPC
UPF
UdG
URV
UdL
UOC
URL
UVic
UIC
UAO
UdA
ESMUC
EUSS
INEFC
ASPB
TERMCAT
CIDEM
FCRI
BAdM
BC
UnescoCAT
CConsultiu
CBUC
CDP
Liceu
VINSEUM
CHV
CSPT
A.1 A.2 A.3 A.4 B.1
B.2
CatSalut
XTEC
B.4
FHAG
XarxaTecla
C.2
PRBB
ParcUdG
C.1
Anella Científica: connected institutionsAnella Científica: connected institutions
BSC
CESCA
CIEMAT
CELLS
CSIC
IDIBELL
Puigvert
SantPau
Dexeus
VHebron
FCRB
FUB
IMIM
FIGTP
BGSE
CESC
CETI
EUPMT
RI
Linked
16-02
≤ 8≥ 10≥ 100≥ 1.000
CTTC
CTFC
IRTA
IGC
IEEC
FBM
i2CAT
ICC
ICIQ
ICFO
IdeG
IDESCAT
IEC
PAM
CRG
IFAE
20-01
21-04
01-01
Anella Científica: projectsAnella Científica: projects
PIC participates in LHC (10 Gbps) i2CAT participates in several European projects:
FEDERICA, Phosphorus,… (10 Gbps) UPC-CCABA participates in EuQoS, MUPBED,… (1 Gbps) New card and 10 Gbps for PASITO Two temporal increases of BW for FCRB at 100 Mbps
(DVTS) Liceu transmits the Opera Oberta course
Anella Científica: EvolutionAnella Científica: Evolution
Time period
Name & logo
Points of access
Technology BW (Mbps) Dedicated circuits
1993-1998 ANELLA CIENTÍFICA
6-8 Ethernet over DQDB
10 (34 Mbps core)
No
1998-2001 15-19 IP/ATM 34-155 (622 Mbps core)
Yes
2001-2003 19-37 IP/ATM 34-155 (2,5 Gbps core)
Yes
2003-2008 37-76 Ethernet over WDM
10-10,000 Yes
2008-2011 76-... IP/DWDM 10-10,000
...
Yes
Changes in the networkChanges in the network
Anella Científica had a central node with separated L3 equipment for institutions type A+B and C.
Level 2 and 3 equipment was the same for most of them. There were several L2 projects involved. In 2008, both the network and the equipment had to be
renewed. Two Calls for tenders to be submitted in July 2007:
one for the network and one for the equipment. Survey among our users to know their needs.
The survey for the last version of Anella CientíficaThe survey for the last version of Anella Científica
Goal: plan the evolution of the RREN (Regional Research and Education Network) and prepare the call for tenders.
Survey was sent in December 2006 to 62 institutions.
We asked about:• Connection needs• Evaluation of our services • Evaluation of our dissemination activities • Planning of new deployments
Results of the survey: BW & points of accessResults of the survey: BW & points of access
2006 2008 2011 2013
Points of access 73 68 69 70
500 Mbps 4 11 17 23
100 - 500 Mbps 16 16 17 14
10 - 100 Mbps 32 30 24 22
10 Mbps 21 11 11 11
Aggregated BW (Mbps) 5,242 11,956 21,625 47,740
2006 2008
Local connections 11 12
10,000 Mbps 0 1
1,000 – 10,000 Mbps 4 5
100 – 1,000 Mbps 2 2
100 Mbps 5 4
Aggregated BW(Mbps) 4,613 19,211
Sustained and progressive growth
Regular traffic connections• Numbers of points of access didn’t
increase considerably• Growth with the access of the Health
and Culture Departments
Special projects
Conclusions of the surveyConclusions of the survey
Requirements for the new Anella Científica:• Increase the BW • Asymmetric and flexible model to cover different needs • Adaptable topology, specially for relevant projects
More reliability: redundancy in the core nodes,in the accesses, etc.
Evolution towards a distributed network:• Two points of presence interconnected• L2/L3 equipment redundant
Connections:• Own fibre• Rent fibre to a operator• Bandwidth from an operator
Anella Científica: 2006Anella Científica: 2006
Internet
Operator
10 Gbps 1 Gbps 100 Mbps 34 Mbps 10 Mbps ≤ 8 Mbps
C. Nord
Anella Científica: 2008Anella Científica: 2008
1. Public and private non-profit Universities2. Official Bodies of Research3. Other non-profit Research centres4. Hospital Research centres
1. Official bodies of R+D management2. Relevant Digital contents institutions3. R+D+i participants4. Special interest for R+D institutions
1. Science and technological parks
2. Other hospital units
A B C
C. Nord Telvent
Operator
Internet
AgendaAgenda
Anella Científica• Introduction• Functionalities• Evolution
The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment
Examples of use
Previous topologyPrevious topology
C. Nord
Catalyst 6513 Level 2/3
A+B institutions
Juniper M320Level 3 (RedIRIS)
Nortel
Level 2 (RedIRIS)
Telvent
Cisco 7200Level 3
C institutions
Internet
Special projects
Localconnections
Operator
Topology: what we asked forTopology: what we asked for
Annexus Telvent
Level 3
equipment
Special projects
CESCA-T
Level 2
Level 3
CESCA-CN
Level 2
Level 3
OperatorLocal connections
RedIRIS
Level 2
RedIRIS
Level 3
The equipment we asked forThe equipment we asked for
It had to:• Be modular, scalable, with redundant power supplies, manageable
via SNMP,…• Support BW reservation, QoS, MPLS, multicast, IPv6, SPAN ports• Have good performance• Permit 802.1q VLAN, jumbo and baby giant frames, VLAN stacking• Have filtering mechanisms like access-lists
Virtualization was an improvement
The equipment we asked forThe equipment we asked for
The old equipment could be re-used, improved, its cards could be replaced… It could be used in any of the nodes, for level 2, level 3 or both.
We explained all its characteristics (in fact, the same we were asking for!)…
…Maybe that was the reason why in the 3 offers we had, the equipment we were offered was from the same vendor, with different configurations, but always with two new equipment and reusing the previous switch/router.
We chose the more advantageous: separated L2 and L3 equipment in one node, same equipment in the other node.
Final topologyFinal topology
C. Nord Telvent
Special projects
OperatorLocal connections
Catalyst 6509
Level 3
Catalyst 6513
Level 2
Juniper M320
Level 3 (RedIRIS)
Nortel
Level 2 (RedIRIS)
Catalyst 6509
Level 2/3
DWDM10 Gbps
Giganet2 x1 Gbps
The new core networkThe new core network
Core with two nodes linked with 10 Gbps DWDM + 2Gbps CESCA-CN
• Segmentation of functionalities level 2 and 3:• Catalyst 6513 Sup 720-3B (L2)• Catalyst 6509 Sup720-3BXL (L3)
CESCA-T• Catalyst 6509 Sup720-3BXL (L2/L3)
Use of Virtual Routers Easy creation of new virtual routers (projects,
management, pre-production, …) Separation of routes Layer 3 Balancing/backup More flexibility, redundancy and reliability
Why virtualization?Why virtualization?
It permitted to have many different routers in each node for: • Different types of institutions (A, B and C).• Pre-production environment.• Special projects.• An institution with many points of access that needed a separated
router for its Campus LAN.
It allowed us to take an ISP approach more than a Campus approach.
We had the backup control on the L3 network. Some parts of the L2 network are under the control of the Operator.
Having more nodes would be easier.
ChangesChanges
C. Nord TelventC. Nord
Virtualized core networkVirtualized core network
CoreNetworkCESCA-CN CESCA-T
@REDIRIS CATNIX
@CESCA A, B INET-ALPI@CESCA C
@REDIRIS CATNIX
@CESCA A, BINET-ALPI@CESCA C
MPLS
OSPF
Providers
Institutions
REDIRIS BCN REDIRIS VALORANGE BCN1 ORANGE BCN2
ISP AISP B
ISP C
Core network
CESCA-CN CESCA-T
@REDIRIS CATNIX
@CESCA, A, BINET-ALPI@CESCA, C
@REDIRIS CATNIX
@CESCA, A, BINET-ALPI@CESCA, C
@CESCA, A, BINET-ALPI@CESCA, C@REDIRIS
Logical architectureLogical architecture
Routing IPv4 (example)Routing IPv4 (example)
Redundancy• Between nodes of CESCA• with RedIRIS (or Orange)• at CATNIX
eBGP Anella RedIRIS-VAL
eBGP Anella RedIRIS-CAT
eBGP Anella CATNIX
eBGP Anella CATNIX
BGP RedIRIS CATNIX
CESCA-CN CESCA-T
RedIRIS Barcelona
RedIRIS
iBGP institutions
OSPF infrastructure
RedIRIS València
Institution
eBGP CESCA-CN Institution
eBGP CESCA-T Institution
The management (I)The management (I)
It is more complex• More equipment involved• More dynamic routing (with providers and institutions)• More interfaces to connect with the operator (from 2 to 20)• More VLAN (from 150 to 270)
New syntax, remember not to forget VRF! Some “false friends”…the legacy commands…
• show ip bgp summary (there are no neighbours!) -> show ip bgp vpnv4 vrf <NAME> summary
• ping 10.1.1.1 (oops! It doesn’t work) -> ping vrf <NAME> 10.1.1.1
The management (II)The management (II)
The real friends: • Aliases!
alias exec bgp-ri-adv sh bgp vpnv4 unicast vrf <NAME> neighbors 10.1.1.1 advertised-routes
alias exec pir ping vrf <NAME1>
alias exec pia ping vrf <NAME2>
alias exec par ping vrf <NAME3>
• Includes, but do not abuse…show vlan | inc <INSTITUTION>
show int desc | inc IP6
…
show ip bgp vpnv4 all | inc 10.1.1.0 (No!!)
• Work with a text file, not with the configuration directly
During the deployment (I)During the deployment (I)
In the Catalyst 6500, by default, all the interfaces have the same MAC address!!!cs-6506-24a#show interfaces | include line | address Vlan1 is down, line protocol is down Hardware is Cat6k RP Virtual Ethernet,
address is 00d0.bcf1.ee5c (bia 00d0.bcf1.ee5c) Internet address is 14.18.2.182/16
Vlan2 is down, line protocol is down Hardware is Cat6k RP Virtual Ethernet, address is 00d0.bcf1.ee5c (bia 00d0.bcf1.ee5c)
If the same equipment has more than one interface on the same VLAN (for instance, one with IPv4 and one with IPv6) there are random flaps, packet loss, etc.
During the deployment (II)During the deployment (II)
Address-family ipv4 multicast is not supported in the VRF:
router(config)#router bgp 13041
router(config-router)#address-family ipv4 multicast vrf TEST ^
% Invalid input detected at '^' marker.
Static routing in only one VRF for multicast.
VRF IPv6 are not recommended:
router(config-router)# vrf upgrade-cli multi-af-mode ...Experimental command, not supported by Cisco.
We keep IPv6 in the physical router rather than in a VRF.
Other opened issuesOther opened issues
Tagging of the routes per-VRF and not per-prefix. Supported by Cisco, but not much documentation:• mpls label mode vrf <vrf> protocol bgp-vpnv4 per-vrf
It would be desirable to “enter the configuration” of each VRF instead of writing the VRF each time you write a command.
The name of the VRF can not be changed!!
After the deploymentAfter the deployment
Two bugs affect the IOS causing a memory leak and increasing the CPU (workaround: avoid “show run” commands ).
Maximum number of routes is approximately 106.If the routes increase considerably, we may face a problem. You can not have so many full-routings.
AgendaAgenda
Anella Científica• Introduction• Functionalities• Evolution
The new core of the network• Topology and equipment• Technology: virtualization in the core• The management• During the deployment• After de deployment
Examples of use
Case study: UABCase study: UAB
VLAN from Bellaterra to CESCA-CN for multicast
UAB–CS
Giganet 100 Mbps
FIGTP
Ethernet 20 Mbps
HUVH (unitat docent)
Giganet 500 Mbps
VLAN from each point of access to CESCA-CN & CESCA-T (internet)
VLAN each point of access to Bellaterra forn internal routing
UAB-IGOP
ADSL 4 Mbps
FUB
ADSL 4 Mbps
PRBB (unitat docent)
Giganet 100 Mbps
CESC
Giganet 100 Mbps
UAB (Bellaterra)
Giganet 1 Gbps
HSP (unitat docent)
Giganet 500 Mbps
HSP (Casa Convalescència)
Giganet 500 Mbps
C. Nord Telvent
VLAN with i2CAT for the Electronic Learning Singing project
i2CAT
VLAN PIC for the LHC project
UAB (IFAE/PIC)
DWDM 10 Gbps
Case study: IRTACase study: IRTA
Intra-institution connections
Internet access
IRTA
INET-ALPI@REDIRIS
IRTA has 9 points of access with different technologies, from ADSL to FastEthernet
The point of access in Barcelona makes NAT and filters all the connections, and all of them go to the internet through it
IRTA-MV
IRTA-CT
IRTA-CA
IRTA-CM
IRTA-MO
IRTA-CO
IRTA-SC
IRTA-VA
IRTACESCA
Thanks for your attention!Thanks for your attention!Questions? Suggestions?Questions? Suggestions?
Top Related