IPv6 R&E Deployment and Campus Science DMZ

Alan Whinery – whinery@hawaii.edu Ken Miller - kdm193@psu.edu

CANS Sept. 22, 2015

CANS Agenda

Telemetry

SDN

Science DMZ

Measuring IPv6 - sFlow

IPv6 Deployment

IPv6 R&E Deployment

IPv6 R&E Deployment v Alan sends his regards v http://www.internet2.edu/

presentations/jt2010feb/20100201-huque.pdf

v http://www.mrp.net/ipv6_survey/#internet2

v How’s your network? v Internet2 Net+ How many are IPv6

ready?

Measuring IPv6 with sFlow

sFlow v Exports truncated packets, together

with interface counters v An sFlow system consists of multiple

devices performing two types of sampling: §  random sampling of packets or application

layer operations §  time-based sampling of counters

sFlow Push vs SNMP counter poll

sFlow can push interface counters out every 20 seconds instead of polling with

SNMP every 1-5 minutes

Interface sFlow Trend sFlow Research Flow Query

Science DMZ Fabric View Border/Science DMZ/Core Weathermap

100G Optical Temp/Power Monitoring

ISCSI Storage Network

Apache httpd

sFlow visibility

L2-L7 Tranparency with sFlow

Including MPLS/VPLS

SDN: Software Defined Networking

Feb 21 2014 – Xbox DDOS

v  14Gbps/20Gbps v  43,322 Sources v  38 minutes long

Border ACL Block Process

• Border SPAN Port • BRO, Accona Tools • NTP, DNS,

CharGen Attacks… • TCP-Syn Floods

Security reviews events/

logs

• New or updated Border ACLs

• BGP Blocking/Black hole • Some events are handled manually

• Some events are handled automatically

Contact NOC or sync with NOC

scripts • Cron • Border ACLs – 5 minutes • SSH Failures – 30 miutes • SynFloods – 5 minutes

• Manual entry • ACLs immediate on • BGP Black Hole

Blocking DDOS

Copyright © InMon Corporation

Customer link traffic

sFlow-RT Flood Protect

Attack

Normal

Flood protection

sFlow-RT Controller

Analytics Control

0 000 Protected link

Upstream traffic

A Research Network based on a Science DMZ Model

v Brocade won the RFP for new core, 100G, MPLS/VPLS, sFlow, SDN

v No Border Firewalls at PennState Customer Edge firewalls.

v Top 10 v4/v6 based on sFlow border data v 2 MLXe Routers – 100G to core v 2 VDX 8770s – vLAG’d to MLX v 12 edge VDX 6740s from Top10 Border

Capacity §  Sequencers, Sensors §  Instruments, Telescopes, Microscopes §  HPC Compute/Storage §  Central Storage

PSU Core

PSU Science DMZ

Secure Researcher On-Ramp

Faster Research Data

to HPC, Storage, and outside Penn

State TNS Provision

Port

SOS Data

Categorization & Survey

SOS MOU

& OLA

TNS Port

Request

Secure Researcher On-Boarding v Engagement

§  Researcher Interview

•  Workflow •  Data Source •  Data Destination •  Data Cat

–  Before Compute –  After

§  Local IT Support •  Policies •  Configuration

v Guidelines/Compliance §  ITS-SOS MOU §  Data Categorization §  Minimum Security

Baseline §  Operating Level

Agreements -OLA •  Vulnerability

Scanning •  Host mitigation

Research Device Security

v Host-Based

v Firewall:iptables,ip6tables,firewalld,

v Host Intrusion Detection OSSEC

v Anti-Virus, Malware

v Network v  Deny All then ACLs built

around researcher requirements

v  IPv6 then RFC1918 private IPv4 to limit public access to workstations and servers

v  Use IPv6 DTN for public IP and data transfer

v  MAC security per port v  Research project/

building specific VLANs

Researcher IT Best Practices

v Department Level

v On-boarding v Policies vs.

central policies v IT directors

reporting to Deans?

v Stop Opt-Out

v Create an enabled exception

Spans- TAPs - Telemetry

TAPs, SPANs, Telemetry •  Review packet monitoring for security, I've been going off of mentioned

requirement of 'All HSBBs' monitored.

•  The first idea is fiber TAPs and individual fiber paths back to SOS. This is what SOS is doing now and usually ad-hoc, timely, costly, and doesn't scale.

•  Secondly, the SPAN-o-matic idea, dedicated fiber patch back from each router back to SOS tools, was what we talked about a few months ago with SOS. The fiber would be in place and all they would need to do is call the NOC and have a HSBB mirrored back to SOS tools.

•  Third is the Brocade Telemetry solution. This takes the SPAN-o-matic idea of dedicated fiber path but scales for HSBBs: all, any, or none, back to a packet broker router. The second advantage is the ability to fan out or filter flows from the packet broker to multiple security tools. It would be a a SOS-self service tool they control where the flows go and to what tools.

Fiber TAPs •  Fiber requested on as needed •  TAPs installed •  Little documentation from CoE requests •  High cost •  Fiber needed for each TAP •  Doesn’t Scale •  Takes Time

SPAN Network

Packet Broker

Telemetry

Costs

TAPs

•  TAPs for each connection

•  Fiber for each TAP •  TAP for both router

uplinks •  $6 million - Gigamon

SPAN

•  10G from each distribution router back to SOS Tools

Telemetry

•  100G from each

distribution router back to Dedicated Brocade MLXe Packet Broker

•  @65% off = $ 871K •  Replacing new core

100G cards with 2 100G •  @65% off = $ 537k

Secure DMZ Network

BGP

ACLs PCFN

performance

L2/ Fabric

Monitoring

Security Status

Next Steps

Edge

Distribution

Core

Border

Host Firewall

sFlow, SPANs

sFlow, TAP Aggregator

ACLs, BRO, BGP Blackhole

What can you learn from outside collaboration

Application Latency v Application Metrics - Do your

protocols support IPv6? Check out SmokePing

v  DNS v  email v  www/http/https v  DHCP v  NTP v  SSH v  LDAP

Netflix

perfSONAR 3.5

perfSONAR 3.5 v Brand new web interface v Toolkit ISO on both CentOS and

Debian/Ubuntu v Enhanced central management and

“auto-configuration” v New Puppet Module

Esnet perfSONAR Mesh

perfSONAR Mesh /MadDash v Baseline – perfSONAR

§  ipv4 bwctl iperf3 mtu 1500 tcp/udp §  ipv4 bwctl iperf3 mtu 9000 tcp/udp §  ipv6 bwctl iperf3 mtu 1500 tcp/udp §  ipv6 bwctl iperf3 mtu 9000 tcp/udp §  loss

I2 Performance Measurement

v PerfSONAR Mesh with ESNet v Hosting a perfsonar workshop in on

campus or REN v sFlow/DDOS webinars in Sept 2014

and June 2015 v Up coming papers:

§  perfSONAR Host Naming Conventions § Campus Deployment Guidelines §  Performance and measurement maturity

model §  Performance to Net+ services

Esnet perfSONAR Globus DTN

Research Route Optimization

Ken Miller – kdm193@psu.edu