Engineering Report

Mark Kosters

Engineering Theme• 2012 success is being aided by

contractors (but not near as many)

• We have one ARIN FTE slot open• Lots of work is done, but there is

much more to do

2 2

Staffing• Operations

– 7 Operations staff + 2 Managers – FULLY STAFFED (up 3 since ARIN 30)

• Development– 7 Developers + Manager – FULLY STAFFED

(up 2 since ARIN 30)– 1 Contractor (down 1 since ARIN 30)

• Quality Assurance– 4 QA + Manager (up 1 since ARIN 30) – ONE OPENING– 2 Contractors (down 2 since ARIN 30)

• Project Management– 1 Project Manager

• Management– 1 (me)

33

Operations• Upgrading end-of-life equipment• Maintaining the various environments we have

running (Production/OT&E/Dev/QA/Staging)• EMC and Oracle challenges• Load Balancer stability• Moving production to the colocation facility• IT support• Billing Integration• Replaced 12 year old PBX with VOIP-based

system

44

5

Core Service Statistics• ARIN Online• Reg-RWS • Whois-RWS• IRR• RPKI

How is ARIN Online used?• 62,998 accounts activated since

inception through Q1 of 2013

20082009201020112012

2013*

Number of Accounts Activated5000 10000 15000 20000

* Through Q1 of 2013

66

Active Usage of ARIN Online

0 1 2 - 5 6 - 10 11 - 15 >160

5000100001500020000250003000035000

Logins

# of

Us

ers

Times logged in

77

Reg-RWS (RESTful Provisioning)

Cumulative totals since April, 2011

REST

Templates

0 200000 400000 600000 800000 10000001200000

Transactions

8

RESTTemplates

0 20 40 60 80 10 12

Transactions

ARIN XXX

ARIN XXIX

REST

Templates

0 500000 1000000 1500000

Today

Transactions

8

Whois-RWS Traffic Loads• Running “normally” now at 420

queries per second (QPS)– Up 18 QPS since last meeting

• RESTful calls have overtaken Port 43 calls since March of 2012

99

Whois-RWS Queries on Port 43

Months

Que

ries

Per

Sec

ond

X100

0

10

2000-01 2001-09 2003-05 2005-01 2006-09 2008-05 2010-01 2011-090

2000000000

4000000000

6000000000

8000000000

10000000000

12000000000

10

Whois-RWS Queries Broken out by Service Type

Months

Tota

l Que

ries

(x

1000

0)

11

2010-09

2010-10

2010-11

2010-12

2011-01

2011-02

2011-03

2011-04

2011-05

2011-06

2011-07

2011-08

2011-09

2011-10

2011-11

2011-12

2012-01

2012-02

2012-03

2012-04

2012-05

2012-06

2012-07

2012-08

2012-09

2012-10

2012-11

2012-12

2013-01

2013-02

2013-030

2000000000

4000000000

6000000000

8000000000

10000000000

12000000000

port43port 80 (/ui)port 80 RESTful

11

Whois-RWS – IPv6To

tal P

er M

onth

Month

12

2009

-01

2009

-06

2009

-11

2010

-04

2010

-09

2011

-02

2011

-07

2011

-12

2012

-05

2012

-10

2013

-030

50000001000000015000000200000002500000030000000

v6 port 80v6 port43

12

IRR Usage over the last two years

1600

1700

1800

1900

2000

Maintainers

13

2011 2012 2013

13

IRR Usage over the last two years

2011 2012 2013350400450500550

17000

18000

19000

20000

21000

Route

14

Inetnums

14

IRR Usage over the last two years

2011 2012 20130

10203040

0

200

400

600

800

Route6

15

Inet6nums

15

16

Production RPKI• Getting the Trust Anchor– 72 people have signed the RPA as of

4/13– This is up from 27 reported at ARIN XXIX

• Signing Resources– 47 Organizations with certified resources– 21 Organizations with ROAS (was 7 at

ARIN XXIX)– 60 ROAS (was 19)– 82 Networks/AS’s (was 30)

17

Production RPKI• Web Delegated RPKI– 0 participants as of Apr 18, 2013– Not unexpected– Only a handful delegated in RIPE’s

region–Why is it not so popular?• Need to setup your own CA and software

suite• Very hard to setup up/down protocol• Not a lot of general interest

Development/QA• Improvements to existing systems• ARIN Online releases since ARIN

XXIX– Integrated Invoices– Fee Calculator–Web-Based Delegated RPKI– Extended Stats Enhancements– Various minor bug fixes

18 18

Initiatives Currently Underway• Improvements to internal billing

systems• Integration of 3rd party election

and meeting registration systems

• Move from Oracle to PostgreSQL• Delegated RPKI using Up/Down

protocol

1919

20

Initiatives Currently Underway• Moving production away from

ARIN HQ to colocation facility• Creating OT&E instance of RPKI

Outage Observations• We have had numerous

maintenance outages–Mostly planned– Two unplanned

• Root Cause– Oracle housed on EMC– EMC has had numerous failures• Secondary systems did not work• Configuration issue on Oracle host

2121

So what, just call tech support• Both vendors are big – perhaps

too big–Multiple handoffs• Same questions asked • Same mitigation techniques demanded

– ARIN is too small• Suboptimal support• Attempt to tell them we are important• Escalation to management is futile

2222

Comments?

2323