July 14, 2002 IEPG @ IETF 54 Slide 2
Agenda
• Lameness
• Why
• (Surprise:) Spotty(?) results
• Approach
• Plans
July 14, 2002 IEPG @ IETF 54 Slide 3
Lameness is...• When an NS RR right-hand-side is a domain
name that• has no address record(s)
• does not respond to queries
• responds negatively for the zone
• Lameness might happen when the domain name• has multiple addresses and at least one fits the
above
• responds non-authoritatively (ie recursively)
July 14, 2002 IEPG @ IETF 54 Slide 4
Why Bother?
• ARIN membership raised the issue of cleaning this up
• Lame delegations cause some popular software to behave badly on the Internet
• Lame delegations can be limited easily
• Intermittent network problems make it infeasible to eliminate it completely
July 14, 2002 IEPG @ IETF 54 Slide 5
Reverse Map• This effort is targeted at ARIN's reverse map
delegations• ARIN's /8's
• Legacy /8's
• Not all /8's - not RIPE's, not APNIC's
• Dependencies are• simplifying assumptions about the parsing of the
zone files
• summary output breaks results into /16's and /24's
July 14, 2002 IEPG @ IETF 54 Slide 6
State of This Work• Code first ran at NANOG 25
• "just in time" development
• Trying at home set me back a bit
• Code now runs at the home office• Results have not been verified
• Results "look" to be valid
• In brief, problem was traffic shaping• Unreliable UDP as a testing mechanism
• Bandwidth bottleneck upstream (i.e., T-1)
July 14, 2002 IEPG @ IETF 54 Slide 7
Early results• Remeber, this is not all of in-addr.arpa...
• Using counts from last run• Number of NS RR's 548,667
• Number of zones 231,240
• Number of name server names 25,047
• Number of IP unique addresses 21,846
• Of three runs made just before leaving for here, two runs had very similar counts, all three had similar %'ages
July 14, 2002 IEPG @ IETF 54 Slide 8
per Zone demographics
• Servers per zone - max 7, avg 2.37
• Addresses per zone - max 26, avg 2.32
• Zones with no addresses 3,062
• Zones with one address 7,365
• All zones have multiple NS RR's
• Some lacked glue for one, some had two names with identical glue, some duplicates slipped through
July 14, 2002 IEPG @ IETF 54 Slide 9
per Name Server
• Zones - max 5772, avg 21.9
• No address - 3,178
• Multiple addresses - 219
• Addresses - max 24, avg not counted
• Longest name 41 chars
• just to tell me how big to make name array
July 14, 2002 IEPG @ IETF 54 Slide 10
per IP
• Zones - max 5772, avg 24.6
• Addresses with multiple domain names pointing to them - 291
• Max number of domain names pointing to an address - 9
• PTR records not checked
July 14, 2002 IEPG @ IETF 54 Slide 11
Results in percentages• Counting by IP addresses:Number of zones 100% 75-99% 50-74% 25-49% <24% dead sample size
any 46% 8% 8% 5% 17% 16% 21,846
1 64% 0% 0% 0% 16% 20% 10,085
2 51% 0% 20% 0% 11% 17% 2,214
3-4 42% 6% 14% 12% 10% 17% 1,778
5-8 34% 13% 12% 9% 17% 15% 1,305
9-16 30% 17% 9% 10% 24% 10% 1,883
17-32 25% 20% 13% 11% 21% 11% 1,624
33-64 14% 29% 16% 11% 19% 11% 1,218
65-128 9% 33% 18% 12% 22% 5% 896
129-1024 7% 33% 17% 18% 18% 7% 811
1025-2048 0% 64% 7% 29% 0% 0% 14
2049-8096 0% 56% 0% 44% 0% 0% 18
July 14, 2002 IEPG @ IETF 54 Slide 12
What the preceding means• 100% servers are those that answered
authoritatively for all of the claimed zones
• 75-99% servers answered positively for almost all (one timed out zone would know a 100%'er to this
• 0-24% servers are likely not answering positively for much
• dead means there was never any reply to a query (not even servfail)
July 14, 2002 IEPG @ IETF 54 Slide 13
Counting by ZonesCategory All /16's /24's
No IP address 1% 1% 1% - unreachable
One IP address 3% 5% 3%
Multi address 95% 94% 96% - "the requirement"
No working 38% 21% 39% - zones not reachable
One working 10% 12% 10%
Multi working 52% 67% 51%
No broken 49% 58% 49% - "perfect" zones
Some broken 13% 21% 12%
All broken 38% 21% 39% - unreachable
July 14, 2002 IEPG @ IETF 54 Slide 14
What the preceding means
• "No working" means that one will never get a reply about that zone (terminal lameness)
• "No broken" means that all NS records lead to good servers (no lameness)
• "Some broken" means that there is some lameness
July 14, 2002 IEPG @ IETF 54 Slide 15
What's Missing
• A good measure of how many NS RR's are faulty
• It dawned on me last week that I hadn't counted this - d'oh!
• Code now dumps results 1:1 with NS RR's
• Has to deal with multiple-address situations
• Need to sort into canonical order for comparisons
• Need to account for changes in NS RR's over time
July 14, 2002 IEPG @ IETF 54 Slide 16
Verifying Results
• A list of all test results is produced
• Just added
• Should be 1:1 to NS RR's but 12K are missing during last run
• Spot checks ought to be done, as testing via UDP is inherently inaccurate
• List of results from different network locations should be correlated
July 14, 2002 IEPG @ IETF 54 Slide 17
Discussion Points
• Test takes 11 hours via a T-1
• Could speed up if servers always answered (djbdns issue)
• UDP congestion control would help
• Coordinating multiple instances of test
• Eliminate false positives
• Not for here: what will ARIN/RIRs do with this?
July 14, 2002 IEPG @ IETF 54 Slide 18
Approach
• Build the following lists from the zone files
Zone RecordNS Domain
NameIP address
NS DomainName
IP address
July 14, 2002 IEPG @ IETF 54 Slide 19
Why?
• This has been seen (1.128.in-addr.arpa):
Zone RecordNS Domain
NameIP address
NS DomainName
IP address
IP address
July 14, 2002 IEPG @ IETF 54 Slide 20
The program
• Runs in two phases
• Reads NS RR's• Builds linked lists
• Uses gethostbyname() to get "glue"
• Runs through IP addresses• Issues SOA queries
• Looks for aa=1, rcode=0, ancount=1
• Both steps print results
July 14, 2002 IEPG @ IETF 54 Slide 21
Impact on the 'net
tester InternetNS
NS
NSNS
NSNS
NSNS
About 1 second apart
Performance hit is close to home
July 14, 2002 IEPG @ IETF 54 Slide 22
Chief Implementation Issue• Speeding up tests
• When there's no answer, I use 3@30 second time outs
• No reason to wait on bad servers
• Queries are parallelized in two dimensions• Multiple IP addresses can be under test
simultaneously
• Multiple zone requests are pipelined to a server
• Wouldn't need to speed this up if down servers could be eliminated quickly
July 14, 2002 IEPG @ IETF 54 Slide 23
Cost of Speeding Up
• Test environment:
tester
router Internet10/100Mb
1.5Mb
`excess` packets?
July 14, 2002 IEPG @ IETF 54 Slide 24
Solution
• Needed to shape the traffic
• Limit number of IP addresses tested
• Stagger pipelined requests (one second apart)
• Seems to slow transmission
• Seems to avoid any rate limiting (if any)
• Watching queries on network shows traffic is smooth, not bursty
July 14, 2002 IEPG @ IETF 54 Slide 25
Next steps
• Finish tweaks to code
• Distribute and run from different locations
• Present observations to membership
• Investigate the use of this data
July 14, 2002 IEPG @ IETF 54 Slide 26
Questions
July 14, 2002 IEPG @ IETF 54 Slide 27
Answers
Top Related