Web100Web100

Basil IrwinBasil Irwin

irwin@ucar.eduirwin@ucar.edu

National Center for Atmospheric ResearchNational Center for Atmospheric Research

Matt MathisMatt Mathis

mathis@psc.edumathis@psc.edu

Pittsburgh Supercomputing CenterPittsburgh Supercomputing Center

Halloween, 2000Halloween, 2000

National Center for Atmospheric ResearchNational Center for Atmospheric Research 22NCARNCAR

The ProblemThe Problem

National Center for Atmospheric ResearchNational Center for Atmospheric Research 33NCARNCAR

Problem ExplanationProblem Explanation

•A major TCP design feature is to compensate for packet/ack latency by sending and buffering multiple consecutive unacknowledged packets

•Packets are buffered for possible retransmission until acknowledgement is received, so the maximum size of the TCP transmit buffer determines the maximum amount of data that can be retransmitted prior to acknowledgement

National Center for Atmospheric ResearchNational Center for Atmospheric Research 44NCARNCAR

Problem Explanation (cont.)Problem Explanation (cont.)

•Transmission will stop if the TCP transmit buffer fills up because none of the transmitted packets have been acknowledged

•To utilized the full bandwidth of a given connection, packets need to be transmitted continuously, i.e., packet transmission must never stop

•Therefore, the TCP transmit buffer must be large enough to avoid completely filling up with unacknowledged packets

National Center for Atmospheric ResearchNational Center for Atmospheric Research 55NCARNCAR

Problem Explanation (cont.)Problem Explanation (cont.)

•To avoid “premature” filling, the TCP transmit buffer must be at least the number of bytes equal to the product of the round trip latency and the bandwidth of the connection

•The problem is that the TCP transmit buffer size is usually defaulted by the application to the kernel default value, and the kernel default is usually sized for low bandwidth LAN connections with very small latency

National Center for Atmospheric ResearchNational Center for Atmospheric Research 66NCARNCAR

Problem Explanation (cont.)Problem Explanation (cont.)

•Also, most apps don’t provide a user-parameter to override the default value

•And even if they did, how does one “know” what the bandwidth of the connection is?

•So the problem is that most TCP apps repeatedly halt data transmission due to mini-sized TCP transmit buffers, and they don’t provide a way for the user to increase the buffer size, and even if they did, the user doesn’t know what size to use

National Center for Atmospheric ResearchNational Center for Atmospheric Research 77NCARNCAR

A SolutionA Solution

National Center for Atmospheric ResearchNational Center for Atmospheric Research 88NCARNCAR

Solution ExplanationSolution Explanation

•Have the OS/kernel transparently, automatically, and optimally set the TCP transmit buffer size (“autotuning”) for the bandwidth and latency conditions detected at all times during every TCP connection based on congestion feedback information available from the TCP stack

•This is easier said than done, and it is what Web100 is all about

National Center for Atmospheric ResearchNational Center for Atmospheric Research 99NCARNCAR

What Web100 IsWhat Web100 Is•Do “autotuning” via the following method:

• Implement per-connection TCP MIB in kernel

•Similar to UNIX netstat information, but per-TCP-connection and more useful information

•Write-variables will allow user-level TCP-connection performance tuning based on real-time congestion feedback from TCP connection

•Also allows multitude of user-level display and diagnostic tools regarding TCP behavior during real-time

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1010NCARNCAR

The PrototypeThe Prototype

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1111NCARNCAR

Implementation InformationImplementation Information

•~1,200 diff lines against Linux 2.2.14

•API is through /proc pseudo-file system

•About 80 defined variables right now

•All counters are cumulative (soon to include statistical measures as well)

•Counters updated continuously in kernel; /proc updates each time accessed

•One instance of data structure for each TCP session in /proc

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1212NCARNCAR

The Coding TeamThe Coding Team

•Matt Mathis at PSC is principal designer

•John Heffner at PSC is currently the main coder

•Andy Adams at PSC is working on prototype autotuning daemon

•John Estabrook at NCSA is working on prototype GUI applications

• (Jeff Semke formerly at PSC was lead software engineer and programmer)

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1313NCARNCAR

128.182.61.238.22 <-> 128.182.61.156.1022 ESTABLISHED--------------------------+-------------------------+---------------------------PktsIn 1974 | PktsOut 1951 | Enabled:DataPktsIn 972 | DataPktsOut 1002 | SACK NAckPktsIn 1975 | AckPktsOut 949 | ECN NDataBytesIn 19823 | DataBytesOut 74651 | Timestamps NDupAcksIn 0 | PktsRetran 0 | | BytesRetran 0 |--------------------------+-------------------------+---------------------------loss episodes 0 | cwnd 1453792 | winscale rcvd 0timeouts 0 | max cwnd 1453792 | rwin rcvd 986816TO after FR 0 | ssthresh 0 | max rwin rcvd 986880 | min ssthresh 0 | winscale sent 0 | max ssthresh 0 | rwin sent 32120 | | max rwin sent 32120---------------------+----+-----------------+-------+-------+-------------------rto (ms) 20 | rtt (ms) 1 | mss 1448 | Ratemin rto (ms) 20 | min rtt (ms) 0 | min mss 1448 | Out (kbps) 0.1max rto (ms) 20 | max rtt (ms) 1 | max mss 1448 | In (kbps) 0.0---------------------+----------------------+---------------+-------------------Overall rate-controlling effects (only valid if we are the sender): aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Receiver:(S)topped,(A)pp,(B)ufsize / Path:(C)ongestion / Sender:(b)ufsize,(a)pp

Sample Demo OutputSample Demo Output

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1414NCARNCAR

Demo Code ReleaseDemo Code Release

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1515NCARNCAR

TimetableTimetable•Alpha release to made today (Halloween)!*Very limited set of handpicked testers

* To test install procedures

* To test API and API documentation

* To provide bug and feature feedback

•Additional alpha testers are possible, but must provide a test plan

•Beta release date unknown*Code is in an extremely early state

* Project just received funding from NSF on 9/15

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1616NCARNCAR

Web100 Project ElementsWeb100 Project Elements

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1717NCARNCAR

SupportSupport

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1818NCARNCAR

Develop Support ProceduresDevelop Support Procedures

•Distribution of source, binary, etc.

•Feedback from users to developers

•Website development

•Answer questions, etc,

•Much of this is being done by NCSA

National Center for Atmospheric ResearchNational Center for Atmospheric Research 1919NCARNCAR

Continued DevelopmentContinued Development

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2020NCARNCAR

Code ImprovementsCode Improvements

•Continuous kernel, MIB, and feature improvements based on user feedback

•What we have now is a concept demonstration just to start the feedback process

•Expect several iterations

•Projected work includes OS-independent library-based API, autotuning the receive buffer, 64-bit counters, statistical variables

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2121NCARNCAR

AutotuningAutotuning

•Use the writeable variables to develop user-level dynamic optimal maximum TCP buffer-size algorithms based on TCP congestion-feedback variables

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2222NCARNCAR

Tool DevelopmentTool Development

•Assist tool developers, collect, package, and distribute tools

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2323NCARNCAR

TCP-MIB DevelopmentTCP-MIB Development

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2424NCARNCAR

TCP-MIBTCP-MIB

•Define, publish, and shepherd new TCP-MIB through the MIB standards process

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2525NCARNCAR

Vendor AcceptanceVendor Acceptance

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2626NCARNCAR

Work with VendorsWork with Vendors

•Extremely important commercial OS vendors pick this code up as quickly as possible

• Integral part of Web100 proposal

•Will work closely with vendors to achieve this

• Includes working with Linux community to have Web100 code included in standard Linux kernel releases

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2727NCARNCAR

Linux DistributionLinux Distribution

•Work with Linux distributors to pick up Web100 tool suite

•Perhaps publish Web100/NSF Linux release based on one or more common commercial releases, perhaps including other work from other NLANR/I2 development projects

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2828NCARNCAR

URLsURLs

National Center for Atmospheric ResearchNational Center for Atmospheric Research 2929NCARNCAR

URLsURLs

•www.web100.org

National Center for Atmospheric ResearchNational Center for Atmospheric Research 3030NCARNCAR

Project StatusProject Status

National Center for Atmospheric ResearchNational Center for Atmospheric Research 3131NCARNCAR

Pending NSF FundingPending NSF Funding

•Major funding for Web100 has been provided from NSF/CISE/ANIR

•Web100 also has received $100,000 grant from Cisco Systems.

National Center for Atmospheric ResearchNational Center for Atmospheric Research 3232NCARNCAR

PartnersPartners

•National Center for Atmospheric Research

•Pittsburgh Supercomputer Center

•National Center for Supercomputing Applications

National Center for Atmospheric ResearchNational Center for Atmospheric Research 3333NCARNCAR

DiscussionDiscussionandand

Feedback!Feedback!