RDMA ENABLED WEB SERVER

Rajat Sharma

Objective

To implement a Web Server serving HTTP client requests through RDMA replacing the traditional TCP/IP data transfer model.

RDMA reduces the CPU overhead at the server side leading to high performance data transfer , Computing and increased Scalabilityin terms of Clients

Known Implementation

Researchers from Ohio Supercomputing Center have implemented what is known as

“ RDMA enabled Apache ”having a special hardware RDMA NIC card as 10 Gbit iWarp (RDMA over TCP) network card.

Summarized the details into a paper

“Accelerating Web Protocols Using RDMA”

“Dennis Dalessandro and Pete WyckoffOhio Supercomputing Center”

Topics of Discussion :

AbstractLot of applications challenge the capacities of server. Increased capacity is achieved through module mod_rdma with unmodified Apache code

Novel Technology

• RDMA vastly improves Performance and Scalability for clients

• Information sent over the network as TCP/IP packets includes multiple copies of the same data resulting in large CPU overhead limiting the server capacity to serve more clients

• TCP cannot serve the need of Supercomputing

Examples : RDMA real life need

• Large data centers : Medical Research Institutes - Access of remote data bases for Researchers -> High Throughput

• Grid Chem Client: Monitoring of Parallel calculations through CGI scripts need large files compressing , sensing or receiving consumes a lot of processing power

• Todays client Common to have 1 MB web content .

• HPC will need high data transfers for dynamic content

Online gaming servers and Streaming Servers will not function well without efficient high speed data transfer

Performance Bottlenecks

• Most of the performance bottlenecks are often found at the server side

• Several measures to remove themRound Robin DNSActive monitoring for load balancing

But data transfer is a biggest limitation -> Nature Of HTTP over TCP

Contd..

Protocol acceptance : Tough Enabling clients with Iwarp software ->No CPU improvement

Why APACHE ??

• Most popular web server• Wide acceptance• Allow third party modules to affect the server

behavior • “Hooks” feature -> used by the Module to

access the web server

Need for RDMA

• Traditional TCP Packet processing• Multiple Copies of data

Direct Data transfer

• Allows adapters to move data from Memory to Memory without CPU

• Advantages– Prevents system call overhead– Hardware interrupts– Latency– Data movement

RDMA and IWARP

RDMA enabled hardware

• RDMA Needs special hardware to be deployed• Knows as RNIC• Main features – Zero Copy

– OS Bypass

IWARP software

• RNICS cards are expensive• NOT viable to have an RNIC card on every client• RNIC enabled Server -----------------

Normal NIC enabled client

HOW ?Solution : IWARP software Protocol

OR IWARP emulator

Efficient Server • RDMA software – not as efficient in Client side

-- No improvement in CPU overhead on client side since no hardware.

RNIC RDMA RDMA Iwarp hardware software

SERVER CLIENT

Soft Iwarp implementations

• User space implementation can be used by applications in user space linking

to library to generate RDMA packets

• Kernel space implementation RDMA calls in kernel space

Soft IWARP : -- Web browser plug in module ..

Design• Web server decision :

other than Apache , all servers lack the third party interaction freedom• Backward Compatibilty

Header Field

RDMA header

RNIC Port Number

Stag

Toffset

Max Len

RDMA data transfer Schemes• GET request and POST request

GET REQUEST Server Writes and Client Reads

Server writes ->

Server ----------------------------------- ClientRDMA write

Client Reads Client ------------------------------------ Server

RDMA read

GET: Server-writes

GET : Client reads

POST Request

• POST requests work in the same way

: Writing data to the server

– Post Request will have

• Client – Writes

• Server – Reads

RDMA connection• Server should be ready to start before the client initiates• Client ---------------------------- Server

– RTT TimeCase Server –Writes

If client GET request Max_ buffer_length < Resource_LenServer ERROR mess

Case client – reads

Max_buffer_len < Resource _client_needsClient Error: Stop RDMA GET request

Implementation

• RDMA mod_rdma module interacts with Apache Server using “hook”

• Child Initrdma_child_init( )

• Pre – connection rdma_pre_connection_handler( )

• Connection Cleanuprdma_connection_cleanup( )

Contd..

• Insert Filterrdma_insert_filter( )

Output Filterrdma_output_filter( )

Ship_file( )

Ship_heap( )

Limitations• RDMA connection cost

• Memory Registration

– Static Registration– Pre –registering size of memory.Mmap () and then register the memory in case of file from disk How ???

Difference b/w memory allocated and memory registered?

– Dynamic Registration

Placing the dynamic content requires memcopy every time to copy the data to RDMA buffer, thus mod_rdma causes one more memory copy

Experimental Results

• Test our web server using 2 processes value in the background , simulating it as an heavily

loaded CPU by using nice function.

15 nodesMain Memory = 2GB10Gbit Iwarp adapterResults shows some hidden costs in terms of

memory

Hidden Costs

Single Client Performance

Highly loaded CPU costs more memory registration as mem_reg = CPU task

Multiple client Performance

Alternate Approach

• Inclusion of Proxy server to implement a Web Server.

Proposed areas in future work

• More requests like PUT • Kernal based implementation of the Web

Server.– Resulting in change of path of data from kernel to

NIC card– Fast implementation