Comprehensive View of a Live Network Coding P2P System

Presented by:

Kenny, W. K. CheungC. M. Fung

Outline of presentation1. Introduction to the paper

2. What is Network Coding?

3. Prototype implementation details

4. Paper Result

5. Topology Construction

6. Connectivity

7. Security

8. Conclusion

Paper Objectives

• Present the first implementation of a P2P content distribution system that uses Network Coding

• Show that Network Coding is practical in a P2P setting (little CPU processing, I/O activity)

Paper Achievement• Present the performance of a live P2P filecasting system from a nov

el set of angles. – E.g. quantity content provider’s saving overtime, the dynamics of topology, the number of un

reachable nodes at any point the time, the overall system efficiency

• Present workers’ field experiences with implementing and using Network Coding. Show that Network Coding is feasible.

• Evaluated various P2P topology construction algorithms.

• Study the influence of unreachable nodes in the system’s efficiency.

• Study the performance of a novel set of security functions to secure Network Coding system. – Secure Random Checksums

Network Coding

• A method of attaining maximum information flow in a network.

• Core notion:– Mixing of data at intermediate network nodes.– Receiver deduces from data packets the

messages that were originally intended for it.

• For each of the k clients, server encode the packets follow the relation

• Mi are packets to be sent originally• Gk , Xk are sent• each encoded message is a linear combination of the file• Linear problem: X = GM• When enough encoded messages are received, client ca

n find M by using Gaussian elimination

A high level explanation on Network Coding

A high level explanation on Network Codingexample (2 packets)

Server

N1

N2

a1x1+b1x2=m1

a2x1+b2x2=m2

a1x1+b1x2=m1

a2x1+b2x2=m2

a2x1+b2x2=m2

2 packets: x1, x2

Server sent:

(a1,b1) and m1 to N1

(a2,b2) and m2 to N2

Network Coding example (attaining maximum information flow)

• Two sources• Two destination nodes• Each edge can carry

only a single value.• No routing scheme can

transmit both A and B simultaneously to both destinations.

Network Coding in this paper

• Use of Network Coding for P2P networks

• Benefits:1. The capacity of the seed server is fully

utilized by constantly serving innovative information

2. Innovative information in a node propagates in optimal time to those nodes needing it, regardless the number of hops between the source and the sink

Prototype Implementation

• Implemented in C#

• Three type of participant:– One or more peers– A registrar– A logger

Prototype Implementation

• Peers:– Sources and sinks for content data– Server: a special peer that contains seeded

content– Seeds: peers that finish the download, but

remain in the system– Each peer maintains 4-8 connections to other

peers. ( one for each direction)– Encodes, decodes, validates, and persists

content data.

Prototype Implementation

• Registrar: enables peer discovery

• Logger: an aggregation point for peer and registrar trace messages

Evaluating Swarming Systems

• Evaluating swarming systems under the right context is critical

• Key issues:– Swarm efficiency with scale– Partial observation are not enough– Impact of arrival patterns

Data Summary

Server upload capacity: 2.5Mbps, also served as registrar and logger.

Individual Peers

• Trial participants were diverse in terms of geographical location, access capacity and access type (e.g. corporate links, DSL/cable home users, wireless links)

Individual Peers

Summary of participating users (Trial-1)

Observation:

Most users had download speed between 550 Kbps and 1.6Mbps

non-NATed node get a much higher throughput than the NATed one.

Individual Peers

Number of upload and download neighbors for each peer in Trial 1

Neighbors:6 to 8 for downloads

4 to 6 for uploads

Individual PeersObservation:Most peers had multiple sessions (some up to 14)

Number of upload and download neighbors for each peer in Trial 1.

System Rates• Overall system throughput = the aggregate download rate.

System Rates

• The download efficiency of a user is the ratio of its average download rate over the maximum, ideally =1

Observation:The efficiency is similar for all groups, implying that nodes used the available resources efficiently even during the early stages of the trial.

Content Provider Savings• The Content Provider Savings are proportional to the ratio of the aggregate download

rate over the upload rate contributed by the server;

fair share ratio:divide server’s upload rate by the rate that the server would have to contribute if all nodes were uploading to their maximum capacity

Content Provider Savings

Peer’s Performance

• Study the performance of a typical peer

• 2.2Mbps downlink, 300Kbps uplink capacity

Time on each activity (for 5 random peers and average)

Resource Consumption

• One typical peer

• Pentium IV @2GHz and 512MB RAM

• CPU usage and disk usage plot

• using interactive applications – e.g. word editing and WWW browsing– no observable decrease in responsiveness

Resource consumption on a typical machine during Trial-4. Left: CPU Activity. Right: Disk Activity.

Server Efficiency

• Network coding ensures optimal resource utilization since every block generated by the server can be used (with high probability) in place of any other block to reconstruct the original.

• encoding entire file impractical due to high encoding and decoding costs.

Server Efficiency

• Solution: Group coding– divide the blocks of the file into groups and we perform

coding only inside the groups.

• New problem arise:– some segments may become unpopular

(like the optimal scheduling of the block propagation)

• Writer’s argument:– segment scheduling is a much easier problem than block

scheduling and penalty of group coding is small

Server Efficiency

• server efficiency: – the number of useful (unique) blocks that are

served by the server in time sufficient to serve one full copy of the file (under optimal scheduling).

• network coding (entire file) : 1.0

• ideal block propagation scheme : 1.0

Server Efficiency• 5 runs per instance• For group coding, scheduling decisions across different s

egments is random

Server Efficiency

• Group coding v.s. Local rarest heuristic– Group coding performance do not varies on s

cale– Local rarest performance depends on number

of neighbors (estimation of the state of the network)

Download Progress

• current peer-to-peer systems have slow performance in:– Beginning (do not have anything to offer (or w

hatever they have is already popular in the system)

– toward the end (cannot find the last missing blocks)

– U-shape in performance

Download Progress• Network coding don’t have this problem

• each encoded block is unique and useful to any node

Topology Construction

• Trial 1,2 :– Each new node picks a random subset of

nodes out of those in the system (registered at the tracker) and connect to them.

– The set of uploading nodes could be different than the set of downloading nodes.

– The maximum number of download connections > The maximum number of upload connections.

Topology Construction• Second eigenvalue : A standard metric measures global connectivity properties.

Topologies with good connectivity have second eigenvalue bounded away from 1.

Topology Construction

• Trial 3 , 4:– The topology construction algorithm periodically

dropped one of its peer at random

– Consideration: dropping frequency (drop one peer after five block are uploaded), selection policies for dropping (drop slow peers)

– Motivation: to force rewiring of connections, hence, construct overlay topologies with better properties.

Topology Construction

• Problem of Trial 3: A large number of Too Busy (cluster) (Dropped peers try to contact other peers in other parts of the network, get continuously rejected since other nodes have their connection budget used)

• Trial 4: to avoid Too Busy problem, allowed a given peer’s peer-set to grow temporarily by some threshold (e.g. max-peer-set +2), creating an “elastic” maximum.

Connectivity

• Un-reachable peers: – Peers behind NATs and firewalls, cannot recei

ve inbound connections. (Exclude peers behind NATs and firewalls configured to allow incoming connections.)

– Can’t exchange content with each other, and, hence, can’t take advantage of the network capacity that exists between them.

Connectivity

• Calculate the optimal throughput of the system assuming all nodes are reachable:

• Calculate the optimal throughput of the system taking into consideration the set of unreachable peers:

• The optimal throughput at time t is computed as the sum of the peak upload rate of all active peers at time t.

Connectivity

The effect of unreachable peers on the overall performance for two trials (left: Trial-1, right: Trial-4). The top curves are computed assuming all nodes are reachable; the lower curves take into consideration the set of unreachable peers over time.

SECURITY

• network coding each intermediate node produces “new” blocks

• Server can’t digitally sign each block

• Secure Random Checksums (SRCs)

Secure Random Checksums (SRCs)

• For each of the clients, server generates:

• vector r = [r1 . . . rm] in Zq

– often q = 16 digits– m : number of symbols per block (=bytes/block)– random elements

• SRC of block i :

• Specific SRC and the seed of the random element vector is sent to the client through secure channel

Secure Random Checksums (SRCs)

SRC of block i : encoded block:

The above equalities should hold if the encode packet received is valid. So ,the client calculates the both sides of the above equations and check if it holds to verify the arriving packet.

Performance of SRCs

• Fast : SRCs are linear operations• On 3.0 GHz Pentium 4 with 1GB of RAM• 2 GB file: 1 sec/SRC

– Done before download commerce– ~ time for once disk read of file

• Client SRC verification : ~1.6 Gbps,

Conclusion• network coding overhead is relatively small, both

in terms of CPU processing and I/O activity. • verification of encoded blocks using SRCs is ver

y efficient.• high utilization of the system resources and larg

e savings for the content provider even during flash-crowd events

• smooth file download progress• Replace fancy block selection algorithms

Our point of view

Network coding in p2p is very applicable from theexperimental results.

With further researches, it can be even more powerful:

• peer-matching algorithms provide much further improvement space

• On-the-fly decoding or parallelisms reduce overhead on decoding

Q & A

Thank you !