Performance evaluation methods for P2P overlays

Performance Evaluation Methods

for

P2P Overlays

Simula Research Laboratory, May 18th, 2009

Knut-Helge Vik Simula Research Laboratory and University of Oslo.

Trial Lecture, Knut-Helge Vik University of Oslo

Outline Introduction

Peer-to-peer (P2P) networks

Peer-to-peer overlays

Performance evaluation

Performance measures of a computer system

Performance measures of a computer network

Performance evaluation methods for P2P overlays

Simulation model

Analytical model

Measurements

Tradeoffs in performance evaluation

Conclusions and summaries


*Stefan Schmid @ HiPC 2007

Many types of networks*…

DISTRIBUTED COMPUTING Internet Graph Web Graph Neuron Networks

Social Graphs Public Transportation Networks

Different properties: Natural vs. Man-made Robustness End-to-end latency Routability ...

Peer to peer networks


Peer-to-peer networks: Peer: Somebody or something who/that is at an equal level

Peer-to-peer system is a network of clients that share resources – Bandwidth!

− File sharing, telephone, video-conference, etc.

Dozens of peer-to-peer applications, many are used to share copyrighted data-files

Peer-to-peer networks differ from client-server networks

− In peer-to-peer networks clients are communicating directly with each other

− Peer-to-peer networks are distributed around the world!

= Router

= Client

= Internet

Peer-to-peer Client-server

= Client = Server


Peer-to-peer overlay networks

An overlay network is a logical network built on top of a physical network

Overlay links are tunnels through the underlying network

Many logical networks may coexist at once over the same underlying network and providing its own particular service

Nodes are often end hosts acting as intermediate nodes that forward traffic, providing a service, such as access to files

- P2P overlays are built on top of the Internet topology

Trial Lecture, Knut-Helge Vik University of Oslo 6


Trial Lecture, Knut-Helge Vik University of Oslo 7


Focus at the application level



Unstructured

− Without prior knowledge of the topology

− Peers are organized according to optimization metrics (if any)

• Freenet

• Gnutella

• FastTrack/KaZaA

• BitTorrent

• Overnet/eDonkey

Structured

− Topology is tightly controlled

− DHT (distributed hash table) enables efficient lookup

• CAN

• Chord

• Tapestry

• Pastry

• Kademlia

• Viceroy

hybrid









Simulation model

Analytical model

Measurements




Performance evaluation in computer systems

Performance: Manner or quality of functioning, fulfillment of a claim, promise, request

Evaluation: Systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards.

Scientific method: Describes the process going from observation to validated theory

System: a collection of entities (e.g., people and machines) that interact over time

Performance evaluation is the application of scientific methods to the study of computer systems [Bob Kinicki].

− We apply scientific methods when we do performance evaluation of computer systems

The goal of performance evaluation is to determine the effectiveness and fairness of a computer system that is assumed to work correctly [Bob Kinicki].

− Find out how well the computer system works based on certain criteria

The goal of computer systems engineers, scientists, analysts, and users is to get the highest performance for a given cost [Raj Jain].

− Best performance, but make it cheap



Metric: a descriptor used to represent some aspect of a computer network’s performance.

Need to measure basic characteristics of a computer network:

− Count how many times an event occurs

− Duration of some time interval

− Size of a parameter, etc.

These characteristics are called performance metrics

Focus is on system-oriented performance metrics – latency, bandwidth, etc.

− Quality of algorithms, protocols, methods

Add up metric values and calculate the average, mean, etc

− Make metrics understandable!

Metric statistics are used to visualize results


Dos and don’ts of performance evaluation

Performance evaluation is often driven by the fact that we want the best!

Unclear goal? No model can solve everything.

Are you biased? Be fair when comparing systems.

Be systematic when comparing.

Understand the problem at hand.

Use the right performance metrics.

Use an appropriate level of detail in the evaluation.

Present results clearly.

State your assumptions.







Performance metrics of a computer network


Simulation model

Analytical model

Measurements




Performance evaluation methods

In general, peer-to-peer network evaluation

makes use of the actual network, an emulated

network or a model of the network.

Model-based techniques

− Accessible system does not exist or is too large (complex, unhandy, not available, etc.)

− Instead, develop a model that mimics a system

Measurement-based techniques

− Accessible system exists

− Run system and evaluate its performance

− PlanetLab is a distributed network of computers used to conduct measurement studies


Model-based performance evaluation

Model-based techniques:

− Simulation Modeling

− Analytical Modeling

− Both modeling techniques tend to rely on queuing theory.

Model-based techniques rely on simplifying assumptions that enable the model to capture important characteristics of networks

These models are usually networks of queues.

Arrivals

Queue Server


Measurement-based performance evaluation Empirical: information gained by means of observation, experience, or

experiment

Empirical measurement of real/actual networks

P2P network measurements can be either active or passive.

Passive measurement studies account for existing traffic

− For example, collect information about packets traversing a network channel.

Active measurement involves purposely adding traffic to the network specifically to facilitate the measurement

− For example, sending packet train probes into the network to estimate the available bandwidth along a flow path.

Sender

Internet

Receiver Packet train

…

Analyze packets interarrival time and calculate available bandwidth


Taxonomy of performance evaluation methods*

*Simulation, Modeling & Analysis (3/e) by Law and Kelton, 2000, p. 4, Figure 1.1


Simulation models Event: a software message indicating that something has happened

Transition: passage from one state to another

A network simulator is a program that predicts the behavior of a network, without an actual network being present.

Event-driven simulation defines a P2P network in terms of states and transitions where events trigger transitions.

There are basically three types of simulators*[Lilja]:

− Emulation

− Static/Monte-carlo simulation

− Discrete-event simulation

*Other simulator types exist, but they are outside the scope

Event loop:


Simulation models: Emulation

An emulator program is a simulation program that runs on some existing system to make that system appear to be something else.

Often used to provide a framework for testing new peer to peer systems.

Emulators can accurately evaluate how overlays behave when there is network traffic

ModelNet is a scalable wide-area network emulator:

− Deploy hundreds of instances of an application over nodes that behave as if they were distributed all over the world

− Widely used for peer-to-peer system evaluation

− De-facto standard for network emulation

− Unlike simulations, models cross traffic and packet loss

− Unlike PlanetLab, gives reproducible results

- Internet emulator nodes (FreeBSD) - Application emulator nodes (Linux)


Simulation Models: Static/Monte Carlo

A static simulation has no time parameter

− Runs until some equilibrium state reached

Used to approximate the probability of certain outcomes by running multiple trial runs, called simulations, using random variables

Driven with random number generator

− So “Monte Carlo” (after casinos) simulation

Used to model physical phenomena, evaluate probabilistic system, numerically estimate complex mathematical expression

Peer to peer overlays: Often used to evaluate and simulate structured overlays[Beaumont].

Advantages: It is a good way of evaluating whether overlays have a chance of getting to undesired states

Drawbacks: No sense of time, we don’t know if it ends

Schematic showing the principal of stochastic uncertainty propagation. (The basic principle behind Monte Carlo simulation.)

Imagine throwing dart at square

− Random x (0,1)

− Random y (0,1)

Count if inside

− sqrt(x2+y2) < 1

Compute ratio R

− in / (in + out)

Can repeat as many times as needed to get arbitrary precision


Simulation Models: Discrete-event

Simulate only at the interesting events

Often a simple way of solving a problem – simulate system and observe results

Discrete-event simulators are

− Stochastic = probabilistic

− Dynamic = changes over time

− Discrete = instantaneous events are separated by intervals of time

Time may be modeled in a variety of ways within the simulation.

− Time as linked events is most common:

Discrete-event simulators classified according to how they generate events[Lilja]*:

− Trace-driven -

− Distribution-driven –

*Other discrete-event simulators exist, but are outside the scope


Discrete-event simulator: Trace-driven Trace: is a record of the sequence of events that occurred when the

system was traced

Application events are recorded in a trace and the trace is used to drive the discrete event simulator

Trace must be independent of the tested algorithms/protocols

Advantages:

− It is easier to sell than random numbers

− Easy validation, accurate workload, fair comparison, similarity to actual implementation

Drawbacks:

− Trace represents only one possible execution path of a system

− Complexity, finiteness , single point of validation

Often used to evaluate existing peer to peer applications

− Determine how the overlays evolve during execution

− Identify inefficiencies of the overlay construction in peer to peer application

− Many peer to peer applications have a naïve (random) overlay construction


Discrete-event simulator: Distribution-driven

Input events are generated by the simulator, sequence of random numbers, etc,

− Common to apply models that generate events that mimic the intended behavior

No assurance that any real application program would produce this input sequence

− run many times with several different input sequences

Common approach for evaluating peer to peer overlays

− For example, in terms of graph theoretical metrics

− OMNet++, ns-2, GloMoSim, Ptolemy

Advantages: Control of the input events and the sequence makes it easier to implement

− Flexible, scalable, supports different event generators

Drawbacks: Harder to trust sequences of random numbers as the foundation of statistics

− Credibility, accuracy

Generic event loop

Ns-2 network


Analytical models A stochastic process is one whose behavior is non-deterministic

Describe an application and the architecture using equations

Analytical models of computer networks tend to be stochastic models built on the theory of stochastic processes associated with independent random variables.

Queuing theory is a commonly used technique

Complexity analysis of algorithms, eliminate exponential time

Advantages:

− Fast & easy analysis of relations

− Allows extrapolation to ridiculous parameters, e.g. thousands of processors

Drawbacks:

− Sometimes infeasible to obtain accuracy (e.g. modelling caches)

− To obtain reasonable accuracy, the models may become very complex (e.g. modelling of network contention)

Peer-to-peer overlays: (complexity analysis) find the theoretical bounds for algorithms


Measurement studies

Measurement Experiments are divided into two major categories:

− Live measurements (observation) − Controlled-traffic measurements


Measurement studies: Live measurement

Live empirical studies measure the performance of the computer network while it is handling real traffic.

One advantage of this type of study is that the measurement involves a real workload.

A drawback of measuring live traffic is being convinced that this measurement involves ‘typical’ traffic for this network.

Another drawback of live traffic measurement is that reproducibility of the exact same traffic workload is usually not possible.

PlanetLab is used for live measurement studies of peer to peer overlays

Issues with trustworthiness, but it is the largest real test-network around


Measurement studies: Controlled traffic Use traffic generator tools or script files to

provide repeatable, controlled traffic workloads on the network being measured.

Chosen when the goal is to evaluate the impact of different versions of a network component, strategy or algorithm on network performance.

Advantage: Controlled, repeatable traffic makes it easier to conduct cause-and-effect performance analysis.

Drawback: the accuracy of the traffic generator tool, and the ability to conduct measurement experiments where the traffic workload choices are adequately varied to provide representative, robust network performance evaluation.

PlanetLab is also used for controlled traffic


Trade-offs in performance evaluation

Trade-off: losing one quality of something in return for gaining another quality.

Analytical models

− Model includes assumptions that may be to tough

− Little or no extra equipment, other than ordinary computers

− May reach eliminating conclusions fairly fast – complexity analysis

Simulation models

− Accuracy and computational intensity

− High vs. low level of detail, often means slow vs. fast

Measurement models

− Tougher to implement, but increases the credibility of the evaluation

Performance evaluations that combine results from analytical, simulation and measurement based studies, often reach more precise and credible evaluations.

How do we compare results from different models?

A common evaluation framework makes the results directly comparable


Comparisons of performance evaluation methods

Comparison of Performance Evaluation Techniques [Raj Jain]



Peer to peer overlays can be evaluated using

− Simulation models

− Analytical models

− Measurements

A common evaluation framework makes the results directly comparable

Performance evaluations that combine results from analytical, simulation and measurement based studies, often reach more precise and credible evaluations.

Questions?


References

[Bob Kinicki] Performance Evaluation of Computer Networks, Presentation, Worcester Polytecnic Institute, 2006

[Willig04] Performance Evaluation Techniques Summer 2004, Dr.-Ing. Andreas Willig, Hasso-Plattner-Institut, Universitat Potsdam, [email protected], April 25, 2005

[Raj Jain], "The Art of Computer Systems Performance Analysis -- Techniques for Experimental Design, Measurement, Simulation and Modeling", Wiley, 1991

Averill M. Law and W. David Kelton, "Simulation Modeling and Analysis, 3rd Edition", McGraw-Hill, 2000

Jerry Banks and John S. Carson and Barry L. Nelson and David M. Nicol, "Discrete-Event System Simulation", Prentice-Hall, 2000

Reuven Y. Rubinstein and Benjamin Melamed, "Modern Simulation and Modeling", Wiley, 1998

[Beaumont] Peer to Peer Multidimensional Overlays: Approximating Complex Structures, Olivier Beaumont, Anne-Marie Kermarrec, and Étienne Rivière, 2007

[Dinh] Large scale distributed simulation of p2p networks, TTA Dinh, M Lees, G Theodoropoulos, R Minson, 2008

[ModelNet] http://issg.cs.duke.edu/modelnet.html

mailto:[email protected]

Performance evaluation methods for P2P overlays

Software

Transcript of Performance evaluation methods for P2P overlays