Chapter 9Multiprocessor

Virtualization (1)

Li Zhenmin

2005.11.14

Multiple Processors

Used in servers and high-end desktop systems

Have large amounts of memory and I/O devices

Web servers : manage huge databases and service requests simultaneously

Computational servers : used for large scientific calculations with TBs of memory and PBs of disk capacity

How to utilize multiprocessor systems efficiently

Mismatch problem:

Reason limitations in the parallelism available in the programs Limitations in the scalability of applications due to the

overhead of communication between processors

Actual numbers of processors Ideal number of

Processors really needed

How to utilize multiprocessor systems efficiently

Solution: make effort in two direction

Parallelism discovery(Hardware, Software)

Parallel programming(OpenMP, MPI)

Multiprocessor systems partitioning

Application vision

system vision

Partitioning of multiprocessor systems

The appearance of a system that may or may not reflect the exact configuration of the underlying physical system

Two dimension of partitioning In time domain (discussed in Ch.8) In space domain Sometimes combined together

Taxonomy of multiprocessor systems

Cluster system Consists of a number of small systems

communicating with each other through high-speed networks interfaces

Shared-memory processing platform Provides a large number of processors that

communicate through shared memory

Taxonomy of multiprocessor systems according to the communication method

Multiprocessorsystems

Shared Memory(communication

Through shared variables)

Interconnection(message passing)

Sharedregisters

SharedMain memory

Direct 1 to 1connection

network

Static network

Cluster computer

SMP

VLIW superscalar

scalability

Low Communication

cost

Basic structure of centralized shared-memory multiprocessor

Need snoopy bus based architecture Need synchronization methods

Cluster interconnection A network of workstation (NOWs) is very attractive a

lternative to the expensive supercomputers and parallel computer systems for high performance computing

Very high performance workstations and PCs readily available at low cost and the latest processors can easily be incorporated

mesh

Our discussion’s focus

The construction of virtual clustered multiprocessor systems on a host SMP platform

Each cluster is an SMP system with small number of processor

Provides an illusion of several virtual shared-memory systems operating simultaneously on a single-memory host system

The partitioning of large shared-memory system

P P P P P P P P P

Virtual Machine Monitor

Memory I/O

Memory Memory MemoryI/O I/O I/O

P P P P P PP P

Virtual Machine 1 Virtual Machine 2 Virtual Machine 3

Shared-memory Multiprocessing Hardware

Motivation : Workload consolidation Nowadays, shared-memory multiprocessors are widely

used in large database servers which are very expensive

Computation workload is distributed in a three-tier model

The three-tier model Database server Application server

Large numbers of workstations of PCs feeding requests to a first level of servers

Make accesses to database servers User PC

Typical three-tier model

Database server

Application server

Application server

Application server

User PC User PC User PC User PC User PC User PC

Consolidation of the application server with database server

User PC User PC

User PC User PC

User PC User PC

Database server

Virtualized App. Server

Virtualized App. Server

Virtualized App. Server

Motivation : Workload consolidation

The consolidation of application servers on a large database servers

The consolidation of multiple workstation users on a large remote server Need to address the problems of privacy

protection and flexibility requirement Virtualization of large servers through partitioning

of physical resources

Motivation : Cluster-based programming model

The clustered applications have been emerged recently

Desire to run cluster-based applications on high-end shared-memory systems

For example The applications written for a cluster system using

Message Passing Interface (MPI) can run on a virtualized shared-memory system

The efficiency can be improved by tuning the message-passing libraries to exploit the shared-memory hardware mechanisms

Motivation:System migration

The process of migration is painful Migration to a new system (OS, middleware) Replace old applications with new one

The partitioned system can be beneficial Allow the testing and verification in separate

partitions The entire system will not be crashed by the

problems discovered in a partition Will speedup the process of migration

Motivation:Reduction of system downtime

The upgrade and installation usually require system shut-down

Partitioned system can perform system downtime in a separate partition without affecting other partitions

An upgraded OS can be completely checked out before the operation to the rest partitions

Motivation:Heterogeneous systems

The ability to run heterogeneous OS in different partitions

Each OS has an illusion of running on a server

Motivation: Improving system utilization

The system can be configured according to the changing workload Configured as a single system image of the whole system

for peak workload Configured as multiple partitions running separate OS for

average workload Capacity planning help to determine the partition

scheme and keep the compute requirement closer to the average workload

Dynamic migration of resources form one partition to another help improving the utilization of resources

Motivation:Multiple time-zone requirements

It is common for some international company to have geographically distributed parts

It is needed to bring down the system for maintenance or upgrade at a convenient local time

Partition system can make decisions independent of the other regions

Motivation:Failure isolation

Most important feature of partitioning Today’s systems are vulnerable to

attacks over the network Malfunctions of software Hardware failures

Partitioning helps isolate the effects of failure Particularly, hardware faults may or may not be local

to a partition Depend on the nature of partitioning Not local for time multiplexed partitioning scheme Local for physically partitioning scheme

Motivation:Failure isolation

The reliability of the partitioned system also depends on the reliability of VMM

VMM is either in software of in microcode, and is assisted by special hardware

Contemporary software VMMs are quite small

The reliability is guaranteed by both software and hardware parts

Mechanisms to support partitioning

VMM is the key part to perform partitioning and can be implemented in different ways Completely hardware Microcode supported by hardware Software supported by hardware

Make hardware modifications to support virtualization ― VMM can operate in a new privilege mode Advantage: avoid the need to run a guest OS in user mode

which will cause performance degradation Disadvantage: enable to virtualize in a recursive manner

more flexible

Types of partitioning techniques

The partitioning spectrum (for a n processors system) Cluster of n nodes Single n-way shared-memory system

The types of partitioning are divided by the implementation of VMM Those use special hardware for the VMM Those don’t

Types of partitioning techniques

With hardwaresupport

Without hardwaresupport

Physical Partitioning

LogicalPartitioning

SVM-Based Approaches

OS-BasedApproaches

MicroprogramBased

HypervisorBased

Same ISA Different ISA

Partitioning Techniques

Techniques with hardware support

Physical partitioning Each OS image uses resources that are physically distinct f

rom the resources used by other OS images

Logical partitioning Can share some of the physical resources in a time-multipl

exed manner More flexible but need additional mechanisms Early implemented in microcode (firmware) Now often implemented as a codesigned firmware-software

layer (hypervisor) Provide an interface to system programs for configuration

Techniques without hardware support

System VM-based approach Technology form System VM Extended to create multiple virtual shared-memory systems Impose overhead Flexible and the only way to achieve different ISA

implementation

OS-based approach Indeed can partition hardware resources among the

processes Is not really virtual machine and lose many desirable features

Physical Partitioning

Simplest and easiest to implement Impose little overhead Control of the configuration of each partition

is mostly in hardware

Central Control Unit

Console of system Administrator

Hardware resources

receive command

Send command

Physical partitioning

The OS is loaded right after a partition is configured

The bootstrapping functions are performed during loading procedure

Completely independent from other partitions

Physical partitioning of a 24-processor system

P P

P P

M

I/O

Disk Disk

P P

P P

M

I/O

Disk Disk

P P

P P

M

I/O

Disk Disk

P P

P P

M

I/O

Disk Disk

P P

P P

M

I/O

Disk Disk

P P

P P

M

I/O

Disk Disk

Partition 1 Partition 2 Partition 3

Practical physical partitioning systems Sun Microsystems : Sun Enterprise 10000 Server

Partitioned into several domain Each physical unit comprises one system board, consisting of 4

Processors, 4GB of memory and 4I/O buses Don’t allow two partitions share a single system board

Hewlett-packard : HP-UX 11i Each system board is referred as a cell (4 p, 16G m, 12 PCI slot

s) Each cell of a partition must be identical to other cells in the partit

ion in terms of the number of processors and amount of memory Fujitsu : PrimePower system

Allows partitions to be smaller than one board Each board of 8 p can contain up to 4 partitions

Advantages of physical partitioning : Failure isolation

Isolation of Software failure The control unit can reset the partition and reboot the OS

for that partition without other partitions observing any effects of the failure

Isolation of Hardware failure Premise: one physical entity (e.g. board) is associated with

only one partition

Can not eliminate single points of failure (catastrophic failure) control unit failure (low possibility) The crossbar switch connecting different boards

Advantages of physical partitioning : Better security isolation

Each partition is protected from the denial-of-service attacks by other partitions

System administrator of one partition can not take unauthorized action in other partition

Advantages of physical partitioning : Better ability to meet system-level objectives

System-level objectives: Result form contracts between owners and users

Physical partitioning create partitions more similar to hardware systems

Techniques used to direct resources to applications in stand-alone systems can be applied more readily and predictably in physical partitioning

Limitations of physical partitioning

Can not achieve better utilization of resources

Usually each of the physical partitions is underutilized

Dynamic workload balancing is difficult because of the physical constraints

Less flexibility in allocating resources to partitions