Distributed Systems

Introduction

2

CS 403 Distributed Systems

D.S. Theory

Peer to peer systemsCloud Computing

Sensor Networks

The Hype! Forrester in 2010 – Cloud computing will go from

$40.7 billion in 2010 to $241 billion in 2020. Gartner in 2009 - Cloud computing revenue will

soar faster than expected and will exceed $150 billion by 2013. It will represent 19% of IT spending by 2015.

IDC in 2009: “Spending on IT cloud services will triple in the next 5 years, reaching $42 billion.”

Companies and even Federal/state governments using cloud computing now: fbo.gov

Many Cloud Providers• AWS: Amazon Web Services

– EC2: Elastic Compute Cloud– S3: Simple Storage Service– EBS: Elastic Block Storage

• Microsoft Azure• Google Cloud• Google Compute Engine• Rightscale, Salesforce, EMC, Gigaspaces,

10gen, Datastax, Oracle, VMWare, Yahoo, Cloudera

• And many many more!

Two Categories of Clouds Can be either a (i) public cloud, or (ii) private

cloud Private clouds are accessible only to company

employees Public clouds provide service to any paying

customer: Amazon S3 (Simple Storage Service): store arbitrary datasets, pay

per GB-month stored Amazon EC2 (Elastic Compute Cloud): upload and run arbitrary

OS images, pay per CPU hour used Google AppEngine/Compute Engine: develop applications within

their appengine framework, upload data that will be imported into their format, and run

Reference Texts

Andrew S. Tanenbaum and Marten van Steen: “Distributed Systems: Principles and

Paradigms” Prentice-Hall

George Coulouris, Jean Dollimore and Tim Kindberg: “Distributed Systems: Concepts and

Design” Addison-Wesley

7

Organization of the course

1 Characterization of DS 2 System Models 3 Networking and Internetworking 4 Interprocess Communication

Foundations

10 Time and Global States11 Coordination and Agreement

Distributed algorithms

6 Operating System Support 8 Distributed File Systems15 Distributed Multimedia Systems16 Distributed Shared Memory18 Mach Case Study

System infrastructure

Middleware

5 Distributed Objects and Remote Invocation 7 Security 9 Name Services17 CORBA Case Study

12 Transactions and Concurrency Control13 Distributed Transactions14 Replication

Shared data

*

8

Can you name some examples of Operating Systems?

9

Can you name some examples of Operating Systems?

…Linux WinXP Vista Unix FreeBSD Mac OSX2K Aegis Scout Hydra Mach SPINOS/2 Express Flux Hope SpringAntaresOS EOS LOS SQOS LittleOS TINOSPalmOS WinCE TinyOS…

10

What is an Operating System?

User interface to hardware (device driver)

Provides abstractions (processes, file system)

Resource manager (scheduler) Means of communication (networking) …

11

Can you name some examples of Distributed Systems?

12

Distributed Systems Examples

Client-server (e.g., NFS) The Internet The Web A sensor network DNS BitTorrent (peer to peer overlays) Datacenters

13

What is a Distributed System?

Definition of a Distributed System

A distributed system is a collection of independent computers that appears to its users as a single system.

A definition (Coulouris, et al)

System of networked computers that

communicate and coordinate their actions only by passing messages

concurrent execution of programs components fail independently of

one another

A definition (Lamport)

“You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done.” inter-dependencies shared state

Definition of a Distributed System

A distributed system organized as middleware.Note that the middleware layer extends over multiple machines.

1.1

Distributed System

Distributed Systems 19

Next Generation Information Infrastructure

Visualization

BattlePlanning

Visualization

Collaborative Multimedia

(Telemedicine)

Collaborative Task Clients

Server farms

BattlePlanning

ElectronicCommerce Distance Learning

Requirements - Availability, Reliability, Quality-of-Service, Cost-effectiveness, Security

DeviceNets &

SensorNets

Wide Area Network (Internet)

Distributed Systems 20

Mobile & ubiquitous distributed systems

Distributed System Example – A Datacenter

Servers

Front Back

In Some highly secure (e.g., financial info)

Power

Off-site

On-site•WUE = Annual Water Usage / IT Equipment Energy (L/kWh) – low is good•PUE = Total facility Power / IT Equipment Power – low is good

(e.g., Google~1.11)

Cooling

Air sucked in from top (also, Bugzappers) Water purified

Water sprayed into air 15 motors per server bank

25

List of Topics we’ve Covered

Distributed system Basics Lamport timestamps File system Name space Failure detectors Replication Clouds and their predecessors (e.g., Grids and

timesharing industry) Sensor networks

Operating System types Centralized Systems

Process management Network System

Share resources Remote access Telnet / FTP No direct control from machine to another

Distributed system Global view of files system Global name Global time….

Example of Distributed Systems Network of workstations in university

It has a single file system with all files accessible from all machines

Workflow information system (automatic order) Orders are placed by means of laptop that are

connected to the system through telephone network

World Wide Web To publish a document by give it unique URL

Distributed Application Examples

Automated banking systems

Tracking roaming cellular phones

Air-traffic control

Example:Embedded Systems Automotive control systems

Mercedes S class carsthese days are equippedwith 50+ autonomous embedded processors

Connected throughproprietary bus-like LANs

Consumer electronics Audio HiFi equipment

Sensor Networks

Organizing a sensor network database, while storing and processing data (a) only at the operator’s site or …

Sensor Networks

Organizing a sensor network database, while storing and processing data … or (b) only at the sensors.

)A (Plants and Animals interacting in the Food Chain

Which is a Distributed System – (A) or (B)?

– Which is a Distributed System(A) or(B)?

(A)

)B (The Internet (Internet Mapping Project, color coded by ISPs)

(B)

34

What is a Cloud? It’s a cluster! It’s a supercomputer!

It’s a datastore! It’s superman!

None of the above All of the above

Cloud = Lots of storage + compute cycles nearby

35

A Sample Cloud Topology

Top of the Rack Switch

Core Switch

Servers

Rack

If higher bandwidth link ,then a “fat tree” topology

36

Timesharing Industry (1975):•Market Share: Honeywell 34%, IBM 15% ,•Xerox 10%, CDC 10%, DEC 10%, UNIVAC 10%•Honeywell 6000 & 635, IBM 370/168 ,

Xerox 940 & Sigma 9, DEC PDP-10, UNIVAC 1108

Grids (1980s-2000s):•GriPhyN (1970s-80s)•Open Science Grid and Lambda Rail (2000s)•Globus & other standards (1990s-2000s)

First large datacenters: ENIAC, ORDVAC, ILLIACMany used vacuum tubes and mechanical relays

P2P Systems (90s-00s)•Many Millions of users•Many GB per day

Data Processing Industry - 1968 :$70 M. 1978: $3.15 Billion.

Berkeley NOW ProjectSupercomputers

Server Farms (e.g., Oceano)

“A Cloudy History of Time” © IG 2010

Clouds

37

Course Projects

1. Multiparty Democracy and distributed systems2. Smartphones for healthy living3. Mobile ad-hoc live streaming4. Social network5. Datacenter topology

6. New transport protocol7. Graph analysis8. Energy-efficient datacenters9. DNS10. VM scheduler11. Instant Messaging12. New cloud services13. Pricing for clouds14. Health information systems