Post on 23-Jan-2016
description
COMP3019 Coursework: Introduction to M-gridSteve Crouchs.crouch@software.ac.uk, stc@ecs
School of Electronics and Computer Science
Objectives To equip students to drive a lightweight
grid implementation to solve a problem that can benefit from using grid technology.
To develop an understanding of the basic mechanisms used to solve such problems.
To develop a general architectural and operational understanding of typical production-level grid software.
To develop the programming skills required to drive typical services on a production-level grid.
Overview Part 1: m-grid
– m-grid: lightweight software illustrating grid concepts in use
– Develop a program with m-grid’s Java API to solve a simple problem, submit it to m-grid with input data, collect results
Part 2: Google MapReduce & GridSAM– MapReduce: framework for distributed processing of
large datasets using many computers– GridSAM: job submission web service interface to a
computational resource (e.g. compute cluster, single machine)
– Extend code stubs to submit jobs to GridSAM and monitor them to completion
– Extend pseudocode that implements a basic MapReduce framework
Where to get stuff/help?
Can obtain coursework materials from website– Ready for Wednesday
Software documentation
Coursework help lecture 19th March
Myself: s.crouch@software.ac.uk
Building 32: Level 4 lab 4067 Bay 23
Background
The Problem
Basically, want to run compute-intensive task
Don’t have enough resources to run job locally– At least, to return results within sensible
timeframe
Would like to use another, more capable resource
Distributed Computing in Olden Times• Small number of ‘fast’ computers
– Very expensive– Centralised– Used nearly all the time– Time allocations for users– Not updated often
Cray X-MP(Cray -1 successor)
Univac 1710
• Punched cards• Wait time huge• MailNet, SneakerNet,
TyperNet, etc…• Mainframes• Cray-1 1976 - $8.8
million, 160 megaflops, 8MB memory
The Present… Now… large number of slow computers:
– Cheap– Distributed
Computation Ownership
– Not used all the time– Exclusive access to users– Updated often– e.g. desktop computers, PDAs, mobile phones
Low utilisation of computing power
e.g.: institutional/university resources…
It’s About Scaling Up…
• Compute and data – you need more, you go somewhere else to get it
• Then… the march towards localisation of computation, the Personal Computer
• Computational Science develops in laboratories
• Is this changing again?
Images: nasaimages, Extra Ketchup, Google Maps, Dave Page
The Grid - a Reminder The grid – many definitions!
“Grid computing offers a model for solving massive computational problems by making use of the unused CPU cycles of large numbers of disparate, often desktop, computers treated as a virtual cluster embedded in a distributed telecommunications infrastructure” – Wikipedia
“A service for sharing computer power and data storage capacity over the Internet.“ – CERN (European Organisation for Nuclear Research)
Two components of grid computing:– Computational/data resource – e.g. computational cluster,
supercomputer, desktop machine– Infrastructure for externalising that resource to others
Some Examples… Grid (i.e. internet-accessible)
examples:– SETI@Home -
http://setiathome.ssl.berkeley.edu/ Process data from Arecibo Radio
Telescope, Puerto Rico 2 million volunteers installed software
– Univa.org- http://www.univa.org/ Projects such as Cancer Research,
Smallpox 2.5 million volunteer systems Sells processing time to organisations
Computational resource (i.e. intranet-accessible):
– Cluster managers, supercomputer, single machine
The Idea - as a Provider… Goal: I want others to access my resources &
applications
I want to provide secure controlled access to:– My applications:
Specify who can access which applications
– My computational or data resources: I can limit external usage of my resources
Provides an interface that allows remote users to access my resources
Enable collaboration with other partners
The Idea - as a User (or Client) Goal: I want to use other resources &
applications
Through a network of service providers I can…:
– Gain access to applications that I do not have installed locally
– Use remote machines [larger resource] with more CPU, memory or storage Process larger problem sizes
– Transparently switch between different service providers No exposure to underlying OS, queuing policy, disk
layout etc.
Cluster Computing & the Grid
Grid is predominantly built on Cluster Computing solutions
University B
University A University C
GridCluster Computin
g
The General Idea…
Abstract ‘virtualisation’ of local network resources– Infrastructure manages many machines– Visualisation as a single resource– Submitted jobs get put on queue(s)
Coordinator Executor
ExecutorClient
Client
……
Condor – Background Begun in 1988, based on Remote-Unix (RU)
project
Predominantly makes use of idle cycles on machines
Condor Components Four main machine ‘roles’ (daemons):
– Submit Client (condor_schedd): used to submit resource requests, monitor, modify and delete jobs.
– Central Manager, Server condor_collector: collects information about
pool resources. condor_negotiator: negotiates (match-makes)
between resources and resource requests.– Job Executor (condor_startd): executes jobs,
advertises resources. Enforces local policy.– (Checkpoint Server (condor_ckpt_server):
services requests to store and retrieve checkpoint files.)
Shared Disk
Condor Architecture
1. Client submits job (executable + input data) to local queue
2. Client schedd advertises job request to server collector
3. Server negotiator gets next priority request from collector
4. Negotiator negotiates w/ client schedd to match resource/job
5. Client removes job from queue and sends it to executor
6. Job runs on executor
7. Job output results returned to client
Client
Server
Executor
Submit client (condor_schedd,
condor_shadow…)
Negotiator (condor_negotiator)
Collector (condor_collector) Executor (condor_startd,
condor_starter…)
Queue
…
2
3
46
5
Queue
…
1 7
M-grid
An overview
Computational Grids - in General
Users supply tasks to be performed via client
Execution nodes contribute processing power
Coordinator node sends tasks to execution nodes, ensuring results returned
Existing grid tech. sophisticated -> significant complexity– To what extent can this be reduced?
Coordinator Executor
ExecutorClient
Client
……
Java Applets?
How about Java applets as a program unit?– Browsers could act as execution nodes
Security concerns?– Web browsers execute foreign code– Java applets executed within a ‘sandbox’ virtual
machine– Stringent security restrictions imposed– In-built security configuration in browsers– Applet can only contact originating server
Risk significantly reduced
M-grid: A Lightweight Grid I
M-grid:– Execution node = Java-applet enabled browser– Client = browser– Coordinator = web server– Tasks distributed as Applets in web pages
Execution node browser opens web page on server Runs task applet Uploads results to server
Coordinator Executor
ExecutorClient
Client
……
M-grid: Overview Implemented on:
– Microsoft’s IIS (Internet Information Server) using ASP– Apache Tomcat – we’ll use this one!
Client– Develops applet class as extension to MGridApplet class– Can run applet locally in appletviewer for testing– Compiles and packages applet with input parameters file into
a jar file– Submits jar to web server via JobSubmit web page– Eventually collects results via ViewJobs web page
Execution node– Requests a job via JobRequest page– Applet submits results from job using SubmitResults page
Security provided by session authentication
Architecture