The Grid-frame Era Richard Hall | Chief Technology Officer, UK 2007 © Avanade. All rights reserved....
-
Upload
lesley-stone -
Category
Documents
-
view
219 -
download
0
Transcript of The Grid-frame Era Richard Hall | Chief Technology Officer, UK 2007 © Avanade. All rights reserved....
The Grid-frame Era
Richard Hall | Chief Technology Officer, UK
2007 © Avanade. All rights reserved. Trademarks acknowledged
ConceptsBusiness opportunityCase studyAvanade Grid ArchitectureExcel Services and GridWho is Avanade?
Entering the Grid-Frame era means the most demanding computational tasks can be delivered at minimal cost and zero incremental hardware
Grid computing concepts
Grid computing is closely associated with ‘on demand’ and ‘utility’ concepts, and can be categorised in two main forms:
High Performance Computing (HPC) where dedicated farms of servers crunch intensive calculations once the preserve of supercomputers, in fields such as aerospace, finance, geophysics, mechanical engineering, pharmaceutical research and weather simulation
Cycle scavenging where highly distributed workstations and servers lend their spare capacity to compute elements of a large workload, effectively forming a virtual supercomputer (a ‘grid-frame’). This pool of distributed resource, whose membership waxes and wanes subject to availability, can then be tasked by a range of different applications
Enterprise wide spare capacity
Enterprise infrastructure
HPC
Internal UtilityGrid power
JobJob
Job
Job
Utility computing Definition
Utility / adaptive / dynamic and ‘on demand’ are all much used terms
For simplicity we will define utility computing to mean:
An infrastructure and application capacity used flexibly by a range of end users (or applications)
Often mediated by web services
Typically built with a Service Oriented Architecture (SOA) approach
Utility computing Internal and external
An internal utility acts as a shared service within an organisation, delivering functionality on tap to different departments and users as required – and possible metering/charging on a per use basis – alleviating silo spending on IT
An external utility acts as a service supplier for the IT fabric and potentially the application functionality/licensing on an ‘on demand’ basis, maintaining the bulk of the infrastructure independently from the client, and potentially shared between multiple organisations. Normally for general purpose IT at a layer below business process outsourcing
Business opportunity
Escaping the high performance niche
Grid computing Business opportunity
Mentioning grid computing often invokes concept of HPC dedicated farms with elaborately coded applications
Either an organisation has developed this – at great expense – or does not see this as affordable route
In both cases grid is seen as only ever having a limited footprint in the organisation
Cycle scavenging turns this thinking on its head, asking instead:
“What applications would you run if no additional hardware was required, and both development and operations were simple?”
The Grid-Frame Era
Computationally intensive tasks can now be addressed for zero incremental hardware costs and minimal development
A new range of business problems are now within reach which previously would not have justified dedicated hardware farms
Typical workloads destined for a mainframe can now be serviced by the grid-frame
As Microsoft tools deliver ubiquitous business intelligence to the desktop users, grid computing will add far deeper insight
Grid solutions will escape the HPC niche and serve the whole organisation
Grid computing Banking scenario example
Risk Management System
Core banking
Front office trading
application Dedicatedserver cluster
Ad hoc Windows grid farm (AGA.NET)
Customer analytics
Scale out
Distributed resource
Workloadgovernance
Monte Paschi de Siena
Evolution of grid within an investment bank
Case study Monte Paschi de Siena
Monte Paschi de Siena (MPS) is one of the Italy’s oldest financial institutions
Consorzio Operativo Gruppo MPS is a shared service provider within the MPS group
Set up as a consortium it manages information technology systems and administrative services for companies within the group
Forward looking IT strategy, striving to improve performance for MPS
Adopted SOA in 1999, and Grid development in 2004
Case study Environment
Every day, the systems of Consorzio Operativo Gruppo MPS execute millions of online transactions and many batch procedures on mainframes of member banks and of the other companies within the consortium
The bank’s mainframe runs a range of batch processes. However, as it has become an increasingly multi-channel architecture, it has evolved to support real time applications as well
Infrastructure estate composed of over 3,000 department servers, 20,000 desktops and 1,000 nodes in the Central Server Farm
Case study Business requirements
Reduce business risk by reducing the workload on the central mainframe (thereby removing capacity and economic constraints)
Improve IT performance and reduce costs related to data processing
Optimise distributive processing capacity across estate
Ensure effective utilisation of the Central Server Farm
Case study Solution architecture
Avanade joined forces with management at Development Division (Architecture Office) and at the Group’s Division staff (Competence Centre) to develop a solution based on Grid Computing called the Avanade Grid Architecture (AGA)
AGA runs on the Microsoft .NET platform designed to enable Service Oriented Architecture solutions
All system functionality exposed as web services
The ‘chat’ between each of the nodes and the central system is SOAP over HTTP
Case study Solution security
AGA leverages the security model defined for WS-Security
Particular attention has been devoted to application security issues. For example, by implementing Code Access Security, it is possible to allow the infrastructure to execute only digitally signed tasks
Alternatively AGA can ensure that only the infrastructure can initiate execution of tasks and prevent additional tasks being initiated or authorised.
The security model is extensible via API
Case study Solution integration
Flexibility and developers can quickly implement new processing tasks thanks to a set of APIs which is provided
Extended to support computing nodes on the Linux platform, realise the value of existing investments and third-party solutions
Incorporate open standards for Grid deployment
Retain manageability
Case study Workload evolution
First deployed for analysis of web server traffic to mine customer insight
Extended to classic batch processing of accounting jobs
As confidence in platform grew, MPS experimented with risk management scenario analysis
Fully integrated into frond and mid-office analytics, delivering daily regulatory risk reporting (Value at Risk, capital adequacy, Basel I)
Integrated with key third-party trading and risk applications
Over 1,000 jobs running on a daily basis
Case study Application functionality
Currently testing a new Portfolio Analysis Workbench application for intra-day analysis
Case study Lessons
Grid meets peak demands for ‘overflow’ processing
Allows timeshifted jobs to optimise utilisation over whole estate
New front end client applications developed to harness the AGA core platform to deliver specific tasks
Provide building blocks for dedicated computational engines, managing persistence of data and logic but able to farm load easily and grab spare capacity
AGA.NET overview
Design principles
AGA.NET design began with the realisation that the majority of nodes available for ad hoc task processing will be running Windows
Must be easy to refactor existing code to run across the Grid
Furthermore, ease of deployment and manageability are vital for realworld adoption
Hence the platform is built to take full advantage of Windows Management Instrumentation (WMI) and Active Directory
Design principles Co-ordination
The Co-ordinating Node performs three key functions
Scheduling: accepting new Task requests
Matching: assigning Task components to appropriate Nodes
Monitoring: policing activity and ensuring Tasks complete or are reallocated
Allows static and dynamic rules to govern allocation of tasks
Design principles Manageability
Controlled allocation of Tasks and managed deployment of all software / configuration (a Package) to computing Nodes
For example, Nodes can be configured by:
o Machine class – group membership
o Maximum complexity – measure of load which can be assigned
o Other measures of dynamic performance
Design principles Grid Security
Two aspects of Grid Security Framework
Infrastructure security:
Governed by Active Directory membership (nodes and admin) Communication secured by IPSEC Polling service on specific TCP port
Code security:
Set at compile time, verified at runtime Digitally signed assemblies
AGA.NET Logical architecture
Persistent Data Storage
Download Manager Cache Manager
Task Factory
Isolated Run-time environment (x task instance)
Generic Task State Management
Instrumentation
Exception Handler
Poller
Computing Node
Coordinating Node
Scheduler
Monitor
Task Request
State Data
External Client
Scheduler Client Interface
Data Source
Tasks Queue
Data Storage Access Layer
Nodes Inventory
Tasks Metadata
State Information
Data SourceData SourceData Source
Launcher
Tasks Package Repository
Package Manager
Matcher
Node State Data Provider
Design principles Resilience
The central Governance Node monitors overall task execution and can easily re-queue incomplete tasks
All state for the Grid and intermediate results are stored in SQL Server database where normal resilience procedures apply e.g. Clustering
Governance Node can be clustered but loss of service will not impact Grid processing already underway
Restarting or failover of the Governance Node allows new scheduling and active monitoring to recommence
Administration Example New package
Design principles Application development
AGA.NET is designed to facilitate deployment of existing business logic across the grid
Unlike academic or language / hardware neutral frameworks, AGA.NET assumes Windows development environment
Must maintain minimum complexity to ensure broadest adoption
Ensure compliance with emerging Grid standards such as WSRF (Web Services Resource Framework) for stateful web services
AGA.NET Sample code
using System; using Avanade.AGA.BaseClasses;using Avanade.AGA.Client; namespace TaskSample{
public class SimpleTask : Task{
public SimpleTask(){}
public override TaskExecutionStatus ExecuteImpl(){
// Reads parameters provided when the task was scheduledInt32 iNum1 = (Int32) Context.InputParameters["Num1"];Int32 iNum2 = (Int32) Context.InputParameters["Num2"];
Int32 iTot = iNum1 + iNum2;
// Saves the result of the processing to the task’s stateContext.State["Tot"] = iTot;
// Commits the contextContext.CommitState();
// Traces a messageLogTaskMessage(MessageType.TRACE, "Done!");
// Signals task’s completionreturn TaskExecutionStatus.Completed;
}}
}
Excel Services and Grid
Office 2007 / Excel Services deployment
Web and device access
Front, middle and back office solutions
(Unix, Linux, Mainframe)
Traders
QuantsDedicated
server cluster
Ad hoc Windows grid farm (AGA.NET)
Excel Services
Scale out
Distribute
Integrate
‘Out of the box’ versus ‘development required’
Function Out of the box Development required
Share and calculate sheets Yes
Audit and secure access Yes
Workflow Yes
Access via web, Excel client Yes
Excel client use all CPU (e.g. 1, 2 or 4) in desktop
Yes
Excel client with >4 CPU No Requires integration with WCC or Grid
Excel Services 1 sheet per server CPU Yes
Run user developed models/functions Partial Likely to need code tuning and some re-development
Access via other application Partial Likely to need integration effort for complex apps
Scale out with Windows Compute Cluster (WCC)
Partial Likely to need conversion to C and tuning
Scale out with Grid No Requires integration to chosen Grid tools
Who is Avanade?Avanade is the leading technology integrator specializing in Windows enterprise services – a unique Joint Venture formed in 2000 by Microsoft and Accenture
Avanade
Deep understanding of Microsoft technology and direction
Assets to accelerate programmes and remove risks
World class expertise with global delivery reach
Cost effective, depth of onshore/offshore resources
Production infrastructure and operations experience
Our skills and experience optimizing the Microsoft Platform: Avanade boosts customers’ operational performance by helping them maximise their IT investment, using our unique insight into Windows technology
Our leading-edge solutions & assets: Avanade partners with customers to define how technology can help them achieve their business vision, without re-inventing the wheel
Our quality people and delivery: Avanade’s people help customers make informed decisions and solve complex problems faster and with reduced risk
Our people help customers around the world maximize their IT investment and create comprehensive solutions that drive business results
Three factors help Avanade provide maximum value to our customers:
Global reach and scale
Avanade Americas
•Seattle•San Francisco•Chicago •Dallas •New York•Toronto•St. Louis•San Antonio•Nashville•Kansas City•Cincinnati
Avanade Europe
• Amsterdam• Barcelona• Brussels• Copenhagen• Edinburgh• Helsinki• Kronberg• London• Milan• Madrid• Malaga• Oslo• Paris• Rome• Stockholm• Zurich
Avanade Australia
• Sydney• Melbourne• Brisbane• Canberra
Avanade Asia• Bangkok• Hong Kong• Kuala Lumpur• Manila• Shanghai• Singapore• Tokyo
AvanadeIndia
• Mumbai• Bangalore• Hyderabad
Over Over 5,000 consultants, $486 million business
Contact details
Richard HallChief Technology Officer
Avanade UK
141 Wardour Street, London, W1F 0UT
Email: [email protected]
Web: www.avanade.com/uk
Blog: richardhall.spaces.live.com