Distributed Raster Processing Framework (DRPF) ISS2011/AO6647/3

Distributed Raster Processing Framework (DRPF) proposal for AO6647

ref: ISS2011/AO6647/3date: 21/02/2011page: 1 of 21

Part B: Technical And Application Proposal

1 TECHNICAL OBJECTIVES

The main objective of the proposed a Distributed Raster Processing Frameworks (DRPR) project is to design and implement generic distributed parallel processing framework specifically targeted at processing of Earth Observation (EO) raster image data in large scale distributed computing environments (clusters of network connected computing nodes, GRIDs, Clouds).

The developed framework shall be deployed in a Cloud environment (selection of the provider shall be part of the project tasks) and the main features shall be demonstrated by a set of use cases relevant for ESA and the EO community.

The key idea of the parallel computing is that a solved problem can be decomposed to several (simpler) sub-problems so that each sub-problem can be solved independently in-parallel and an operation joining results of the particular sub-problem to a single solution equivalent to the one of the original problem exists. Since the recursive nature of this idea (sub-problems can be further divided to sub-sub-problems, etc.) several layers of exploitable parallelism can be generated for a particular problem.

This general theoretical concept of parallel computing may look very simple but the praxis has to deal with many additional issues. Beside the fact, that not all algorithms can be simply decomposed to independent sub-problems according to the key idea, a significant issue is the development cost of the parallelised SW. Ad-hoc parallelisation of particular algorithms increases complexity of the developed SW, the optimal performance requires careful optimisation for the target HW platform and yet there are many HW platforms changing in time requiring re-engineering of the parallel SW every time a new HW comes along. This is the main reason why historically parallelisation had been adopted by application where other consideration outweigh the expense.

Until recently, the parallel computing was at margin of the SW developers who could rely at increasing speed of the sequential HW. The situation changed dramatically, however, as the clock speed of the single core CPUs reached its limit and parallel HW become affordable commodity (multi-core CPUs, GP-GPU accelerators, cheaper and faster networking, the concept of Cloud computing).

In order to bridge the gap between the SW and HW, the computer science has been concerned many years by decoupling semantics of programs from the particular parallel implementation. The role of the bridge in this attempt is played by a model of parallel computation. A model provides abstract machine to the programmer while allowing implementations on different HW architectures. The complex details of the HW infrastructure is handled by the generic SW framework implementing the computation model while the application programmer can focus on the specific algorithms.

This approach is particularly beneficial for calculation in large scale distributed environments, focused on by this proposed project, such as Cloud and large data-centres with a need for efficient, scalable and reliable exploitation of the parallel resources.

2 REQUIREMENTS

The preliminary list of high-level requirements is as follows:

Requirement 1: The framework (FW) shall employ generic computing model suitable for processing of raster image data while hiding the details of the computing infrastructure and of the complex network communication and allowing implementation/deployment of broad range of algorithms.

Requirement 2: FW shall be designed to allow exploitation of parallel processing capabilities of the employed worker nodes (e.g., multi-core CPU, GP-GPU computing).

Requirement 3: FW shall be scalable without a limit of the number of employed



worker nodes or any performance bottle-neck.

Requirement 4: FW shall be elastic, i.e., it should be able adapt dynamically the number of employed worker nodes based on actual load and availability of the computing resources (prerequisite for efficient deployment in the Cloud environment).

Requirement 5: FW shall be tolerant to worker node / network failures with minimal interception of the running tasks and with no lost of the stored data (prerequisite for deployment in large scale computing environment).

Requirement 6: FW shall be secure, preventing any unauthorised use of the framework / stored data, interception of the running distributed tasks or corruption of integrity of the data (prerequisite for deployment in shared/public computing infrastructure).

Requirement 7: FW shall be efficient, employing the dynamical load balancing maximizing utilisation of the available computing resources.

Requirement 8: FW shall be designed as multi-user, i.e., it shall allow more than one user using the FW at time and fairly share the available resources among the users within the limit (quota) and access restrictions.

Requirement 9: The performance of the FW shall be demonstrated by relevant use cases in Cloud environment.

Requirement 10: The target Cloud infrastructure shall be assessed by the project.

3 TECHNOLOGY READINESS LEVEL

The initial technology readiness level of the distributed parallel framework is 2-3, i.e., part of the envisioned are available to ISS as prove-of-concept prototypes, although, some of the features exists in form well generally known concepts and/or published algorithms.

The anticipated target technology readiness of the final product is 4.

4 ENGINEERING APPROACH

The project shall follow the typical SW development procedure (see also Part C, Sec. 4 work-logic flow chart) starting by SW System Specification and formalisation of requirements, followed by the specification of the architectural design and finalised by the SW detailed design and implementation. The developed product shall be subject to intensive testing, validation and verification against the specification and requirements. It is expected the already the early phase systems specification and design shall be accompanied by rapid prototyping proving the key concepts.

The basic principles of the parallel processing framework is not new and we admit that the concepts of the proposed framework is inspired/influenced by the existing frameworks such as Google MapReduce [RD01], Apache Hadoop [RD02] and SectorSphere [RD03]. All these frameworks share the idea of data centric processing, i.e., it is cheaper to move processing rather than the processed data.

The key differences of the proposed framework from the above mentioned ones is the direct support for processing of collections of raster image data (described in detail in Sec. Key Features of the Proposed Framework below). Further, a strong attention is paid to the failure tolerant and elastic (Cloud compliant) design, thus the new proposed framework cannot be considered as a simple reimplementation of the existing work.

Distributed File System

The key pillar of the above mentioned frameworks and also of the proposed one the special purpose Distributed File System keeping the track of the location of the processed data.

Assuming the distributed infrastructure consists of large number of computers (worker nodes) each of them having its local storage. The working nodes of the processing infrastructure are used not only for processing but they store the fragments of the processed data forming large distributed file-system. The file system tracker holds the whole view of the distributed file-



system and the relevant meta-data. When the parallel data processing is triggered the computing framework is able to locate the data fragments and preferably start the processing at the worker nodes where data are stored. As long as the worker nodes holding the data are not overloaded the data do not need to be transferred (replicated) over network speeding-up significantly the processing.

In large computing infrastructures (hundreds or thousands of worker nodes) the probability of a HW failure which renders a part of the system dysfunctional approaches certainty. In order to assure the reliability of the system and avoid data losses the data fragments must be kept in several copies. In case of single work-node failure one or more alternative replicas exists. The file-system tracker monitors the systems and manages the data replication/recovery to keep the required number of replicas. The replication adds a benefit of alternative worker nodes simplifying the scheduling and load balancing of the processing tasks.

Although the distributed file-system provides the analogy of the traditional file-systems (e.g., POSIX file systems), i.e., it allows to structure the data in trees of directories/folders holding the files, it relaxes some of the properties of the standard file-system such as the ability to modify once stored files. The distributed file-system is thus primary designed to support efficient processing rather than replace the traditional file-systems.

Different frameworks vary in details of the implementation such as whether stored files can be fragmented (e.g., Google MapReduce splits files to 16-128MiB blocks) or the files are stored as whole (e.g., SectorSphere) and the optimal data fragmentation is controlled by the client application. Further, some of the frameworks (such as Apache Hadoop) allow only single tracker to be present in the file-system leading to a single-point of failure and long recovery times in case of failures.

Parallel Execution

Second pillar, beside the data storage, is the is data processing model. The model hides the complex details of interaction of the different framework components while the user focuses on the functionality of his/her application.

Common approach is that the framework user prepares a client application and worker modules distributed to all or limited selection of the worker nodes. The client application uploads the data for processing (if not already stored by the framework), triggers processing of the selected data and collects the results. The frameworks takes care about the interaction between the client application (linked with framework client interface library), trackers and worker nodes.

The processing is split to smaller sub-tasks which are executed in parallel by the worker nodes. The

Figure 1: Simplified scheme of the key components of the proposed framework with indicate lines of interaction.

Figure 2: Simplified scheme of the key interfaces of the proposed framework.



execution of the sub-task needs to be tracked in order to keep the integrity of the processing and to minimise the total execution time and to assure efficient use of the computing resources. The task tracker needs to decide the optimal set of target worker nodes to which it submits the processing. Analogically to the data storage the task execution needs to cope with possible failures. In case of non-responding worker node (indicating possible failure) the assigned sub-task needs to be submitted for processing to an alternative location.

The different existing frameworks differ in way of implementation of the work trackers and schedulers, whether they employ special work trackers (Apache Hadoop) or they keep the task tracking at the client side (enclosed with the framework library linked to the client application).

Model of Parallel Computing

The Google Map-Reduce and the open source Apache Hadoop employ the Map-Reduce [RD04] computing model based on the modified map and reduce functional programming operations. This model assumes the input data are in form of a list (sequence) of key-value pairs. While map operation transforms a list to another list of key-value pairs the reduce operation groups and aggregates the values by key. The map and reduction operators are implemented as user-provided modules executed by the worker nodes.

The Map/Reduce frameworks process sequences of input records of arbitrary type. Typical Map/Reduce operation parse the set of input files to the key-value sequences which then can pass through additional map operations finished by one or more parallel reduction steps. The Map/Reduce frameworks were primarily designed for processing of textual information such as indexing of web pages, classification of records or extremely large data-bases and these are the applications for which these frameworks provide the best performance. The operation over arbitrary binary data is in principle possible and use cases of processing of raster data exist.

The Sector/Sphere framework uses similar though more generic model of User Defined Functions (UDF) allowing arbitrary operation over set of input files. It allows full implementation of the Map/Reduce model as the Google Map-Reduce or Apache Hadoop.

Key Features of the Proposed Framework

As it was already mentioned, the proposed framework shall reuse the key concepts of the existing frameworks, namely it shall employ the concept of the distributed file-system and the projection (map) and aggregation (reduce) operations as over collection of image fragments. But rather than operation over linear sequences, the framework shall focus on processing image data fragments (typically 2 dimensional matrices of pixels) arranged in 4-dimensional grid. The grid fragments define the

Figure 3: Map and reduce list operation operation. The map-filtering and map-expand extension to standard map operation (allowed by the Map/Reduce frameworks) are displayed.

Figure 4: Four dimensional image collection - an illustration.



granularity of the parallel processing.

The distributed file-system shall be able to keep the 4-dimensional extent (bounding box) of data-fragments (tiles) able to hold time series of 2-dimensional multi-band (multi-primitive-feature) images. Queries to the file-system tracker shall return set of data-fragment of the particular collection falling into the queried 4-dimensional bounding box.

The file system shall hold the image fragments as whole files allowing the users to store the data in arbitrary file format and data granularity. The content of the files shall be interpreted by the user defined worker modules and the framework shall threat them as atomic.

The generic projection operation shall be able to transform a selection of input data to a new collection or image layers (applicable to, e.g., geometric transformations, primitive feature extractions, feature space transformations, image filtering, etc.). The aggregation operation then shall be able to extract the information and accumulate the sub-result to one final result (applicable to extraction of statistical parameters of the data or geometrical vector features).

The distributed data storage and the affinity of processing to the location of the data fragments shall allow easy repeated/iterative calculations in cases of iterative calculations (e.g., clustering algorithms) and/or when several consecutive processing steps needs to be applied.

Regarding the failure tolerance we propose a model of multiple lightweight peer trackers guarding consistency of the distributed file-system and keeping the track of the available worker nodes. These trackers shall exchange the update messages and keep the consistent view of the system, all the trackers shall provide the same functionality and if needed to replace the dysfunctional peer. The trackers shall employ the Eventual Consistency model [RD04 and RD05].

The task tracking shall be primarily employed by the client application, the client application shall query the tracker to provide informations about the available worker nodes and location of the data. The rest of communication shall be carried out between the client application and the worker nodes. The changes in the data storage data shall be reported by the worker nodes to one of the trackers. The worker node shall replicate the data to another worker nodes and vice versa it shall download missing data from the other worker nodes when required for processing.

The worker nodes shall be able to roll-back the operation and clean-up the temporary data of an orphan task in case of the client application failure.

This briefly described architecture shall guarantee efficient system functionality without a single point of failure.

As stated in the requirements the processing shall not prevent application of local parallel processing such as multi-threading on multi-core CPUs and GP-GPU computing. The utilisation of the local worker node parallel processing capabilities shall responsibility of the user provided modules. However the worker node shall control the use of the local parallel capabilities in order to exploit them efficiently and prevent possible contentions. We suggest that the local worker node shall provide basic WS primitives such as shared thread pool, thread safe message queues and/or dedicated GP-GPU context allowing efficient concurrent exploitation of the parallel resources by more than one instance of the user modules.

Cloud Infrastructure Binding

The system shall provide a basic management and monitoring interface manage the users of the system, to upload user modules to selected nodes and control the operation of the worker nodes and trackers.

The proposed architecture of the system allows the dynamical change of the number of employed worker and tracker nodes. The newly added worker node tracker needs to know at least one functional tracker to get the informations about the peer trackers and worker nodes and quickly become part of the system. The worker nodes and trackers shall allow soft showdown operation when they will stop accepting new and commit the pending operation, sign-off themselves form a near-by tracker and shut-down gracefully.



This elasticity is beneficial for Could environment where the number of employed instances can be changed based on the work-load demand.

We propose a special Cloud binding component integrated with the Cloud middle-ware which shall monitor both the framework and Cloud state and take an action to dynamically start new or shut-down running instances. Beside the work-load adaptation this subsystem could also optimise the Cloud costs by employing cheaper spare (spot) instances of the Cloud nodes when available. E.g., Amazon E2C Cloud infrastructure provides interface providing the instant costs of the spot instances and allows register call-back functions triggered when costs drop below a desired threshold.

The elasticity has its limits. The worker node instances cannot be released in large groups as the replication could not assure data consistency. The model of distributed storage may also fail for very small numbers of worker nodes. A possible solution to this issue is to use the could storage to hold backup replicas of the data. By applying the rule of one backup replica kept always in the large and slow storage element the number of active worker nodes can drop to zero without lost of data.

It is planned in this proposal that the developed framework shall be deployed in Cloud infrastructure with the Cloud specific binding implemented. In the beginning (WP200) a short survey on the public available Cloud public infrastructures shall be performed. The survey should also focus on availability of GP-GPU enabled Cloud infrastructures. The survey should lead to a trade-offs selection of the target platform. The Amazon E2C platform is considered to be the default option.

Identity Management

As the distributed framework shall be able to serve possible large number of concurrent users an identity management subsystem needs to be considered. The trackers and worker nodes need to be able to validate the incoming requests and the system shall be able to apply limits on the processing resources provided to a single or group of users.

We do not intend to implement this subsystem ourselves but rather reuse an existing standard based secure solution allowing interoperability with a third party Identity Management System. It is envisioned that the Shibboleth open source SW [RD07] shall be used for this task.

Further isolation of the work space for each individual user shall be provided by the file distributed system to prevent file-name collisions within the file-system. On the other hand, the system shall allow data sharing among the defined groups of users. For this task we propose a system of isolated data buckets, i.e., set of isolated name-spaces each having an unique identifier and access control list applied to all files with in the bucket. Each user shall have a default private unique data bucket, the default file storage, and right to create new buckets and share them with the other users.

Security

When used on a public / shared computing infrastructure the security must be inevitably considered. There may be many different ways how an unsecured system could be corrupted, starting by the unauthorised use of the framework / stored data, interception of the running distributed tasks or integrity of the data. To mitigate the security risks the proposed framework shall be able to use SSL (Secure Socket Layer) standard for secure network communication.

On the other hand the application of strong encryption implies significant performance overhead, especially for large bulk data transfers. Therefore the use of the secured networking shall be configurable. It could be disabled in cases where the security risks are negligible and the performance is the clear priority, e.g., in virtual private networks or private HW infrastructures.

We also consider a semi-encrypted mode using encryption of relatively small meta-data only and allow the large data payload to be transferred transported unencrypted assuming these non-annotated data would have little to none meaning for any potential eavesdropper.

Use Cases

The performance of the FW shall be demonstrated by relevant use cases. It is expected that



the final selection of use cases shall be subject of negotiation between ISS and the Agency. For the time being we propose following list of use cases based on algorithms developed by ISS in recent projects:

● K-means clustering – simple but demanding iterative clustering algorithm

● Gaussian Mixtures – Expectation Minimisation clustering algorithms

● Simple Pan Sharpening – image fusion algorithm (as implemented in [RD08])

● Road Extraction algorithm (as implemented in [RD08])

● Water Shed image segmentation (as implemented in [RD08])

● Ortho-Rectification (as implemented in [RD08])

● Large Scale Mosaicking

These algorithms had been thoroughly studied in the previous projects. Performance optimised versions (SIMD vectorised, multi-threaded, or GP-GPU enabled) of these algorithms has been developed by ISS can be reused for purpose of the Use Cases.

The definition of suitable datasets shall be part of the design phase WP300.

Reference Documents

RD01 http://labs.google.com/papers/mapreduce.html

RD02 http://hadoop.apache.org

RD03 http://sector.sourceforge.net

RD04 http://en.wikipedia.org/wiki/MapReduce

RD05 http://en.wikipedia.org/wiki/Eventual_consistency

RD06 http://en.wikipedia.org/wiki/CAP_theorem

RD07 http://en.wikipedia.org/wiki/Shibboleth_%28Internet2%29

RD08 http://www.orfeo-toolbox.org/otb

5 TECHNOLOGICAL FEASIBILITY AND DEVELOPMENT RISK

The ISS competence to fulfil the task were proved by the portfolio successfully completed projects. Based on the company and personnel experience we deem the proposed project goal feasible and identify the following risks.

Risk Assessment

Risk – 1: Cloud infrastructure costs escalation

Mitigation:

The costs for Cloud storage and machine time shall be continually monitored by the Project Manager. In case of indication of possible costs escalation the Cloud utilisation shall be reduced in order to fit the allocated budget limit.

6 APPLICATION OF TECHNOLOGY DEVELOPMENT

The processing of air-borne or space-borne raster image data is the crucial step in the field of Image Information Mining (IIM) and EO data processing. In order to obtain semantic content of an image sophisticated algorithms with high processing cost must be usually applied. Beside this baseline processing cost, the analysis of EO data is further complicated by continually increasing data volumes thanks to the trend towards very high-resolution sensors. Moreover, the amount of the processed data can increase furthermore when the image time series is considered, i.e., when several acquisition systems, large set of multi-band images, and multiple acquisitions times are analysed. In order to make these calculations feasible

http://labs.google.com/papers/mapreduce.html

http://www.orfeo-toolbox.org/otb

http://en.wikipedia.org/wiki/Shibboleth_(Internet2)

http://en.wikipedia.org/wiki/CAP_theorem

http://en.wikipedia.org/wiki/Eventual_consistency

http://en.wikipedia.org/wiki/MapReduce

http://sector.sourceforge.net/

http://hadoop.apache.org/



efficient parallel processing must be inevitably taken into account, at least, for the most computationally expensive 'bottle-neck' algorithms involved in the data analysis. Notable examples of these core algorithms are feature space dimensionality reduction, synthesis, and similarity clustering. In addition, it is a well known fact, that only very small percentage of information contained by space-borne EO data is currently exploited. In order change this situation significant effort has been put to develop EO-IIM (Earth Observation – Image Information Mining) algorithms and processing tools to foster EO data applications.

The DRPF framework could be beneficial for EO-IIM community providing a tool for exploitation of large computing infrastructure for fast and efficient data processing. It shall be especially well suited for emerging concept of Cloud computing.

There is a potential for application of the developed SW framework as the key building block of EO data processing services, one of key areas of interest described in the Czech National Space Plan document.

The DRPF framework may significantly enrich the portfolio of ISS products providing new prospect in future commercial project not only in the EO area but generally in applications where large image dataset need to be processed.

7 INTELLECTUAL PROPERTY RIGHTS

The Intellectual Property Rights (IPRs) shall be subject to General Clauses and Conditions for ESA contracts. Namely, the IPR's related to the developed SW product and its documentation are owned by ISS and ISS grants the Agency irrevocable, non-exclusive, world-wide free licence, with the right to grant sub-licences, to use, for the Agency's own requirements, any intellectual property right, software (including both object and source code), information or data produced by the project.

Background Intellectual Property Rights

The SW product shall be written completely by ISS without contribution of source or object code from any other subject.

The implementation will be based on publicly available technical documents and these will be listed as references in the documentation.

Only public domain algorithms shall be considered for implementation.

It is foreseen that the SW product will not require any paid proprietary SW tool or operating system, either during the compilation of the object code from sources or during the run-time.

The only proprietary component currently considered to be used for the project is the Nvidia's GPU SW Development Kit (SDK) available under royalty-free license.

To fulfil the objectives, the project shall use Open Source SW tools and operating system (e.g., GNU/Linux operating system, GCC compiler, Python runtime environment, etc.). These tools are available free of charge and without any IPR restriction and shall not be part of the PalDMC SW product.

In case any additional BIPR items should become identified during the execution of project, these items shall be documented and reported to the Agency.

8 CONDITIONS RELATING TO EXPORT/IMPORT LICENCES/AUTHORIZATIONS AND RELATED DOCUMENTATION

There are no export or import restrictions the prime contractor and/or subcontractors would be subject to.



Part C: Financial, Management And Administrative

1 BACKGROUND AND EXPERIENCE OF THE COMPANIES

Iguassu Software Systems, a.s.

Iguassu Software Systems (ISS) was founded in 1994 as the Czech subsidiary of Science Systems (now SciSys). As such it had been involved in ESA and Eumetsat projects and even before the Czech PECS accumulated over 40 man years in space work. ISS Prague based FFP project for SciSys, to develop a test tools suite for the MSG ground segment, was on the critical path of a major Scisys delivery to EUMETSAT. The successful execution by ISS won acclaim in the industry survey, which ESA commissioned in 2002. The company worked on sites in ESOC and ESRIN as well as in UK, in satellite control and payload data processing.

ISS continued the space work after becoming and independent Czech SME in 1999, including contracts for CAM GmbH (Eumetsat) and Scisys (Galileo). At the same time it designed and developed s/w for technology clients such as Agilent Technology, HP and Thermoking.

ISS is a member of the Indra consortium which won development of the search and rescue capability of Galileo and contributed to the development in Indra Barcelona.

ISS was the most active and successful company in the Czech PECS (as stated by the CSO).

Six ESA PECS projects were successfully carried out

● EGNOS SISNET for ESA GNSS in Tolouse

● the establishment and operation (ongoing) of the 1st CEE EGNOS monitoring station

● PECS-GRID – porting of Synthetic Aperture Radar (SAR) processing software to Grid and development and integration of pilot services in the ESA g-Pod Grid infrastructure, involving over one year work on site in ESRIN.

● EGNOS SISNET tools and server development

● PECS-IIM-TS – the first Czech participation in a winning competitive bid for Information Mining in Time Series (IIM-TS) project a members of consortium led Advanced Computer Systems, Spa. (ACS) and CCN extension.

● EGNOS Education tools project (staff embedded in ESA GNSS team in Toulouse)

In the first open Czech call (AO6052), ISS won two projects, currently being concluded

● PalDMC – Parallel Data Mining Components (with ACS as subcontractor)

● RTPMT – Real-Time GNSS Performance Monitoring Tool

We also participated in two international bids, both of which were successful

● AO 6149 Interference Monitoring System for GNSS Reference Stations (Astrium prime)

● AO 6143 O3S - Open-standard Online Observation Service (EOX Austria as prime)

ISS developed good working relationship with ESA suppliers, notably ACS, GMV, Indra, EOX and Astrium.

The MD has been working in space technologies since 1975, including 12 years as ESOC staff member. He has a good track record of identifying Czech engineers who prove competent in ESA projects. This will enable ISS to implement the goals of the task force and embark on more extensive space work. If more work is won, it will use its 6 ESA experienced staff to train young highly skilled Czech engineers.

The MD also initiated and presides the industrial association the Czech Space Alliance, which with 18 members, covers the majority of CZ space industry.



2 TEAM ORGANISATION AND PERSONNEL

Iguassu Software Systems – company structure

Team organization

The Project Manager (PM) is the project team leader and he is responsible for communication with ESA technical officer. The contract officer is responsible for all the communication with ESA contract officer. The senior engineers in charge of the assigned work packages report to the PM and they may be responsible for one or more junior engineers.

Project Management Plan

MANAGEMENT OBJECTIVES AND PRIORITIES

The main objectives of the project and quality management are:

● ensure overall compliance of the ISS team the with the goals, requirements, schedule and costs, as per ESA contract

● timely completion and handover of deliverables

● ensure compliance with the contractually agreed subset of software engineering standards

● regular reporting of progress, deviations or anomalies, as well as tracking and resolution of action items

● agreeing with ESA technical manager resolution of any possible deviations from plan



An electronic tool will be used by the project management to:

● maintain the document list recording all the documents produced by the project in Document List;

● record and track resolution of all action items in Action Item List (AIL);

● maintain Risk Register including the risk's status and proposed means of mitigation;

● maintain up-to-data bar-chart schedule of the project and particular WPs

● perform the configuration management of all documents, software components, and documentation produced by the project.

ISS, as the responsible toward the Agency for the project, will make all reasonable effort to fulfil these goals.

ASSUMPTIONS, DEPENDENCIES AND CONSTRAINTS

not applicable

RISK MANAGEMENT

The identified risks, potentially affecting the achievement of the project's goals, are presented in Sec. 5 of the technical proposal together with the evaluation of their impact and proposed means of their mitigation.

The risks will be tracked during the lifetime of the project and their status will be reported in the progress reports.

INFORMATION AND DOCUMENTATION FLOW

All the formal communications between ESA and prime contractor regarding financial and contractual matters have to be exchanged in hard copy. If necessary, electronic copies of these documents may be delivered in parallel via e-mail or fax.

The use of electronic documents such as e-mails, MS Word and PDF documents for formal communications regarding the managerial and technical matters will be accepted.

The informal communication will take place as needed by phone, fax, and electronic means.

During the lifetime, the project's deliverables (presentations, documentation and SW) shall be delivered in electronic form. They can be either delivered via e-mail or made available for download from web pages.

At the end of the project, the final accepted deliverables shall be delivered in electronic form stored on suitable storage medium (CD, DVD, etc.).

Document exchange will be freely allowed, but only correspondence and documents signed by the relevant prime contractor's or ESA personnel are binding.

Any formal documents exchanged between the members of the consortium shall be written in English and shall be made available for ESA if requested.

QUALITY MANAGEMENT

The adherence to standards and plans is monitored by the Project Manager. Formal checks take place during the reviews. The project shall adhere to relevant subset of ECSS-E-40B agreed with and approved by ESA.

QA/QC Activities During the Procurement

All procedures, developed code and document deliverables to ESA, will follow the relevant subsets of ECSS-Q-80B agreed with and approved by ESA.



Deliveries and Reviews

Before the review packages will be delivered to ESA, the Project Manager shall review and check their completeness and their adherence to the required standards.

The list of review and progress meetings is presented in Sec. 5 of this document. The deliverables are listed in Sec. 8 and also indicated in individual WBD Sec. 4 of this document.

Auditing

At ESA discretion.

PROJECT CONTROL

The projects costs and schedule are tracked in the context of the project control activities.

The Project Manager is responsible for the project control activities, namely, he is obliged to monitor progress and adherence to established project plans, identify potential problems and risks, and suggest corrective actions.

The project schedule shall be based on the current WBS. Any changes to WBS, individual WBD, or dates of key events cannot be made without ESA approval.

PROGRESS REPORTING

Progress Reports

The project manager shall provide monthly Progress Reports to the ESA Technical Representative. These Progress Reports shall cover the status of the accomplished work, deliverables and pending actions, and significant event/problem occurred during the reported period. The reports shall also include the activities planned for the following time period.

Other brief progress reports may be provided by other means, e.g., via e-mail or by telephone, as necessary.

A template for the progress reports shall be proposed by the prime contractor by the start of the project.

Meetings and Reviews

Progress Meetings shall be held approx. every three months and they shall cover the entire scope of the contract, namely, it shall include contractual, technical and schedule matters.

When applicable, Progress Meetings will be held jointly with Reviews.

Progress Meetings may also be combined with working meetings to discuss the details of the planned near-term work.

The ESA Technical Representative shall be informed about date and agenda of the planned meetings and reviews at least two weeks in advance in order to enable to organise ESA participation. In case of Review meeting, ESA shall receive the reviewed documentation package at least five working days in advance.

The course of all formal meetings shall be documented by minutes. The results of the actions arising from each meeting will be uniquely identified, have an expected completion date and be tracked/reviewed in the following meetings until closed. The conclusion of the meetings and list of action items, shall be agreed, written and signed by all participants at the end of each meeting.

The list of planned meetings and their anticipated date and location are presented in Sec. 5 of this document.



SETTLING DISAGREEMENTS

A possible disagreement regarding the technical and managerial matters between the members of the consortium shall be resolved by the Project Manager.

External disagreements shall be settled in accordance with the ESA contract and practices. These disagreements shall be resolved as per Clause 35 of the draft contract.

ISS Key Personnel

MARTIN PAČES – PROJECT MANAGER

In 1999 Master degree from Chemical Engineering, major Mathematical Modelling and Informatics, Institute of Chemical Technology (ICT) Prague. 1997-2004 department of Chemical Engineering, Centrum of Nonlinear Dynamics, ICT, Prague. 2002-2003 Max Planck Institute for Dynamics of Complex Technical Systems, Magdeburg. Working in area of mathematical modelling of complex reaction transport processes in electrical field.

Since 2005 employee of Iguassu Software Systems, a.s., where he has worked as a software developer in area of Earth Observation, Grid computing and algorithm parallelisation. He successfully carried out 3 PECS projects: PECS-GRID project (2005-2007) focusing on porting of Synthetic Aperture Radar (SAR) processing software to Grid and development and integration of pilot services in the ESA g-Pod Grid infrastructure; PECS-IIM-TS project (2007-2008) studying possible interconnection of ESA g-Pod and KEO infrastructure; and PECS-IIM-TS-2 (2008-2008) implementation of parallelised versions of selected core information mining algorithms. During the PECS project he spent 21 months working on-site in ESRIN/ESA and 8 months working on Advanced Computer Systems, spa. (ACS, IIM-TS prime) premises in Rome.

MP has been assigned to currently being concluded PalDMC – Parallel Data Mining Components (with ACS as subcontractor) project (the first open Czech call AO6052) which deals with performance optimisation and parallelisation of IIM algorithms. The scope of this project has been on ESA request extended for implementation of GP-GPU enabled versions of the algorithms.

MP is also part time assigned to O3S - Open-standard Online Observation Service GSTP project (AO6143, EOX Austria as prime) dealing with design and implementation of OGC based services for processing and distribution EO data.

MP has more than 12 years of experience in area of numerical calculations, High Performance Computing (HPC), and SW development in Unix/Linux environment. Further, he has 6 years active experience with design and implementation of performance optimised and parallelised SW and services. His detailed knowledge covers parallel processing from the low level CPU SIMD vector processing, multi-threading on multi-core CPUs, GP-GPU calculations, up to large scale distributed calculations.

Thanks to PECS projects, he has also gathered managerial experience required by small ESA projects.

3 FACILITIES

Iguassu Software Systems

ISS premises in Prague are equipped with the infrastructure necessary for software design & development, and communications with ESA, as has been demonstrated in over 6 years of working on ESA projects. These include

● h/w - computers, printers, high speed internet connectivity (plus back-up wireless), telephones, and fax with email forward.

● s/w – systematically updated office software and firewall, virus and password protection.



● the ECSS standards and other necessary technical documentation

● Full time administrative support at the disposal of the project.

Any additional equipment that may be required for specific tasks shall be costed in the PSS forms. The location of the company’s premises is Evropská 120, 160 00 Prague, Czech Republic. The company is 10 mins from the international airport and in close vicinity of good quality hotels.

4 WORK DESCRIPTION

Work logic flow-chart

Work Breakdown Structure

WP100 Project ManagementWP200 SW System SpecificationWP300 SW Architectural Design WP400 SW Detailed Design and Implementation WP410 Core Framework Implementation WP420 Use Cases Implementation WP430 Cloud IntegrationWP500 SW Validation and VerificationWP600 Final Assessment

Work Package Description

__________________________________________________________________________

Identifier: WP100Title: Project ManagementManager: Martin Pačes Company: ISSStart Date T0 Start Event: KO/M0End Date: T0+20 End Event: FP/M6Inputs:

● Accepted Project Proposal● KO Meeting Agreements● Contract

Tasks:● Organise the communication and information flow within the consortium● Coordinate the tasks execution (managerial and technical) ● Identify potential problem areas affecting cost and/or schedule



● Maintain Contract Inventory, Document List, Action Item List, Risk Register, and Bar-char Schedule.

● Perform the configuration management● Archive all produced documentation and quality records● Maintain close contacts with the Agency to ensure proper coordination, feed back,

and problem notification● Organise meetings and prepare the agenda. ● Produce Progress reports ● Review and approve all project documentation ● Ensure quality and timely delivery of all deliverables items● Produce minutes of meetings ● Document activities, procedures, standards and quality indicators used during project

life-cycle● Organise the end of project activity presentation● Prepare an overview of the work performed ● Provide a comprehensive analysis of the results achieved● Carry out presentations

Outputs: ● Progress Reports (PR)● Minutes of Meetings (MoM), Minutes of Teleconferences● Action Items List (AIL)● Executive Summary● Final Presentation● Delivery of Technical Data Package at FP

__________________________________________________________________________

Identifier: WP200Title: SW System SpecificationManager: Martin Pačes Company: ISSStart Date T0 Start Event: KO/M0End Date: T0+3 End Event: SSR/M1Inputs:

● Accepted Project Proposal● KO Meeting Agreements

Tasks:● System Specification and collection of SW Requirements of DRPF framework, use

cases and cloud binding. ● Preparation of Validation and Verification Plan.● Survey on available Cloud infrastructures proposing the target Could platform for

Cloud integration and use cases deployment.● Rapid prototyping proving feasibility of selected requirements.

Outputs: ● SSS – Software System Specification document● SVVP – Software Validation and Verification Plan● TN – Selection of the Target Cloud Infrastructure

__________________________________________________________________________

Identifier: WP300Title: SW Architectural DesignManager: Martin Pačes Company: ISSStart Date T0+3 Start Event: SSR/M1End Date: T0+6 End Event: CDR/M2Inputs:

● SSS – Software System Specification document● TN – Selection of the Target Cloud Infrastructure

Tasks:



● Architectural design of DRPF framework, use cases and the Cloud binding.● Specification of DRPF interfaces.● Rapid prototyping proving feasibility of selected design concepts.● In addition during the design phase the final set of use case algorithms and test

dataset shall be specified in the DPRF Design Document.Outputs:

● TN – DPRF Design Document● ICD – DRPF Interface Control Document

__________________________________________________________________________

Identifier: WP400Title: SW Detailed Design and ImplementationManager: Martin Pačes Company: ISSStart Date T0+6 Start Event: CDR/M2End Date: T0+16 End Event: PM6/M4Inputs:

● SSS – Software System Specification document● TN – Selection of the Target Cloud Infrastructure● TN – DPRF Design Document● ICD – DRPF Interface Control Document

Tasks:● WP410 – Core Framework Implementation.● WP420 – Use Cases Implementation.● WP430 – Cloud Integration.● The activities will be carried partially in parallel.

Outputs: ● DRPF SW package● DRPF Use Cases SW Package● DRPF Cloud Binding SW Package● DRPF functional cloud instance● TN – DRPF Test Report ● SUM – SW User Manual● updated TN – DPRF Design Document● updated ICD – DRPF Interface Control Document

__________________________________________________________________________

Identifier: WP410Title: Core Framework ImplementationManager: Martin Pačes Company: ISSStart Date T0+6 Start Event: CDR/M2End Date: T0+13 End Event: PM4/M3Inputs:

● SSS – Software System Specification document● TN – Selection of the Target Cloud Infrastructure● TN – DPRF Design Document● ICD – DRPF Interface Control Document● KO Meeting Agreements

Tasks:● DRPF Detailed Design● DRPF Implementation● Testing of DRPF both in-house and within the Cloud infrastructure

Outputs: ● DRPF SW package● contribution to SUM – SW User Manual● contribution to updated TN – DPRF Design Document● contribution to updated ICD – DRPF Interface Control Document



● contribution to TN – DRPF Test Report

__________________________________________________________________________

Identifier: WP420Title: Use Case ImplementationManager: Martin Pačes Company: ISSStart Date T0+11 Start Event: PM3End Date: T0+16 End Event: PM6/M4Inputs:

● SSS – Software System Specification document● TN – Selection of the Target Cloud Infrastructure● updated TN – DPRF Design Document● updated ICD – DRPF Interface Control Document● DRPF SW package● SUM – SW User Manual – preliminary version

Tasks:● Detailed Design of DRPF Use Cases● DRPF Implementation of DRPF Use Cases● Testing of DRPF Use Cases both in-house and within the Cloud infrastructure

Outputs: ● DRPF Use Cases SW package● contribution to SUM – SW User Manual● contribution to updated TN – DPRF Design Document● contribution to updated ICD – DRPF Interface Control Document● contribution to TN – DRPF Test Report

__________________________________________________________________________

Identifier: WP430Title: Cloud IntegrationManager: Martin Pačes Company: ISSStart Date T0+11 Start Event: PM3End Date: T0+16 End Event: PM6/M4Inputs:

● SSS – Software System Specification document● TN – Selection of the Target Cloud Infrastructure● updated TN – DPRF Design Document● updated ICD – DRPF Interface Control Document● DRPF SW package● SUM – SW User Manual

Tasks:● Detailed Design of DRPF Cloud Binding● DRPF Implementation of DRPF Cloud Binding● Testing of DRPF Cloud Binding within the Cloud infrastructure

Outputs: ● DRPF Cloud Binding SW Package● DRPF functional cloud instance● contribution to SUM – SW User Manual● contribution to updated TN – DPRF Design Document● contribution to updated ICD – DRPF Interface Control Document● contribution to TN – DRPF Test Report

__________________________________________________________________________

Identifier: WP500Title: SW Validation and VerificationManager: Martin Pačes Company: ISS



Start Date T0+16 Start Event: PM6/M4End Date: T0+19 End Event: AR/M5Inputs:

● SSS – Software System Specification document● SVVP – Software Validation and Verification Plan● DRPF SW package● DRPF Use Cases SW Package● DRPF Cloud Binding SW Package● DRPF functional cloud instance● SUM – SW User Manual● TN – DPRF Design Document● ICD – DRPF Interface Control Document

Tasks:● Validation of the developed DRPF framework, use cases and cloud binding.● Verification of the developed DRPF framework, use cases and cloud binding.

Outputs: ● SVVR – Software Validation and Verification Report

__________________________________________________________________________

Identifier: WP600Title: Final AssessmentManager: Martin Pačes Company: ISSStart Date T0+19 Start Event: AR/M5End Date: T0+20 End Event: FP/M6Inputs:

● DRPF SW package● DRPF Use Cases SW Package● DRPF Cloud Binding SW Package● SUM – SW User Manual● TN – DPRF Design Document● ICD – DRPF Interface Control Document

Tasks:● Summarise the achievements of the project● Prepare the Final Presentation.● Prepare the final data package.

Outputs: ● FP – Final Presentation● DP – DRPF Data Package

5 PLANNING

Gantt Chart



Anticipated Key Personnel Allocation

Martin Pačes

as project manager 10%

as senior engineer 40%

In addition to the key personnel one SW Engineer shall be assigned to the project by ISS with 100% time allocation for the duration of the project.

Milestone, Meting, and Review Plan

date mile-stone

event description meeting location

T0 M0 KO Kick Off ESRIN / ESA

T0+1½ – PM1 Progress Meeting teleconference

T0+3 M1 SRR System Requirements Review ISS / Prague

T0+5 – PDR Preliminary Design Review teleconference

T0+6 M2 CDR Critical Design Review ESRIN / ESA

T0+8½ – PM2 Progress Meeting teleconference

T0+11 – PM3 Progress Meeting teleconference

T0+13 M3 PM4 Progress Meeting ISS / Prague

T0+14½ -- PM5 Progress Meeting teleconference

T0+16 M4 PM6 Progress Meeting ESRIN / ESA

T0+17½ -- PM7 Progress Meeting teleconference

T0+19 M5 AR Acceptance Review ISS / Prague

T0+20 M6 FP Final Presentation ESRIN / ESA

6 COST PRICE DATA

PSS Forms

PSS Forms A1, A2, and A8 are attached to the proposal document.

Exchange rate applied to the NC (CZK) is 22.42 CZK per Euro. Based on the long-term re-valuation trend of CZK we consider this exchange rate to be a reasonable middle rate estimation for the expected duration of the projects.

The details of the method of the CZK/EUR exchange rate development estimate is described in additional annex. In principle it is the extrapolation of the rate from last year over the duration of the project.

Cost Breakdown per Work Packages

WP Id WP title Price

WP100 Project Management 16,493 EUR

WP200 SW System Specification 25,515 EUR

WP300 SW Architectural Design 26,002 EUR

WP400 SW Detailed Design and Implementation 90,474 EUR

WP410 Core Framework Implementation 51,705 EUR



WP Id WP title Price

WP420 Use Case Implementation 19,385 EUR

WP430 Cloud Integration 19,385 EUR

WP500 SW Validation and Verification 27,627 EUR

WP600 Final Assessment 8,786 EUR

TOTAL 194,897 EUR

Milestone Payment Plan

PRIME CONTRACTOR MILESTONE PAYMENT PLANMilestone Description Scheduled

Dates% of Total Price

Payments from ESA to Prime Contractor

ADVANCE: Upon signature of the Contract by both parties.

T0 19.5% 38,000 EUR

PROGRESS: Upon successful M2/CDR completion and the Agency’s acceptance of planed deliverable items.

T0 + 6 30.3% 59,000 EUR

PROGRESS: Upon successful M3/PM4 completion and the Agency’s acceptance of planed deliverable items.

T0 + 13 30.3% 59,000 EUR

FINAL 1: Upon the Agency’s acceptance of all deliverable items due under the Contract and the Contractor’s fulfilment of all other contractual obligations, including performance of the Final Presentation.

T0 + 20 14.9% 29,000 EUR

FINAL 2: Upon successful completion of the 6 month guarantee period.

T0 + 26 5.1% 9,897 EUR

TOTAL 100% 194,897 EUR

Subcontractor Milestone Payment Plan

not applicable

7 TRAVEL AND SUBSISTENCE PLAN

Travels envisaged for the execution of the contract

WP id desti-nation

purpose com-pany

nr.persons

nr. days

subsist.allowance

travel cost

total cost

WP100 ESRIN KO/M0 ISS 1 2 200 600 800WP100 ESRIN CDR/M2 ISS 1 2 200 600 800WP100 ESRIN PM6/M4 ISS 1 2 200 600 800WP100 ESRIN FP/M6 ISS 1 2 200 600 800

Note: The subsistence allowance is the sum of the daily “catering” allowance stipulated for each country by the Czech labour law (e.g., 45 EUR for Italy) plus an estimated hotel rate of 55 EUR per night.

The flight and land transfers are estimated at 600 EUR per trip.



8 DELIVERABLES

List of Deliverables

deliverable type on event to be due

Progress Report Template PR KO/M0 T0

Software System Specification document SSS SSR/M1 T0+3

Software Validation and Verification Plan SVVP SSR/M1 T0+3

Selection of the Target Cloud Infrastructure TN SSR/M1 T0+3

DPRF Design Document – preliminary version TN PDR T0+5

DPRF Interface Control Document – preliminary version ICD PDR T0+5

DPRF Design Document TN CDR/M2 T0+6

DPRF Interface Control Document ICD CDR/M2 T0+6

SW User Manual – preliminary version SUM PM4/M3 T0+13

DPRF Design Document – update TN PM4/M3 T0+13

DPRF Interface Control Document - update ICD PM4/M3 T0+13

SW User Manual SUM PM6/M4 T0+16

DPRF Design Document – update TN PM6/M4 T0+16

DPRF Interface Control Document - update ICD PM6/M4 T0+16

DRPF Test Report TN PM6/M4 T0+16

Software Validation and Verification Report SVVR AR/M5 T0+19

Technical Data Package DP FP/M6 T0+20

Executive Summary TN FP/M6 T0+20

Final Presentation TN FP/M6 T0+20

Regular Monthly Reports PR monthly T0+i

Distributed Raster Processing Framework (DRPF) ISS2011/AO6647/3

Documents

Transcript of Distributed Raster Processing Framework (DRPF) ISS2011/AO6647/3