Service platform integration and deployment in cloud ... · 5.19 Geoportal ... Figure 13: OpenDJ...

This project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the

Competitiveness and Innovation Framework Programme by the European Commission under

grant agreement no. 621074

COMPETITIVENESS AND INNOVATION FRAMEWORK PROGRAMME

CIP-ICT-PSP-2013-7 Pilot Type B

WP3 – Service platform integration and deployment in cloud infrastructure

D3.6.3: Deployment and integration report

Deliverable Lead: PSNC

Deliverable due date: 28/02/2017

Actual submission date: 28/02/2017

Version: 1.5

D3.6.3 Deployment and integration report

http://www.foodie-project.eu Copyright © FOODIE Project Consortium. All Rights Reserved. Grant Agreement No.: 621074 Page:2 / 71

Document Control Page

Title Deployment and integration report

Creator Raul Palma (PSNC)

Description

This document describes the infrastructure that has been put in place for the development, deployment and integration of the different software components that will comprise the FOODIE platform, as well as the guidelines and procedures for using such infrastructure. In particular, this document includes: a description of the tools selected and configured to support the software development tasks; a set of guidelines and best practices for developers; a description of the current state of FOODIE cloud; the procedures for the software integration tasks; an overview of the components and base technologies which have been deployed in the cloud; the cloud domain and applications endpoints; the cloud security framework; and the procedures to be carried out for the delivery of resources and services with High Availability.

Publisher FOODIE Consortium

Contributors

Fryderyk Raczyk (PSNC) Marcin Wolski (PSNC) Michał Rej (PSNC) Mario Nuñez Jimenez (ATOS) Tomas Resnik (WRLS)

Creation date 01/02/2016

Type Text

Language en-GB

Rights copyright “FOODIE Consortium”

Audience internal public restricted

Review status

Draft WP leader accepted Technical Manager accepted Coordinator accepted

Action requested

to be revised by Partners for approval by the WP leader for approval by the Technical Committee for approval by the Project Coordinator

Requested deadline



Revision History

Version Date Modified by Comments

0.1 01/02/2015 Raul Palma Document creation

0.2 11/02/2015 Fryderyk Raczyk New content provided (Software Development, Software Integration, Cloud Infrastructure, Service Platform Components, High Availability)

0.3 19/02/2016 Mario Nuñez Jimenez Added section (Security Components)

0.8 24/02/2016 Raul Palma Final reading and edition

1.0 28/03/2016 Miguel Ángel Esbrí (ATOS) QA

1.1 19/01/2017 Mario Nuñez Jimenez (ATOS) Update security architecture and Rasdaman

1.2 02/02/2017 Raul Palma (PSNC) Update of Cloud infrastructure, marketplace and semantic services section

1.3 15/02/2017 Tomas Reznik (WRLS Update telemetry service

1.5 24/02/2017 Miguel Ángel Esbrí (ATOS) QA

Note

This deliverable is subject to final acceptance by the European Commission.

Disclaimer

The views represented in this document only reflect the views of the authors and not the views of the European Union. The European Union is not liable for any use that may be made of the information contained in this document.

Furthermore, the information is provided “as is” and no guarantee or warranty is given that the information is fit for any particular purpose. The user of the information uses it at its sole risk and liability.



Table of Contents

Note ........................................................................................................................................................................... 3

Disclaimer .................................................................................................................................................................. 3

Table of Contents ....................................................................................................................................................... 4

Glossary...................................................................................................................................................................... 9

Abbreviations and Acronyms ..................................................................................................................................... 10

Executive Summary ................................................................................................................................................... 11

1 Introduction ....................................................................................................................................................... 12

2 Software development ...................................................................................................................................... 14

2.1 Software development infrastructure .............................................................................................................. 14

2.1.1 Git ............................................................................................................................................................. 14

2.1.2 JIRA ........................................................................................................................................................... 15

2.1.3 FishEye ...................................................................................................................................................... 16

2.2 Software development guidelines .................................................................................................................... 17

2.2.1 Using source code repository ................................................................................................................... 17

2.2.2 Issue management guidelines .................................................................................................................. 18

2.2.3 Coding guidelines ...................................................................................................................................... 20

2.2.4 Working with Apache Maven ................................................................................................................... 22

2.2.5 Release management practices ................................................................................................................ 23

3 Cloud infrastructure ........................................................................................................................................... 25

3.1 Infrastructure as a service ................................................................................................................................ 26

3.2 Database as a Service ....................................................................................................................................... 31

3.2.1 Relational database specialized for geographical and spatial data .......................................................... 31

3.2.2 Semantic triplestore ................................................................................................................................. 33

3.3 Rasdaman ......................................................................................................................................................... 34

3.4 Additional storage infrastructure ..................................................................................................................... 34

4 Software integration .......................................................................................................................................... 35

4.1 VM provisioning ................................................................................................................................................ 35

4.1.1 General information ................................................................................................................................. 35

4.1.2 Requesting the new VM ........................................................................................................................... 35

4.1.3 Requesting maintenance of the VM ......................................................................................................... 36

4.2 Database access ................................................................................................................................................ 37

4.3 Deployment and integration of FOODIE platform components ....................................................................... 38

4.4 Administration and maintenance ..................................................................................................................... 38

5 Service platform components ............................................................................................................................ 39

5.1 Mapserver ......................................................................................................................................................... 40



5.2 Geoserver ......................................................................................................................................................... 40

5.3 Rasdaman Petascope add-on ........................................................................................................................... 41

5.4 Sensor networks components .......................................................................................................................... 41

5.5 Micka ................................................................................................................................................................ 42

5.6 Notification broker ........................................................................................................................................... 42

5.7 Spring integration framework .......................................................................................................................... 43

5.8 R suite ............................................................................................................................................................... 43

5.9 NETCAD suite .................................................................................................................................................... 43

5.10 Fleet Transport statistics service ...................................................................................................................... 44

5.11 Yield potential ................................................................................................................................................... 45

5.12 Pesticides recommender .................................................................................................................................. 45

5.13 Semantic annotation service ............................................................................................................................ 46

5.14 Silk framework .................................................................................................................................................. 46

5.15 Marketplace ...................................................................................................................................................... 47

5.16 Dashboard......................................................................................................................................................... 47

5.17 Moodle ............................................................................................................................................................. 47

5.18 Swagger ............................................................................................................................................................ 48

5.19 Geoportal .......................................................................................................................................................... 48

5.20 SmartV .............................................................................................................................................................. 48

5.21 IPM-DSS ............................................................................................................................................................ 48

6 FOODIE cloud access and security ...................................................................................................................... 49

6.1 Domain and application endpoints ................................................................................................................... 49

6.2 Security framework .......................................................................................................................................... 49

6.2.1 Security Architecture ................................................................................................................................ 50

6.2.2 LDAP Server – OpenDJ .............................................................................................................................. 51

6.2.3 IAM – OpenAM ......................................................................................................................................... 56

6.2.4 PA – OpenAM ........................................................................................................................................... 62

6.2.5 Policies definition ..................................................................................................................................... 65

7 High Availability ................................................................................................................................................. 67

7.1 Physical data redundancy ................................................................................................................................. 67

7.2 System monitoring ............................................................................................................................................ 67

7.3 Backup procedures ........................................................................................................................................... 67

7.3.1 System backups ........................................................................................................................................ 67

7.3.2 Database backups ..................................................................................................................................... 67

7.3.3 Other backups ........................................................................................................................................... 67

7.4 Service Level Agreement .................................................................................................................................. 67



7.4.1 General SLA for FOODIE assumptions ...................................................................................................... 68

7.4.2 Hardware infrastructure ........................................................................................................................... 68

7.4.3 Service....................................................................................................................................................... 68

7.4.4 Maintenance ............................................................................................................................................. 68

8 Conclusions ........................................................................................................................................................ 69

References ................................................................................................................................................................ 70



Index of Figures

Figure 1: Pulling from Git repository (TortoiseGit example) ............................................................................................ 15

Figure 2: JIRA Agile (Scrum) board example ..................................................................................................................... 16

Figure 3: JIRA Burndown (Scrum) Chart example ............................................................................................................. 16

Figure 4: FishEye exemplary view ..................................................................................................................................... 17

Figure 5: JIRA basic workflow ........................................................................................................................................... 19

Figure 6: FOODIE Cloud Architecture ............................................................................................................................... 25

Figure 7: FOODIE Database topology ............................................................................................................................... 32

Figure 8: Requesting new VM using JIRA .......................................................................................................................... 36

Figure 9: Visualization of mapserver configuration .......................................................................................................... 40

Figure 10: Visualization of geoserver configuration ......................................................................................................... 41

Figure 11: Visualization of Netcad configuration ............................................................................................................. 44

Figure 12: FOODIE cloud secure access description ......................................................................................................... 51

Figure 13: OpenDJ installation I ........................................................................................................................................ 52

Figure 14: OpenDJ installation II ....................................................................................................................................... 52

Figure 15: OpenDJ installation III ...................................................................................................................................... 53

Figure 16: OpenDJ installation IV...................................................................................................................................... 53

Figure 17: OpenDJ installation V....................................................................................................................................... 54

Figure 18: OpenDJ installation VI...................................................................................................................................... 54

Figure 19: OpenDJ installation VII..................................................................................................................................... 55

Figure 20: OpenDJ installation VIII.................................................................................................................................... 55

Figure 21: OpenDJ installation IX ...................................................................................................................................... 56

Figure 22: IAM installation ............................................................................................................................................... 57

Figure 23: IAM configuration I .......................................................................................................................................... 58

Figure 24: IAM configuration II ......................................................................................................................................... 58

Figure 25: IAM configuration III ........................................................................................................................................ 60

Figure 26: IAM configuration IV ........................................................................................................................................ 61

Figure 27: IAM configuration V ......................................................................................................................................... 61

Figure 28: IAM configuration VI ........................................................................................................................................ 62

Figure 29: PA configuration I ............................................................................................................................................ 63

Figure 30: PA configuration II ........................................................................................................................................... 63

Figure 31: PA configuration III .......................................................................................................................................... 64

Figure 32: PA configuration IV .......................................................................................................................................... 64

Figure 33: PA configuration V ........................................................................................................................................... 65

Figure 34: Policies definition I........................................................................................................................................... 65

Figure 35: Policies definition II.......................................................................................................................................... 66



Index of Tables

Table 1: Abbreviations and Acronyms .............................................................................................................................. 10

Table 2: Initial toolkit of software development infrastructure ....................................................................................... 14

Table 3: The most important Git’s terms.......................................................................................................................... 18

Table 4: Characteristics of the FOODIE instance of OpenStack ........................................................................................ 26

Table 5: Specification of the compute node ..................................................................................................................... 26

Table 6: Configuration of VM1 (foodie-vm1.man.poznan.pl) ........................................................................................... 27




Table 10: Configuration of VM5 (foodie-vm5.man.poznan.pl) ......................................................................................... 28





Table 15 Configuration of VM10 (foodie-vm10.man.poznan.pl) ...................................................................................... 30



Table 18 Postgres-XL configuration .................................................................................................................................. 32

Table 19 Characteristic of the nodes ................................................................................................................................ 32

Table 20 PostgreSQL configuration .................................................................................................................................. 33

Table 21 FOODIE’s OpenLink Virtuoso configuration ....................................................................................................... 33

Table 22 References ......................................................................................................................................................... 71



Glossary

The glossary of terms used in this deliverable can be found in the public document “FOODIE_Glossary.pdf” available at: http://www.foodie-project.eu

http://www.foodie-project.eu/



Abbreviations and Acronyms

Abbreviation / Acronym

Description

API Application Programming Interface

CPU Central Processing Unit

DBA Database Administrator

DBaaS Database as a Service

DDD Domain-Driven Design

DNS Domain Name System

GTM Global Transaction Manager

HA High Availability

HDD Hard Disk Drive

HTTP Hypertext Transfer Protocol

IaaS Infrastructure as a Service

IDE Integrated Development Environment

JAR Java ARchive

MPP Massive Parallel Processing

OS Operating System

POM Project Object Model

RAM Random Access Memory

RDBMS Relational Database Management System

RDF Resource Description Framework

SLA Service Level Agreement

SQL Structured Query Language

TCP Transmission Control Protocol

TDD Test – Driven Development

VM Virtual Machine

API Application Programming Interface

Table 1: Abbreviations and Acronyms



Executive Summary

This document describes the infrastructure that was provided for the development, deployment and integration of the different software components that will comprise the FOODIE platform, as well as the guidelines and procedures for using such infrastructure.

In particular, the document introduces the technologies that were selected (by FOODIE technical partners) and deployed in PSNC in order to support the tasks of distributed and collaborative software development. It also includes a set of complementary guidelines for software development that were prepared and shared with project partners, covering:

Source code repository usage

Issue management guidelines

Coding best practices

Build process tasks

Software release management

Additionally, the document presents the current and final state of the cloud infrastructure that was configured and deployed in PSNC in order to support the deployment and integration of FOODIE platform components. This presenta-tion also includes a description of how this infrastructure is used for the integration of software components, and an overview of the set of components and base technologies that were deployed. Furthermore, the document introduces FOODIE cloud domain and endpoints, followed by a detailed description of the security framework.

Finally, the document describes the procedures and tasks for the provision of services with High Availability, including a template for a Service Level Agreement that would enable in the future to expand FOODIE cloud by including addi-tional nodes without compromising the quality of service.

Further notice: D3.6.3 is a self-contained document that supersedes previous version (D3.6.1 and D3.6.2). Therefore, content from the previous document has been reused and updated according to the latest developments. The main novel contributions of this document include updated cloud domain and services endpoints section (Section 6.1), the cloud security framework (Section 6.2), as well as an updated description of the current state of the cloud infrastructure (Section 3) and the set of components and technologies deployed (Section 5).



1 Introduction

One of the key objectives of FOODIE was to build an open and interoperable platform hub on the cloud for the manage-ment of agriculture and farming related data, integrated from heterogeneous (public and private) sources, that would enable the provision of specific and high-value applications and services supporting the planning and decision-making processes of different stakeholders’ groups related to these domains.

This platform should integrate a set of re-usable components following a loosely coupled approach that would enable large possibilities for research, industry and SMEs actors in Europe to create and deploy new types of “Application Level” geospatial and agricultural services, as specified in D2.2.3 [00].

These components are based in most cases on existing solutions and base technologies, likely adapting and extending them, but in some cases it has been necessary to define and work on new domain-specific components for agriculture, according to the requirements analysis.

Hence, the implementation of FOODIE platform was a continuous process of integration, testing and deployment of selected technologies, extension modules, and other components developed in the technical tasks, which co-evolved with the specification of the platform architecture. Moreover, the implementation of such composite platform involved developers from different teams, residing in distributed geographical locations. In such distributed environment, soft-ware development tasks required appropriate, efficient but not hampering, coordination and support. Lack of coordi-nation and support could have result in ineffective usage of resources, multiplication of overlapping efforts and inade-quate accessibility of source code and documentation [08].

The first part of this document describes the software development infrastructure that we deployed in order to support the development tasks in FOODIE. The selection of the particular tools was made in agreement among the project tech-nical partners, taking into account their suitability to an agile software development methodology and to support the collaboration of a distributed developer teams, as well as the preferences and experiences of the partners with the tools. The infrastructure included Git for source code repository, Jira for issue management, Fisheye+Crucible for source code browsing and review, Jenkins for continuous integration, and Maven for the automatic build process.

Moreover, in order to unify and increase the quality of the software development tasks, we prepared a set of guidelines and best practices covering:

The usage of source code repositories

Issue management

Coding best practices

Build process tasks

Software release management

These guidelines were shared with the project partners since the start of the project through the project Wiki (link).

The second part of this document describes the final state of FOODIE cloud infrastructure, which was configured in PSNC in order to support the deployment and integration of FOODIE platform components. This infrastructure was imple-mented according to the architecture specification in D2.2.3 [00], that is, delivering two different service models: IaaS+ (IaaS+DbaaS) and SaaS, and deployed as a private cloud built on top of PSNC HPC resources. In this document, we de-scribe the components at each of these two cloud types, their configuration, and their final usage state. In particular, for the IaaS+ cloud, we describe the configuration of OpenStack for the provision of IaaS, as well as the relational data-base with spatial extension and Virtuoso for the provision of DbaaS, including the VMs that were provisioned to deploy platform components. Then we explain how these components were used for the integration tasks, and introduce the final set of components and base technologies, building blocks of the SaaS cloud, which have been deployed on top of the VMs. These include MapServer, GeoServer, Moodle, semantic annotation service, and others.

The third part of the document discusses the accessibility and security aspects of FOODIE cloud. During the previous period, a DNS domain has been registered for FOODIE cloud to provide homogeneous access to the services and appli-cations, and to foster sustainability of the platform beyond the project lifetime. This (unique) entry point also facilitates



the implementation of the cloud security framework described afterwards. In particular, this framework is based on two existing software components: OpenAM and OpenDj.

Finally, the fourth part of this document describes the procedures and tasks for the provision of services with High Availability (HA). In particular, we describe the backup procedures that are performed to avoid data losses. Additionally, we present a template for a Service Level Agreement, covering the general conditions based on the project assumptions, and the requirements for hardware, service and maintenance, which should be agreed between users and providers. In addition to gain confidence and reach the user's expectations, an SLA would enable in the future to expand FOODIE cloud by including additional nodes without compromising the quality of service.

The remainder of this document is organized as follows: Section 2 presents the software development infrastructure (2.1), guidelines and best practices (2.2); Section 3 describes the current state of FOODIE cloud infrastructure, particu-larly the IaaS (3.1) and DbaaS (3.2) components, and then Section 4 explains how to use these components for the software integration, including VM provisioning procedures (4.1), database access (4.2) and administration and mainte-nance (4.3); next, Section 5 introduces the current set of components and base technologies deployed for FOODIE plat-form; Section 6 describes the cloud access and security aspects, including the domain and applications endpoints (6.1), and a detailed description of the security framework (6.2); finally Section 7 describes the HA procedures, including the SLA template (7.2), and we conclude in Section 8.



2 Software development

High quality of the developed software is undoubtedly one of the most important factors of successful IT projects. Using common software development infrastructure and following shared guidelines and best practices is crucial in large and innovative projects where number of developers cooperate in distributed environment and increases the quality of the software development process.

The common software infrastructure supports not only developers but also project managers providing them with tools supporting work scheduling and development progress monitoring.

2.1 Software development infrastructure

Table 2 lists initial toolkit of the software development infrastructure.

Name of the tool Main purpose Address of the service

Git Source code management https://git.man.poznan.pl/stash/projects/FO

OD

JIRA Issue management https://jira.man.poznan.pl/jira/browse/FOOD

FishEye + Crucible

Read-only user friendly view of the source code aligned with peer code

reviewing tool.

https://fisheye.man.poznan.pl/fisheye/

Jenkins Continuous integration https://maven.man.poznan.pl/jenkins/view/F

OODIE/

Maven Build automation tool https://maven.man.poznan.pl/repository/

Table 2: Initial toolkit of software development infrastructure

During the course of the project, around 800 issues were created in Jira, nine repositories in Git, and three jobs in Jenkins. The following table summarises the issues created in Jira during the project lifetime. Note that most of the unresolved issues, were part of the backlog of desired functionalities. The critical (must/should) issues were in general resolved.

Period Created Issues (Unresolved) Created Issues (Resolved)

2014 18 7

2015 58 309

2016 98 300

2017 2 16

The software development guidelines and best practices refer to the abovementioned tools in order to provide better readability and maintainability of the software developed in the FOODIE project. Both tools and guidelines have a great impact on other aspects of the software development, i.e. methodology of work or quality assurance.

2.1.1 Git

Git is free and open source distributed version control system. In opposite to centralized version control systems it supports multiple workflows (creating local branches that are independent on each other) and local staging. Because there is no a central server, each developer possesses entire copy of the whole repository. Elimination of the single point of failure significantly increases the security of the repository. Local staging means that even different parts of a single file can be committed independently [13]. Since majority of operations is performed locally, Git is definitely faster than other centralized systems. Because of all its features, Git is not recommended for projects implemented by a single developer.

https://git.man.poznan.pl/stash/projects/FOOD


https://jira.man.poznan.pl/jira/browse/FOOD

https://fisheye.man.poznan.pl/fisheye/

https://maven.man.poznan.pl/jenkins/view/FOODIE/

https://maven.man.poznan.pl/jenkins/view/FOODIE/

https://maven.man.poznan.pl/repository/

https://jira.man.poznan.pl/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+FOOD+AND+created+%3E%3D+2014-01-01+AND+created+%3C%3D+%222014-12-31+23%3A59%22+AND+resolution+%3D+Unresolved

https://jira.man.poznan.pl/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+FOOD+AND+created+%3E%3D+2014-01-01+AND+created+%3C%3D+%222014-12-31+23%3A59%22+AND+NOT+resolution+%3D+Unresolved









Figure 1: Pulling from Git repository (TortoiseGit example)

Git repository of the FOODIE project is available at: https://git.man.poznan.pl/stash/projects/FOOD

2.1.2 JIRA

JIRA is a tool developed by Atlassian company. The core functionality of JIRA is related to issue tracking [14]. Other features combined with numerous add-ons extend the possibilities of JIRA so it can be used for example as a:

Project management tool

Source code review tool

Workflow tool

Task board

Test planner

Source of software development metrics

Release notes generator

JIRA reveals all its features when it is used regularly and consequently (each developer follows the agreed workflow). Recommended workflow and useful best practices can be found in following chapter.




Figure 2: JIRA Agile (Scrum) board example

Figure 3: JIRA Burndown (Scrum) Chart example

JIRA for the FOODIE project is available at: https://jira.man.poznan.pl/jira/browse/FOOD

2.1.3 FishEye

FishEye is a tool developed by Atlassian company. It provides a user-friendly interface to source code repository [15]. FishEye smoothly integrates with JIRA, so commits (new or changed code) related to particular tickets can be analysed individually. Source code metrics and activity charts and reports are features that support quality assurance and man-agement procedures. In order to integrate FishEye with JIRA, developers must provide appropriate comment when committing to the source code repository. The structure of the comment is described in software development guide-lines.

Crucible is a tool developed by Atlassian company. It supports lightweight, distributed source code reviewing by means of iterative reviews, threaded comments and inline discussions. It can be easily integrated with JIRA, so remarks and comments of the reviewers can by automatically turned into JIRA tickets. Regular source code reviews increase the quality of the final product and promote knowledge sharing among developers.

Crucible supports project managers and quality assurance managers by metrics of source code review coverage.

https://jira.man.poznan.pl/jira/browse/FOOD



Figure 4: FishEye exemplary view

FishEye and Crucible for FOODIE project are available at: https://fisheye.man.poznan.pl/fisheye

2.2 Software development guidelines

2.2.1 Using source code repository

Source code repository is the primary area of storage for the project artefacts and should contain:

Source code

Test scripts

Database scripts

Installation scripts

Project schemas

External libraries (if and only if Maven dependency management cannot handle them)

Developers must not commit to the repository the following artefacts:

Results of builds (.jar, .war files, logs)

External libraries that can be handled by Maven dependency management

IDE’s project and configuration files

Git version control system is very effective when working with independent branches. Branches are used to manage work that may be later merged back into the main trunk or originating branch. There are two basic scenarios for using branches:

Feature branches

Bugfix branches

The first scenario should be applied when particular feature that is being implemented, may not be completed before the next release. In such a case, name of the branch should start with word “feature” and should be suffixed with identifier of the JIRA ticket that relates to the implemented functionality (e.g. “feature_FOODIE-123”). The appliance of the latter scenario requires creating new branch as a result of each new release. Such a branch contains stable snapshot of a software at the time of release and is a base for further maintenance releases. Bugfix branches names should start with word “bugfix” and should be suffixed with the particular version number associated with release (e.g. “bugfix_2_1”).

https://fisheye.man.poznan.pl/fisheye



When particular snapshot is decided to be referential (thus non-editable) it should be marked with tag. Tags are used to mark specific releases or phases of development, in order to be able to retrieve and inspect a past release exactly as it was. Unlike branches, tags do not impose any maintenance costs, so these should be used liberally in order to track the release history.

The following table (Table 3) lists and explains the most important terms related to Git [16].

Term Explanation

Blame Describes the last modification to each line of a file: displays the revision, author and time.

Branch It is a parallel version of a repository that is contained within the repository, but does not affect the primary or master branch allowing you to work freely without disrupting the "live" version. When you've made the changes you want to make, you can merge your branch back into the master branch to publish your changes

Clone

It is a copy of a repository that lives on your computer instead of on a website's server somewhere, or the act of making that copy. With your clone, you can edit the files in your preferred editor and use Git to keep track of your changes without having to be online. It is, however, connected to the remote version so that changes can be synced between the two. .

Commit It is an individual change to a file (files). Each commit effects in creation of a unique ID (or "hash") that allows to keep record of what changes were made when and by who. Commits usually contain a commit message which is a brief description of what changes were made. It is also known as a “revision”.

Diff It is the difference in changes between two commits, or saved changes.

Fetch It means getting the latest changes from an online repository without merging them in. Once these changes are fetched you can compare them to your local branches (the code residing on your local machine).

Fork

It is a personal copy of another user's repository that lives on your account. Forks allow you to freely make changes to a project without affecting the original. Forks remain attached to the original, allowing you to submit a pull request to the original's author to update with your changes. You can also keep your fork up to date by pulling in updates from the original.

Merge It means taking the changes from one branch (in the same repository or from a fork), and applying them into another one.

Pull It means fetching in and merging the changes.

Push It refers to sending the committed changes to a remote repository so others can access them.

Remote It is a version that is hosted on a server and can be connected to local clones, so all changes can be synchronized.

Repository It is the most basic element of a Git. It contains all of the project files (including documentation), and stores each file's revision history.

Upstream It is the primary branch on the original repository is often referred to as the "upstream", since that is the main place that other changes will come in from. The branch/fork you are working on is then called the "downstream".

Table 3: The most important Git’s terms

2.2.2 Issue management guidelines

The core functionality of JIRA is related to issue tracking. In order to maximise its usefulness JIRA ought to be used regularly (on daily basis) and consequently (each developer should follow the agreed workflow). Following picture (Figure 5) presents the basic (yet recommended) issue lifecycle [17].



Figure 5: JIRA basic workflow

An issue can exist in one status only at any point in time. In the basic workflow, there are 5 statuses:

Open – by default, each newly created issue is Open. Issues related to new features should be created by a person responsible for development of given module. Bugs can be reported by anyone. As soon as possible the issue should be assigned to particular developer who will be responsible for its resolution.

In progress – when developer starts to work on the issue, he should change its status to “In progress”.

Resolved – once the resolution is found or developed, the issue can be resolved. Developers should provide a reference to unit test or document describing the testing scenario (this should be stored in the project source code repository). This information will be the base to verify whether implemented features works properly during the preparation of release. The assigned developer resolves the issue and indicates status of the reso-lution. JIRA provides a predefined list of statuses [18]:

o Fixed – a fix for this issue has been implemented and committed o Won’t fix – this issue will never be fixed (relevant comment is crucial) o Duplicate – this issue is duplicated (identifier of the original ticket should be provided) o Incomplete – there is not enough information to work on this issue (appropriate comment on what is

missing should be provided) o Cannot reproduce – the issue could not be reproduced in particular environment o Invalid – the request is not relevant to the project o Not a bug – the requested issue is not a bug but a designed feature o Will not be fixed – this issue will never be fixed (relevant comment is crucial). In comparison to won’t

fix, this status may have different origin (e.g. business or analysis) o Done – the resolution had already been implemented

Closed – if the resolution meets requester’s expectations, he should close the issue. In case of bugs, the issue can be closed by tester who is responsible for verification of the correctness of the implementation.

Reopened – if the resolution does not meet requester’s expectations, he should reopen the issue. In case of bugs, the issue can be reopened by tester who is responsible for verification of the correctness of the imple-mentation.

When working with JIRA one has to remember that:

descriptions should be clear and precise,

required additional materials (e.g. logs, screenshots, database dumps) are provided,

one issue is created per one bug,

no bug is too trivial to report,

before starting any code changes, the corresponding JIRA ticket needs to be created.



2.2.3 Coding guidelines

Source code is written only once, but us read numerous times. Thus, in order to provide readability and maintainability of the source code, developers should follow coding guidelines described in this chapter.

2.2.3.1 General rules All FOODIE sources (source code, comments and configuration) should be in-line with following recommendations:

Use English - since the FOODIE project will be used and maintained in multilingual context, all sources (imple-mentation, comments and configuration) must be in English;

Avoid hacks - developers should avoid hacks and unclear and unnecessary optimizations. If there are specific (functional or non-functional) requirements justifying these unobvious optimizations, these should be refer-enced in source code comments;

Apply TDD – Test-Driven Development (TDD) assumes that before any business logic is implemented, all tests are already developed. Using TDD increases the quality of the implementation and limits the overall number of the lines of code (not tested, thus unneeded, functionality is not implemented);

Write integration tests – follow your implementation with automated integration tests that can be included in standard maven lifecycle. Automated integration tests, combined with unit tests and continuous integration mechanisms, significantly increase the quality of the project (by testing if independent modules integrate with each other) and provide verification of regression;

Do not commit when it does not compile – committing the source code that does not compile is not only annoying to other developers, but also seriously disturbs continuous integration mechanisms;

Do not copy and paste code – when you identify source code that must be copied and pasted to be reused in another place, it means that re-factorization is needed;

Rely on Maven – rely on Maven mechanisms when external dependencies are required. Use stable versions pulled from official repositories. Avoid attaching JARs manually;

Use @Author tag – in all files stored in repository

Each class in a separate file – avoid creating files with definitions of numerous classes. Each class should be placed in its own separate file, named after the class. You should also avoid using white spaces in the names of all files.

Use ASCII codes - all source code files should be encoded using only ASCII (Latin-1) characters (if necessary use native2ascii for Java classes).

2.2.3.2 Assure code readability Simplicity of the source code is not a trivial issue. First of all, it requires developers to follow the agreed rules and practices. It also requires code reviews and regular refactoring, so one can be sure, that readability of the source code is continuously taken care of. The reasonable basic rules to be followed by developers are:

Java Code Convention [19] – for Java Developers

Ruby Style Guidelines [20] – for Ruby developers

Following list contains the most important recommendations to be applied in the project:

Apply DDD – Domain-Driven Design assumes that both terminology and classes hierarchy reflect the reality;

Avoid abbreviations – do not hesitate to use long names of variables if this increases the level of clarity;

Avoid negations in variables’ names – always double check if negations in variables names are necessary;

Avoid complex conditional expressions – presence of complex if statements suggest the need of redesign or re-factorization

Avoid magic numbers – use named constants instead;

Use left-hand comparison style – for constants or expressions to avoid null-pointer exceptions;

Use code formatters – before committing code into source code repository;

Use analysing tools – in order to receive hints and recommendations related to the quality of the source code (e.g. Sonar / PMD / Findbugs);

Follow order of declarations – order of declarations in each file should be the same:



o Class or interface statement o Class variables

Public Protected Package Private

o Instance variables Public Protected Package Private

o Constructors o Methods

2.2.3.3 Logging and exception handling It is recommended to use appropriate logging facility (e.g. log4j or java.util.logging) in order to investigate potential issues and to track back application’s behaviour. Developers are recommended to apply following general rules:

Use meaningful exceptions – exceptions with meaningful names and messages should be thrown. Developers must not return NULL values as an indicator of an error or incorrect behaviour;

Catch specific exceptions – avoid catching general exceptions (e.g. java.lang.Exception) and handle them ap-propriately;

Do not double log entries – catch block should not both log error message and throw another one;

Do not use System.out and System.err – developers must not use standard system outputs, especially in web applications;

Use single message – when logging, all related data should be logged in a single message;

Provide date / time of occurrence – log needs to contain precise information about date and time of the event;

Log reasonable – avoid logging at frequently or iteratively executed spots. Log relevant and meaningful infor-mation only.

All logging systems introduce several levels of logging. It is important to understand the meaning of each of those levels. The most important are described below:

Fatal - serious errors that are likely to lead to premature application termination, e.g., lack of memory, lost database connection;

Error – errors in the regular operation of the system which prevent normal program execution but are likely to allow the application to continue running;

Warn – should be used to record transient problems or undesirable, unexpected or potentially harmful situa-tions that are not necessarily errors;

Info - describes interesting runtime events, like startup, shutdown, receipt of user request. These messages should inform on the progress of the application at coarse-grained level during its normal operation;

Debug - these messages are produced by code added to debug an application for tracing its flow or internal data.

2.2.3.4 Comments It is recommended to provide compact yet meaningful documentation (in a form of Javadoc comments) for all public classes and public and protected methods. Code comments should be used to provide an overview of the code and additional information that is not readily or available in the code itself.

Developers are recommended to apply following best practices when providing comments:

Use English - since the FOODIE project will be used and maintained in multilingual context, all sources (imple-mentation, comments and configuration) must be in English;

Think ahead – avoid comments that are likely to become outdated when the code evolves;



Remove unneeded comments – if a comment no longer applies, you should modify or remove it;

Code meaningfully – it is always better to provide meaningful source code then extensive comment to unclear and tricky implementation;

Remove “commented out” implementation – all source code that has been “commented out” should be re-moved from revisions that will be transitioned into operational versions;

Provide formally required comments – e.g. copyright header, author’s name, etc.

2.2.4 Working with Apache Maven

Apache Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information.

The pom.xml file is the core of a project's configuration in Maven. It is a single configuration file that contains the ma-jority of information required to build a project in just the way you want. The POM is huge and can be daunting in its complexity, but it is not necessary to understand all of the intricacies just yet to use it effectively [21]

In the most basic form, a POM file has the following structure [22]:

Listing 1: POM structure – basic elements

The following list contains explanation of POM’s elements:

Project – this is top-level element in all maven pom.xml files;

modelVersion – indicates which version of the object model this POM is using. The version of the model itself changes rarely but it is mandatory in order to ensure stability of use if and when the Maven developers deem it necessary to change the model;

groupId – indicates the unique identifier of the organization or group that created the project;

artifactId – indicates the unique base name of the primary artifact being generated by this project. The primary artifact for a project is typically a JAR file. Secondary artifacts like source bundles also use the artifactId as part of their final name. A typical artifact produced would have the form <artifactId>-<version>.<extension> (for example, myapp-1.0.jar);

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns:xsi="http://www.w3.org/2001/XML-Schema-instance" xmlns="http://ma-

ven.apache.org/POM/4.0"

xsi:schemaLocation="http://maven.apache.org/POM/4.0 http://maven.apache.org/maven-

v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>org.apache</groupId>

<artifactId>maven</artifactId>

<packaging>jar</packaging>

<version>2.0</version>

<name>Maven 2.0</name>

<url>http://maven.apache.org</url>

</project>



packaging – indicates the package type to be used by this artifact (e.g. JAR, WAR, EAR, etc.). This not only means if the artifact produced is JAR, WAR, or EAR but can also indicate a specific lifecycle to use as part of the build process;

version – indicates the version of the artifact generated by the project;

name – indicates the display name used for the project;

url – indicates where the project's site can be found;

description - provides a basic description of the project.

POM file can be very complicated, since in can provide a lot of information. For clarification purposes it can be assumed, that the whole file can be decomposed into three different sections:

project – related properties – provides information that precisely identify the project (also about developers, organisation, collaboration tools, etc.);

process – related properties – defines how the build is performed (including definitions of plugins supporting different lifecycle phases and dependencies to be resolved by maven)

environment – related properties – stores information about locations of build artifacts and dependencies of various types

Following list contains maven’s default lifecycle phases:

Validate – validate the project is correct and all necessary information is available;

Compile – compile the source code of the project;

Test – test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed;

Package - take the compiled code and package it in its distributable format, such as a JAR;

Integration-test - process and deploy the package if necessary into an environment where integration tests can be run;

Verify - run any checks to verify the package is valid and meets quality criteria;

Install - install the package into the local repository, for use as a dependency in other projects locally;

Deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.

2.2.5 Release management practices

Software release is the distribution of a software product, and it usually includes not only the executable program, but also software code, documentation, and support materials, e.g., release notes, configuration data.

The adoption of an agile software development methodology, as it is planned in FOODIE, usually increases the number of release events. Hence, in FOODIE it is planned to have many internal or minor releases, and at least two major re-leases, one for each project development phase. Thanks to usage of tools like JIRA and Maven it is possible to keep cost of release management on a very low level.

Detailed procedure of FOODIE software releases heavily depends on the architecture definition, because this will deter-mine which artefacts will be produced as an outcome of FOODIE development.

However, software release procedure should be associated with general procedures that ensure that the developed product executes what it is supposed to do. All found bugs should be fixed and changes related to them should be committed to the source code repository. The next step is to create tags and branches which will reflect the state of the code at the time of the release.

When the source code is ready, a person responsible for releases (Release Manager) should build the final project arte-facts. This can be done manually or using continuous integration server. When creation of artifacts is finished, software should be deployed into a copy of the production environment. Information from issue tracker should be used to create the list of features implemented in a given release. A dedicated person or a group of people should verify the software quality using the list of implemented features and testing procedures provided by developers in the description of the



JIRA issue. If any bugs are identified, changes should be committed into source code repository and the whole procedure of building and verification of release should be repeated. When previous steps are completed, all project artifacts should be placed in all required locations (including for instance local mirrors of Maven repositories) and, in case of public releases, the project website. After successful deployment of the release, appropriate configuration files (e.g. POM files) should be updated in order to reflect that the team moves forward to work on the next release.

Following list contains best practices related to release management [23]:

Two weeks before a release preparation, a notification must be made through the FOODIE developers list about the planned code freeze. The actual release date would include additional time for testing, preparation and any last-minute changes, at least a week later.

After these 2 weeks, the code freeze is announced.

The Release Manager (i.e., the person responsible for routine maintenance and processing a release) checks out the entire source tree from the relevant build project.

The source tree is branched according to the release version: o The Release Manager, working on the new branch, updates the dependencies by removing the snap-

shot versions. o Modules which are not updated for this release, are instead pulled in from their previously tagged/de-

ployed versions. o The Release Manager makes sure the code builds and runs locally. o Then the new POMs, and other files such as plugin.xml files, are now committed. The Release Manager

makes a note of the commit changes (using VCS logs, automated emails, etc.).

A build is made for this particular version, checking out from the branch. This build forms the first internal release candidate which is distributed for testing.

o At this point any bug-fixes or last minute tweaks are applied to the trunk, and merged across to the release branch.

o After a release is prepared, and packaged, it is handed to the tester users.

When a release is ready and sent out, the Release Manager does a tag on the branch to mark the release.

The build is modified to build from the release tag - this forms a golden master (i.e., a stable-versioned tagged version).

The installers are created and uploaded from the golden master built.



3 Cloud infrastructure

FOODIE cloud was deployed and configured according to the architecture specified in D2.2.3 [00]. In summary, it con-sists of two types of clouds, delivering different service models, as depicted in Figure 6. The first one, called the devel-opers cloud, delivers an Infrastructure as a Service (IaaS) with the addition of a Database as a Service component, which combined deliver an IaaS+ solution. This cloud is aimed for FOODIE partners, and in the future potentially for other application providers, to develop and deploy their software components in FOODIE platform. The second cloud, called the end-users cloud, delivers a Software as a Service (SaaS). It comprises all the software components developed by FOODIE partners and delivered as services to end-users, i.e., farmers and other stakeholders.

FOODIE cloud has been initially deployed as a private cloud built on top of PSNC HPC resources.

Figure 6: FOODIE Cloud Architecture

In the remainder of this section we describe the two core components of the developers’ cloud: IaaS and DbaaS, and in section 4 two we explain how these components are used for the integration of software components. After that in section 5, we introduce the initial set of components deployed in FOODIE cloud, representing the building blocks for the end-user cloud.



3.1 Infrastructure as a service

Infrastructure as a Service (IaaS) in FOODIE is based on OpenStack. OpenStack is an open source software for creating private and public clouds [09]. The characteristic of the FOODIE instance of OpenStack is presented in Table 4.

Characteristic Value

Distribution Mitaka

Compute nodes 16

Controller node 2

Network node 2

Cinder node 2 x Ceph storage

Compute node specification

CPU: 2x E5 2670 v2 / 2697 v3 @ 2.6GHz

RAM: 256GB

storage: 40TB connected via Fibre Channel, 50 TB Ceph via 10Gbit eth

Table 4: Characteristics of the FOODIE instance of OpenStack

Table 5 presents specification of the compute node.


CPU 2x E5 2670 v2 / 2697 v3 @ 2.6GHz

RAM 256 GB

Storage 40TB connected via Fibre Channel, 50 TB Ceph via 10Gbit eth

Table 5: Specification of the compute node

Following services have been installed on the FOODIE instance of the OpenStack:

Keystone – OpenStack Identity Service

Nova – OpenStack Compute Service

Cinder – OpenStack Block Storage Service

Horizon – OpenStack Dashboard Service

Neutron – OpenStack Network Service

Glance – OpenStack Image Service

Currently there are twelve VMs with the characteristics specified in tables: Table 6, Table 7, Table 8, Table 9, Table 10, Table 11, Table 12, Table 13, Table 14 and Table 15.


CPU 8 Intel class core x64

RAM 13GB

HDD 250GB + 345GB (external volume)

Operating system Ubuntu Linux 14.04, Kernel version: 3.13.0-24, libc6 version: 2.19

Network configuration external interface (configured on OpenStack): 62.3.168.78




DNS name: foodie-vm1.man.poznan.pl

Purpose DbaaS node (including semantic triplestore), GIS application server node,

Virtuoso server

Main services/applications Postgres-XL (data node) for experimentation, Virtuoso, Mapserver, Geoserver

Table 6: Configuration of VM1 (foodie-vm1.man.poznan.pl)



RAM 13GB

HDD 250GB +345GB (external volume)




Purpose DbaaS node, GIS application server node

Main services/applications Postgres-XL (data node) for experimentation, Mapserver, Geoserver, IPM-DSS




RAM 8GB

HDD 250GB




Purpose DbaaS node, application server

Main services/applications Postgres-XL coordinator (access node) for experimentation, PostgreSQL, Semantic Annotation service, Moodle, Swagger, Marketplace, Pesticides

Recommender, SILK




RAM 13GB

HDD 250GB




Operating system CentOS 7.0.1406



Purpose R application server

Main services/applications R environment, Rstudio desktop, Rstudio server




RAM 8GB

HDD 250GB

Operating system Windows Server 2012 R2 Standard



Purpose Netcad application server

Main services/applications Netcad GIS, Netgis server, Netcad base




RAM 13GB

HDD 25GB + 250GB (external volume)

Operating system Ubuntu Linux 14.05 LTS



Purpose Storage node, data harvester

Main services/applications Satellite imagery storage, SFTP, Policy Agent 2 (PA)




RAM 4GB

HDD 25GB




Operating system Ubuntu Linux 14.04



Purpose Endpoint for https://foodie-cloud.org, security server

Main services/applications Apache web server, Security components (openAM, openDJ), Policy Agent 1

(PA), Postfix mail server, proxy




RAM 4GB

HDD 25GB




Purpose Notification server

Main services/applications Notification Broker




RAM 4GB

HDD 50GB




Purpose Dashboard application server

Main services/applications SmartV (Widgets Dashboard), Spring, RabbitMQ




RAM 4GB

HDD 150GB




Operating system Debian 8



Purpose Geoportal, catalogue

Main services/applications Geoportal, catalogue (micka), senslog, maplog

Table 15 Configuration of VM10 (foodie-vm10.man.poznan.pl)



RAM 16GB

HDD 250GB




Purpose Rasdaman server

Main services/applications Rasdaman with a WCS and WMS services, satellite imagery harvester and

processing scripts (landsat8)




RAM 26GB

HDD 250GB




Purpose Rasdaman server

Main services/applications Rasdaman with a WCS and WMS services, satellite imagery harvester and

processing scripts (Sentinel2)




3.2 Database as a Service

This component delivers scalable and reliable Cloud Database as a Service (DBaaS) provisioning functionality for both relational and non-relational database engines. In particular, the DBaaS model provides database functionality on the basis of well-known database systems [10].

This component runs on top of the IaaS infrastructure, thus delivering in combination an IaaS+ solution. However, given the particular FOODIE requirements, this component is not based on the DbaaS component of Open-Stack. Instead, it comprises specialized database engines for managing the key data types to be used in the project, including at the moment, geographical and spatial data, and RDF data, while satellite imagery data is stored directly on the file system.

3.2.1 Relational database specialized for geographical and spatial data

In FOODIE, DBaaS for geographical and spatial data is delivered by Postgres. Originally, we selected and deployed Post-gres-XL, a scalable open source PostgreSQL-based database cluster [11], but after starting using it in production mode, we received several complains from the users that the database is not behaving properly and we also found some sta-bility issues. We even tried compiling the latest sources of Postgres-XL available at sourceforge1, but didn’t solve all the issues. Hence, we decided to wait for the next stable release, and in the meantime, we are using PostgreSQL for pro-duction. In the reminder of this section we describe both the Postgres-XL (now only for experimentation), and the Post-greSQL installations

3.2.1.1 Postgres-XL

Postgres-XL provides the following features:

Fully cluster-wide ACID support

o Atomicity – it means that in an atomic transaction a series of database operations either ALL occur or NOTHING occurs. The series of operations cannot be divided apart and executed partially from each other.

o Consistency – it means that any database transaction must only change affected data in allowed ways. Any data written to the database must be valid according to all defined rules.

o Isolation – it defines when the data changes made by one operation becomes visible to others.

o Durability – it means that transactions that have committed will survive permanently (even if the sys-tem crashes).

Cluster-wide consistency – the Global Transaction Monitor (GTM) is responsible for issuing transactions iden-tifiers and snapshots as part of Multi-Version Concurrency Control. The Coordinator manages the user sessions and interacts with GTM and the data nodes. The Coordinator parses and plans queries, and sends down a serialized global plan to each of the components involved in a statement. Actual data is stored in nodes (Data Nodes). The distribution of the data can be configured by the database administrator (DBA).

Multi-tenant security – multi-tenancy is an architecture in which a single instance of software application serves multiple consumers (tenants). Tenants are allowed to customize some parts of the application, but their adjustments are not visible to other consumers.

Massive Parallel Processing – MPP is an architecture with large number of processors performing a set of coordinated computations in parallel.

Postgres-XL installation in FOODIE cloud has the topology depicted in Figure 7. It consists of one access node, and two data nodes. Connections to the database can be established only by using the coordinator port on the access node (Node 3).

1 https://sourceforge.net/p/postgres-xl/postgres-xl/ci/master/tree/

https://sourceforge.net/p/postgres-xl/postgres-xl/ci/master/tree/



Figure 7: FOODIE Database topology

The latest information about Postgres-XL installation is summarized in Table 18 Postgres-XL configuration

, while he individual configuration of the nodes is provided in Table 19 Characteristic of the nodes

.


Configuration 2 data nodes + 1 access node

Access port 5432

Postgres-XL version 9.2 Release Candidate (based on PostgreSQL 9.4.0)

Postgis version 2.0.6

Total cluster capacity Up to 450 GB

Table 18 Postgres-XL configuration

Node Characteristic

Node 3 (access node)

IP 62.3.168.80 (DNS name: foodie-vm3.man.poznan.pl)

Coordinator (responsible for access to cluster by clients, TCP port: 5432)

Global Transaction Manager (responsible for transaction management, not accessible directly to clients, TCP port 6668)

Node 2 (data node)


Coordinator (responsible for access to cluster, TCP port: 5432)

Datanode (not accessible directly to clients, TCP port 5433)

Node 1 (data node)


Coordinator (not accessible directly to clients, TCP port: 5432)

Datanode (not accessible directly to clients, TCP port 5433)

Table 19 Characteristic of the nodes



3.2.1.2 PostgreSQL

PostgreSQL is a powerful, open source object-relational database system (ORDMS). Its main features include:

Fully ACID compliant

Data integrity features including (compound) primary keys, foreign keys with restricting and cascading up-dates/deletes, check constraints, unique constraints, and not null constraints.

SQL implementation conforming to the ANSI-SQL:2008 standard. It has full support for subqueries, read-com-mitted and serializable transaction isolation levels. It supports most SQL:2008 datatypes, and the storage of binary large objects, including pictures, sounds, or video.

Support for compound, unique, partial, and functional indexes which can use any of its B-tree, R-tree, hash, or GiST storage methods.

Table inheritance, rules systems, and database events.

Native programming interfaces for the most popular programming languages

Advanced features such as Multi-Version Concurrency Control (MVCC), point in time recovery, tablespaces, asynchronous replication, nested transactions (savepoints), online/hot backups, a sophisticated query plan-ner/optimizer, and write ahead logging for fault tolerance.

Additionally, PostgreSQL can be configured in High Availability mode, where multiple database servers can work to-gether to allow, for instance, a second server to take over quickly if the primary server fails (high availability), or to allow several computers to serve the same data (load balancing).

The current installation of PostgreSQL is running in one node (as summarized in Table 20 PostgreSQL configuration

), and we are currently working on configuring the High Availability mode.


Configuration One node (High Availability will be available soon)

Node and port IP 62.3.168.80 (DNS name: foodie-vm3.man.poznan.pl) Port: 5432

PostgreSQL version 9.5.1

Postgis version 2.2.1

Total cluster capacity Up to 450 GB

Table 20 PostgreSQL configuration

3.2.2 Semantic triplestore

The OpenLink Virtuoso (open source edition of Virtuoso Universal Server) is a hybrid product that combines the func-tionality of RDBMS, ORDBMS, XML, RDF, web application server and content management tool [12]. In FOODIE, we are mainly interested in the RDF store for the provisioning of a semantic database (triplestore) as a service for the storage of semantic annotations and other RDF data, as well as for their publication as linked data. The current FOODIE’s OpenLink Virtuoso instance is configured as described in Table 21 FOODIE’s OpenLink Virtuoso configuration.


Configuration Single-node configuration

Access node IP 150.254.165.102 (DNS: actinidia.man.poznan.pl, foodie-vm1.man.poznan.pl)

Version Open-source edition 6.1.6

Access ports SQL port: 1111

HTTP port: 8890

Table 21 FOODIE’s OpenLink Virtuoso configuration



3.3 Rasdaman

There are 2 instances of Rasdaman (version 9.3.1) installed at VM11 and VM12 that can be accessed through the proxy in VM7:

https://www.foodie-cloud.org/rasdaman/ows

https://www.foodie-cloud.org/rasdaman2/ows

Rasdaman is third party software providing Array DBMS functionality allowing storage of multi-dimensional arrays. For FOODIE purposes, it provides storage for raster imagery data with time dimension of satellite imagery from Landsat8 and Sentinel2.

Rasdaman deploys several components:

Rasmgr is the central Rasdaman request dispatcher.

Rascontrol allows to interactively controlling Rasdaman server.

Rasspasswd allows to administrate Rasdaman database user logins.

RasServer is the Rasdaman server engine.

RasBase is the array database where imagery is stored.

Rasdl and Rasql are command line tools to maintain relational database schema and to query relational data-base data, respectively.

PostgreSQL is a 3rd party relational database server that Rasdaman uses to store Petascope metadata.

Petascope is a component implementing OGC interface standards WCS, WPS and WMS. It maintains additional metadata in a separate PostgreSQL relational database.

Geo-Services is a set of OGC compliant services as Web Feature Service (WFS), Web Map Service (WMS) and Web Processing Service (WPS). They work as Java servlets under Tomcat.

SECORE is a Semantic Coordinate Reference System Resolver, identical to the one deployed online by OGC. It works under Tomcat. It is a server which resolves CRS URLs into full CRS definitions represented in GML 3.2.1. SECORE stores and queries XML data in a BaseX XML database.

3.4 Additional storage infrastructure

The approach for the storage and management of satellite imagery relies on a couple of small bash scripts configured with Ubuntu Cron to run periodically. These scripts use a small file, currently with all the path rows for the Spanish, Polish, Turkish, Czech and Latvian pilots, and using landsat-tools and PEPS utilities, they search for each of the path rows for Landsat8 and Sentinel2 satellite imagery results. Every match results on the download of all the bands rasters, and additionally an RGB raster is generated by combining some of the bands through a GDAL library command. Then the scripts by using GDAL, transforms each raster to a standard coordinate reference system (EPSG:4326), and are tempo-rary stored on the same folder until all are converted and then pushed into Rasdaman, then the complete folder is deleted. The storage space for these images is under a drive attached to VM11 and VM12 (see Tables 16 and 17). A more detailed description of this approach is available in D3.1.3[01].

Additionally, we have configured an sftp server for the storage of resources necessary for the partners to carry out their tasks. Each technical partner has login and password to access sftp server which has been provided individually (atos, seresco, netcad, ctic, wrls, progis and psnc). Each technical partner has its own directory name after their login where they can read/write data and also, they have access to a shared PUBLIC directory accessible to everybody.





4 Software integration

4.1 VM provisioning

We introduced a JIRA-based approach for the provision of Virtual Machines. Note that, instead of providing access to OpenStack dashboard to FOODIE IaaS users to create their own VMs, we followed a more centralised approach for the VM provisioning because of the following reasons: (i) the number of VM to be deployed is not significantly high. Most of the VMs needed by partners have been already deployed. Future external parties that may want to integrate their applications will need to be evaluated and assigned resources based on the business model adopted; (ii) most of the partners do not know the process for the creation and configuration of a VM in OpenStack, and if they will need to do it only once or twice, it would not worth for them to spend important efforts in learning such process.

4.1.1 General information

The process of requesting a virtual machine (VM) is managed and maintained by the Help Desk and is based on a JIRA project. In particular, the Help Desk service enables the following actions:

Creating a Virtual Machine with well-defined capacity and features.

Running the VM in the requested time frame.

Suspending the VM outside a requested time frame.

Closing or deleting a VM.

Creating a snapshot of a VM in a specific state.

Loading/reloading a VM with a new instance of an operating system.

Loading/reloading a VM from a specific snapshot.

Re-assigning resources to a VM, if such resources are available.

It is assumed that the Help Desk provides facilities for managing the VM at the level of virtual machines. All aspects related to the maintenance of the particular operating system running on the assigned VM are handled by an appointed member of a specific development team who is given operating system root level privileges.

4.1.2 Requesting the new VM

In order to request a VM, one should create a JIRA ticket (using dedicated JIRA project) and specify following details:

Name, surname and affiliation

Processor assignment

RAM memory assignment

HDD space

OS Type

User to be assigned OS root privileges

To request an action to be carried out on an existing VM (e.g. a maintenance action), its necessary to modify this VM's issue by creating sub-tasks for it. The sub-tasks cover the following actions:

Requesting a VM snapshot.

Requesting loading/reloading of a VM with a snapshot or fresh image.

Changing resource assignment for a particular VM (GHz, RAM, HDD).

Requesting a VM to be suspended/closed.



Figure 8: Requesting new VM using JIRA

It is recommended to provide the description of the requested action (i.e. the text of one of the bullets listed above) as sub-task summary. When a VM is closed, its corresponding issue is closed in JIRA.

4.1.3 Requesting maintenance of the VM

4.1.3.1 Requesting snapshot of the VM In order to request a snapshot of a particular VM one should create a subtask for an issue that is related to this particular VM. The procedure is as follows:

1. Browse the submitted issues (or use search form) until you find an issue that corresponds to the VM you want to take the snapshot of;

2. Create sub-task (e.g. using More Actions menu) for the issue; 3. Set the issue type to Help Desk enter “Create snapshot of the VM” in the Summary field; 4. In the Description field, specify the date you want the snapshot to be created; 5. Click Create to submit your request.



4.1.3.2 Requesting reload of the VM In order to request a VM to be loaded from a particular snapshot one should create a subtask for an issue that is related to this particular VM. The procedure is as follows:

1. Browse the submitted issues (or use search form) until you find an issue that corresponds to the VM you want to reload from snapshot;

2. Create sub-task (e.g. using More Actions menu) for the issue; 3. Set the issue type to Help Desk enter “Load a snapshot” in the Summary field; 4. In the Description field, provide the name of the snapshot you want to be loaded; 5. Click Create to submit your request.

4.1.3.3 Changing the resource assignment for particular VM Once the resources are available, it is possible to increase the capacity of a virtual machine in terms of processor (GHz) and assigned RAM memory. This can be done where justified, i.e. when the history of VM utilisation/load clearly shows that the VM resources are extensively used. To request resources to be extended:

1. Browse the submitted issues (or use search form) until you find an issue that corresponds to the VM you want to change;

2. Create sub-task (e.g. using More Actions menu) for the issue; 3. Set the issue type to Help Desk enter “Extension of resources” in the Summary field; 4. In the Description field, specify to what capacity you would like to extend and provide a reason for the exten-

sion; 5. Click Create to submit your request.

4.1.3.4 Suspending the VM If a particular VM is no longer going to be used, one should request the VM to be suspended in order to free up its resources for other requests. The procedure is as follows:

1. Browse the submitted issues (or use search form) until you find an issue that corresponds to the VM you want to suspend;

2. Create sub-task (e.g. using More Actions menu) for the issue; 3. Set the issue type to Help Desk enter “Suspend the VM” in the Summary field; 4. Click Create to submit your request

4.2 Database access

Database creation and access configuration is conducted on demand. In particular, PSNC operators supervise and per-form these operations upon user’s request. We evaluated the need of providing database services via a dedicated API, similar to OpenStack approach, as proposed in D2.2.3 [00], but decided that it was not necessary based on the feedback from the DBaaS users, and their needs (e.g., frequency, amount, knowledge, etc.).

We decided to have separated databases for each of the pilots, even though they all share the same core schema based on FOODIE data model. This enables the possibility for each of the pilots to extend the model if needed. For example, the Spanish and Polish pilots required extensions to the core FOODIE data model, that were not needed for the others pilots. Additionally, we have three more databases, two with the FOODIE core schema (one common database to be used by all the generic services and one test database), and one database for Landsat images. In particular, our data-bases are the following:

Test database: o user: data_model o db: data_model_db

Main database: o user: foodie-main o db: foodie-main_db



ES database: o user: foodie-es o db: foodie-es_db

CZ database: o user: foodie-cz o db: foodie-cz_db

DE database: o user: foodie-de o db: foodie-de_db

PL database: o user: foodie-pl o db: foodie-pl_db

TR database: o user: foodie-tr o db: foodie-tr_db

LANDSAT database: o user: foodie-landsat o db: foodie-landsat_db

4.3 Deployment and integration of FOODIE platform components

It is assumed, that each partner is responsible for the deployment and configuration of their software components, including selected base technologies and any extension modules that they will need to develop.

Service components may implement one or more standard interfaces (e.g. OGC WCS, WFS) and restful APIs for the integration with other components in FOODIE platform. Applications, like web applications or desktop applications, will use those services to provide their functionality to the end-users.

4.4 Administration and maintenance

Once the VM has been created, it is the requesting person (default administrator) the responsible for installation, con-figuration and management of the whole VM software stack, which characterizes an IaaS type of cloud.

All requests related to the administration of the infrastructure (e.g. upgrading resources, opening ports, system issue, etc.) are managed and maintained by the Help Desk and are based on JIRA.

The ticket related to opening ports for the application should follow the following template:

IP address of the involved node

Identifier of the VM

Port identifier

Address of the network traffic source (e.g. node, network, Internet)

The ticket related to the issue with operating system on the machine should follow the following template:

Identifier of the VM

System part identifier (e.g. web server)

Problem description with examples and/or logs

Requests related to accessing the individual VM are managed by the default VM administrator. All of these procedures are described in the project wiki: https://confluence.man.poznan.pl/community/pages/viewpage.ac-tion?pageId=44106115.

https://confluence.man.poznan.pl/community/pages/viewpage.action?pageId=44106115

https://confluence.man.poznan.pl/community/pages/viewpage.action?pageId=44106115



5 Service platform components

In FOODIE project, components and applications are deployed in separate VMs. The following list presents the final set of components and applications that were deployed and configured (categorized based on the classification introduced for the architecture definition – see D2.2.3 [00]):

Data access services o Mapserver o Geoserver o Rasdaman Petascope add-on o Sensor network components

Senslog Maplog

Catalogues and registries o Micka

Event processing and notification o Notification broker

Data harvester o Landsat8 and Sentinel2 data harvester

Processing and fusion services o Spring integration framework

RabbitMQ o R suite o Netcad suite o Fleet transports statistics service o Yield potential o Pesticides recommender

Semantic tagging and publication tools o Semantic annotation service o Data semantization service o SiLK

Visualisation and presentation services and applications o Geoserver o Mapserver o Marketplace o Widgets Dashboard o Geoportal o Moodle o Swagger

End-user products o SmartV o HSLayerNG o IPM-DSS

Business oriented services o OpenAM o Marketplace

Additionally, we have deployed the security components, which will be described in detail in the section 6.2.



5.1 Mapserver

Currently, there are 2 identical instances of Mapserver (version 6.4.1) installed:

M1: http://foodie-vm1.man.poznan.pl/mapserver/

M2: http://foodie-vm2.man.poznan.pl/mapserver/

Mapservers services should be accessed through the proxy, which is configured on the Node 3: https://www.foodie-cloud.org/mapserver . To serve the same content over proxy, both instances should use with identical configuration files (Mapfile), which should be stored in both Node 1 and Node 2 in directories: /home/mapserver/www/cgi. Directory is accessible for user "mapserver".

Note that Mapserver uses PostgreSQL database, and thus its original configuration has been modified after the migra-tion from Postgres-XL to PostgreSQL. In the current configuration, there is only one database node (depicted in Figure 9 ).

Figure 9: Visualization of mapserver configuration

5.2 Geoserver

Currently, there are 2 identical instances of Geoserver (version 2.6.1) installed:

G1: http://foodie-vm1.man.poznan.pl/geoserver/

G2: http://foodie-vm2.man.poznan.pl/geoserver/

Geoserver configuration is made through the Web Administration Interface accessed directly from geoserver URL. Con-figuration consists of datastores, workspaces and services definitions, user roles, application settings, etc. Geoserver active clustering extension plugin (broker: http://www.geo-solutions.it/blog/advanced-clustering-geoserver/) is in-stalled and activated on both instances. The plugin synchronies configuration between instances automatically. At the moment both instances are configured to be in 'master' and 'slave' modes. In this configuration changes made to one of nodes are propagated to the other, therefore changes to the configuration could be made through the G1 or G2 or Proxy Web Administration Interface. Plugin takes care of configuration, not the data itself. Therefore, to use data which are stored locally, like images, shapefiles etc., local data directory should be additionally synchronized between G1 and G2, ensuring the same file structure. Geoserver services should be accessed through the Proxy, which is configured on the Node 3: https://www.foodie-cloud.org/geoserver .

Note that Geoserver also uses PostgreSQL database, and thus its original configuration has been modified after the migration from Postgres-XL to PostgreSQL. In the current configuration, there is only one database node (depicted Fig-ure 10).

http://foodie-vm1.man.poznan.pl/mapserver/

http://foodie-vm2.man.poznan.pl/mapserver/

https://www.foodie-cloud.org/mapserver

https://www.foodie-cloud.org/mapserver

http://foodie-vm1.man.poznan.pl/geoserver/

http://foodie-vm2.man.poznan.pl/geoserver/

https://www.foodie-cloud.org/geoserver



Figure 10: Visualization of geoserver configuration

5.3 Rasdaman Petascope add-on

This is an add-on component of Rasdaman implementing OGC interface standards WCS, WPS and WMS. It maintains additional metadata in a separate PostgreSQL relational database. The service is accessible from the two instances of Rasdaman via:

https://www.foodie-cloud.org/rasdaman/ows (port 443)

https://www.foodie-cloud.org/rasdaman2/ows (port 443)

See section 3.3 for additional details.

5.4 Sensor networks components

Sensor observations produced by external devices are processed by SensLog or MapLog. MapLog is focused on mobile devices and live tracking and SensLog processes any other producers of sensor observations.

MapLog - The main task of MapLog is to determine the positions of the devices on the earth in real time. Observed positions are being accepted directly from mobile devices primarily, stored to RDBMS and processed to continuous trajectory. Mobile devices can be equipped with any number of additional sensors e.g., temper-ature, light conditions, the state of the wearer units (for human e.g. cardiac pressure, for automobiles e.g. tire pressure etc.). These observations from additional sensors are being processed as same sensors observations but are bound to positions where they were observed. MapLog contains proprietary RESTful API to exchange messages in JSON format to publish data, mainly mobile devices tracking data. Data from additional sensors can be publish by same proprietary API in JSON format or can be used SensLog to publish data in standardized form.

SensLog - The task of SensLog is to receive measured sensor data either directly from sensor devices or indi-rectly from any Front-End elements, store them properly in RDBMS, pre-process them for easier queries if necessary and publish data for other Front-End elements through the system of web-services. SensLog contains proprietary REST API in JSON format or standardized form using OGC SOS version 1.0.0. Standardized interface to publish data provides possibility to publish data to third-party applications.

These components are accessible via: http://portal.foodie-cloud.org/SensLog/



http://portal.foodie-cloud.org/SensLog/



5.5 Micka

Micka is the software for spatial data / services metadata management according to ISO, OGC and INSPIRE standards.

Supported standards

The system is able to implement any standard based on XML document. For implementation, separate administration module is required. The following standards are implemented:

Spatial data metadata (ISO 19115)

Spatial services metadata (ISO 19119)

Feature catalog (ISO 19110)

Dublin Core metadata (ISO 15836)

Profiles

Additional metadata elements and user metadata profiles can be used. These following profiles are default in the sys-tem:

ISO 19115 mandatory elements

ISO 19115 core elements

ISO 19115 full standard

INSPIRE 19115/19119

MICKA profile (INSPIRE + ISO core elements and some additional to cover the most user requirements)

User defined

The user may change the profile during editing.

Language environment

The application interface is multi-lingual, currently 12 languages are supported. The offered language list is set in con-figuration file. The interface language may be selected by clicking the flag on the page header. The metadata itself may be also multilingual. The user defines main language and other languages during record creation or record administra-tion. In all cases UTF-8 codepage is used.

Additional information can be found at: http://foodie.wirelessinfo.cz/php/metadata/?ak=help

The Micka instance for FOODIE is available at: http://portal.foodie-cloud.org/php/metadata/csw/

5.6 Notification broker

FOODIE Notification Broker Service was initially developed for issuing hazard-related warnings (e.g., earthquake and tsunami alerts) but has been adapted to be application domain independent, therefore allowing for instance to issue notifications with agriculture-based information that is relevant for FOODIE platform subscribers.

In a general, the service allows registered users in FOODIE platform subscribe to different subscription “topics” defined by the service (e.g., meteorological alerts, specific crop plague alerts, monthly reporting notifications, etc.). The gener-ation of these notifications is initiated by some data pushed to the notification service (referred to an existing topic) by some other component located in FOODIE platform (e.g., recommenders, sensors, processing services, etc.).

Notifications are customized and delivered to topic subscribers through different communication channels (according to the users’ subscription preferences), being currently supported email and SMS message notifications.

More information is available in D4.3.3 [31].

The notification broker is available at: http://foodie-cloud.org/notification-broker .

http://foodie.wirelessinfo.cz/php/metadata/?ak=help

http://portal.foodie-cloud.org/php/metadata/csw/

http://foodie-cloud.org/notification-broker



5.7 Spring integration framework

Spring Integration extends the Spring programming model to support the well-known Enterprise Integration Patterns. It enables lightweight messaging within Spring-based applications and supports integration with external systems via declarative adapters. Those adapters provide a higher-level of abstraction over Spring's support for remoting, messag-ing, and scheduling.

The primary goal of this component is to provide a simple model for building enterprise integration solutions while maintaining the separation of concerns, which is essential for producing maintainable, testable code.

This framework is deployed as a Spring Web App (deployed on a Tomcat server) using Spring Integration to manage messages and RabbitMQ as Queue Broker. Spring Integration is motivated by providing a simple model for implement-ing complex enterprise integration solutions, facilitating asynchronous and message-driven behaviour within a Spring-based application. Spring Integration is guided by some basic principles: components should be loosely coupled for modularity and testability and the framework should enforce separation of concerns between business logic and inte-gration logic.

These components have been deployed in http://foodie-vm9.man.poznan.pl/. In particular, RabbitMQ is available at: https://www.foodie-cloud.org/rabbitmq/ .

5.8 R suite

R is a well–known language for data analysis. The characteristics of FOODIE R application server are:

R environment: 3.2.0 (free software environment for statistical computing and graphics)

Rstudio desktop 0.98 (IDE for R program language)

Rstudio server 0.98 (tool to provide access to RStudio via a web browser)

FOODIE R application server is available at: http://foodie-vm4.man.poznan.pl/, endpoint at: https://www.foodie-cloud.org/rstudio/

5.9 NETCAD suite

The FOODIE instance of the NETCAD suite is the base technology for the Data Fusion methods described in D3.4.3[04].

Netcad GIS

Netcad GIS, is an advanced CAD and GIS software designed for Engineering and GIS users especially working with Maps. NETCAD GIS has a powerful raster data management infrastructure in addition to its CAD and GIS features. Thanks to this structure, it can process 1 band - N band images in the range of 1 bit - 64-bit pixel formats. Many raster and grid formats such as TIF, geoTIF, ECW, MrSID, JPEG, PNG, DEM, SRTM, DTED are also supported. NETCAD GIS, besides its raster infrastructure, image rectification, wide variety of image processing capabilities, also works with Netcad Archi-tect. In addition to raster imaging operations, switching between Vector and Raster spaces can also be easily done using Netcad Architect.

Installed Components:

Netcad Analist

Netcad WPS Module

Netcad Architect

Ports used: 80, 475, 476

Netgis Server

http://foodie-vm4.man.poznan.pl/

https://www.foodie-cloud.org/rabbitmq/

http://foodie-vm4.man.poznan.pl/

https://www.foodie-cloud.org/rstudio/

https://www.foodie-cloud.org/rstudio/



NETGIS Server is the common name of Netcad Enterprise Solutions. NETGIS Server Family includes many compo-nents/modules. These components provide data in OGC WPS, WMS, WFS, WFC, WMTS, KML standards. The solutions are database independent. It allows Application Development. A web-based map presentation and inquiry site can be created with NETGIS Server.

Installed Components:

Netgis Server WPS Handler

Netcad NCC Proxy

Netcad Base

Netcad Base is a family of server applications that correspond to infrastructural needs of Netcad applications. The in-stalled components include:

Authentication

Logserver

StateServer

ParameterServer

SchedulerServer

Netcad suite configuration is depicted in the figure below.

Figure 11: Visualization of Netcad configuration

The FOODIE instance of Netcad suite is available at:

https://www.foodie-cloud.org/netgiswps

https://www.foodie-cloud.org/geoprocessing

https://www.foodie-cloud.org/fusion_map

5.10 Fleet Transport statistics service

Fleet Transport Statistics Service was developed as a tool to obtain, process and analyse data coming from agricultural machinery fleet monitoring, i.e. tractors and application machines. Monitoring devices from companies Teltonika and Fons were used to track any tractor/application machine through ISO/CAN BUS no matter the brand (Case, Steyer, Zetor as well John Deere machines were successfully monitored). Analysed data are then used as economic evidence for a farmer.

https://www.foodie-cloud.org/netgiswps

https://www.foodie-cloud.org/geoprocessing

https://www.foodie-cloud.org/fusion_map



The Fleet Transport statistics service has a loose integration since:

it contains extremely sensitive data that may lead to bankruptcy of a farmer if misused,

end users are not willing to use the machinery monitoring if their data would be stored in a different country than a country where a farmer cultivates his/her plots (i.e. when cultivation and data would not be under the legal framework of the same country).

it is more tightly connected to back-end (board computer of a tractor and monitoring hardware mounted on a tractor and/or application machine) than any other FOODIE component.

As a result, it was decided that the whole machinery fleet monitoring including the Fleet Transport statistics service is being operated independently. It also means, that the second authorisation level runs in parallel with the FOODIE plat-form’s authorisation. The Fleet Transport statistics service is called centrally from the FOODIE platform under the URL http://foodie-cloud.org/telemetry, which initiates a proxy service with endpoint at WRLS’s premises as the second au-thorisation level to protect farmer’s data as much as possible.

The service is available at: http://foodie.lesprojekt.cz/MapLog/AnalystService

5.11 Yield potential

Yield potential is a special kind of an offline service even if its online deployment was foreseen during the second year of the project. The main reason for providing an offline service are computational complexity and volume of required data.

Satellite images from the second half of vegetation period of last 8 years are needed in order to compute yield potential for a certain location. One scene in the best available resolution of Sentinel-2 has volume of 6 GB. If we for instance count surrounding of Brno city in the Czech Republic, we reach up to 170 days of vegetation period. Its half is 85 days when in total 51 GB of Sentinel-2 data are needed a year (second half of vegetation period). Eight-year series means processing of 408 GB of Sentinel-2 satellite data, which is almost the full memory capacity of one cluster dedicated to the FOODIE platform. When using also Sentinel-2/B together with Sentinel-2/A, we reach 5 days’ temporal resolution that results in a need of one TB of memory for one location.

Moreover, such volumes of satellite data need to be processed. For instance, scenes need to be combined, CF Mask has to be applied, vegetation and moisture indices (NDVI, EVI) need to be computed, yield potential with its median value has to be computed, temporal consistency of each plot needs to be checked, reference amount of a fertilizer needs to be set, yield potential has to be re-computed etc. It is obvious that such service cannot be within storing and processing capabilities of the FOODIE project operated on-the-fly neither for the whole Europe.

5.12 Pesticides recommender

The pesticides recommender is a prototype (proof of concept) service that wraps the domain knowledge and SWRL rules stored in the Pesticides ontology and provides the mechanism for giving recommendations of products appropri-ate for the specified situation. It can recommend the particular pesticide and its dose to be applied for a given situation (crop type, disease/pest, plot size), or general recommendations if some information is no provided (e.g., no crop type provided, no disease/pest provided, etc.).

Technologies and frameworks

Pesticides Recommender Service is a Java web application. It requires at least Java 1.8 for development and for runtime environment.

Pesticides Recommender Service uses Spring v.4.1.4 as main application framework, taking advantage of its bean management and event processing functionalities as well as support for REST API implementation.

Pesticides Recommender Service is build using Apache Maven v.3.1.3. All project dependencies and deploy-ment process are defined and managed by Maven.

Uses Protégé SWRL API v.1.1.4 to extract data from ontology using SQWRL queries

http://foodie-cloud.org/telemetry

http://foodie.lesprojekt.cz/MapLog/AnalystService



The building process creates a war file that can be deployed on any web container. Currently Apache Tomcat v.7.0 is used for the development and production deployment.

Source code of the application is available on git repository: https://git.man.poznan.pl/stash/pro-jects/FOOD/repos/pesticides_recommender/browse

Provides a simple graphical user interface based on Swagger, which can be used to test the functionality.

More information is available in D4.1.3 [30].

The service can be tested via: http://www.foodie-cloud.org/swagger_ui/?url=http://foodie-cloud.org/swag-ger_api/Pesticides_Recommender_API.yaml

5.13 Semantic annotation service

This component provides a REST API enabling the creation of semantic annotations. The latest version of this component is based on AgroTagger and Babelfly semantic annotation tools. The first one uses a subset of AGROVOC vocabulary for annotating the text (limited to items of type skos:concept, and including only their English definition), while the second one uses several purpose knowledge bases, particularly DBPedia. The semantic annotation API provides:

possibility to annotate text in Web resources

possibility to annotate textual files (PDF, DOC, XLS and HTML)

possibility to provide manual annotations

creation of annotation in RDF format

storage of RDF triples in the triplestore

access to created annotations

The component is deployed on Apache Tomcat server and is available at: https://www.foodie-cloud.org/semanticAn-notation/. The component is built up into WAR package so it should be possible to deploy it on a different server. A client application is available via: http://www.foodie-cloud.org/swagger_ui/?url=http://foodie-cloud.org/swag-ger_api/Semantic_Annotation_API.json. A detailed description of this component is available in D3.3.3 [03]

5.14 Silk framework

Silk is the tool selected for the implementation of the linked data publication service. It enables the discovery and gen-eration of links between related data items within different Linked Data sources. In our case, Silk is will enable the generation of RDF links from our triplestore data to other data sources on the Web in order to publish five-start Linked data.

Silk provides both a Graphical User Interface and a REST API, so that it can be used programmatically. It is available at http://silk.foodie-cloud.org/ .

A detailed description of this component is available in D3.3.3 [03].

https://git.man.poznan.pl/stash/projects/FOOD/repos/pesticides_recommender/browse

https://git.man.poznan.pl/stash/projects/FOOD/repos/pesticides_recommender/browse

http://www.foodie-cloud.org/swagger_ui/?url=http://foodie-cloud.org/swagger_api/Pesticides_Recommender_API.yaml

http://www.foodie-cloud.org/swagger_ui/?url=http://foodie-cloud.org/swagger_api/Pesticides_Recommender_API.yaml

http://silk.foodie-cloud.org/



5.15 Marketplace

The marketplace is a web application covering the following functions in FOODIE ecosystem:

marketplace for exchange of data, applications and services;

collaboration hub for organizations.

The current prototype of marketplace is available at: https://www.foodie-cloud.org/marketplace/ .

A detailed description of this component is available in D3.5.2 [05].

5.16 Dashboard

Web dashboard designed to empower users to navigate and explore the information by means of visual instruments, such as maps, charts and graphs. The dashboard would be customizable by means of selecting different visualizations or functionalities implemented as widgets.

These UI components will be reusable and modular, so they can be easily assembled to build data-driven applications from the FOODIE platform. For instance, the components can be pieced together comprising a dashboard to monitor a given farm or plot of land, or an interactive report result of a data analysis process. In this context, the components will be materialized as HTML5 widgets and will rely on standard services and protocols in order to provide access to appli-cation data.

Technologies:

Ext JS: Dashboard and widgets developed ander this JavaScript application framework.

JBoss: application server hosting the foodie dashboard.

These components are available at: https://foodie-cloud.org/widgets-dashboard .

The dashboard is described in detail in D4.2.3[06]

5.17 Moodle

Moodle is a learning platform designed to provide educators, administrators and learners with a single robust, secure and integrated system to create personalised learning environments [24]. Moodle is provided freely as Open Source software and can be extended by numerous plugins and add-ons.

At the moment of writing our Moodle instance includes 6 courses grouped in two categories:

Methods of farm management o Production methods o Management methods o Agro-meteorological information o Market analysis

Software components and tools of farm management o Data management o Technical solution

The FOODIE instance of Moodle service is available here: https://www.foodie-cloud.org/moodle/ .

This component is one of the key components for the delivery of training materials, as described in D5.4.3[07].

https://www.foodie-cloud.org/marketplace/

https://foodie-cloud.org/widgets-dashboard

https://www.foodie-cloud.org/moodle/



5.18 Swagger

Swagger provides a standard interface to REST APIs that allows humans and computers to understand the capabilities of a service without accessing the source code.

It allows developers to write the API definition once, and automatically generates server code, client code, automated test cases, and the final documentation. The documentation generated allows the developers to try calling API end-points without having to write any code.

Swagger uses YAML but has also full support for specifications written in JSON.

The FOODIE instance of Swagger can be accessed from: http://www.foodie-cloud.org/swagger_ui/

This component is the base of D3.2.3 [02].

5.19 Geoportal

Geoportal is a Web application enabling:

the visualization of data in a map

filters the type of information to be displayed on a map

select the data sources to be displayed on the map

Adding additional services or database accessible to standard protocols like WMS

Select layers to be displayed

Save the customized view

Access to a metadata catalogue based on INSPIRE metadata schema.

The Geoportal is available at: http://portal.foodie-cloud.org/

5.20 SmartV

This is the Spanish pilot final product, and is accessible via the dashboard (see Section 5.16). More information availa-ble in D5.5.3 [32].

5.21 IPM-DSS

This is the Polish pilot final product, and is accessible via http://dss.foodie-cloud.org/ .

The pilot was focussed on the development of a web-based Decision Support System (DSS) supporting the application of the general principles of integrated pest management (IPM) by all professional users of plant protection products, obligatory for the EU member states since 1st January 2014 as outlined in detail in the provisions of Art. 14 Directive 2009/128/EC and art. 55 of Regulation No 1107/2009/WE.

Such DSS is useful in determining the optimal terms of crop protection, and thus allows obtaining high efficiency of these treatments while reducing the use of chemical pesticides to a minimum.

More information available in D5.5.3 [32].

http://www.foodie-cloud.org/swagger_ui/

http://portal.foodie-cloud.org/

http://dss.foodie-cloud.org/



6 FOODIE cloud access and security

This section describes the accessibility and security aspects of the cloud. In particular, we describe the official domain for the cloud and the URL template to access the different services and applications deployed. Then, this section de-scribes the security framework to manage the access to the cloud.

6.1 Domain and application endpoints

As part of the accessibility and sustainability plan of FOODIE cloud platform, we have registered the DNS domains foodie-cloud.org and foodie-cloud.com, which point to the same location.

The main motivation was to provide homogeneous access to all FOODIE services and applications, and to make trans-parent to the users the cloud infrastructure provider. Accordingly, we support the following URL format for the compo-nents:

foodie-cloud.org/servicex, e.g., foodie-cloud.org/marketplace

Due to the fact that some applications could have different security needs we provided a second entry point to the cloud:

marketplace-admin.foodie-cloud.org

This is specially added for the Marketplace needs but other applications can be deployed aligned with this domain name in order to provide a different security approach as described in Section 6.2 depending on application specific needs.

Additionally, this simplified the implementation of the security framework as all the requests go through a proxy server which maps the provided URL to the actual endpoint, as described in Section 6.2. The previous section provided the endpoints for each of the services and applications. Additionally, the information is available in the FOODIE wiki. Note that some application that do not require access through the security component can have endpoints with another URL format: service.foodie-cloud.org e.g., silk.foodie-cloud.org/

Furthermore, we have designed a welcome page(s) to FOODIE cloud platform at https://www.foodie-cloud.org/ , which is the main entry point to the services and applications developed in FOODIE.

Finally, this domain will support and foster the sustainability of FOODIE cloud platform beyond the project lifetime. As the project website may be discontinued in the future, the goal is that the platform will be sustained and will be available far beyond.

6.2 Security framework

This section describes the security architecture and the different components providing Authentication and Authoriza-tion to the entire FOODIE cloud as well as how to deploy them.

FOODIE cloud security framework is based on OpenAM [28] and OpenDJ [29], which are third party software suiting FOODIE security needs:

• OpenAM is an open source access management, entitlements and federation server platform developed by ForgeRock.

• OpenDJ is an open source Lightweight Directory Access Protocol (LDAPv3) and Directory Service Markup Language (DSMLv2) compliant directory service written in the Java programming language.

https://www.foodie-cloud.org/



6.2.1 Security Architecture

The usual access to the entire FOODIE cloud is under Firewall restrictions but virtual machine 7, where the security system is deployed, represents the main access point to the cloud.

This virtual machine is mapped by two domain names:

www.foodie-cloud.org this is the main domain to access any resource in the FOODIE cloud

Iam.foodie-cloud.org this is the specific domain to access the Identity Access Manager (IAM).

Virtual machine 7 has certificates to enable HTTPS through port 443 in order to have secure data transfers with the users.

Security is provided by two components IAM and Policy Agent (PA). While IAM is running under Tomcat7 on port 443, PA is running on an Apache2 web server and listening in port 443.

So, any access to port 443 will be handled by Apache2 web server but there are two active network interfaces enabled for this virtual machine:

• Eth0: any request to iam.foodie-cloud.org will be forwarded by Apache2 web server to the IAM. • Eth1: any request done to www.foodie-cloud.org will be checked by the Policy Agent.

Virtual machine 6 provides another entry point to FOODIE cloud and it is monitored by another Policy Agent instance. This other virtual machine is mapped by a different domain name:

Marketplace-admin.foodie-cloud.org this is the access point to reach more restricted services within the FOODIE cloud.

It is also using certificates to provide https secure access through port 443 and there is only one networking interface:

Eth0: any request done to marketplace-admin.foodie-cloud.org will be checked by the second Policy Agent.

The purpose of having two different access points is to be able to provide different security behaviours de-pending on service’s needs. Direct security Domain is secured by default but exception URLs will be not secured. This result in PA monitoring every request into the domain but the exception URLs.

Inverse security Domain is not secured by default, only specific URLs. This means Policy agent will ignore every request unless it is one of the specific URLs.

To summarize there are two entry points to the FOODIE cloud depending on the domain name used.

Marketplace-admin.foodie-cloud.org this access point uses second policy agent running on virtual machine 6 with direct security, that means any access to the domain is monitored by the PA as default. But there is the chance of defining specific URLs to avoid Policy Agent from secure them.

www.foodie-cloud.org this is the common access point for every user and most of the services within the foodie cloud. The access is done through virtual machine 7 with inverse security, that means every resource is not secured unless a specific exception URL is created for the Policy Agent to monitor. This way a new resource in the domain needs an exception rule to be added on IAM in order to secure the access to it.

Accessing any resource in the www.foodie-cloud.org or marketplace-admin.foodie-cloud.org domains without session token will result in corresponding PA forwarding that request to the IAM.

The Identity Access Manager (IAM) grants access through providing a session token to the user after a successful login operation. It also determines depending on the Policies if the user has access to the specific resource or not and then IAM will forward the request to the resource URL or will show a restricted access message. In the end, Policies determine which resources (URL) are allowed for which users or groups and for this purpose there is a LDAP instance where IAM stores all information needed regarding security such as users, groups, policies, etc.

The diagram below depicts how security is working and how accesses are handled.

http://www.foodie-cloud.org/

http://iam.foodie-cloud.org/






Figure 12: FOODIE cloud secure access description

6.2.2 LDAP Server – OpenDJ

LDAP is one of the broadest used and tested solutions to store the user information within security and privacy envi-ronments hence LDAP has been chosen as the most suitable option for the OpenAM requirements. Between the suite of tools provided by OpenAM, OpenDJ provides LDAP storing capabilities and the associated services to connect to the end user data. Therefore, in order to fully integrate the authorization and authentication services the OpenDJ tool is needed to be installed and configured.

In the following we describe how to install this component

6.2.2.1 OpenDJ installation

To install it, download OpenDJ software from one of the following locations. • The ForgeRock Enterprise Downloads page has the latest stable, supported release of OpenDJ and the other

products in the ForgeRock identity stack. • The Nightly Builds page posts links to the latest nightly builds of OpenDJ software. Note that these builds are

the working version from the trunk and are not for use in a production environment. • The Community Archives page includes stable community builds for previous releases of OpenDJ software.

QuickSetup uses Java WebStart to let you perform an installation of OpenDJ directory server starting with a click in your web browser, which can be a great way to try OpenDJ directory server for the first time, or to do a quick test installation.

If the WebStart installation does not work in your browser, copy the WebStart URL, ending in QuickSetup.jnlp, from the OpenDJ download page. Next, pass the link as an argument to the javaws command in a terminal window to start the installer.

The WebStart installer corresponds to what you start if you download OpenDJ-2.7.0-SNAPSHOT.zip, unzip the file, and then run opendj/setup (UNIX).

• $ export PATH=/path/to/java/bin:$PATH

• $ javaws URL-to-QuickSetup-Installer



Figure 13: OpenDJ installation I

Figure 14: OpenDJ installation II



Figure 15: OpenDJ installation III

Figure 16: OpenDJ installation IV



Figure 17: OpenDJ installation V

Figure 18: OpenDJ installation VI



Figure 19: OpenDJ installation VII

Figure 20: OpenDJ installation VIII



Figure 21: OpenDJ installation IX

To launch OpenDJ Control Panel again later, you run one the following command (in UNIX): /path/to/opendj/bin/con-trol-panel (this command depends on the host system).

6.2.3 IAM – OpenAM

The main tool for the Security in FOODIE is going to be the Identity Access Manager. This component is going to bring together the Authentication and Authorization. In the following we describe how to install this component.

6.2.3.1 OpenAM installation

The steps needed to install it are as follows:

The iam-1.0.0-SNAPSHOT.war file contains all Identity Provider server components and samples. How you deploy the .war file depends on your web application container.

1. Deploy the .war file on your container.

1. For example, copy the file to deploy on Apache Tomcat.

2. Check that you see the initial configuration screen in your browser at openam.example.com:8080/iam.

• $ cp openam/iam-1.0.0-SNAPSHOT.war.war /path/to/tomcat/webapps/iam.war

Figure 22: IAM installation

6.2.3.2 Configure IAM

First of all, it is needed to set up certificates for HTTPS in Tomcat:

1. Stop Tomcat. 2. Uncomment the SSL connector configuration in Tomcat's conf/server.xml, specifying your key store file and

password. 3. Start Tomcat

Next IAM deployment is done as described in the next steps: 1. In the initial configuration screen, click Create New Configuration under Custom Configuration. 2. Provide a password having at least 8 characters for the OpenAM Administrator, amadmin.

  <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" keystoreFile="/path/to/tomcat/conf/keystore.jks" keystorePass="changeit"

• clientAuth="false" sslProtocol="TLS" />



Figure 23: IAM configuration I

3. Make sure the server settings are valid for your configuration.

Figure 24: IAM configuration II

• Server URL Provide a valid URL to the base of your OpenAM web container, including a fully qualified domain name (FQDN).



In a test environment, you can fake the FQDN by adding it to your /etc/hosts as an alias. The following excerpt shows lines from the /etc/hosts file on a Linux system where OpenAM is installed.

• Cookie Domain Starts with a dot (.).

• Platform Locale Supported locales include en_US (English), de (German), es (Spanish), fr (French), ja (Japanese), ko (Korean), zh_CN (Simplified Chinese), and zh_TW (Traditional Chinese).

• Configuration Directory Location on server for OpenAM configuration files. OpenAM must be able to write to this directory.

4. In the Configuration Store screen, you can accept the defaults to allow OpenAM to store configuration data in

an embedded directory. The embedded directory can be configured separately to replicate data for high avail-ability if necessary. You can also add this OpenAM installation to an existing deployment, providing the URL to reach an existing OpenAM instance. Alternatively, if you already manage an OpenDJ or DSEE deployment, you can choose to store OpenAM config-uration data in your existing directory service. You must, however, create the suffix to store configuration data on the directory server before you configure OpenAM. OpenAM does not create the suffix when you use an external configuration store.

Note: When you create a new OpenAM custom configuration that uses an external LDAP directory server for the config-uration data store, you must use a root suffix DN with at least two domain components, such as dc=example,dc=com. The root suffix must be dc. You will not be able to move to the next configuration screen if both of these conditions are not met.

5. In the User Store screen, you configure where OpenAM looks for user identities.

OpenAM must have write access to the directory service you choose, as it adds to the directory schema needed to allow OpenAM to manage access for users in the user store.

• 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 lo-

calhost6

• 127.0.1.1 openam openam.example.com



Figure 25: IAM configuration III

• User Data Store Type

If you have a directory service already provisioned with users in a supported user data store, then select that type of directory from the options available.

• SSL/TLS Enabled To use a secure connection, check this box, then make sure the Port you define corresponds to the port on which the directory listens for StartTLS or SSL connections. When using this option, you also need to make sure the trust store used by the JVM running OpenAM has the necessary certificates installed.

• Directory Name FQDN for the host housing the directory service

• Port LDAP directory port. The default for LDAP and LDAP with StartTLS to protect the connection is port 389. The default for LDAP over SSL is port 636. Your directory service might use a different port.

• Root Suffix Base distinguished name (DN) where user data are stored

• Login ID Directory administrator user DN. The administrator must be capable of updating schema and user data.

• Password Password for the directory administrator user

6. In the Site Configuration screen, you can set up OpenAM as part of a site where the load is balanced across multiple OpenAM servers. For your first OpenAM installation, you can accept the defaults. If you have a load balancer, you can enable session high availability persistence. This will store sessions in case of a server failure, so that when the server



is restored, users will be returned to their sessions without having to login again. If you would like to enable session high availability persistence or if you plan to setup additional instances, enter the Site Name, Load Balancer URL, and click the Enable Session HA Persistence and Failover.

Figure 26: IAM configuration IV

If you plan to set up an additional instance but are not ready yet, you do not have to set it up now. The site configuration can be set up during configuration of your additional instances.

7. In the Agent Information screen, provide a password having at least 8 characters to be used by policy agents to connect to OpenAM.

Figure 27: IAM configuration V

8. When the configuration completes, click Proceed to Login, and then login as OpenAM admin.



Figure 28: IAM configuration VI

9. Restrict permissions to the configuration directory (by default $HOME/openam, where $HOME corresponds to

the user who runs the web container).

6.2.4 PA – OpenAM

1. Once the apache2 web server is installed and SSL certificates are configured:stop apache 2. unzip Policy Agent package 3. Create a password for the PA

4. Run the installer

5. Once the installer is running, you have to provide the following parameters: o Apache folder: /etc/apache2-foodie o IAM URL: https://iam.foodie-cloud.org:443/iam o Agent name: FoodieAgent o Agent URL: https://www.foodie-cloud.org:443 o Agent password: /home/ubuntu/tmp/pwd.txt

echo <agent_password> /home/ubuntu/tmp/pwd.txt

sudo bash web_agents/apache24_agent/bin/agentadmin --install



Note: Edit /home/ubuntu/web_Agents/Agent_00X/config/OpenSSOAgentBoostrap.properties file and check the prop-erty com.sun.identity.agents.config.organization.name is matching your realm in the PA or leave it as ‘/’.

6. Start apache

6.2.4.1 Configure Policy Agent

To configure the Policy Agent head to https://iam.foodie-clouf.org/iam from the web browser and login into the system. Once there, select the ‘Access Control’ tab and select the Top Level Realm if you are using the default one.

Figure 29: PA configuration I

Now select the ‘Agents’ tab, and then the FoodieAgent in the table below.

Figure 30: PA configuration II

https://iam.foodie-clouf.org/iam



Select the tab ‘OpenAM Services and check the realm is ‘correct’ and the ‘Application’ depending on the application name you will use to define policies.

Figure 31: PA configuration III

Now head back to the realm and fill the ‘Realm/DNS Aliases’ list with the domains FODDIE cloud is using:

Figure 32: PA configuration IV

Once done go back to the ‘Agents’ tab and select FoodieAgent and then select the ‘Application’ tab to fill the ‘Not Enforced URLs’ with those URL’s you want to give free access to everyone.



Figure 33: PA configuration V

6.2.5 Policies definition

To define which URL’s are protected by OpenAM there is need to create policies defining which URLs and which kind of HTTP operations and users/groups are allowed. From the ‘Agents’ tab, select the agent and then ‘Policies’ tab. There you have to define an application; this means to define the URL’s you want to protect.

Figure 34: Policies definition I

Once an application is defined click on the paper icon next to it to display its policies, and there you can define:

Specific URL patterns

HTTP actions

User/groups rules

Environment conditions (regarding time, ips, etc…)



Define response attributes

Figure 35: Policies definition II



7 High Availability

In information technology, High Availability (HA) refers to a system that is continuously operational for a desirably length of time. Since computer system consists of many parts in which all parts usually need to be resent in order for the whole to be operational, much planning for high availability centres around backup and failover processing and data storage and access [25].

7.1 Physical data redundancy

Each virtual machine has a virtual disk assigned from a mass storage array. Each virtual disk is protected by redundancy provided by storage array.

7.2 System monitoring

We have added FOODIE VMs to Nagios monitoring system deployed in PSNC. Nagios offers monitoring and alerting services for servers, switches, applications, and services, alerting administrators when things go wrong and when the problem has been resolved.

In particular, our Nagios instance checks FOODIE server live status via ICMP ping. It also monitors the ssh service for linux based systems and RDP for windows based systems. We also add other services, like database or web applications monitoring. As Nagios sends email to administrators as soon as it detects some problem with services, we will be able to address the issues timely, avoiding long downtime of the services provided.

7.3 Backup procedures

7.3.1 System backups

Virtual machines with Linux systems are backup at files level. The main goal of this backup is to store a copy of all files on other storage system. The copy can be used to restore data in case of accidentally removing or unaware modification. The backup script crates a compressed archive file (with tar and gzip programs) and store it on local disk. Each virtual machine has virtual disk assigned from mass storage array. Virtual disk is protected by redundancy provided by storage array. After completion of this process the archive file is uploaded to Platon-U4 storage service [26] via sftp connection. Backups are made every Sunday at 04:00:00 (CET).

7.3.2 Database backups

All databases are backed up every week. Backups are stored with usage of Platon-U4 storage service [20].

7.3.3 Other backups

By default, the VM administrators are responsible to take care of the data stored on their machine.

7.4 Service Level Agreement

SLA (Service Level Agreement) is a part of a service contract where particular aspects of the service (e.g. scope, quality, responsibilities) are agreed between the service provider and the service user. A common feature of an SLA is a con-tracted delivery time [27].

The most important objectives of SLA are as follows:

Creating an environment that ensures effective support of end users

Describing the responsibilities of all parties of the agreement

Defining the commencement of the agreement, its initial term and the provision for reviews

Defining all details of the service to be delivered thus reducing the risk of misunderstanding



Providing a common understanding of service requirements/capabilities and of the rules involved in the meas-urement of service levels

7.4.1 General SLA for FOODIE assumptions

The SLA contract that will be signed by all involved parties shall include following assumptions:

The agreement will commence on agreed date and will continue until the end of the project;

The agreement will be reviewed at agreed date by all involved parties. The review will cover services provided, service levels and procedures. Changes to this agreement must be approved by all parties;

The SLA summary reports will be delivered on a regular basis;

The service provider shall not be liable for any loss, which are consequences of natural disasters, as well as network outages that are consequences of problems in another ISP’s network. Responsibility is limited to actual loss. Parties shall not be responsible for the loss of benefits as a result of the incident;

The service provider reserves the right to immediately disconnect any or all of the devices from power supply in case of any risk associated with continued operation (especially fire hazard).

The service provider reserves the right to block network traffic to any or all of the systems in case of a reason-able suspicion of an attack carried out from one of the systems;

The service provider will accept incident requests 24/7. Standard priority requests will be handled during busi-ness hours. The service provider will make every reasonable effort to resolve any “high priority” (e.g. complete inaccessibility of hardware or loss of network connectivity) issues as soon as possible.

7.4.2 Hardware infrastructure

Hardware related issues are resolved on the best effort basis, however, if the issue resolution involves actions covered by supplier warranty package, the resolve time is not less than the support request resolve time offered by hardware supplier support package and described in support package warranty terms and conditions.

The provider hosts the aforementioned hardware in its premises, providing also redundant and uninterruptible power supply and network infrastructure. The provider shall not be liable for network and power outages which are conse-quences of natural disasters, as well as network outages which are consequences of problems in another ISP’s network.

7.4.3 Service

The goal of the service is to ensure availability of aforementioned hardware at the agreed level not counting planned maintenance times. The availability metric will be measured by a rolling 6 months period. The administrative tasks and responsibilities of provider’s admin group include:

Supervising and monitoring the running servers,

Identifying issues and problems related to hardware and network connections

7.4.4 Maintenance

In case of maintenance hardware and software activities these will be announced to FODDIE community:

Two days in advance if the maintenance action expects the reduction of available resources only

Seven days in advance if all of the equipment will be inaccessible during the maintenance.



8 Conclusions

FOODIE project aimed at providing a cloud-based platform, specialized for the agri-food domains, providing high-value applications and services for different stakeholders’ groups related to these domains. This platform integrates different components, which build in most cases on existing technologies, adapted and extended according to the analysis of user requirements.

In order to build such platform, tasks related to software development and integration were very intensive and dynamic throughout the project lifetime, and required collaboration among heterogeneous and distributed teams. In order to facilitate these development tasks, and to support an efficient collaboration among teams, an appropriate infrastruc-ture was put in place for developers so they could focus on the quality of their product. In this context, the “quality” means: efficiency of the software, readability of the source code, accuracy of the documentation and maturity (and repeatability) of the software development process. This document described the tools selected by FOODIE consortium to support the development tasks, and included a set of guidelines and best practices that would help developers to use this infrastructure and to deliver high-quality results.

Additionally, the deployment of these software components in a cloud environment required an adequate provisioning of resources and a set of procedures ensuring High Availability on the provided services, so that both authors of appli-cations (developers) and their customers can rely on the whole environment.

Accordingly, this document presented the final state of the cloud infrastructure that was configured to support the deployment and integration of FOODIE platform components, including the procedures for using this infrastructure, and an overview of the set of components and base technologies that were deployed. Additionally, this document dis-cussed FOODIE cloud accessibility aspects and described in detail the security framework for managing authentication and authorization to the cloud platform.

Complementary, the document described the procedures and tasks for the provision of services with High Availability, including a template for a Service Level Agreement that would enable in the future to expand FOODIE cloud by including additional nodes without compromising the quality of service.



References

00 Esbrí M., Palma R., Rodríguez J., Campos A., Charvat K., Ogut G., “Service Platform Specification”. D2.2.3 FOODIE deliverable. October 2016

01 Esbrí M., Nunez M., “Heterogeneous data repositories and related services”. D3.1.3 FOODIE delivera-ble. February 2017

02 D3.2.3 FOODIE deliverable. February 2017

03 Krystek M., Palma R., Wolniewicz M., “Sematic tagging and open data publication tools”. D3.3.3 FOODIE deliverable. February 2017

04 Sümer A., “Data Fusion Tools”. D3.4.3 FOODIE deliverable. February 2017

05 Łabędzki M., Promiński P., Rybicki A., Palma R., Brahma S., “Marketplace”. D3.5.2 FOODIE deliverable. February 2017.

06 Rubiera E., “Advanced Rich Interfaces”. D4.2.3 FOODIE deliverable. February 2017.

07 Charvat K., et al. “Training materials”. D5.4.3 FOODIE deliverable. September 2016.

08 Kędziora, Lewandowski, Marović, Mazurek; “Bringing governance into distributed R&D software devel-opment - GÉANT case study”, TNC 2012

09 http://www.openstack.org/

10 http://whatis.techtarget.com/definition/Database-as-a-Service-DBaaS

11 http://www.postgres-xl.org/

12 http://virtuoso.openlinksw.com/

13 http://git-scm.com

14 https://www.atlassian.com/software/jira

15 https://www.atlassian.com/software/fisheye/overview

16 https://help.github.com/articles/github-glossary/

17 https://confluence.atlassian.com/display/JIRA052/Configuring+Workflow

18 https://confluence.atlassian.com/display/JIRA/What+is+an+Issue; https://confluence.atlas-sian.com/display/JIRA/Defining+Resolution+Field+Values

19 http://www.oracle.com/technetwork/java/codeconv-138413.html

20 http://www.caliban.org/ruby/rubyguide.shtml

21 http://maven.apache.org/

22 http://docs.codehaus.org/pages/viewpage.action?pageId=47832

http://www.openstack.org/

http://whatis.techtarget.com/definition/Database-as-a-Service-DBaaS

http://www.postgres-xl.org/

http://virtuoso.openlinksw.com/

http://git-scm.com/

https://www.atlassian.com/software/jira

https://www.atlassian.com/software/fisheye/overview

https://help.github.com/articles/github-glossary/

https://confluence.atlassian.com/display/JIRA052/Configuring+Workflow

https://confluence.atlassian.com/display/JIRA/What+is+an+Issue

https://confluence.atlassian.com/display/JIRA/Defining+Resolution+Field+Values

https://confluence.atlassian.com/display/JIRA/Defining+Resolution+Field+Values

http://www.oracle.com/technetwork/java/codeconv-138413.html

http://www.caliban.org/ruby/rubyguide.shtml

http://maven.apache.org/

http://docs.codehaus.org/pages/viewpage.action?pageId=47832



23 http://dev.mygrid.org.uk/wiki/display/developer/Development%2C+release+and+testing+procedures

24 https://docs.moodle.org/28/en/About_Moodle

25 http://searchdatacenter.techtarget.com/definition/high-availability

26 https://storage.pionier.net.pl/

27 http://en.wikipedia.org/wiki/Service-level_agreement

28 https://www.forgerock.com/platform/access-management/

29 http://opendj.forgerock.org/

30 Reznik T., “Advanced Decision Support tools”. D4.1.3 FOODIE deliverable. February 2017.

31 Esbri M., “Event-based notification services”. D4.3.3 FOODIE deliverable. February 2017.

32 Suarez I., “Pilots Execution Progress Reports”. D5.5.3 FOODIE deliverable. February 2017.

Table 22 References

http://dev.mygrid.org.uk/wiki/display/developer/Development%2C+release+and+testing+procedures

https://docs.moodle.org/28/en/About_Moodle

http://searchdatacenter.techtarget.com/definition/high-availability

https://storage.pionier.net.pl/

http://en.wikipedia.org/wiki/Service-level_agreement

https://www.forgerock.com/platform/access-management/

http://opendj.forgerock.org/

Service platform integration and deployment in cloud ... · 5.19 Geoportal ... Figure 13: OpenDJ...

Documents

Transcript of Service platform integration and deployment in cloud ... · 5.19 Geoportal ... Figure 13: OpenDJ...