TEFIS D5.4 - CORDIS · TEFIS - D5.4.1– Connector implementation and documentation for LivingLab....

TEFIS - D5.4.1– Connector implementation and documentation for LivingLab

Deliverable Title D5.4.1 Connector implementation and documentation for Living Lab

Deliverable Lead: Annika Sällström (LTU-CDT)

Related Work package: WP 5

Author(s): Annika Sällström (LTU-CDT), Santiago Martínez (SQS), Itziar Ormaetxea (SQS), Ainhoa Gracia (SQS), Jonathan Gonzalez Rios (SQS), Gabriele Giammatteo (ENG), Robert Samuelsson (LTU-CDT)

Dissemination level: PU

Due submission date: 31/05/2011

Actual submission: 30/09/2011

Project Number 258142

Instrument: IP

Start date of Project: 01/06/2010

Duration: 30 months

Project coordinator: THALES

Abstract This document summarizes the results for the initial work with the Living Lab connector of TEFIS. This deliverable will be updated when the initial Living Lab use-case is finalized and the Open call experiments have been evaluated.

of 32


Project funded by the European Commission under the 7th European Framework Programme for RTD – ICT theme of the Cooperation Programme.

License

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 License.

To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.

Project co-funded by the European Commission within the Seventh Framework Programme (2008-2013)

Copyright by the TEFIS Consortium

of 32


Versioning and Contribution History

Version Date Modification reason Modified by

0.1 05/15/2011 Creation Annika Sällström

0.2 06/03/2011 Contribution Santiago Martínez

0.3 06/03/2011 Review Itziar Ormaetxea, Ainhoa Gracia

1.0 - Final version Annika Sällström

2.0 28/09/2011 Updated version Annika Sällström, Itziar Ormaetxea, Ainhoa Gracia, Santiago Martínez, Jonathan Gonzalez Rios, Gabriele Giammatteo, Robert Samuelsson

of 32


TABLE OF CONTENT

Executive Summary ....................................................................................................................................... 5

1 Introduction........................................................................................................................................... 6

2 Living Lab Connector functionality ........................................................................................................ 7

3 Generic specification of the Botnia Living Lab connector workflow..................................................... 7

4 The Living Lab connector and the IMS‐Botnia use‐case........................................................................ 8

5 Connector Usage ................................................................................................................................. 10

5.1 FIRST PHASE – request/Booking of a resource .......................................................................... 10

5.2 SECOND PHASE – Execution order ............................................................................................. 14

5.3 THIRD PHASE – Management of data ........................................................................................ 17

6 Botnia connector implementation ...................................................................................................... 19

6.1 Resource Management.............................................................................................................. 20

6.2 Resources Execution .................................................................................................................. 26

6.3 Data Management ..................................................................................................................... 30

7 Status of the actual implementation of the Botnia Living Lab connector........................................... 32

8 Benefits for the TEFIS Living Lab use‐case of the Living Lab connector .............................................. 32

9 Future Work ........................................................................................................................................ 32

of 32


Executive Summary This document summarizes the results for the initial work with the Living Lab connector of TEFIS. For this

implementation we have identified four different Botnia resources of relevance for TEFIS.

They are:

‐ User involvement expertise

‐ Methodology for user involvement

‐ The Botnia Living Lab user database

‐ Business model expertise

To design and implement the first version of the Botnia Living Lab connector we have chosen to

elaborate the connector specification via the IMS‐Botnia use‐case to investigate the potential solution

for the Living Lab connector. The deliverable includes the initial design and workflow of the Botnia

connector to cover Resource management, Resource execution and Data management. In the design we

have taken into account the Botnia infrastructure and the TEFIS architecture to optimize the integration

of Living Lab resources for TEFIS users. This deliverable will be updated when the Living Lab facility has

been integrated into the upcoming experiments of the Open call where the Living Lab resources are

expected to be one of requested kind of resources by the experimenters in combination with resources

from other testbeds of TEFIS.

of 32


1 Introduction The main purpose of the Living Lab connector is to provide different Living Lab resources from Botnia

Living lab for TEFIS access.

There are four main Botnia Living Lab resources identified for TEFIS access based upon the initial use‐

case needs (from the IMS‐Botnia use‐case):

1. User‐involvement expertise One resource of Botnia to be accessible via TEFIS is the expertise in user involvement activities. This resource consists of research expertise in the field of user centred design and evaluation and they support experimenters in setting up, and running user involvement activities. 2. Methodology for user‐involvement For the process of user involvement one resource of Botnia Living Lab is the Form‐IT methodology. This is an iterative and interactive process in several steps for user‐engagement in all phases of the development of an IT‐based service/product – from need finding to beta‐trial and pre‐market launch. Different methods and tools are used to provide professional support for user‐involvement. Often we combine qualitative and quantitative methods for the best results like web‐based questionnaires to investigate specific areas among a bigger user‐group and observations and interviews to go into depth around specific issues and to get answers on why and how. For user‐involvement, it is very important to recruit the right users matching the purpose of your experiment. 3. The Botnia Living Lab Users Data Base This is a database of 6000 creative end‐users (individuals) from 18 years of age and older in Sweden and access to end‐users around the world via 3rd parties. The Botnia user database is currently implemented as a MySQL‐database where End‐user basic data for end‐user involvement are stored.

4. Business model expertise

This resource consists of research expertise in the field of business model development and evaluation.

Their involvement is supported by a business model design scheme to support the business model

development.

The expected results for an experimenter when using Botnia Living Lab resources are:

Redesign of products/services;

Decisions for implementation of new functions;

New target user groups;

New ideas as result of user involvement;

Increased knowledge among experimenters/developers;

Established relations with new business partners;

Faster innovation process(shortened time for development) by support from end‐users for decision making;

Business model evolution

of 32


2 Living Lab Connector functionality

The communication protocol for the Living Lab connector is initially foreseen to be as follows:

The Living Lab connector will be installed within the web infrastructure of Botnia Living Lab as a back‐end application,

Access to resources and result data will be mainly manual considering the resources to get access to including result data cannot be controlled by any software, configured, deployed or monitored by others than the Living Lab providers.

In the long‐term, if “virtualization” would be possible, there might be some possibilities to make some

resources available for direct access and configuration remotely.

3 Generic specification of the Botnia Living Lab connector workflow Here follows a workflow to describe the functionality of the connector in the different phases of an

experiment. In next section we will describe this workflow more in details in relation to the IMS‐Botnia

use‐case.

1. User registration:

Experimenter creates a TEFIS user‐account

Action from Botnia Living Lab connector: None

2, Experiment definition: Experimenter looks for available testing resources in the TEFIS Experimental manager Action from Botnia Living Lab connector: None 3. Experiment Configuration Experimenter specify resources to be used and design the tasks involving Botnia resources Action from Botnia Living Lab connector: None 3. Planning Experimenter makes a request for Botnia Living Lab resources by the Experimental manager. Action from Botnia connector: The Request is sent by e‐mail to the Living Lab manager including the requested configuration for resources and the experiment description. Upon the agreement with the experimenter the final configuration is decided. 5. Execution Tasks are performed using Botnia Living Lab resources. Status of execution of tasks is handled in the Botnia Drupal platform (testplats.com). Action from Botnia Living Lab connector: The connector asks the Drupal platform of the execution and parameter (queued, running and finished) . 6.Reporting Botnia tasks are finalized and results are stored in Drupal(testplats.com) and results are uploaded in iRods.

of 32


of 32

Action of Botnia Living Lab connector: None (but it might be depending on the actual functionality of iRods) 7. Knowledge sharing Experimenter wants to share experience from this experiment with other experimenters. Information to share is how the experiment was performed and experience of using Botnia resources. Action of Botnia Living Lab connector: None

4 The Living Lab connector and the IMSBotnia usecase To specify and implement the Botnia Living Lab connector the IMS‐Botnia use‐case has been the baseline to identify the different requirements of the connector. This experiment is focused on a mobile application for content sharing and is divided into three different

phases of the service development life‐cycle: concept development, prototype development and

Business model definition. This experiment addresses the three main issues facing mobile applications

today. First, this experiment will explore end‐user feedback to check if the application is suitable for

them. In the second step, the testbed facilities are used as a validation tool, and in the third step, the

correct business model for long‐term sustainability is investigated. In this use‐case we are combining

experimental resources from two different testbeds independent on place: The SQS IMS testbed in

Spain1 and the Botnia Living Lab in Sweden2.

In this experiment the TEFIS testing service is used for designing, planning, management of experimental

workflow, configuration assistance, experimental data management, reporting, and knowledge sharing

with other experimenters. For this purpose the Botnia Living Lab connector is used to create easy

connection between the TEFIS platform and the Living Lab resources.

The access to Botnia Living Lab is through the Tefis Platform and Botnia Connector. Tefis Platform is also

connected with IMS testbed, so the users of Botnia LivingLab will have access directly to IMSTestbed

1 http://www.tefisproject.eu/media/upload/11‐2445‐SQS‐korr_110204_3.pdf 2 http://www.tefisproject.eu/media/upload/11‐2445‐blad‐botnia‐korr_1102041.pdf

http://www.tefisproject.eu/media/upload/11-2445-SQS-korr_110204_3.pdf

http://www.tefisproject.eu/media/upload/11-2445-blad-botnia-korr_1102041.pdf


TEFIS PLATFORM

Figure 1: IMS-Botnia use-case overview

LIVING LAB SQS TESTBED

Phase 1 Phase 2 Phase 3

LIVING LAB + SQS TESTBED

- Number of users - Q-Mobile testing - Max network bandwidth

- Users profiles - Mobile devices - Max network bandwidth by user - Utility - Functional correct

tests - Users profiles

PHASE1: TEFIS as concept evaluation tool

PHASE2: TEFIS as validation tool PHASE3: TEFIS as

business validation tool

of 32


5 Connector Usage The of thuse e connector in TEFIS will be divided in three phases.

5.1 FIRST PHASE – REQUEST/BOOKING OF A RESOURCE In the first phase, once the experimenter has defined an experiment, the Experiment Manager sends the

order to Teagle for booking the resources. Teagle, in turn, sends an order for each resource to the

corresponding connector.

The flow of this job for the IMS‐Botnia use‐case is presented in a schematic way in the next graphic

(Fig.2):

Experimenter ExperimentManager

Teagle(Booking Processor)

CONNECTOR

Botnia Manager

Botnia’s Resources Database

Creation of the

ExperimentBook the resources

Book theresources

TEFIS

Storage the resources and their configurations

Send an email with information about booked resources

Figure 2: First phase flow

of 32


In the figure below (Fig. 3) the experiment definition for the IMS‐Botnia use case is shown. The

experimenter will book the Botnia Living Lab resources to use:

User involvement Methodology;

User database;

User Evaluation Expertise .incl Business Model Creation Expertise;

The IMS testbed’s resources will be booked in the same step.

Figure 3: Advanced Experiment Definition in Tefis Platform

When the Botnia’s connector receives the order to book a resource, it sends an e‐mail to Botnia

Manager. This e‐mail shall contain all the information related to the requested/booked resource with

their multiple and different configurations. Next, the connector saves the data in the resources database,

so it will be possible to have access to the information in the future.

In this use‐case the connector sends an email for each booked resource (User Involvement Methodology, User Database, Business Model creation and Evaluation Expertise) to the Living Lab manager with the correspondent configurations and stores all parameters in the database.

of 32


In Fig.4, the list of experiments can be seen.

Figure 4: Experiment lists in TEFIS Platform (the Experiment manager)

In Fig. 4, the use‐case is represented in TEFIS Platform when an experiment is created.

In the use‐case there are four different tasks in different steps and in each task different resources are

being used.

In this situation the connector will receive the order from TEFIS to book the requested resources.

The three tasks involving Botnia resources are:

Conecpt evaluation task: User Involvement Methodology, User database and User Evaluation Expertise

resources;

Usability testing: User Involvement Methodology resource;

Business Model Selection : ser Involvement Methodology, User Database, Business Model Expertise and

User Evaluation Expertise resources

The connector sends one mail for each booked resource to the Living Lab manager with the resource

name, type and configuration parameters. This information is also stored in a Database in the Living Lab

connector

of 32


Figure 5: Resources to book in Tefis Platform

As it is already explained in this document, when an experimenter has requested the resources, the way

the Living Lab connector works is to send an email to the LivingLab Manager acting like a trigger for an

internal mechanisms of Botnia to act on the request.

In those mails, as we can see in Fig. 6, the type of the resource, its name and all parameters of

configuration are included.

This is an example of a typical mail the Botnia Testbed responsible manager will receive when an

experimenter has requested a resource:

From: [email protected] To: [email protected] Subject: A new resource has been booked Mail: Resource type: Botnia User Database Configuration: Name: /userdatabase-UserDataBase_1 Users: 150 Age: 25-30 Gender: Male Language: English

Figure 6: E-mail sent after the requesting of a resource

Through the connector all resources has been booked with the configuration that we want without any

human interaction. But the step for resource requesting is in interaction between experimenter and the

Living Lab provider and the change of availability of each resource will be handled via the connector and

the Drupal/testplats.com to change the status of the request from pending to rejected or accepted or

invalid.

of 32

mailto:[email protected]

mailto:[email protected]


5.2 SECOND PHASE – EXECUTION ORDER

The second and third phases are closely related.

The second phase is the Execution of the experiment part, and in it, once the experimenter sends the

execution order via TEFIS portal, the Experimenter Manager sends the execution order to the Scheduler.

The Scheduler then sends one order to the Connector for each resource involved in the task

The connector, in turn, sends an update request for Drupal(testplats.com) to change the status of the

experiment. This request is firstly handled as quing until it´s activated manually in testplats.com as

started. This is reported back via the Living Lab connector and the status of the experiment is changed to

running in the connector. (See Figure 7 and 8 below)

The Botnia Manager takes then control of its own resources.

of 32


.

Figure 7: Second phase flow

of 32


Below is a screenshot of the form in testplats.com where the actual status of an experiment is updated

manually by the Livivng Lab manager. This information is then being reported back to the Living Lab

connector of TEFIS as described before.

Figure 8: Screenshot from status in testplats.com

of 32


5.3 THIRD PHASE – MANAGEMENT OF DATA The third phase is the management of the data.

The Botnia Living Lab Manager, is responsible for sending the files and data to its resources (input data)

and saves the results of the resources (output data) in the Data Manager. (See Figure 9)

Figure 9: Third phase flow

of 32


As a summary, the whole process can be seen in the next graphic:

Figure 10: Botnia connector involvement in the life cycle of an experiment

of 32


6 Botnia connector implementation The main mission of Botnia Living Lab is to serve as a facility to perform the development of ICT‐based products and services in real‐life contexts with real users. Botnia's objectives include the generation of new knowledge, methods and tools, for open user‐centric research and innovation.

The Botnia Connector, even if was developed to manage a particular facility, has to manage three

different features as any connector:

Resource Management;

Resource Execution;

Data Management;

Botnia’s Living Lab has a different behavior if a comparison is made between its connector and the

connectors of the other testbeds included in TEFIS: it has to manage a human made work. In this context,

and to facilitate the interaction between the testers responsible of reviewing the input provided by the

experimenters, e‐mail alerts are used to provide notifications to the people in charge of the Botnia Living

Lab environment.

Thus, when a resource is instantiated, the connector sends an e‐mail to a specific person so they can

generate the resource and put it in the database with its configuration parameters.

Figure 11: Botnia Living Lab layers

of 32


After that resource is executed, the connector has to send to the contact person the execution order by

e‐mail. At the same time, it generates inside database table that order, assigning pending status.

The execution process management, the update of the execution’s status is completely manual. So, it is

necessary to change manually the status in its field on the database.

For each of these features three different programming modules can be distinguished, which names

have a direct correspondence with them.

The programming language used to develop the connector is Python and MySQL is the database.

For the connector implementation MySQLdb, Smtplib and Re modules of Python have been used. These

modules can be downloaded from Python official website: http://python.org/

Besides this, Linux has been chosen as the system where the connector should be executed and

implanted. Botnia Connector is nowadays operative, running in Ubuntu 10.10. Ubuntu can be

downloaded from this URL: http://www.ubuntu.com/download/ubuntu/download

It is possible to access to the Botnia Connector using this URL: http://62.99.76.22:8010/

Figure: 12 The Botnia Living

6.1 RESOURCE MANAGEMENT

Lab connector instances

“Resources are the main concern of connectors: in fact, the main objective of having testbeds connected

to the TEFIS System is to allow TEFIS users to use the resources provided by testbeds for their own

purposes.

TEFIS includes different types of testbeds (e.g. network‐oriented testbeds, services‐oriented testbeds,

and living labs) that have different concepts of resource. To give an idea of the range of resource types

of 32

http://python.org/

http://www.ubuntu.com/download/ubuntu/download

http://62.99.76.22:8010/


of 32

that TEFIS will integrate, consider that in PACAGrid or PlanetLab a resource is a virtual machine, while in

Botnia a resource is a document, a questionnaire, or even a team of people who are “allocated” for an

experiment. Therefore, the very first requirement for connectors is to homogenize the concept of

resource so that the TEFIS System can deal with those resources in a uniform manner.

The lifecycle of a testbed resource is made of these five steps: i) the resource is required by the system,

ii) the resource is created (or, if it already exists, access is grant to the requester), iii) the resource is

configured, iv) the resource is used, v) the resource is released. This is a general and abstract lifecycle,

but each testbed has its own actual resource lifecycle. Connectors are responsible for mapping the actual

testbed resource lifecycle onto the more general TEFIS resource lifecycle. To do so, connectors should

expose an interface offering a set of functions to manage the various steps of the resource lifecycle.

The TEFIS Middleware and TEFIS Portal will leverage functions exposed by connectors to control the

usage of testbed resources in experiments carried out in TEFIS during their different phases: plan,

request, provision, deploy, run and evaluate (see TEFIS deliverable D6.1.1 for further information on

experiment lifecycle).”3

The implementation for the resources management has five different modules with a class each:

Class bbdd.py: this class manages the negotiation with the database

BtUserDBAdapter.py: which inherits from the first class, only exists the definition of the

functions and calls to the first class. It manages “Users Data Base” resources.

BtUserEEAdapter.py: which inherits from the first class, only exists the definition of the

functions and calls to the first class. It manages “User Evaluation Expertise” resources.

BtUserIMAAdapter.py: which inherits from the first class, only exists the definition of the

functions and calls to the first class. It manages “User Involvement Methodology” resources.

BtBMAdapter.py: which inherits from the first class, only exists the definition of the

functions and calls to the first class. It manages “Business model creation and evaluation

expertise” resources.

3 From Document 5.1.1‐Generic and Specific Connector specification


classes structure

In the Botnia specific case, three kind of resources exist. The resource types to be booked from Botnia are:

Users Data Base: It is a Database with End‐users for Future Internet service development and evaluation;

User Evaluation Expertise: It is an Expertise for Human centred design processes;

User Involvement Methodology: It is a Living Lab methodology for service innovation processes by user involvement in all phases from needfinding to pre‐market launch;

Resource Management Module

The functions implemented for the resource management are the following:

list_resources:

Input: the function receives a type of resource

Logic implemented: the purpose of the function is to consult the list of current available

instances that are registered in the database.

Output: it returns all found instances of the specified type of resource defined in the Botnia

Living Lab.

have_resource:

Input: the function receives a type of resource

Logic implemented: the function checks whether or not the specified type of resource exists

Output: it returns True or False

get_resource:

Input: the function receives a name of an instance

Logic implemented: the function checks whether or not the specified instance of a specific type

of resource exists

Output: it returns the name of the instance found

Among others, this function is a complement for other functions considering it is useful

to see configurations.

of 32


add_resource:

Input: name of a new type of resource and configuration

Logic implemented: a new entry in the database for an instance linked to a type of resource is

created with the given name and configuration

Output: an e‐mail is sent to a specific contact person so the Botnia Living Lab Testbed provider is

notified that a new instance has been created.

delete_resource:

Input: name of the instance to be deleted

Logic implemented: the referred instance is deleted from the database

Output: None

get_configuration:

Input: name of the instance

Logic implemented: the function obtains the defined configuration (parameters and current

values) for a specific instance

Output: list of configuration data

get_attribute:

Input: name of the instance and specific attribute (parameter)

Logic implemented: the function obtains the current value contained in specified attribute that is

stored in the database

Output: Current value of the attribute

set_configuration:

Input: list of configuration parameters with their corresponding values and the name of the

instance they are referred to

Logic implemented: the current configuration defined for a specific instance is updated with the

data received

Output: None

of 32


set_attribute:

Input: specific attribute, the value the attribute is intended to have, and the name of the instance it

is referred to

Logic implemented: the attribute or parameter provided is updated with the value received

Output: None

Resource Management Database

The Resources management is based on the database created in the LivingLab. This DataBase will keep

all the information related to the LivingLab resource and any instance booking or execution query.

The Resource Management Database has two tables, one for managing the resources and other one for

managing the number of instances.

The structure of the table to manage the resources module is presented below:

Structure of the table for the resource management

Any type of resource defined for a testbed should have the corresponding code inside a Resource

Adapter class, so it is possible to establish the communication between that resource and the module of

the TEFIS platform that wants to read or provide information from or to the testbed.

In this context, a table should be created in the database to contain different kind of data about the

resource and configuration of instances.

The structure of such table is more detailed below, including the names of the columns defined and the

information they contain.

id

Type of data: integer

of 32


It is the primary key, it is a unique and not null identifier of a specific row of the database. It is

used to have access to a row so searches are possible, and it does not contain useful information

name

Type of data: string

It is the name used to identify a specific instance of the resource. Its structure is:

/Botnia‐Resource_X

common_name


It is the name given by the user to identify a concrete instance.

age

Type of data: string that identifies ranges, eg. 22‐30

Configuration parameter of the resource that contains the age range of the working people

inscribed in the Living Lab.

gender


Configuration parameter that represents the gender of the users of the Living Lab.

users


Configuration parameter that corresponds to the number of the intended users of the Living Lab.

language


Configuration parameter that corresponds to the language of the intended users of the Living

Lab.

Resource_type_id


Configuration parameter that corresponds to the kind of resource.

of 32


The structure of the table to manage the number of resources module is presented below:

Resource Type

PK id

common_namenumber_of_instances

Structure of the table to manage the number of resources

id


It is the primary key, it is a unique and not null identifier of a specific row of the database. It is

used to have access to a row so searches are possible, and it does not contain useful information

common_name


It is the name given by the user to identify a concrete resource

.

Number_of_instances


It is the number of instantiated resources. If the resource has to have infinite instances this

number always will be 0.

6.2 RESOURCES EXECUTION

“A TEFIS experiment is made up of a number of steps that will be executed, in parallel or sequentially,

each one in one of the testbeds connected to the TEFIS System. Of course, testbeds offer heterogeneous

interaction paradigms and usage methods and, therefore, the definition of what a step of an experiment

is is liable to change depending on the testbed in which it is executed. In particular, it is interesting in this

section to analyze the level of execution automation provided by the testbeds. With regards to the

testbeds in the TEFIS project, it is easy to infer that the different testbeds have different levels and types

of 32


of 32

of automation: from the complete manual execution that takes place in Botnia, to the completely

scriptable PACAGrid executions.”4

The functionality for the current and future resources execution of the Botnia Living Lab consists in

alerting (by email) people in charge of executing the task and status changes in the testplats.com

environment. The actual execution of the previously created and booked resource, and this functionality

is executed manually by the Living lab staff.

In order to achieve this goal, a database table has been created. In this table, it is saved the execution

state and the corresponding used folders with the data of that execution for a given experiment.

This functionality is implemented using a class with two functions in BotniaConnector.py module.

+get_info()+execute()+get_execution_status()

BotniaConnector

Structure of the class for resource execution

Next, the functions implemented in that module are described:

get_info:

Input: nothing

Logic implemented: the function obtains the name and version of the connector

Output: The name and version of the connector are returned

execute:

Input:

a context execution object that could consist on the id of the experiment and the folders

used to take and save the data

the list of resources selected for the experiment

the execution script in case it exists

Logic implemented: the function performs the execution of the experiment for the given context,

with the instances selected that should be configured and it shall be in charge of executing a

script if it is provided

Output: the execution unique identification

get_execution_status:

4 From Document 5.1.1‐Generic and Specific Connector specification


Input: execution identification

Logic implemented: the function obtains the status of the given execution

Output: the current status of the execution. The status can be:

Running

Failed

Pending

Success

Resource execution database

The resource execution table structure is presented below:

Execution Table

PK id

exec_idexperiment_idinput_data_folderoutput_data_folderexperiment_root_data_foldertask_run_input_data_foldertask_run_output_data_folderexecution_scriptresourcesstatus

The structure of the Execution table is more detailed below, including the names of the columns defined

and the information they contain:

id

Data type: Integer

It is the primary key for each execution. It is a unique and not null identifier of a specific row of

the database. It is used to have access to a row so searches are possible.

exec_id

Data type: Integer

It is the identifier for a specific execution.

of 32


experiment_id

Data type: String

It is part of the context execution object. It represents the identifier of the experiment to be

executed or that has been already executed, that includes a set of resource instances and

configurations

input_data_folder

Data type: String

Folder where the needed or input data, provided among others by the experimenter, that makes

possible the execution of a context is going to be located. It is part then of the context of an

execution object.

output_data_folder

Data type: String

Folder where the result data obtained after the context execution for a set of resource instances

with specific configuration is going to be located. It is part of the context execution object.

experiment_root_data_folder

Data type: String

General folder of the experiment. It is part of the context execution object.

task_run_input_data_folder

Data type: String

Folder with the execution tasks defined by the experimenter. These tasks shall contain a list of

resources and have a specific order of execution. It is part of the context execution object.

task_run_output_data_folder

Data type: String

Folder where the data obtained after the execution of a given task within an experiment will be

saved. It is part of the context execution object.

execution_script

Data type: string

It contains the execution script

resources

Data type: String

of 32


List of resources that is going to be executed

status

Data type: String

Status of the process of execution. The status can be:

Running

Failed

Pending

Success

6.3 DATA MANAGEMENT “TEFIS will have to manage different types of data related to experiments: experiment definition data,

experiment configuration data, experiment input data and experiment output data. An analysis of the

characteristics of that data (type, scope, amount, flows) has been carried out during the initial stages of

the architectural definition summarized in TEFIS deliverable D6.1.1.

The analysis of data that flows in the TEFIS System has highlighted that not all data will be stored in the

TEFIS infrastructure. For instance, data generated within testbeds during experiment runs will be left in

the testbed facilities since the effort to move and store that data in the TEFIS infrastructure is too high.

With this respect, the amount of data generated during an experiment can be very high (for example a

log file for a web server under stress during a performance test).

Considering these issues, the experimental data management system in TEFIS shall be distributed and

there shall be available several data storage services in the system:

The central data service (the Research Platform Repository Service ‐ RPRS) that is deployed in the TEFIS

infrastructure and that communicates with native data storages from testbeds, via the Testbed

Infrastructure Data Service (TIDS).

With this architecture, most of the data generated during the experiment will remain in the data storage

of the testbed and will be accessed by TEFIS directly from there. Connectors must support the

communication between the TEFIS data management system and the testbed data storage offering a

common interface through which the RPRS can interact with the latter to retrieve experimental data.

The fact of being a facility manually managed is, also, an important point on this implementation; the

Botnia Living Lab has not been automated, and the lab providers shall be the intermediates between the

experimenters using the TEFIS platform and the users working on the Living Lab.

It has been decided that any data activity will be managed using get_input_file() and put_output_file()

from Connector.py class:

get_input_data:

of 32


Input: id of the experiment, and file to has got.

Logic implemented: the function get the file stored in the data manager.

Output: None

put_output_data:

Input: id of the experiment, and file to has let.

Logic implemented: the function let the result of execution file in the data manager

Output: None

To make it possible to use these functions is necessary install an iRODS server in the same machine as

the connector runs. It can be downloaded from: https://www.irods.org/index.php/Downloads

Also, it is necessary install PyRods, that is a python frameworks to communicate with iRODS server. It can

be downloaded from: http://code.google.com/p/irodspython/downloads/list

The configuration of the Tefis iRODS Server is:

[data‐management]

Irods‐server=tefis6.inria.fr

Irods_port=1247

Irods_zone=TefisTest

Irods_user=tefisUser1

Irods_password=tefisUser1

of 32

https://www.irods.org/index.php/Downloads

http://code.google.com/p/irodspython/downloads/list


of 32

7 Status of the actual implementation of the Botnia Living Lab connector

After the completion of the first phase, a first implementation of the Living Lab connector is available to be used for the IMS‐BOTNIA use‐case in the first version of the TEFIS platform. A Living Lab use‐case is running and the Botnia Living Lab connector is working into the overall architecture of TEFIS platform and with respect to the Data Manager (with limited functionality, as said), Experiment definition, etc. The Living Lab connector presented in this deliverable is considered to be very different from other TEFIS testbed connectors because it deals with humans and the specific peculiarities of its implementation have been explained. In the near future, it has been considered to implement the first final version of the connector for the Open call experiments and they will be the first beta‐testers of the Living Lab connector.

8 Benefits for the TEFIS Living Lab usecase of the Living Lab connector

To conclude, the Botnia Living Lab connector is really needed for the IMS‐Botnia use‐case described

before as it allows to keep the Living Lab Manager informed of the entire process of the experiment,

sending the relevant work for execution, storing of resource configurations and keep track of execution.

Additionally, the connector will give the experimenter the ability of requesting, booking, configuration

and execution of the Living Lab resources and to support the interaction with the Botnia Living Lab team

and it also makes the Living lab resources to be accessible and used in parallel with other testing

resources.

9 Future Work In future work we plan to investigate the monitoring capabilities for Botnia Living Lab experiments and

further integration between the Botnia Drupal platform (testplats.com), the Living Lab connector and the

Data management functionality of iRods. What also is necessary to develop further is the functionality of

the connector when a task involves resources from different testbeds including Botnia Living Lab. We

also need to make a solution for how to design the functionality of the Experiment manager to handle

parallel tasks and dependencies between tasks and how this might influence next version of the Living

Lab connector design and its implementation.

TEFIS D5.4 - CORDIS · TEFIS - D5.4.1– Connector implementation and documentation for LivingLab....

Documents

Transcript of TEFIS D5.4 - CORDIS · TEFIS - D5.4.1– Connector implementation and documentation for LivingLab....