DELIVERABLE D2.6 Evaluation Report of the bIoTope...

DELIVERABLE

This project has received financial support from the European Union Horizon 2020 Programme under grant agreement no. 688203.

D2.6 Evaluation Report of the bIoTope Pilots

Project Acronym: bIoTope

Project title: Building an IoT Open Innovation Ecosystem for Connected Smart Objects

Grant Agreement No. 688203

Website: www.bIoTope-project.org

Version: 1.0

Date: 30 June 2017

Responsible Partner: BIBA – Bremer Institut für Produktion und Logistik

Contributing Partners: BMW, CIRB, ControlThings, CSIRO, Enervent, FVH, Greater Lyon, IRISNET

Dissemination Level: Public X

Confidential – only consortium members and European Commission Services

Ref. Ares(2017)3317345 - 03/07/2017


© 688203 bIoTope Project Partners 2 30th of June 2017

Revision History

Revision Date Author Organization Description

0.1 29/05/2017 Dirk Werthmann BIBA Initial draft

0.2 12/06/2017 Dirk Werthmann BIBA Final draft for reviewing

0.3 20/06/2017 Robert Hellbach BIBA Inserting remarks and comments from

reviewers

1.0 28/06/2017 Robert Hellbach BIBA Final Version

Every effort has been made to ensure that all statements and information contained herein are accurate, however the bIoTope Project Partners accept no liability for any error or omission in the same.



Table of Contents

1. Introduction ..................................................................................................................................... 8

1.1. Objective of the Deliverable ........................................................................................................................ 8

1.2. Structure of the Deliverable ......................................................................................................................... 9

1.3. Evaluated subjects within bIoTope .............................................................................................................. 9

1.3.1. bIoTopes’ Core Components .................................................................................................................... 10

1.3.2. bIoTopes’ Pilots ........................................................................................................................................ 10

2. Progress of development within bIoTope ........................................................................................ 12

2.1. Evaluation by assessing the Technology Readiness Level as well as the Integration Readiness Level within

bIoTope .................................................................................................................................................................. 12

2.2. Evaluation of Core components ................................................................................................................. 18

2.3. Evaluation of Technologies developed within the Cross-domain Smart City Pilots..................................... 19

2.3.1. Pilot: Brussels – Capital Region ................................................................................................................. 19

2.3.2. Pilot: FVH – Forum Virum Helsinki ............................................................................................................ 19

2.3.3. Pilot: Greater Lyon .................................................................................................................................... 19

2.4. Evaluation of Technologies developed within the Domain-specific Pilots .................................................. 20

2.4.1. Pilot: Smart Mobility ................................................................................................................................. 20

2.4.2. Pilot: Smart Building and Equipment ........................................................................................................ 20

2.4.3. Pilot: Smart Air Quality ............................................................................................................................. 20

3. Matching of KPIs and Requirements ................................................................................................ 21

3.1. Evaluation Methodology for Matching of KPIs and Requirements ............................................................. 21

3.2. Evaluation of Cross-domain Smart City Pilots ............................................................................................ 23



3.2.2.1. Use Case: Electrical Car charging service using IoT BnB concept ......................................................... 25


3.2.3.1. Use Case: Bottle Banks ......................................................................................................................... 27

3.2.3.2. Use Case: Heat Wave ........................................................................................................................... 29

3.3. Evaluation of Domain-specific Pilots .......................................................................................................... 31




4. Preparation of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots ....................... 42

4.1. Evaluation of bIoTope Pilots based on User Acceptance Testing ................................................................ 42

4.2. Evaluation of Cross-domain Smart City Pilots ............................................................................................ 49



4.2.2.1. Use Case: EV Charging & Parking App .................................................................................................. 51


4.2.3.1. Use Case: Bottle Banks ......................................................................................................................... 53

4.2.3.2. Use Case: Heat Wave ........................................................................................................................... 55

4.3. Evaluation of Domain-specific Pilots .......................................................................................................... 57




4.3.4. Preparation of the KPI Test Scenarios for ex-post Evaluation of the bIoTope Pilots ................................ 63

5. Conclusion ...................................................................................................................................... 66

6. References ..................................................................................................................................... 67

7. Annex 1 .......................................................................................................................................... 68



List of Tables

Table 1: Technology Readiness Levels (TRL) definition [U.S. Government Accountability Office] ................... 13

Table 2: Integration Readiness Level (IRL) [Sauser] .......................................................................................... 14

Table 3: Core Components assessed by using TRL ............................................................................................ 15

Table 4: Technologies/ Software Components assessed by using TRL ............................................................. 15

Table 5: TRL and IRL of bIoTopes’ Core Components ........................................................................................ 18

Table 6: TRL and IRL of Pilot: Helsinki ................................................................................................................ 19

Table 7: TRL and IRL of Pilot: Greater Lyon ....................................................................................................... 19

Table 8: TRL and IRL of Pilot: Smart Mobility .................................................................................................... 20

Table 9: TRL and IRL of Pilot: Smart Building and Equipment ........................................................................... 20

Table 10: TRL and IRL of Pilot: Smart Air Quality............................................................................................... 20

Table 11: Structure of the spreadsheet for matching KPIs and Requirements ................................................. 22

Table 12: Matching of KPIs and Requirements for Pilot: Brussels – Capital Region ......................................... 23

Table 13: Matching of KPIs and Requirements for Pilot: Forum Virium Helsinki - Electrical Car charging service

using IoT BnB concept ............................................................................................................................... 25

Table 14: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Bottle Banks ..................... 27

Table 15: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Heat Wave ....................... 29

Table 16: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Charging Station Selection

Service ....................................................................................................................................................... 31

Table 17: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Route Planning Service . 33

Table 18: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Electric Car Gear Service34

Table 19: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart

Building Interaction with Smart Mobility .................................................................................................. 35

Table 20: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart

Equipment ................................................................................................................................................. 38

Table 21:Matching of KPIs and Requirements for Pilot: Smart Air Quality – Identify polluted areas .............. 39

Table 22: Matching of KPIs and Requirements for Pilot: Smart Air Quality – Scenario Provide councils view of

their city ..................................................................................................................................................... 41

Table 23: Example of a Test Condition Matrix .................................................................................................. 46

Table 24: Example of a Test Schedule ............................................................................................................... 46

Table 25: Start dates and end dates for User Acceptance Testing within bIoTopes’ Pilots .............................. 48

Table 26: Test Condition Matrix for Pilot: Brussels – Use Case: Safety around School .................................... 49

Table 27: Test Schedule for Pilot: Brussels – Use Case: Safety around School ................................................. 50

Table 28: Test Condition Matrix for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App...... 51

Table 29: Test Schedule for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App .................. 52

Table 30: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Bottle Banks ........................................... 53

Table 31: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Heat Wave .............................................. 55

Table 32: Test Condition Matrix for Pilot: Smart Mobility ................................................................................ 57

Table 33: Test Schedule for Pilot: Smart Mobility ............................................................................................. 58

Table 34: Test Condition Matrix for Pilot: Smart Building and Equipment ....................................................... 59

Table 35: Test Schedule for Pilot: Smart Building and Equipment .................................................................... 60

Table 36: Test Condition Matrix for Pilot: Smart Air Quality ............................................................................. 61

Table 37: Test Schedule for Pilot: Smart Air Quality ......................................................................................... 62

Table 38: Progress of the preparation for the KPI Test Scenarios for ex-post Evaluation ................................ 63

Table 39: Start dates and end dates for executing the KPI Test Scenarios for ex-post Evaluation within bIoTopes’

Pilots .......................................................................................................................................................... 64

Table 40: Template for the KPI-based ex-post Evaluation of the bIoTope Pilots .............................................. 65



List of Figures

Figure 1: Initial Evaluation Procedure of the Business Services within bIoTope according to D2.2 “Evaluation

Methodology for Pilots Validation” ........................................................................................................... 43

Figure 2: Status of User Acceptance Testing at month 18 of bIoTope [Hambling and von Goethem] ............. 44



Executive Summary

This deliverable (D2.6) reports the evaluation of technologies developed within bIoTope, especially the domain-specific pilots,the cross-domain smart city pilots and furthermore used technologies developed in WP3-5 and used in the pilots. D2.6 keeps track of the development of the project by assessing the Technology Readiness Level (TRL) as well as the Integration Readiness Level (IRL) for the core components, the cross-domain city pilots and the domain-specific pilots. The deliverable also evaluates the matching of Key Performance Indicators (KPIs) and requirements for the cross-domain smart city pilots and the domain-specific pilots. Finally, D2.6 prepares the ex-post evaluation of the bIoTope pilots based on user acceptance testing and for the cross-domain smart city pilots.

Based on the previous deliverable D2.2 “Evaluation Methodology for Pilots Validation” from the track of Task T.2.E “Evaluation Methodology and Pilot Validation” the evaluation methodology is applied amongst the project results, reflecting an intermediate status of the bIoTope project due to month 18. Within the evaluation, the project partners evaluated each technology according to the status of the ongoing development as well as defined a target level (based on TRL and IRL), which they want to reach until the end of the project. Pilots as well as technologies that are developed in WP3, 4 and 5, which are fundamental for an interoperation in an IoT ecosystem, are considered. Those technologies (see Table 5 and regarding deliverables for further details) rely to the Everything-as-a-Service (XaaS) approach, support the interconnection / orchestration of heterogeneous services and establish business value inside the bIoTope ecosystem.

After a short introduction to this deliverable in the first chapter, each pilot is quantitative evaluated based on the TRL and IRL description. Summarized, current cross-pilot TRL assessment turned out legitimately high, while IRL provides more innovation potential. This is actually the representation of a problem statement, bIoTope address in the current landscape of IoT, whereas already existing smart objects (products with high TRL) need to be orchestrated in a way that seamless information flow between vertical silos can be achieved. The problem representation consequently is directly characterized in the IRL indications of the pilots and XaaS connected technologies.

Facing previous results, the following chapter documents the mapping of recorded requirements from deliverable D2.1 with KPIs stated in deliverable D2.2. This shall ensure an alignment of the evaluation procedure to the intended project results according to the pilots. The relationship of requirements to KPIs is n to 1, which addresses a measurability in the evaluation process. Then, the already prepared evaluation procedure starts with the preparation of User Acceptance Testing. This considers test conditions and schedule for each pilot and gives the pilot case owners a guideline for evaluating their developments. Future work is and conclusions are stated in the last chapter, while changes that will occur during the agile development will be documented in the following deliverable D2.9.



Terms

Big Data as a Service (BDaaS) A core component of bIoTopes’ ecosystem to handle the amount of

services and respectively data in the ecosystem.

Billing as a Service (BiaaS) A core component of bIoTopes’ ecosystem to buy and pay for services.

Business Service (BS) A digital service helping end users to perform a task in a better way.

Context as a Service (CoaaS) A core component of bIoTopes’ ecosystem to respond intelligently to

context-related events.

Development Test & Evaluation (DT&E) A testing procedure, which is conducted during the engineering and

development process to ensure that the specifications have been met.

Everything-as-a-service (XaaS) Approach used in bIoTope to define functionalities as services used in

a system of systems landscape.

Information as a Service (IaaS) A core component of bIoTopes’ ecosystem to discover services within

the ecosystem.

Integration Readiness Level (IRL) A metric of the integration maturity between two or more

components.

Internet of Things (IoT) Interconnection and -action (collecting and exchange of data) of things

(physical devices) embedded with electronics, software, sensors,

actuators and connectivity.

Internet of Things service PuBlication

and Billing (IoTBnB Marketplace/ Service

Catalogue)

A core component of bIoTopes’ ecosystem that acts as a service

catalogue for discovering and consuming data, connected to a billing

framework that allows micropayments for the value of data and / or

services.

Knowledge as a Service (KaaS) A core component of bIoTopes’ ecosystem to interpret and analyse a

huge amount of data.

Key Performance Indicator (KPI) An indicator to evaluate the success of organisations or activities.

National Aeronautics and Space

Administration (NASA)

A governmental organisation of the United States of America, being

responsible for aeronautics research, explores the earth and the whole

universe, development and operation of space technology as well as

commercial spaceflights.

Open Source Software (OSS) Licensed source code that is available for studying, change and

distribution to anyone for any purpose.

Operational Test & Evaluation (OT&E) A field test, which is conducted under realistic conditions.

Security as a Service (SeaaS) A core component of bIoTopes’ ecosystem to ensure a trust-based

communication of new business models to protect its stakeholders

and their data / services..

System of systems (SoS) Collection of different systems, which can co-exist as stand--alone

version, but are able to pool resources and capabilities.

Technology Maturation Plan (TMP)

A plan, which is aiming at bringing immature critical technology up to a desired TRL.

Technology Readiness Assessment (TRA) A systematic that evaluates the maturity of technologies.

Technology Readiness Level (TRL) A measurement scale used to rate technologies according to their maturity.



1. Introduction

bIoTope aims to improve economic, ecological and societal sustainability by supporting innovation and cocreation of services in the Internet of Things (IoT). To achieve this, bIoTope develops an open, interoperable, secure and highly context-sensitive Systems-of-Systems (SoS) platform for IoT that will enable developers to publish, consume and compose services without any programming.

Within the project, work package WP2 “Requirements, Methods and Pilot Validation” is responsible for producing the technical and Open Source Software (OSS) specifications of the bIoTope IoT based SoS platform collected and processed from all bIoTope stakeholders. Stakeholders can be cities, application integrators, developers, end-users, and others. The structured and methodical collection and analysis of requirements allows not only the definition and subsequent development of the City Pilots in the involved cities, but also the definition of a proper methodology for pilot evaluation and validation throughout the project duration. Thus, WP2 also supports the software development process and ensures that coding procedures and conventions are respected throughout the project. The overall goal of WP2 is to ensure, that the bIoTope project will reach the objectives presented in the Part B of the proposal.

To summarize,WP2 has to achieve the following tasks:

Elicit and analyse requirements from all the stakeholders involved in the bIoTope ecosystem, including the involved cities, infrastructure providers, application integrators, platform developers, end-users, open source communities.

Define the most relevant City Pilots according to the requirement analysis outcomes.

Produce technical specifications of the bIoTope SoS platform for the IoT in terms of functional and non-functional features. The structuring specification also addresses the open source project.

Specify an evaluation methodology and validation plan of the bIoTope pilots.

1.1. Objective of the Deliverable

In the context of WP2, deliverable D2.6 reports part of the results of Task 2.E “Evaluation Methodology and Pilot Validation”. Task 2.E develops a framework for the evaluation of bIoTopes’ Core Components and Pilots. The periodic evaluation in this task enables agile adjustment of the validation arrangements where needed, peer learning and knowledge transfer, as well as best practice sharing across the Use Cases. The aim of the evaluation framework is to assess the Use Case internal arrangements and validation processes, as well as their impact against the KPIs assigned for the validation outcome.

The evaluation includes ex-ante-evaluation and ex-post-evaluation. The ex-ante evaluation focuses on actors, processes and interfaces in the Use Cases. The ex-post evaluation will focus on policy, economic, societal, technical and environmental impact of the Use Cases. Such evaluation methodology allows refining RTD specifications all along the project. This version of deliverable D2.6 is mainly concerned with ex-ante-evaluation, while ex-post-evaluation will be performed at a later stage. However, proper preparation of the ex-post-evaluation is an important topic in this version as well.

The evaluation work reported in deliverable D2.6 is based on the evaluation methodology developed in deliverable D2.2 “Evaluation Methodology for Pilots Validation” and the general description of the quality methodology within deliverable D1.2 “Quality plan”. Deliverable D1.2 specifies the objectives and initial set of Key Performance Indicators (KPIs) in general. Thereupon, Deliverable D2.2 describes evaluation methodologies for assessing bIoTopes’ Core Components and Pilots. Prominent among the system assessment methodologies are the Technical Readiness Level (TRL) and the Integration Readiness Level (IRL) assessment. Moreover, User Acceptance Testing is part of the evaluation as well. These methodologies have been used in order to prepare the evaluation reported in this deliverable.

Deliverable D2.2 also describes the state-of-the-art in the Cross-domain Smart City Pilots and the Domain-specific Pilots, as well as the benefits of bIoTopes’ implementations within these pilots.



Deliverable D2.6 also considers the results of deliverable D2.1 “Ecosystem Stakeholder Requirements Report and Pilots Definition” and deliverable D2.4 “bIoTope SoS Reference Platform Specification” (version 1.1 from 20 January 2017).

Deliverable D2.1 reports the requirements of the various bIoTope stakeholders, such as infrastructure providers, application integrators, and application service providers, according to the envisaged value chains. Deliverable D2.4 transforms the fundamental requirements into a specification of the overall architectural framework and the specifically detailed Pilots. It describes the development of the SoS platforms and service components, as well as the implementation strategies of the Pilots and their interrelations. Thus, the requirements of each Use Case described in deliverable D2.1 have been detailed in user stories and use cases descriptions.

What is evaluated in deliverable D2.6 are Core Components and Pilots. On the one hand D2.6 evaluates technologies, which means the critical services and Core Components that bIoTope develops. On the other hand, it evaluates real world Use Cases within the Cross-domain Smart City Pilots and the Domain-specific Pilots, in which bIoTope implements the developed technologies.

1.2. Structure of the Deliverable

The deliverable consists of five main chapters, each comprising several sections.

Chapter 1 describes the objectives of the deliverable within the context of work package 2 (section 1.1) and the structure of the deliverable (section 1.2)

Chapter 2 keeps track of the development progress within bIoTope by assessing the Technology Readiness Level (TRL) and the Integration Readiness Level (IRL), the core components (services), the cross-domain city pilots and the domain-specific pilots. Section 2.1 assesses the Technology Readiness Level (TRL) and the Integration Readiness Level (IRL); section 2.2 assesses the core components. Section 2.3 evaluates the technologies developed within the cross-domain smart city pilots and section 2.4 evaluates the technologies developed within the domain-specific pilots.

Chapter 3 evaluates the matching of Key Performance Indicators (KPIs) and requirements. Section 3.1 assigning the defined KPIs to the functional requirements. Based on this assignment, section 3.2 evaluates the matching of KPIs and requirements for the cross-domain smart city pilots, while section 3.3 evaluates it for the domain-specific pilots.

Chapter 4 preparation of the ex-post Evaluation of the bIoTope Pilots. Section 4.1 prepares the evaluation by assessing the improvements based on user acceptance testing, whereas section 4.2 prepares the evaluation of cross-domain smart city pilots.

Chapter 5 is concluding the presented results. Further actions are described and explicit procedure documented in the previous evaluation deliverable D2.2.

1.3. Evaluated subjects within bIoTope

As mentioned before deliverable D2.6 is about the evaluation of technologies developed within bIoTope, mainly the Core Components. The Use Cases make use of the Core Components, in terms of combining them and adding individual services in order to realize Business Services being of value for the end-users. bIoTopes’ Core Components are presented in chapter 1.3.1 to provide sufficient information for the reader of deliverable D2.6. In addition, deliverable D2.6 is about the evaluation of bIoTopes’ Use Cases. Those Use Cases are based on the technologies developed within bIoTope and will demonstrate the improvements which can be achieved by bIoTope. Chapter 1.3.2 explains the Use Cases as being part of different Pilots in which the Test Scenarios will make the improvements assessable.



1.3.1. bIoTopes’ Core Components

The overall architectural framework of bIoTopes’ SoS is based on an “Everything-as-a-Service” (XaaS) paradigm, where six key XaaS topics, considered as essential for any Software-as-a-Service (SoS) Platforms are created and tested:

IaaS - information-as-a-Service: This service includes a standards-based framework, around Open API standards, for the publication of information provided and consumable by heterogeneous Smart Objects.

KaaS - knowledge-as-a-Service: This service creates knowledge from data and information, making use of semantic web, data analytics and machine learning tools.

CoaaS - context-as-a-Service: This service, also referred to as “knowledge broker” invokes software components to discover, predict, validate and supply relevant ‘context(s)’ to applications and/or entities requesting it.

SeaaS - Security-as-a-Service: This includes offering of security related applications, such as e.g. anti-virus software, delivered externally. It includes development of new systems that are “context-sensitive”, i.e. able to self-adapt protection and privacy policies based on one or more “Context dimensions”.

BiaaS - Billing-as-a-Service: This service includes provision of core billing operations over a cloud infrastructure.

BDaaS - BigData-as-a-Service: Big data is an increasingly important paradigm that is driven by the pervasive diffusion and adoption of smart connected objects, mobile devices, social media tools, and other forms of information systems. The owner of big data may conduct data storage, management, and analysis and provide Web APIs for users to access the service-generated big data or the analyzed results; or may outsource the big data processing (or part of it) to a third party.

The IoTBnB - Marketplace and service catalogue: This service acts as a catalogue for discovering and consuming data. Inherent of this component is a discovery engine and a directory service to link several services to a new business process.

Each XaaS will result in one or more standards-based software core components, forming in sum the bIoTope XaaS Suite.

1.3.2. bIoTopes’ Pilots

bIoTope develops seven pilots that ensure innovation. Four pilots are so called Domain-specific Pilots to ensure innovation, dissemination and exploitation impact through the involved companies. One pilot (Smart Waste Management, see following bullet points) is provided by an associated partner of the consortium, which leads to special constellation regarding reporting and timing. In addition to these Domain-specific Pilots, bIoTope has established three Cross-domain Smart City Pilots. These Cross-domain Smart City Pilots serve as key proofs-of-concept of horizontal system interoperability across multiple application domains. Moreover, three cross-domain Smart City pilots in bIoTope offer the possibility to compare and evaluate variances in requirements and capabilities, as different legislation. The following paragraphs describe each pilot briefly.

Domain-specific Pilots

Smart Mobility: This Domain-specific Pilot involves consists of three Use Cases such as a Charging Station Selection Service, a Route Planning Service and an Electric Car Gear Service. Every Use Cases wants to make the use of electric vehicles more convenient. Therefore, the involved project partners will develop new Business Services based on cloud and IoT technologies.

Smart Building and Equipment: This Domain-specific Pilot consists of two Use Cases such as Smart Building Interaction with Smart Mobility (in particular, e-cars), and Smart Equipment (in particular, air handling units). The main objectives of the Pilot are to make home automation solutions more



common by developing solutions, which are based on a reduced complexity and can be implemented easily in order to reduce costs. Moreover, the Pilot wants to come up with open interfaces to avoid lock-in effects to just one manufacturer and to allow interaction with other smart devices.

Smart Air Quality: The smart air quality Domain-specific Pilot involves Use Cases such as identification of polluted areas, an overview about the pollution across the whole city for the city council, and a better experience to push bike riders. Crowd-sensing by citizens carrying mobile sensors can achieve the necessary granularity for realizing the planned Business Services.

Smart Waste Management*: This Domain-specific Pilot serves as test environment for a waste management system. Smart garbage bins (SGBs) around the urban area of St. Petersburg are equipped with sensors and actuators. Using micro-billing mechanisms, citizens are motivated to deliver waste to those SGBs. The SGBs help to monitor their status and offering a broad overview on the waste situation in the area of St. Petersburg that helps to properly plan waste management (truck routing, capacities, maintenance, etc.)

* The Smart Waste Management pilot is provided by the ITMO University from St. Petersburg, whom is an associated partner of the bIoTope project. Their access to the project started in month 11 and the Russian ministry realizes the founding for this partner. Therefore, a synchronization / alignment of created results is not feasible. A proper consideration of the pilot regarding the evaluation might be possible for the upcoming version of this deliverable planned for month 36 at the end of bIoTope. Nevertheless synergies between the pilots (especially similarities to Lyons bottle bank use case) are valuable, even though reporting on evaluation is not synced.

Cross-domain Smart City Pilots

Brussels – Capital Region: This Cross-domain Smart City Pilot serves as test environment for the Use Case for establishing safe roads for children and bikes’ green light. The Pilot will make use of already existing sensor networks and connect the data to realize the necessary business services.

FVH – Forum Virum Helsinki: This Cross-domain Smart City Pilot serves as test environment for the Use Case for an electrical car charging service using the IoT BnB concept. (private charging stations appended to public systems).

Greater Lyon: This Cross-domain Smart City Pilot serves as test environment for the Use Cases for bottle bank services and heat wave services. Improving the services for bottle bank emptying will need to implement new sensors into the bottles banks and make the data available to the city council and the citizens. Services based on hyperlocal weather data make use of already available sensor data for triggering actions like watering of plants.

For the pilot cases, deliverable D2.1 has described basic requirements, whereas deliverable D2.4 has further refined them in order to derive specifications. Each pilot has been specified by user stories, descriptive specifications, enumerated data sources and ArchiMate models, which serve as a formal notation of the intended functionalities that are revealed to the bIoTope ecosystem.



2. Progress of development within bIoTope

Based on the approach described in chapter 2.1. of deliverable D2.2 “Evaluation Methodology for Pilots Validation”, the assessment of bIoTopes’ Core Components as fundamentals for an ecosystem and the technologies/ software components developed within bIoTopes’ pilots is described in the following chapters. Before presenting details of the evaluation of each Core Component and the technologies/ software components, the progress of development is described in section 2.1, in general. Section 2.2 is considering bIoTopes’ Core Components, whereas section 2.3 is looking on the technologies/ software components developed within bIoTopes’ cross-domain smart city pilots and chapter 2.4 is doing the same considering the domain-specific pilots. The assessment is done by calculating the Technology Readiness Level (TRL) according to the U.S. Government Accountability Office [U.S. Government Accountability Office] and the Integration Readiness Level (IRL) according to Sauser [Sauser].

2.1. Evaluation by assessing the Technology Readiness Level as well as the Integration

Readiness Level within bIoTope

The project management members uses the assessment based on TRL and IRL to monitor the progress of the development of bIoTopes’ Core Components as well as the different technologies/ software components developed within bIoTopes’ pilots. Based on the results of the evaluation the project management members take necessary actions, in order to achieve the objectives planned. Furthermore, the project management members will use the evaluation to review the technical approaches of bIoTope and refine bIoTopes’ Research and Technological Development (RTD) specifications, if needed.

Status of the development within bIoTope

During the intermediate evaluation of all bIoTope technologies, which are critical for bIoTopes’ success, all projects partners being responsible for the development of a critical technology executed a self-evaluation. Within the evaluation, the project partners evaluated each technology according to the status of the ongoing development as well as defined a target level, which they want to reach until the end of the project. The results of this evaluation process are presented in the subsequent chapters.

Before presenting the results, the following paragraphs describe the steps and actions of the evaluation process. The TRL process, as defined by the Government Accountability Office’s (GAO) [U.S. Government Accountability Office] was taken as reference, when assessing bIoTopes’ Core Components and the technologies/ software components. Along with the TRL assessment the IRL assessment was executed. Section 2.1.1 of deliverable D2.2 describes this process, which the whole consortium has approved. The description of the TRL and the IRL assessment in the following sections is based on this approach.

Step 1 Design the overall technology and integration maturity assessment strategy for the programme or project.

As already mentioned, section 2.1.1 of deliverable D2.2 describes the TRL assessment strategy, which has been chosen as a process template for bIoTope. These include the definitions of the TRLs according to the U.S. Government Accountability Office [U.S. Government Accountability Office], the TRL questionnaire according to the Air Force Research Laboratory [Air Force Research Laboratory] and the assessment process according to the Government Accountability Office’s (GAO) [U.S. Government Accountability Office]. Table 1 lists the TRL definitions once again. Based on the TRL questionnaire according to the Air Force Research Laboratory [Air Force Research Laboratory], an excel spreadsheet was designed to support the project partners being responsible for the assessment selecting the appropriate TRLs.

Regarding the IRL assessment strategy the assessment team has chosen the IRL definitions and their detailed descriptions according to Sauser [Sauser]. Table 2 gives an overview about these IRL descriptions. More information about the different IRL levels can be found in section 2.1.2 of deliverable D2.2. Based on the information provided by deliverable D2.2 and a spreadsheet designed by the assessment coordinator BIBA, the project partners selected suitable IRLs for each critical technology.



Table 1: Technology Readiness Levels (TRL) definition [U.S. Government Accountability Office]

TRL – Technology Readiness

Level

Description

1 Basic principles observed

and reported

Lowest level of technology readiness. Scientific research begins to be

translated into applied research and development (R&D). Examples

might include paper studies of a technology's basic properties.

2 Technology concept and/or

application formulated

Invention begins. Once basic principles are observed, practical

applications can be invented. Applications are speculative, and there

may be no proof or detailed analysis to support the assumptions.

Examples are limited to analytic studies.

3

Analytical and experimental

critical function and/or

characteristic proof of

concept

Active R&D is initiated. This includes analytical studies and laboratory

studies to physically validate the analytical predictions of separate

elements of the technology. Examples include components that are not

yet integrated or representative.

4

Component and/or

breadboard validation in

laboratory environment

Basic technological components are integrated to establish that they

will work together. This is relatively “low fidelity” compared with the

eventual system. Examples include integration of “ad hoc” hardware in

the laboratory.

5

Component and/or

breadboard validation in

relevant environment

Fidelity of breadboard technology increases significantly. The basic

technological components are integrated with reasonably realistic

supporting elements so they can be tested in a simulated environment.

Examples include “high-fidelity” laboratory integration of components.

6

System/subsystem model or

prototype demonstration in

a relevant environment

Representative model or prototype system, which is well beyond that

of TRL 5, is tested in a relevant environment. Represents a major step

up in a technology's demonstrated readiness. Examples include testing

a prototype in a high-fidelity laboratory environment or in a simulated

operational environment.

7

System prototype

demonstration in an

operational environment.

Prototype near or at planned operational system. Represents a major

step up from TRL 6 by requiring demonstration of an actual system

prototype in an operational environment.

8

Actual system completed

and qualified through test

and demonstration.

Technology has been proven to work in its final form and under

expected conditions. In almost all cases, this TRL represents the end of

true system development. Examples include developmental test and

evaluation (DT&E) of the system in its intended operational

environment to determine if it meets design specifications.

9

Actual system proven

through successful

operations in an operational

environment.

Actual application of the technology in its final form and under

operational conditions, such as those encountered in operational test

and evaluation (OT&E). Examples include using the system under

operational conditions.



Table 2: Integration Readiness Level (IRL) [Sauser]

IRL – IntegrationReadiness

Level Description

1

Semantic

An interface between technologies has been identified with sufficient

detail to allow characterization of the relationship.

2 There is some level of specificity to characterize the Interaction (i.e.

ability to influence) between technologies through their interface.

3 There is compatibility (i.e. common language) between technologies to

orderly and efficiently integrate and interact.

4

Syntactic

There is sufficient detail in the quality and assurance of the integration

between technologies.

5 There is sufficient control between technologies necessary to establish,

manage, and terminate the integration.

6 The integrating technologies can accept, translate, and structure

information for its intended application.

7 The integration of technologies has been verified and validated and an

acquisition/insertion decision can be made.

8

Pragmatic

Actual integration completed and qualified through test and

demonstration, in the system environment.

9 Integration is proven through successful operations in the system

environment.

Step 2 Define the individual TRL’s and IRL’s purpose; develop a TRL and IRL plan and assemble the assessment team.

The purpose of using the TRL assessment was to monitor the progress of the development of bIoTopes’ Core Components and the technologies/ software components, as described in chapter 3 of D2.2. In addition, the IRL assessment is aiming at measuring the integration of bIoTopes’ Core Components and the technologies/ software components into the SoS developed within bIoTope. The TRL and IRL assessment was used to evaluate the status of the TRL at the moment the Intermediate Evaluation was executed and to define the intended TRL and IRL at the end of the bIoTope project. According to Part B of the proposal, the assessment was coordinated by BIBA, being the leader of Task 2.E, and executed by the project partners, being responsible for the WPs 3 – 5 and the six Pilots.



Step 3 Select critical technologies

The selection of critical technologies was done in two way. Selecting the Core Components has already been done within deliverable D2.4., Table 3 shows the selected Core Components. The leader of WPs 3, 4 and 5 are responsible for the evaluation of those technologies. Within the six pilots, use case specific technologies/ software components are developed. Within Table 4 the pilots and the technologies/ software components developed especially for the different use cases are displayed. The Pilot leaders selected the technologies/ software components, which are of great importance for demonstrating the benefits of bIoTope within the use cases.

Table 3: Core Components assessed by using TRL

Core Components

KaaS (WP4)

BiaaS (WP3)

IaaS (WP3)

SeaaS (WP3)

Open API for O-DF (WP5)

Dashboard Framework – UiaaS (WP5)

CoaaS (WP4)

BDaaS (WP4)

Table 4: Technologies/ Software Components assessed by using TRL

Pilots Technologies/ Software Components


Brussels – Capital Region Android & IoS APP

Back end (O-MI Node)

API Manager (http://API.Brussels)

BigData Platform (Warp10)

FVH – Forum Virum Helsinki Back end (O-MI Node) for management

Charging pole, physical installation with software

Android app

Greater Lyon Data Platform

Lora Sensors


Smart Mobility BMW Connected Drive

Smart Building and Equipment Aalto app to Mist

IoTBnB to Mist

Smart Air Quality Air Quality Monitoring and Prediction



Step 4 Evaluate critical technologies

The project partners, being responsible for the WPs 3, 4 and 5 and the six Pilots 1 , also executed the assessment. They used the definitions listed in Table 1 and the provided excel-based questionnaire to select the appropriate TRL of each critical component. In accordance with the assessment team at BIBA, the questionnaire was not used in a dogmatic way. The decision to interpret the TRL definitions in a pragmatic way was made by the people involved in the assessment team. The decision was motivated by the fact, that bIoTope is a research project. Moreover, the software developed within bIoTope is not as critical as technologies developed for rockets or other flying equipment, for which the NASA TRL assessments originally introduced. Instead, the use of the TRL assessment within bIoTope should guide the developers to improve the software tools used within bIoTope. The superior objective using TRL assessments was to help the companies being involved in bIoTope to offer innovative software solutions based on the innovations made within bIoTope. In order to sum it, the questionnaire was used in a pragmatic way by the assessment team and not every statement within the questionnaire had to be answered with “YES” to reach a specific TRL level.

The same project partners responsible for the TRL assessment executed the IRL assessment. They used the information provided in chapter 2.1.2 of deliverable D2.2 and the questionnaires provided for the documentation of the selected IRLs. Due to the fact, that not such a strict questionnaire was provided, the project partners had some more leeway for choosing the appropriate IRL. However, the basic idea using IRL for the Evaluation of the technologies is about the same as for using TRL. The assessment should guide the developers towards the objective of bIoTope to create a SoS, which enables the exchange of information across different domain-specific platforms. By designing bIoTopes’ SoS in the way, companies can integrate new Business Services into the SoS in a very efficient way, the system will enable those companies to offer new Business Services for Users.

Step 5 Prepare, coordinate and submit TRA reports

After filling the current and the intended TRL and IRL questionnaires for each critical technology, the partners, being responsible for the WPs 3 – 5 and the six Pilots, sent the questionnaires to BIBA being the coordinator of the whole assessment. BIBA generated tables, summarizing the results of the Evaluation. The tables show the current and the intended TRL and IRL Core Components and the technologies/ software components. Chapter 2.2 presents the results of the evaluation of the Core Components. The results of the evaluation of the technologies developed within the Cross-domain Smart City pilots can be found in chapter 2.3 and the results of the evaluation of the technologies developed within the Domain-specific Pilots can be found in chapter 2.4.

Step 6 Using TRA results and developing a Technology Maturation Plan (TMP)

By having identified the current TRL and the intended TRL at the end of bIoTope, the partners being responsible for the development of the different Core Components and the technologies/ software components as well as the project management knows, which steps need to be taken to reach bIoTopes’ objectives. Especially the questionnaires lead the path for further development by listing relevant steps to achieve the intended TRL and IRL. Aligned with the planned development documented and measured objectively by using the TRL as well as the IRL approach a detailed plan for the ongoing development of the Core Component and the technologies/ software components is documented in the deliverables of WPs 3 – 5 and the deliverables of the six Pilots. The documentation of the paths of future development within these deliverables is within the meaning of a Technology Maturation Plan (TMP). All relevant deliverables explaining the status of the development and the plan for future development and are presented in the tables showing the evaluation results in sections 2.2 - 2.4.

1 Seventh pilot (Smart Waste Management) is excluded in this report and reasoned in the previous chapter (section 1.3.2)



Summary of the development within bIoTope

As already mentioned before, the responsible project partners within the WPs 3 – 5 and the six Pilots filled out the spreadsheets to evaluate the status of each critical technology and to define the level they want to reach until the end of bIoTope. So the Intermediate Evaluation of bIoTopes’ technologies was completed in time and the results of the evaluation considering the Core Components are presented in 2.2. Section 2.3 and 2.4 present the results of the evaluation of technologies/ software components developed within bIoTopes’ Cross-domain Smart City Pilots and Domain-specific Pilots.

The following paragraphs present a summary of the evaluated TRL and IRL levels for bIoTopes’ Core Components and technologies/ software components. Based on the determined levels conclusions considering the status and the planned developed are presented.

As Table 5 shows, most of the current TRLs for bIoTopes’ Core Components are medium level values ranging from level 4 to level 7. The project partners intend to raise these values by roughly two levels on average, resulting in levels of 6 to 9. The TRL evaluation of the Billing-as-a-Service and the BigData-as-a-Service are outliers to the side of low levels, both in its current and its intended levels. These results are reflecting the generally lower technical maturity of Micro-Billing Services as well as Big Data Technologies. Within both topics, a lot of research is still ongoing, because they are of big importance for an economically successful IoT. Billing-as-a-Service is needed to monetize the Business Services and BigData-as-a-Service is of great importance to analyse the huge amount of data generated and made accessible by the IoT. As also shown in Table 5, the current IRLs assigned to bIoTopes’ core components are somewhat lower than the TRLs, ranging from levels 2 to 6. Due to the fact that bIoTopes’ main objective is to interconnect different technologies within the IoT, the IRLs rise slightly more than the TRLs. Most project partners plan to raise the IRLs of the technology they are responsible for by 2 or 3 levels. As a result, the most IRLs should reach the level 8, just two Core Components will probably be below, reaching an IRL of 4 and 6. In order to sum it up, the Core Components are on a good way to reach the necessary maturity levels in order to support the Use Case specific Business Service with the required IoT infrastructure.

As Table 6 up to Table 10 show, the current TRL of the technologies/ software components developed especially for bIoTopes’ Pilots are widely spread, reaching from level 2 to level 5. Even the intended levels at the end of bIoTope differ a lot. The reason for the wide spread TRLs are, that the technologies developed within the Use Cases are very specific developments; even their maturity differs a lot from each other. However, this is not causing issues for the whole project, because they are just of importance for the Use Cases. In the end, this will result in different time to market for the developed Business Services. The main objective of the Pilots to proof the applicability of bIoTopes’ approach is not at risk, this is also possible with prototypes having a very high maturity level. Regarding the IRLs, the situation is about the same. Across all technologies/ software components developed within bIoTopes’ Pilots the IRLs are widely spread. The current IRLs reach from level 1 to level 4 and the intended IRLs at the end of bIoTope reach from level 3 to level 7. Due to the fact, that the worst IRL will be level 3, it is ensured, that each Use Case is able to demonstrate in a prototypical way, how the interconnection of different technologies within bIoTope will work. According to the IRL definitions applied within the Intermediate Evaluation Level 3 represents the minimum required level to realize a successful integration.

The Intermediate Evaluation based on the TRL and IRL methodology, which quantified the maturity levels of the different technologies used within bIoTope, not just generated a report for the project management team, it also provided support to the project partners being responsible for technology development. By going through the TRL and IRL evaluation process, they were able to reflect the status of the work on the technologies at the end of the first half of the project and come up with ideas how they need to proceed to reach the intended status of their technology. By using TRL and IRL as Evaluation methodologies, the development of each technology was assessed on the one hand. On the other hand, interfaces across the different technologies to realize cross-domain Business Services were assessed as well. Especially the development of those interfaces in the context of the IoT across different ecosystems is in the focus of bIoTope. In order to reach bIoTopes’ objectives the different development teams within bIoTope do have now



a detailed overview about the status and have a kind of guideline, based on the TRL and IRL definitions, how to proceed to reach the intended levels.

In respect of the results of the TRL and IRL evaluation, the deliverables of WPs 3 – 5 considering the Core Components describe actions how to they will proceed will the further development. Considering the technologies/ software components developed by the Pilots, the actions for making each Use Case a successful demonstrator of bIoTopes’ objectives are described in the deliverables D6.1 – D6.14.

Decisions deduced from status of the development within bIoTope

Based on the approach deliverable D2.2 “Evaluation Methodology for Pilots validation” presents in section 2.1 the evaluation team and the project management office came up with the following decisions:

Monitoring of the IRLs should be in the focus of the evaluation team and the project management team in order to guarantee bIoTopes’ success. This is due to the fact, that bIoTopes’ main objectives are inspired by a cross-ecosystem data exchange and the number of levels between current IRLs and intended IRLs at the end of bIoTope are quite challenging.

TRL and IRL definitions should be in mind of the development teams to guide them during the development process to the intended maturity levels.

TRL and IRL progresss of each critical bIoTope technology need to be presented in front of the consortium members at each meeting of the technical team (initiated by WP 4 and accompanied by WP3 and 5) by using the provided spreadsheets to keep track of the development process and to compare the current maturity levels with the intended levels at the end of bIoTope.

2.2. Evaluation of Core components

The overall architectural framework of bIoTopes’ SoS is based on an “Everything-as-a-Service” (XaaS) paradigm, where six key XaaS topics, considered as essential for any SoS Platforms are created and tested. Each XaaS results in one or more standards-based software core components.

Table 5 lists the Technology Readiness Levels as well as the Integration Readiness Levels of the bIoTope core components, which have resulted from the evaluation of the stakeholders:

Table 5: TRL and IRL of bIoTopes’ Core Components

Core Components

Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress

KaaS (WP4) 7 8 6 8 D4.2 & D4.7

BiaaS (WP3) 2 4

3 6

(7 in best case)

D3.4

IaaS (WP3) 7 9 D3.2

SeaaS (WP3) 7 8 D3.1

Open API for O-DF (WP5)

4 6 5 8

UIaaS - Dashboard Framework

(WP5)

4 6 5 8 D5.3 & D5.6

CoaaS (WP4) 2 4 2 4 D4.5 & D4.9

BDaaS 9 9 6 8 D4.1



2.3. Evaluation of Technologies developed within the Cross-domain Smart City Pilots

In preparation of this deliverable, the technologies used within the cross-domain smart city pilots developed in bIoTope have been evaluated from user perspective. As part of the state of the art evaluation, stakeholders influencing and influenced by the pilot have been identified. The objectives of each pilot were named by the “Stakeholder Viewpoints” of each pilot within D2.1 “Ecosystem Stakeholder Requirements Report and Pilot Definition”. The identified stakeholders include both, stakeholders in general as well as potential end users. The evaluation itself was mainly prepared by members of each Pilot as well as the project partners being responsible for WP 3 - 5. The results are reported in the following sections 2.3.1-2.3.3.

2.3.1. Pilot: Brussels – Capital Region

Technology Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress

Android & IoS APP 9 9 2 7

D6.5 & D6.11

Back end (O-MI Node)

3 6 2 6

API Manager (http://API.Brussels)

8 9 8 9

BigData Platform (Warp10)

8 9 2 7

2.3.2. Pilot: FVH – Forum Virum Helsinki

Table 6: TRL and IRL of Pilot: Helsinki


Back end (O-MI Node) for management

5 6 4 6

D6.6 & D6.12 Charging pole, physical installation with software

2 4 2 4

Android app 3 4 3 4

2.3.3. Pilot: Greater Lyon

Table 7: TRL and IRL of Pilot: Greater Lyon


Data Platform 2 7 1 4 D6.4 & D6.10

Lora Sensors 1 5 1 3

http://api.brussels/



2.4. Evaluation of Technologies developed within the Domain-specific Pilots

The technologies used within the domain-specific smart city pilots have been evaluated from user perspective as well. The identified stakeholders again include both, stakeholders in general as well as potential end users. The evaluation itself was again mainly prepared by project management members. The results are reported in the following sections 2.4.1-2.4.3.

2.4.1. Pilot: Smart Mobility

Table 8: TRL and IRL of Pilot: Smart Mobility


Pilot as a whole

(BMW Connected Drive – MobiVoc – O-MI node, see D2.1 and D2.4)

2 4

(5 in best case)

2 6 D6.3 & D6.9

2.4.2. Pilot: Smart Building and Equipment

Table 9: TRL and IRL of Pilot: Smart Building and Equipment


Aalto app to Mist (WP3)

Not applicable (just integration)


1 7

D6.2 & D6.8 IoTBnB to Mist (WP3)



3 7

2.4.3. Pilot: Smart Air Quality

Table 10: TRL and IRL of Pilot: Smart Air Quality


Air Quality Monitoring and Prediction

2 3 2 3 D6.1 & D6.7



3. Matching of KPIs and Requirements

In order to ensure, that the Business Services developed within bIoTope address the needs of the intended users within each Use Case the project partners execute different assessments. These assessments are based on different criteria, motivated by the users’ needs as well as technical requirements. Finally, bIoTopes’ Business Services have to meet both kinds of criteria. On the one hand, they have to fulfil the users’ needs, motivating the customer to pay for the service. On the other hand, the services have to work reliably, otherwise the customer will not buy it neither. Therefore, the services have to meet the technological requirements.

3.1. Evaluation Methodology for Matching of KPIs and Requirements

During the preparation of deliverable D2.2, every Pilot defined KPIs for its Use Cases. Those KPIs are based on user stories defined for each bIoTope Use Case. Each KPI illustrates how the developed Business Services improve the daily life of Europe’s citizens. The KPIs detail bIoTopes’ superordinate objectives, defined by the consortium partners within bIoTopes’ proposal Part B of the Description of Actions. This kind of perspective Based on the proposal the KPIs were clustered as follows:

Planned Economical Improvements

Planned Societal Improvements

Planned Technical Improvements

Planned Environmental Improvements

The defined KPIs are a kind of non-functional requirements for the Business Services developed within bIoTope. They are expressing the planned improvements of the Use Case’s processes. Finally, the project partners will evaluate the KPIs in Test Scenarios after the implementation of the needed IoT infrastructure. Based on the results of the Test Scenarios, it can be said, whether bIoTope is a success from user perspective. Chapter 5 of deliverable D2.2. described the first version of the test scenarios. Within chapter 5 of the deliverable being on hand updates triggered by the Use Case Owners as well as bIoTopes’ development teams are shown. In order to make the KPIs tangible they should the SMART criteria listed in deliverable D2.2 and repeated here:

S – Specific: The criterion should make the need for an explicit goal so that there is no room for misinterpretation. A specific goal will usually answer the six 'W' questions: Who, What, Where, When, Which and Why.

M – Measurable: The next criterion means that goals must be measurable based on concrete, operational criteria. Usually, the entire goal statement is a measure for the project, but there are several short-term or smaller measurements built into the goal.

A – Achievable: This criterion should make sure that the goals are realistic and attainable. That means it must be possible to achieve the goal. However, they need to be stretching, ambitious and not too easy, because it will not be motivating.

R – Relevant: The fourth criterion Relevant means to choose goals that matter. Goals that are relevant for the employees, the team or the organization will receive needed support and drive the team forward.

T – Time-bound: The last criterion Time-bound is about setting the timeframe within which any identified objectives should be achieved. A commitment to a deadline helps a team focus their efforts towards completion of the goal.

In contrast to the approach for defining non-functional requirements based on KPIs, bIoTopes’ deliverable D2.1 defines formal requirements. bIoTopes’ software and hardware developer use those formal requirements to design each component according to the needs of the users as well as according to the needs



resulting from bIoTopes’ SoS approach. The Pilot owners developed those requirements based on ArchiMate models, being a technical standard from The Open Group based on the IEEE 1471 standard. In general, the standard provides methodologies to model business processes and IT infrastructures. Based on the available set of methodologies within ArchiMate bIoTopes’ project partners had decided on a number of so-called viewpoints. Each viewpoint is a selection of a relevant subset of ArchiMate’s methodologies offering the possibility to describe, analyse and visualise business processes and IT infrastructures from a certain perspective. For bIoTopes’ purposes, the project partners decided to use the following Viewpoints give an overview of high-level requirements regarding architectural aspects of each Pilot.

Organization Viewpoint

Layered Viewpoint

Stakeholder Viewpoint

Goal Realization Viewpoint

In order to ensure, that the KPIs, representing non-formal requirements are aligned with the formal requirements defined by the use of the ArchiMate modelling approach, the following evaluation procedure was executed. By using the structure shown in Table 11, all Pilots assigned the requirements presented originally in chapter 3 and 4 of deliverable D2.1 to the KPIs defined originally in chapter 5 of deliverable D2.2. Therefore, BIBA, as task leader of Task 2.E prefilled Excel spreadsheets with all KPIs defined. Afterwards, the responsible project partners had to go through the document and select the suitable requirements being needed to realize the KPI from an already prepared dropdown menu. It was possible to assign more than one requirement to a KPI. This is due to the fact, that even more features of the developed Business Service are needed to fulfil the KPI. At minimum one requirement needs to be assigned to a KPI. If no requirement is assigned to a particular KPI, the KPI is not addressed by the Business Service. This means the Business Service will not be able to achieve this KPI. If such a circumstance occurs, the project partners have to think about, whether the additional requirement is needed on order to be realised by the Business Service, or whether the KPI is not relevant any more. Therefore, the project partner executing this evaluation had to go through the KPIs and the requirements as well and update them if necessary. Chapter 5 shows the updates made to the KPIs during the Intermediate Evaluation.

Table 11: Structure of the spreadsheet for matching KPIs and Requirements

Summary of the Matching of KPIs and Requirements within bIoTope

As the following sections show, the software and hardware development process within bIoTopes’ is addressing the needs of the Use Case’s processes. Therefore, all KPIs are addressed by a minimum of one formal requirement. Hence, it will be possible to evaluate all KPIs within the Test Scenarios during bIoTopes’ final Evaluation in order to assess, the improvements which can be realized by bIoTope in Use Cases inspired by real world processes.

Pilot's Name Use Case's Name

Nr. KPI Group KPI Requirements

1 Requirement 1

Requirement 2

Requirement 3

Requirement 4

Requirement 5

Requirement 6

Requirement 7

Requirement 8

Requirement 9

Requirement 10

https://en.wikipedia.org/wiki/IT_infrastructure

https://en.wikipedia.org/wiki/IT_infrastructure



3.2. Evaluation of Cross-domain Smart City Pilots

In order to compare the KPIs and requirements of each Cross-domain Smart City Pilots developed in bIoTope have assigned to each other. Sections 3.2.1 - 3.2.3 present the evaluation results in detail.


Table 12: Matching of KPIs and Requirements for Pilot: Brussels – Capital Region


1 Planned Economical Improvements Lowering the amount of accidents in school zones and on the itinerary of the children Req_Brussel_2_1 Predict peak hours Requirement 1

Req_Brussel_2_2 Predict traffic Requirement 2

Req_Brussel_2_3 Identi fy Safer Routes Requirement 3

Req_Brussel_3 Local ize s tudents Requirement 4

Req_Brussel_4 Reduce cars around schools Requirement 5

Req_Brussel_5 Reduce traffic speed Requirement 6

Req_Brussel_7 Promote safer & greener mobi l i ty Requirement 7

2 Shortening the average daily commute time of children and their parents Req_Brussel_2_1 Predict peak hours Requirement 1



Req_Brussel_4_2 Inform drivers about school zone Requirement 4

Req_Brussel_6 Organize co-mobi l i ty Requirement 5

Req_Brussel_5_2 Alert drivers (via navigation systems)[1] Requirement 6

Req_Brussel_5_3 Alert drivers (via dynamic message panels ) Requirement 7

3 Lowering traffic jams in around schools Req_Brussel_2_1 Predict peak hours Requirement 1




Req_Brussel_6 Organize co-mobi l i ty Requirement 5





4 Planned Societal ImprovementsIncrease the effective security of school children, to be measured via perceived sense of

securityReq_Brussel_1 Col lect rea l -time data Requirement 1

Req_Brussel_2_3 Identi fy Safer Routes Requirement 2

Req_Brussel_1 Col lect data Requirement 3

Req_Brussel_1_2 Col lect data about s tudents Requirement 4

Req_Brussel_1_3 Col lect perceived sense of securi ty Requirement 5

5 Increase usage of other transport methods than the personal carReq_Brussel_6_2 Organize co-mobi l i ty for pedestrians (APP-

based)Requirement 1

Req_Brussel_6_3 Organize co-mobi l i ty for bikes (APP-based) Requirement 2

Req_Brussel_6_4 Organize co-mobi l i ty for publ ic transport (APP-

based)Requirement 3

Req_Brussel_7 Promote safer & greener mobi l i ty Requirement 4

6 Lowering the amount of accidents in school zones Req_Brussel_2_3 Identi fy Safer Routes Requirement 1


Req_Brussel_4_1 Deviate traffic Requirement 3


Req_Brussel_5 Reduce traffic speed Requirement 5

Req_Brussel_5_1 Adapt dynamical ly speed traffic s igns Requirement 6



7 Planned Technical Improvements Bundling different sources (sensors) via O-DF/O-MI Req_Brussel_1_1 Col lect data from Telecom Operator Requirement 1

Req_Brussel_1_2 Col lect data about s tudents Requirement 2

Req_Brussel_1_3 Col lect perceived sense of securi ty Requirement 3

8Planned Environmental

ImprovementsRoute planning of children keeps into account overall pollution along possible itineraries Req_Brussel_7 Promote safer & greener mobi l i ty Requirement 1

Req_Brussel_7_1 Reward green behaviour Requirement 2

Req_Brussel_7_2 Inform about green mobi l i ty modes Requirement 3

Req_Brussel_7_3 Inform about safer and greener paths Requirement 4




3.2.2.1. Use Case: Electrical Car charging service using IoT BnB concept

Table 13: Matching of KPIs and Requirements for Pilot: Forum Virium Helsinki - Electrical Car charging service using IoT BnB concept


1 Planned Economical ImprovementsProvide equal possibilities to the companies, individuals, and organizations to participate in

creation of ecosystems for electrical carsReq_FVH_1 Publ ishing charging s tation Requirement 1

Req_FVH _1_1 Workflow to integrate new charging systems Requirement 2

Req_FVH_1_1_1 Deploying service information and metadata in

easy and s tandardized wayRequirement 3

Req_FVH_1_1_2 Connecting to the supplementary services Requirement 4

2 Reduce risks of monopolization of charging services by utilities companiesReq_FVH_1_1_1 Deploying service information and metadata in


Req_FVH _1_1 Workflow to integrate new charging systems Requirement 2

3Tremendous increase in the amount of charging stations – less distances to charge, less

energy consumption Req_FVH_2 Providing Charging Service Requirement 1

Req_FVH_2_1 Integration to car navigation system or mobi le

devicesRequirement 2

Req_FVH_2_1_1 Integration to car navigation system Requirement 3

Req_FVH_2_1_2 Providing fi l tering poss ibi l i ties Requirement 4

4 Increase sales of electrical cars Req_FVH_1 Publ ishing charging s tation Requirement 1

Req_FVH_2 Providing Charging Service Requirement 2

5 Planned Societal ImprovementsEnables mass roll out of electrical cars in Northern countries that was somewhat slow due

to a lack of suitable charging stationsReq_FVH _1_1 Workflow to integrate new charging systems Requirement 1


6 Less traffic due to better navigation and improved serviceReq_FVH_2_1 Integration to car navigation system or mobi le


Req_FVH_2_1_2 Providing fi l tering poss ibi l i ties Requirement 2

Req_FVH_2_1_3 User mobi le or web access Requirement 3

7 Public and private assets optimization Req_FVH _1_1 Workflow to integrate new charging systems Requirement 1




8 Planned Technical Improvements Single entry point for all charging stations, easy and fast to searchReq_FVH_1_1_1 Deploying service information and metadata in


Req_FVH_1_1_3 Service provis ioning Requirement 2

Req_FVH_2_1 Integration to car navigation system or mobi le


9 Easy to pay Req_FVH_2_2 Integration of payment system (optional ) Requirement 1

Req_FVH_2_3 Authentication methods Requirement 2

10 Seamless adding of new stations and suppliers without programming and integration effort Req_FVH _1_1 Workflow to integrate new charging systems Requirement 1

Req_FVH_1_1_1 Deploying service information and metadata in



ImprovementsLess CO2 emission due to increased amount of electrical vehicles Req_FVH_1 Publ ishing charging s tation Requirement 1

Req_FVH_2 Providing Charging Service Requirement 2




3.2.3.1. Use Case: Bottle Banks

Table 14: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Bottle Banks


1 Planned Economical Improvements Collection costs lowered, due to less use of trucks Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1

Req_Lyon_1_3 Cons ider banks ful lness rate and other cri teria

in col lection schedul ingRequirement 2

Req_Lyon_1_4 Cons ider traffic predictions and real time traffic

in col lection schedul ing and road mapsRequirement 3

Req_Lyon_1_5 Providing generic API for bottle bank Information

exchangeRequirement 4

2Bottle bank costs lowered, due to less emptying operations on the banks and control of

bottle banks handlingReq_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1





Req_Lyon_1_6 Dai ly bottle banks dashboard Requirement 4

3Better equality between operators in tender procedures regarding the discarded glass

collectionReq_Lyon_1_6 Dai ly bottle banks dashboard Requirement 1

4 Planned Societal ImprovementsCitizens satisfied because they are informed of bottle bank availability, they can report

problems and most of the time bottle banks are not full



Req_Lyon_1_6 Dai ly bottle banks dashboard Requirement 2

5 Less trucks in the streets, less traffic Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1



6 The metropolis has a better knowledge of the waste glass collection activity Req_Lyon_1_6 Dai ly bottle banks dashboard Requirement 1

7 Planned Technical Improvements

Interaction between a digital system owned and managed by the metropolis (the bottle

banks’ sensors network) and a digital system owned and managed by a subcontractor (the

trucks navigation system)



Req_Lyon_1_8 Data integration from datalyon.com Requirement 2




ImprovementsLess noise in the streets Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1



9 Less noise in residential areas, respect of “quiet” time slotsReq_Lyon_1_2 Cons ider res identia l area in col lection

schedul ingRequirement 1

10 Less pollution due to the reduction of the use of trucks through collections optimization Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1





3.2.3.2. Use Case: Heat Wave

Table 15: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Heat Wave


1 Planned Economical Improvements Less water consumption, use of rainwater Req_Lyon_2_10 Watering scheduler Requirement 1

Req_Lyon_2_9 Rainwater tank level and water debit

meterRequirement 2

Req_Lyon_2_6 Soi l humidity sensors Requirement 3

Req_Lyon_2_7 Evapotranspiration sensors Requirement 4

Req_Lyon_2_8 Weather forecast Requirement 5

Req_Lyon_2_4 Humidity and temperature sensors Requirement 6

2 Less trees in bad health to replace Req_Lyon_2_7 Evapotranspiration sensors Requirement 1

Req_Lyon_2_10 Watering scheduler Requirement 2

3 Planned Societal Improvements Increased citizen’s well-being Req_Lyon_2_10 Watering scheduler Requirement 1

Req_Lyon_2_3 Temperature information Requirement 2

Req_Lyon_2_1 Citizen porta l Requirement 3

4 A better life quality due to green spaces in good health Req_Lyon_2_10 Watering scheduler Requirement 1

5 The metropolis demonstrates its commitment in climate change adaptation Req_Lyon_2_1 Citizen porta l Requirement 1


6 Planned Technical ImprovementsSmart watering solution involving various sensors and actuators and external

data/informationReq_Lyon_2_10 Watering scheduler Requirement 1

7 The solution could be extended easily to others streets Req_Lyon_2_10 Watering scheduler Requirement 1




ImprovementsHeat wave mitigation based on a natural solution Req_Lyon_2_6 Soi l humidity sensors Requirement 1

Req_Lyon_2_7 Evapotranspiration sensors Requirement 2

Req_Lyon_2_4 Humidity and temperature sensors Requirement 3

Req_Lyon_2_9 Rainwater tank level and water debit

meterRequirement 4




3.3. Evaluation of Domain-specific Pilots

Sections 3.3.1 - 3 show the results of the evaluation in order to ensure, that all KPIs are addressed by formal requirements leading to the development of appropriate features of the Business Services developed by each Pilot.


Table 16: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Charging Station Selection Service


1 Planned Economical ImprovementsLess energy consumption, resulting in less costs due to the reduction of saved kilometres

searching for appropriate charging stationsReq_SmartM_1 Parking Spot Discovery Requirement 1

Req_SmartM_3 Charging Station Database Requirement 2

Req_SmartM_5 Pre-Conditioning System Requirement 3

2Increased number of sold electric vehicles due to a more convenient search for appropriate

charging stationsReq_SmartM_3 Charging Station Database Requirement 1

Req_SmartM_3_1 Charging Stations Requirement 2

Req_SmartM_4 Uniform Payment System Requirement 3

3 Increased number of sold charging stations due to the increased number of electric vehicles Req_SmartM_3 Charging Station Database Requirement 1



4 Single payment method for at least 3 different charging station providers Req_SmartM_4 Uniform Payment System Requirement 1

5 Planned Societal Improvements Less traffic caused by searching for appropriate charging stations Req_SmartM_1 Parking Spot Discovery Requirement 1

Req_SmartM_2 Predict Traffic Si tuation Requirement 2



Req_SmartM_6 Driver Identi fication Requirement 5

Req_SmartM_6_1 Driver Data Requirement 6



6 Better usage of space and existing charging stations Req_SmartM_1 Parking Spot Discovery Requirement 1



7 Planned Technical ImprovementsReduced complexity to integrate a charging station selection service into navigation

systems due to standardized data interfacesReq_SmartM_3 Charging Station Database Requirement 1



8 Reduced complexity to integrate different payment solutions into the charging stations Req_SmartM_4 Uniform Payment System Requirement 1

9Charging stations taken into account are from different providers (min 3; max (amount of

providers in area))Req_SmartM_3 Charging Station Database Requirement 1



ImprovementsLess carbon dioxide emissions due to the increased number of electric vehicles Req_SmartM_3 Charging Station Database Requirement 1



Req_SmartM_5_1 Cars & Bui ldings Requirement 4

11 Better air quality in the city due to the increased number of electric vehicles Req_SmartM_3 Charging Station Database Requirement 1



12Less resource consumption due to less traffic caused by the search for appropriate charging

stationsReq_SmartM_1 Parking Spot Discovery Requirement 1








Table 17: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Route Planning Service


1 Planned Economical Improvements Better use of existing infrastructure Req_SmartM_1 Parking Spot Discovery Requirement 1



2Increased number of sold vehicles / navigation systems due to more convenient route

planning services

Req_SmartM_1_1 Real Time Parking Information

SystemRequirement 1

Req_SmartM_2_1 Real Time System Requirement 2


3 Planned Societal Improvements Less unnecessary traffic in specific areas; e.g. school zones Req_SmartM_2 Predict Traffic Si tuation Requirement 1

4 Less stress impact on drivers due to unnecessary hazards Req_SmartM_2_1 Real Time System Requirement 1

5 Planned Technical Improvements Reduced effort to integrate new route-relevant data sources into the service Req_SmartM_1 Parking Spot Discovery Requirement 1




6 Higher accuracy of data can be used for additional service generationReq_SmartM_1_1 Real Time Parking Information

SystemRequirement 1

Req_SmartM_2_1 Real Time System Requirement 2




ImprovementsLess pollution due to reduced unnecessary driving and traffic jams Req_SmartM_1 Parking Spot Discovery Requirement 1



Table 18: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Electric Car Gear Service


1 Planned Economical ImprovementsEfficient use of energy, as the preconditioning is triggered based on the actual events and

no fixed time scheduleReq_SmartM_5 Pre-Conditioning System Requirement 1

Req_SmartM_5_1 Cars & Bui ldings Requirement 2

2 Reduced overall costs for batteries in particular hot or cold countries and urban mobility Req_SmartM_5 Pre-Conditioning System Requirement 1

3 Planned Societal ImprovementsElectric Vehicles have a bigger effective range, reducing the range anxiety and increasing

the attractiveness of electric vehicles in urban areasReq_SmartM_3 Charging Station Database Requirement 1



4 Future combination of Smart Home and Smart Mobility Req_SmartM_5_1 Cars & Bui ldings Requirement 1

5 Planned Technical Improvements Reducing the cost for creating cross-domain based services Req_SmartM_4 Uniform Payment System Requirement 1



ImprovementsBetter usage of space and existing charging stations Req_SmartM_1 Parking Spot Discovery Requirement 1

Req_SmartM_1_1 Real Time Parking Information

SystemRequirement 2




Table 19: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart Building Interaction with Smart Mobility


1 Planned Economical ImprovementsTremendous increase in the amount of parking (and power outlet) alternatives – less

distances to parking, less energy consumptionReq_SmartBE_1_1 A private parking space which can be shared Requirement 1

2Provide equal possibilities to the companies, individual, and organizations to participate in

creation of ecosystems for parking

Req_SmartBE_1 Parking spot as a shared space of a smart

bui ldingRequirement 1

Req_SmartBE_1_1 A private parking space which can be shared Requirement 2

Req_SmartBE_1_1_1 A service description publ icly access ible to

anyone within des ignated groupRequirement 3

Req_SmartBE_1_2 A communication infrastructure between

parking space provider and parking space consumer (car driver)Requirement 4

3 Improved competition between parking alternatives Req_SmartBE_1_1 A private parking space which can be shared Requirement 1







4 Planned Societal Improvements Increased parking alternatives, more likely to find close to destinationReq_SmartBE_1 Parking spot as a shared space of a smart









5 Less traffic due to more parking alternatives and less parking challengesReq_SmartBE_1 Parking spot as a shared space of a smart







6 Public and private assets optimizationReq_SmartBE_1_2_1 Open standardized communication

channel between des ignated groupsRequirement 1



7 Planned Technical Improvements Single entry point for all charging stations, easy and fast to searchReq_SmartBE_1_2_1 Open standardized communication


Req_SmartBE_1_3_2 UI for discovering services Requirement 2

Req_SmartBE_1_3_4 Feature for selecting des ired service Requirement 3

8 Easy to pay Requirement 1

9 Seamless adding of new parking suppliers without programming and integration effortReq_SmartBE_1_5_1 Feature for providing a name for the

parking s lotRequirement 1

Req_SmartBE_1_5 Providing attributes for a parking spot Requirement 2

Req_SmartBE_1_5_4 Optional feature for describing additional

equipmentRequirement 3

Req_SmartBE_1_3_1 UI for describing a service Requirement 4

Req_SmartBE_1_2_1 Open standardized communication



ImprovementsImproved utilisation of parking lots










11Less CO2 emission due to reduced kilometres resulting originally from driving round and

searching for available parking lots









12Less resource consumption due to less traffic caused by the search for appropriate charging

stationsReq_SmartBE_1_3_4 Feature for selecting des ired service Requirement 1


Req_SmartBE_1_3_3 Feature for fi l tering services (optional ) Requirement 3



Table 20: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart Equipment


1 Planned Economical Improvements Reduced service costs Req_SmartBE_2_7_1 Abi l i ty to ad hoc connect new stakeholders Requirement 1

Req_SmartBE_2_7 Remote monitoring and control of a i r

handl ing unitRequirement 2

Req_SmartBE_2_6_2 Fi rmware release hosting service Requirement 3

Req_SmartBE_2_6_1 Fi rmware compatibi l i ty fi l tering Requirement 4

Req_SmartBE_2_6_6 Feature for flashing an a i r handl ing unit

with a new fi rmwareRequirement 5

2 Reduced energy costsReq_SmartBE_2 Air handl ing unit control in the field of smart


3 Planned Societal Improvements Increased human productivity due to better indoor air qualityReq_SmartBE_2 Air handl ing unit control in the field of smart


4 Healthier buildingsReq_SmartBE_2 Air handl ing unit control in the field of smart


5 Planned Technical Improvements Smooth connectivity due to Mist/WiFi technology (O-MI/O-DF) Req_SmartBE_2_5 Ad hoc IoT connectivi ty Requirement 1

Req_SmartBE_2_1 Generic connectivi ty Requirement 2

Req_SmartBE_2_2_3 Flexible support for new sensor types ,

without hardware modificationsRequirement 3

Req_SmartBE_2_3_1 Support for remote (internet relayed), as

wel l as loca l (peer-to-peer) communicationRequirement 4

Req_SmartBE_2_3 Smartphone UI for interaction with a i r

handl ing unitRequirement 5


ImprovementsLess energy consumption, less CO2 emissions

Req_SmartBE_2 Air handl ing unit control in the field of smart


Req_SmartBE_2_3_2 Feature for visual is ing / communicating a i r

qual i ty information to end userRequirement 2




Table 21:Matching of KPIs and Requirements for Pilot: Smart Air Quality – Identify polluted areas

Smart Air Quality Identify polluted areas


1 Planned Economical Improvements Complementary AQMP measurements Req_SmartAQ_1 Air Qual i ty Requirement 1

Req_SmartAQ_6 Compute Air Qual i ty Requirement 4

2 Revenue stream for local councils aware of and finding air polluters Req_SmartAQ_3 Uti l i ty Computation Requirement 1

Req_SmartAQ_1 Air Qual i ty Requirement 2

3 Planned Societal Improvements Increased trust of air quality measurements by EPA and local councils Req_SmartAQ_1 Air Qual i ty Requirement 1

Req_SmartAQ_4 Provide heat maps Requirement 2

Req_SmartAQ_2 Push Noti fications Requirement 3


4 Personalised heat maps for healthier environments Req_SmartAQ_2 Push Noti fications Requirement 1


5 Cleaner air for communities Req_SmartAQ_1 Air Qual i ty Requirement 1


Req_SmartAQ_5 Efficiency Requirement 3

6 Planned Technical Improvements Crowd-sourced complementary measurements of air quality Req_SmartAQ_1_2 pocket level Requirement 1

Req_SmartAQ_3 Uti l i ty Computation Requirement 2

Req_SmartAQ_4_2 Web Vers ion Requirement 3

7 Personalised awareness of air quality through a smartphone app Req_SmartAQ_2 Push Noti fications Requirement 1



8 Evolutionary learning and improvement of air quality predictions Req_SmartAQ_6 Compute Air Qual i ty Requirement 1


ImprovementsIncreased density of air quality measurements Req_SmartAQ_1 Air Qual i ty Requirement 1



10 Cleaner air as the result of continuous monitoring and awareness of air quality Req_SmartAQ_1 Air Qual i ty Requirement 1








Smart Air Quality Identify polluted areas




2 Revenue stream for local councils aware of and finding air polluters Req_SmartAQ_3 Uti l i ty Computation Requirement 1

Req_SmartAQ_1 Air Qual i ty Requirement 2

3 Planned Societal Improvements Increased trust of air quality measurements by EPA and local councils Req_SmartAQ_1 Air Qual i ty Requirement 1




4 Personalised heat maps for healthier environments Req_SmartAQ_2 Push Noti fications Requirement 1





6 Planned Technical Improvements Crowd-sourced complementary measurements of air quality Req_SmartAQ_1_2 pocket level Requirement 1


Req_SmartAQ_4_2 Web Vers ion Requirement 3

7 Personalised awareness of air quality through a smartphone app Req_SmartAQ_2 Push Noti fications Requirement 1



8 Evolutionary learning and improvement of air quality predictions Req_SmartAQ_6 Compute Air Qual i ty Requirement 1


ImprovementsIncreased density of air quality measurements Req_SmartAQ_1 Air Qual i ty Requirement 1



10 Cleaner air as the result of continuous monitoring and awareness of air quality Req_SmartAQ_1 Air Qual i ty Requirement 1








Table 22: Matching of KPIs and Requirements for Pilot: Smart Air Quality – Scenario Provide councils view of their city

Smart Air Quality Provide councils a view of their city




2 Revenue stream for local councils aware of and fining air polluters Req_SmartAQ_1 Air Qual i ty Requirement 1


3 Planned Societal Improvements Personalised heat maps for healthier environment Req_SmartAQ_2 Push Noti fications Requirement 1





5 Aware and informed about air quality city authorities Req_SmartAQ_2 Push Noti fications Requirement 1


6 Planned Technical Improvements Personalised dashboards Req_SmartAQ_4 Provide heat maps Requirement 1

7 Near real-time air quality monitoring alerting and predicting Req_SmartAQ_4 Provide heat maps Requirement 1


8Near real-time air quality monitoring

alerting and predictingIncreased density of air quality measurements Req_SmartAQ_6 Compute Air Qual i ty Requirement 1



4. Preparation of User Acceptance Testing for ex-post Evaluation within

bIoTope Pilots

Based on the approach described in section 2.2. of deliverable D2.1 “Evaluation Methodology for Pilots Validation”, the evaluation of the Business Services developed within bIoTope is described in the following sections. Before presenting details of the Evaluation of each Use Case, the preparation status of the ex-post Evaluation of bIoTopes’ Pilots is described in general in section 4.1. Section 4.2 addresses bIoTopes’ Cross-domain Smart City Pilots, whereas section 4.3 addresses bIoTopes’ Domain-specific Pilots. The methodology used to execute the ex-post Evaluation of bIoTopes’ Pilots is “User Acceptance Testing” according to Hambling and von Goethem [Hambling and von Goethem], as described in section 2.2 of deliverable D2.2.

4.1. Evaluation of bIoTope Pilots based on User Acceptance Testing

In order to ensure that the Business Services developed within bIoTope are addressing user’s KPIs, described in chapter 5 of deliverable D2.2 “Evaluation Methodology for Pilots Validation”, User Acceptance Tests are executed within bIoTope. The definition of the KPIs is based on user stories defined for each bIoTope Use Case. Those Use Cases exemplify how bIoTope can improve the daily life of Europe’s citizens. Improved process flows for each bIoTope Use Case are described in chapter 5 of deliverable D2.1. In addition, a number of different Key Performance Indicators (KPIs) expresses the improvements bIoTope wants achieve through the realization of several Business Services. Each KPI is contributing to the planned achievements described in bIoTopes’ Part B of the Description of Actions. For clustering the KPIs the following criteria were used:

Planned Economical Improvements

Planned Societal Improvements

Planned Technical Improvements

Planned Environmental Improvements

In fact, the improvements are not assessed directly through the KPIs within User Acceptance Tests. In order to evaluate non-functional requirements represented by the KPIs, based on the process perspective and functional requirements, based on the technical perspective, also the KPIs were translated into requirements. The technical perspective is needed as well to make sure, that the Business Services are running reliably. Nevertheless, it needs to be ensured, that all KPIs are represented by at least one requirement. For this reason, the Evaluation presented in chapter 3 of this deliverable was executed to show how the KPIs are represented by the requirements. In order to sum it up, the User Acceptance Tests are evaluating the requirements listed within D2.1 “Ecosystem Stakeholder Requirements Report and Pilot Definition”.

Regarding the procedure to evaluate the Business Services developed within bIoTope, the plan illustrated in Figure 1 was set up. The plan was presented in section 2.2 and 3 of deliverable D2.2 and approved by the whole consortium. As visualized, it was planned to finish the first User Acceptance Tests of already running Business Services until the Intermediate Evaluation, which is documented in this deliverable and illustrates the status of bIoTopes’ Pilots at month 18 of the project.



Figure 1: Initial Evaluation Procedure of the Business Services within bIoTope according to D2.2 “Evaluation Methodology for

Pilots Validation”

User Acceptance Testing is possible only when the software development is almost completed and the software is running reliably. Until the Intermediate Evaluation bIoTopes’ Business Services are still under development. An insight into the progress within the deployment of the Business Services is provided by deliverable D6.1 up to D6.14. The deliverables dealing with the Domain-specific Pilots “Smart Mobility”, “Smart Building and Equipment” and “Smart Air Quality” will be submitted at month 18 of bIoTope. Deliverables presenting the current and future work within Cross-domain Smart City Pilots “Brussels – Capital Region”, “FVH – Forum Virum Helsinki” and “Greater Lyon” will be submitted at month 25 of bIoTope. Nevertheless, the developers of each Business Service have already executed some tests based on the User Stories described in chapter 5 of D2.2. Those tests are a regular part of agile software development, which is used for developing the Business Services within bIoTope.

Status of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots

In the following, the status of User Acceptance Testing is described. Therefore, the process of User Acceptance Testing within bIoTope and the documents generated during this process are shown in Figure 2. Due to the fact, that all bIoTopes’ Business Services are still under development, no User Acceptance Tests could be implemented and reported until month 18 of bIoTope, when this deliverable is submitted. Nevertheless, the preparation of the test has already been going on. Therefore, a “Test Condition Matrix” and a “Test Schedule” have already been designed for every bIoTope Use Case. In the following the status of the preparation is described in detail.



As shown in Figure 2 the “Set Up/ Plan of the Testing” as well as the “Design of the Tests” has been finished successfully. So almost all processes for preparing the initial tests have been terminated. Shaping, planning and preparing of the User Acceptance Tests was already finished and described in section 2.2.1 of D2.2 in order to finish the first step “Set Up/ Plan of the Testing”. During the first step, a rough schedule, presented in section 3.1 of D2.2 was set up, the procedure was defined and the associated tables were prepared and presented in section 2.2 of D2.2.

Figure 2: Status of User Acceptance Testing at month 18 of bIoTope [Hambling and von Goethem]

In addition, the “Design of the Tests” is finished and presented in this deliverable. Section 4.2 is presenting the documents regarding the Cross-domain Smart City Pilots, whereas section 4.3 is presenting the documents regarding the Domain-specific Pilots.

During the “Design of the Tests”, the test conditions were identified for each Use Case and documented in the

spreadsheet named “Test Condition Matrix”. Test conditions are the first step in the creation of tests. Test

conditions are logical statements that represent some aspects of a requirement. In other words, a single test

condition means "there is something that can be tested". All test conditions are addressing the requirements

described in deliverable D2.1. Each test condition represents a single feature of a Business Service that can be

assessed as either true or false. Consequently, the feature is correctly implemented if and only if all conditions

are true. Within the evaluation procedure, every requirement has to be tested at least once. For the

preparation of User Acceptance Testing, every bIoTope Use Case has filled out the “Test Condition Matrix”

illustrated in

Table 23.

In order to ensure that the test cases are of relevance for using the developed Business Services in a real world scenario they have been designed based on the real world processes of bIoTopes’ Use Cases described in chapter 4 and 5 of deliverable D2.2. The project partners being responsible for the evaluation of bIoTopes’



Pilots will use the state of the art processes as well as the test processes described chapter 4 and 5 of deliverable D2.2 to evaluate the KPIs. In order to execute bIoTopes’ evaluation procedures in the most efficient way, the User Acceptance Tests and the KPI assessment will be executed in the same Tests Scenario in parallel, if appropriate.

In order to execute a meaningful evaluation procedure of all test cases, it is necessary to set up a test schedule and a test logging mechanism. In Table 24 the structure of the “Test Schedule” for evaluating a Business Service is presented. The order of the tests should be according to the importance of the requirements, to make sure that the most important tests are examined during the evaluation procedure.



Table 23: Example of a Test Condition Matrix

Table 24: Example of a Test Schedule

Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4 Condition 5

1. Creating

contactsXXX

Create a contact

category1.1

Create a contact

category on excels ior-

contacts

Create a contact

category page

Create a contact

category webs iteAdd a l ink to a contact

Check category status is

new

Create contacts

sub-category1.2

check main

contacts1.3

Sequence no. Requirement no. Service Test case no.Test case description Input data Expected output Test script ID Tester ProcessDate completed

[dd.mm.yy]

1. XXXGeneral

Functional i ty SecurityGF-S1.54

Veri fy that the team

member can log on

to excels ior

Team

member 1

logon

Excels ior home

page appearsSY1.5 Doe John

New

contact

approval

07.03.17



Summary of the status of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots

As already mentioned before, the responsible project partners within the Pilots filled out the spreadsheets “Test Condition Matrix” and “Test Schedule” in the preparation of User Acceptance Testing within the Use Cases. Section 4.2 shows the spreadsheets considering the Cross-domain Smart City Pilots. Spreadsheets considering the Domain-specific Pilots are part of section 4.3. Because no Use Case is running at the time the Intermediate Evaluation is due, the Use Case Owners could not fill the spreadsheets for documentation and reporting of the User Acceptance Tests, yet. Nevertheless, working on the spreadsheets “Test Condition Matrix” and “Test Schedule” was most helpful for the Use Case Owners. That is why; the spreadsheets forced them to deliberate about the implementation of the features of the Business Services. Therefore, the spreadsheets supported the developers to consider all requirements belonging to the Use Case. The Use Case Owners did this by describing the variations of the Conditions, each one representing a feature of a Business Service, belonging to the different requirements.

For guaranteeing a smooth execution of the User Acceptance Tests, the Use Case Owners need to update the “Test Condition Matrix” and “Test Schedule” along with the ongoing development of the Business Services. The reason for this is that the spreadsheets are just representing the status of software development. Especially within a research and innovation project like bIoTope, software development is even more agile than during common software development processes. Therefore, updates of the “Test Condition Matrix” and the “Test Schedule” are necessary to guarantee that the tests cases fit to the developed software prototypes. This enables the Use Cases to adjust themselves according to the findings of bIoTope.

Based on the documents, listed at the bottom of Figure 2, it is obvious that work on the last four documents has not started, when bIoTopes’ Intermediate Evaluation was executed. The reason for that is: it only makes sense to proceed with the preparation of the User Acceptance Tests, when the software development is almost done and the “Test Condition Matrix” and “Test Schedule” are almost finished. When this moment will be reached, the Use Case Owners have to fill the “Test Script”. The “Test Script” is a corner stone for the execution of the User Acceptance Test within each Use Case.

Along with the execution of the User Acceptance Tests the rough “Test Log” and the detailed “Incident Report” can be filled out. Finally, a summary of each User Acceptance Test can be documented in a “Completion Report”.

Decisions deduced from status of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots

Based on the approach deliverable D2.2 “Evaluation Methodology for Pilots validation” presents in section 2.2 and the delays regarding User Acceptance Testing for ex-post Evaluation within bIoTope Pilots the evaluation team and the project management office came up with the following decisions:

Timing of User Acceptance Testing in WP2 and Pilot Deployment in WP6 has to be harmonized more

intensively by defining the start dates and the end dates of User Acceptance testing in accordance with

the Pilots. Table 25 presents the dates agreed.

“Test Condition Matrix” und “Test Schedule” need to be detailed on time, based on the dates

presented in Table 25, in order to define one test case for each test condition within the “Test Script”

as well right before starting User Acceptance Testing.

Tester need to be selected and a detailed timetable needs to be agreed with the tester according to

the high-level schedule of each Pilot.

Documentation of User Acceptance Tests need to be send over to BIBA, being the task leader of Task

2.E, in order to ensure a proper evaluation of the project and for including the results into the final

deliverable D2.6 at month 36 of bIoTope.



Completion reports need to be presented in front of the consortium members at the meeting

following each User Acceptance Test. At the end of the presentation the consortium has to discuss

about the incidents occurred and whether some additional actions are needed across the whole

consortium to solve them. Moreover, ongoing developments within bIoTope can consider the

evaluation results, to improve the impact of bIoTope.

Start and End dates of User Acceptance Tests have to be monitored with emphasis by BIBA as leader

of Task 2.E and by the project management office.

Table 25: Start dates and end dates for User Acceptance Testing within bIoTopes’ Pilots

Pilot Start of User Acceptance Testing End of User Acceptance Testing


Brussels – Capital Region M23 M27

FVH – Forum Virum Helsinki M22 M27

Greater Lyon - Bottle Banks M23 / M25 (staged) M26 / M28 (staged)

Greater Lyon - Heat Wave Mitig. M21 / M28 (staged) M24 /M31 (staged)


Smart Mobility M26 M33

Smart Building and Equipment M20 M23

Smart Air Quality M24 M28



4.2. Evaluation of Cross-domain Smart City Pilots

The following sections 4.2.1 – 3 show the “Test Condition Matrix” and the “Test Script” for the preparation of the User Acceptance Tests within each use cases of the cross-domain Smart City pilots.


Table 26: Test Condition Matrix for Pilot: Brussels – Use Case: Safety around School

Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3

1. Manage issuesReq_Brussel_2_

3Identi fy Safer Routes 1.1 Dashboard enables consultation of i ssues

2. Organise co-mobi l i ty

sess ionReq_Brussel_6 Organize co-mobi l i ty 2.1 Co-mobi l i ty groups are created

Co-mobi l i ty i tinerary are

created

Req_Brussel_6_

5_1Start a co-mobi l i ty sess ion 2.2

Other members of the group receives a

noti fication

Co-mobi l i ty sess ion is

vis ible on a map

Req_Brussel_6_

5_2Join a co-mobi l i ty sess ion 2.3 Co-mobi l i ty sess ion is vis ible on a map

Req_Brussel_6_

5_3Stop a co-mobi l i ty sess ion 2.4 Parents receives a noti fication (optional )

3. Inform students about

safety

Req_Brussel_1_

2Col lect data about s tudents 3.1

Req_Brussel_2_

3Identi fy Safer Routes 3.2

Waze information are col lected and

analysed

beMobi le information

are col lected and

Safety routes are

analysed

Req_Brussel_7_

3

Inform about safer and

greener paths3.3


analysed


are col lected and

Safety routes are

analysed

4. Ca lculate safer

i tinerariesReq_Brussel_1 Col lect data 4.1


analysed


are col lected and

Safety routes are

analysed

Req_Brussel_2_

2Predict traffic 4.2 Traffic data are analysed

5. Send/Receive

noti ficationsReq_Brussel_9 Manage noti fications 5.1

Noti fication are exchanged between

students , school management, parents

6. Manage Gamification

points

Req_Brussel_7_

1Reward green behaviour 6.1 Gamification points are ca lculated



Table 27: Test Schedule for Pilot: Brussels – Use Case: Safety around School

Sequence

no.Requirement no. Service Test case no. Test case description Input data Expected output Test script ID Tester Process

Date completed

[dd.mm.yy]

1Req_Brussel_8 -

Manage IssuesIssues Bxl -001

Create an issue via

mobi le app

Student

geolocal isationIssue is created

2Req_Brussel_6 -

Organize co-mobi l i tyIssues Bxl -002

Issues are vis ible

for s tudentsIssues

A comobi l i ty group is

created

3Req_Brussel_6 -

Organize co-mobi l i tyCo-mobi l i ty Bxl -002 Plan co-mobi l i ty

A comobi l i ty group and de

predefined i tinerary are

created

4Req_Brussel_6 -

Organize co-mobi l i tyCo-mobi l i ty Bxl -004

Start de co-mobi l i ty

sess ionGroup & i tinerary

Other group members

receives a noti fication

5Req_Brussel_6 -


Accept invi tation to

join the sess ion

Co-mobi l i ty

Sess ion

Leader recieves a

noti fication

6Req_Brussel_6 -


Decl ine invi tation

to join the sess ion

Co-mobi l i ty

Sess ion

Leader recieves a

noti fication

7Req_Brussel_6 -


Start joining the

group

Co-mobi l i ty

Sess ion

8Req_Brussel_6 -


Stop de co-mobi l i ty

sess ion

Co-mobi l i ty

Sess ion

Sess ion is s topped &

(optional ) parent receives a

noti fication

9

Req_Brussel_7 -

Promote safer &

greener mobi l i ty

Itineraries Bxl -009Calculate Safer

i tinerary

Issues , analysed

data

Influence i tineraries based

on data col lected on

Brussels Region

10

Req_Brussel_7 -

Promote safer &

greener mobi l i ty

Safety Informations Bxl -010Show Safety

information

Issues , analysed

data

Issues and safer routes are

vis ible on a map

11Req_Brussel_9 -

Manage noti ficationsNoti fication Bxl -011 Send a noti fication

Recipients receives

noti fication

12

Req_Brussel_7_1 -

Reward green

behaviour

Gamification Bxl -012

Receive

gamification points

for i ssue creation,

green mobiti ly

usage, …

Is sues ,

geolocal isationAdded Gamification points




4.2.2.1. Use Case: EV Charging & Parking App

Table 28: Test Condition Matrix for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App


1. 1 Insta l l & run app 1.1Download the app from

web page

Insta l l the App on

AndroidRun the app View main page

2. 2 Login 1.2 Select login page View proper login page Type login credentia ls Veri fy successful login

3. 3 Map interface 1.3 View a map Pan & zoom mapChoose location on

map

2. 4View parking

spots2.1

Find parking spots on

mapChoose parking spot

View

avai lable/reserved

parking spots on map

View info on parking

spot

3. 5 Configure profi le 3.1 Choose profi le View profi le pageEnter EV socket

parametersView updated profi le

4. 6Find suitable

charging spots4.1

View map with profi le

compatible charging

spots

Choose target parking

spot

View map with target

selected

5. 7 Routing 5.1 Have target selected Cl ick on routing

View a route from

current pos i tion to

target on map

6. 8Approaching

target6.1 View a route to target

View updates on route

whi le moving

View stop routing,

target found message

Acknowledge parking

s tarted

View updated parking

spot s tatus on map

7. 9 Access charger 7.1View instructions on

access ing chargerOpen charger l id Connect charger to car Observe charging s tatus

8. 10 Stop charging 8.1 Select end charging Detach charger cable Close charger l id Observe charging s tatus



Table 29: Test Schedule for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App


[dd.mm.yy]

1. App insta l l

Insta l lation of EV

Charging & parking

app

App-1Can user insta l l the

app succesfuly.apk fi le

App can be

s tarted on

device

30.06.17

2. Map view

Show avai lable

parking & charging

spots

Map-1

Are avai lable

parking & charging

spots vis ible and

understandable on

map to user

App

interaction

Map shows

spots30.06.17

3. Info viewShow info on

selected spotMap-2

Test that app can

show data on target

App

interaction

Info view on

target spot30.06.17

4. Target selection Define target spot Map-3

Test that target

selection on map

works

App

interaction

Map shows a

route to target30.06.17

5. Route fol lowing Routing Map-4Show advancement

on mapGPS Updated route 30.06.17

6. Charger permiss ionRelease charger for

usePole-1

User can open

charging pole with

reservation

credentia ls

App

interactionCharger opens 30.06.17

7. Charger charges Actual charging of EV Pole-2User connects cable

to car for charging

Measured

current on

pole

Charging s tatus

view30.06.17




4.2.3.1. Use Case: Bottle Banks

Table 30: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Bottle Banks


1.Col lection

schedul ing and

road maps

optimization

Req_Lyon_1_1

Generate the l i s t

of bottle banks to

empty

1.1

Bottle banks under the

ful lness l imit are

miss ing

Bottle banks above the

ful lness l imit are

present

Establ ish the

road maps1.2

Road map is relevant

regarding bottle banks

to empty

Road map cons ider the

truck capacity

Road map

adjusted during

col lection

1.3

New road map

proposed when a bottle

bank next to the truck i s

ful l

2.Cons ider

res identia l

area

Req_Lyon_1_2

Road map and

tour schedul ing

cons ider

forbidden time

s lots in

res identia l areas

2.1

Define a new

res identia l area ->

impact on the tour

schedul ing/road map

Modifiy a time-s lot for

a res ientia l area ->

impact on the tour

schedul ing/road map

3.Banks

ful lness rate,

weather

forecast,

coming events

Req_Lyon_1_3

The proposal of

l i s t of banks to

empty learn from

his torica l data on

ful lness rate

3.1

Fulness prediction

improved after 2 weeks

of operations

Fulness prediction


of operations

Fulness prediction


of operations

Weather forecast

i s cons idered in

the prediction of

ful lness

3.2

Fulness prediction

modified i f warm

temperature forecasted

Fulness prediction

modified i f ra inee

week-end coming

Coming events

are cons idered in

the prediction of

ful lness

3.3

Fulness prediction

modified i f an event i s

coming

Fulness prediction

modified i f an event i s

cancel led



4.Traffic

predictions

and real -time

traffic

Req_Lyon_1_4

Road maps are

modified in “real -

time” cons idering

the real - time

road events

4.1

Road map modified

depending on trafic

jams

One-hour traffic

prediction is used

to optimize the

road maps

4.2

Road map modified

depending on trafic

conditions in the next

hour

5.Optimization

in terms of

dis tances , time

Req_Lyon_1_5

Road maps are

ca lculated in

order to minimize

the dis tance

traveled by the

trucks and the

duration of each

road map

5.1

6. Information

exchange with

ci tizen porta l

Req_Lyon_1_7

Status of each

bottle bank is

ava i lable

6.1Consultation of ful l

bottle banks

Consultation of

ava i lable bottle banks

Consultation of oi t of

order bottle banks

Citizens can

parameter his

favouri te porta l

and a lerts

6.2Reception of an a lert i f

"my" bottle bank is ful l

Ci tizens can

report problems

on bottle banks

6.3Reporting of a ful l

bottle bank

Reporting of an out of

order bottle bank

7. Dai ly bottle

banks

dashboard and

a lerts

Req_Lyon_1_8Data for each

bottle bank7.1

Dashboard enables

consultation of

detai les and his torica l

data for each bottle

bank

Consol idated

data7.2

Dashboard enables

consultation of dayly,

monthly, yearly

consol idated data

Alerts 7.3Bottle banks about to

be ful l are vis ible

8. IoT BnB

compl ianceReq_Lyon_1_9

Bottle bank

sensors avai lable

in IoT BnB

8.1Access to data sensors

via IoT BnB / O-MI node



4.2.3.2. Use Case: Heat Wave

Table 31: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Heat Wave


1.Citizen porta l Req_Lyon_2_1

Various cl imatic

informations are

avai lable on the

porta l

1.1

Access to the cl imatic

conditions at pi lot

location on the ci tizen

porta l

Ci tizen can post

on the porta l his

perception about

temperatures/hu

midity conditions

1.2

Creation of a

perception on the

porta l

2. Temperature

profi le Req_Lyon_2_2

Sensors data are

processed to

produce various

indicators

2.1

Temparature profi le i s

ava i lable in the pi lot

Dashboard and

relevant

3. Temperature

information Req_Lyon_2_3

Local current

temperature is

ava i lable at any

time publ ished

on O-MI node

3.1

Temperature

information is

ava i lable in the pi lot

Dashboard and

relevant

Sensors present in IoT

BnB

4. Sensors

network - Ai r

temperature

and humidity

sensors

network

Req_Lyon_2_4

Data avai lable

form

crowdsources and

metropol i tan

sensors

publ ished on O-

MI node

4.1Sensors present in IoT

BnB

5. Soi l humidity

sensorsReq_Lyon_2_6

Tens iomanager

data publ ished in

O-MI node


BnB

6. Trees

evapotranspira

tion sensors

Req_Lyon_2_7

Pepipiaf data

publ ished in O-MI

node


BnB

7. Weather

forecast Req_Lyon_2_8

Weather forecast

avai lable7.1



8. Rainwater

tank level Req_Lyon_2_9

Tank data

publ ished in O-MI

node

8.1Information present in

IoT BnB

9. Watering

scheduler Req_Lyon_2_10

Watering is

triggered

according to the

operating rules

9.1



4.3. Evaluation of Domain-specific Pilots

The “Test Condition Matrix” and the “Test Script” prepared for the Use Cases of the Domain-specific Pilots in order to execute the User Acceptance Tests are shown in the sections 4.3.1 - 3.


Table 32: Test Condition Matrix for Pilot: Smart Mobility


Find suitable parking

locationsReq_SmartM_1

Connect to discovery

service

Lis t a l l parking locations in

area

Retrieve detai led information

about avai lable parking

poss ibi l i ties

Req_SmartM_1_1Connect to discovery

service

Lis t a l l parking locations in

area

Query additional

information for the parking

s tations

Information is retrieved

as l inked data

Information contains

payment options

Find information sources for

traffic dataReq_SmartM_2


service

Lis t traffic data sources in

areaRetrieve s tatic data

Get rea l -time information

about trafficReq_SmartM_2_1


service

Lis t traffic data sources in

area

Retrieve l ive data

described as l inked data

data is geo-referenced

us ing a known and

suitable format

Find suitable charging

s tationsReq_SmartM_3


service

Lis t a l l charging s tations in

area

Retrieve s tatic data about

charging s tations

Charging s tation avai labi l i ty Req_SmartM_3_1Connect to discovery

service

Lis t a l l charging s tations in

area

Retrieve s tatic data about

charging s tations

Retrieve avai labi l i ty of

charging s tations

Uniform Payment system

avai lableReq_SmartM_4

Identi fy poss ible

payment methods for

enti ties

Perform a secure transaction

(payment process)

Pre-Conditioning of vehicle Req_SmartM_5

Secured service for pre-

conditioning offered by

vehicle

Pre-Conditioning service can

be triggered by authorized

person

Combine Smart Home and

VehicleReq_SmartM_5_1

Pre-Conditioning

service can be triggered

by smart home (m2m)

Driver identi fication Req_SmartM_6

Secure identi ty

management in

ecosystem avai lable

Make use of driver related

dataReq_SmartM_6_1

Secure identi ty

management in

ecosystem avai lable

Relevant mobi l i ty data can

be used for driver-related

services



Table 33: Test Schedule for Pilot: Smart Mobility


[dd.mm.yy]

1. Req_SmartM_1Find suitable

parking locations

Lis ted Parking spots can be

used for parking services

2. Req_SmartM_2

Find information

sources for traffic

data

Traffic sources can be used

for traffic and routing

systems

3. Req_SmartM_3Find suitable

charging s tations

Retrieved charging s tations

can be used for charging

the electric car

4. Req_SmartM_4Uniform Payment

system avai lablePayment poss ible (/done)

5. Req_SmartM_5Pre-Conditioning of

vehicle

Car s tarts preconditioning

on-demand




Table 34: Test Condition Matrix for Pilot: Smart Building and Equipment


Device commiss ioning to identi ty-

based network

Req_SmartBE_2_3

_1

Commiss ion Mist device to

identi ty-based network

Connect to device us ing identi ty-

based communication over mobi le

network

Publ ish expert to directoryReq_SmartBE_1_3

_2

Create identi ty for the

expert

Publ ish expertise identi ty (with

description) to a cata logue service

Expert discovery from directoryReq_SmartBE_2_7

_3

Lis t experts (discover their

identi ties )

Connect to expert (us ing identi ty-

based communication)

Expert access to device Req_SmartBE_2_7 Send request to expertExpert achieves connection for

monitoring and control

Publ ish parking service to directoryReq_SmartBE_1_3

_2

Publ ish Mist device to

directory

Discover parking service from

directory

Req_SmartBE_1_3

_2

Lis t publ ished services ,

discover their identi ties

Connect to parking service Req_SmartBE_1_2

Connect to discovered

parking service provider

(us ing identi ty-based

communication)

Send activate/deactive commands

to parking serviceReq_SmartBE_2_7 Start parking meter Stop parking meter



Table 35: Test Schedule for Pilot: Smart Building and Equipment

Sequence no. Requirement no. Service Test case no. Test case description Input data Expected output Test script ID Tester Process

Date

completed

[dd.mm.yy]

1.Req_SmartBE_2_3

_1

Device commiss ioning

to identi ty-based

network

Device has generated i tsel f an

identi ty, i s commiss ioned to the

network, and the device admin role

i s cla imed by another identi ty

01.06.17

2.Req_SmartBE_1_3

_2

Publ ish expert to

directory

An expert description a long with

his identi ty i s included in a service

catalogue

01.08.17

3.Req_SmartBE_2_7

_3

Expert discovery from

directory

An app is able to l i s t experts (and

their identi ties ) from a service

catalogue

01.09.17

4. Req_SmartBE_2_7 Expert access to device

Expert i s able to establ ish identi ty-

based communication to the

device

01.10.17

5.Req_SmartBE_1_3

_2

Publ ish parking service

to directory

A parking service description a long

with the service provider identi ty i s

included in a service catalogue

01.08.17

6.Req_SmartBE_1_3

_2

Discover parking

service from directory

An app is able to l i s t parking

services from a service catalogue01.09.17

7. Req_SmartBE_1_2Connect to parking

service

An identi ty-based communication

to the parking service provider i s

achieved

01.10.17

8. Req_SmartBE_2_7

Send activate/deactive

commands to parking

service

The customer has an abi l i ty to

activate / deactive the parking

service

01.11.17




Table 36: Test Condition Matrix for Pilot: Smart Air Quality

Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4

1. Access ing Sensor Network Req_SmartAQ_1 Air Qual i ty 1.1

Req_SmartAQ_1_1 Macro Level 1.2 Lis t ava i lable Sensorsidenti fication and

loca l isation of sensorsclass i fy sensors by region

Req_SmartAQ_1_2 pocket level 1.3 Lis t ava i lable Sensorsidenti fication and

loca l isation of sensors

class i fy sensors by type (mobi le

objects )

Update rate and

connectivi ty of sensors

2. Push Noti fications Req_SmartAQ_2 Push Noti fications 2.1Val idate constra ints (IFTTT-l ike

express ions)

identi fy (user) location /

condition

Receive noti fication

Req_SmartAQ_2_1 O-MI/O-DF 2.2val idate compl iance to O-MI / O-

DFIdenti fy users current context

3. User integration to

publ ish dataReq_SmartAQ_3 Uti l i ty Computation 3.1

Enable users to publ ish data

(crowd sourcing)

Integration of micro-

payments

4. Create heat maps Req_SmartAQ_4 Provide heat maps 4.1

Heat maps with di fferent degree

of detai l (zooming between

macro and micro view)

Req_SmartAQ_4_1 Mobi le vers ion 4.2

communicating with the sensor to

capture and contribute a i r qual i ty

data (see ref. 1.3)

a l lows user to personal ise

the a i r qual i ty subscriptions

(see ref. 2.1)

track user's context and match

subscriptions on the move (see

2.2)

provide visual isation

components to present

a i r qual i ty data to the

user

Req_SmartAQ_4_2 Web Vers ion 4.3 Real -time a ir qual i ty provis ioning

5. Ensure efficency Req_SmartAQ_5 Efficiency

Req_SmartAQ_5_1 Energy 5.1control l the duty cycle of the

sensor

use a publ ish subscribe

approach where data i s only

contributed i f there is a need

use mobi le device onboard

analytics and s torage

Req_SmartAQ_5_2 Communication 5.2us ing rea l -time data reduction on

the mobi le device

provide loca l s torage and

query process ing instead of

sending a l l the data

reduce the communication

between the external sensor and

the mobi le device by restricting

i t to certa in times when data i s

required

Req_SmartAQ_5_3 Duty Cycle 5.3 us ing duty cycle control

use context via mobi le

devicce to determine when to

request data from the sensor

Req_SmartAQ_5_4 Cost 5.4accuracy of the data provided by

the service

determine cost of hosting the

AQMP service (cost of external

sensor and uti l i ty)

6. Compute Air Qual i ty Req_SmartAQ_6 Compute Air Qual i ty 6.1use of models to estimate a i r

qual i ty

use of predictive models to

determine a i r qual i ty

Req_SmartAQ_6_1 Data Fus ion 6.2

efficiently and effectively fuse

data to del iver greater accuracy to

users



Table 37: Test Schedule for Pilot: Smart Air Quality

2

2 AQMP – Air Quality Monitor Provisioning

Sequence

no.Requirement no. Service Test case no.Test case description Input data Expected output Test script ID Tester Process

Date

completed

[dd.mm.yy]

1. Req_SmartAQ_11. Acces Sensor

NetworkAQMP.1

Air qual i ty data i s gathered from

external sensors in the field

Triggering sensors (e.g.

method ca l l )

Sensor va lues from static and

mobi le enti ties (might be

synthes ised fi rs t)

AQMP.script.1

2. Req_SmartAQ_2 2. Push Noti fications AQMP.2Noti fications are released /

triggeredn by constra ints

IFTTT-l ike constra int as

user input

O-MI / O-DF compl iant

noti ficationsAQMP.script.2

3. Req_SmartAQ_33. User integration to

publ ish dataAQMP.3

Pipe external sensor data to the

bIoTope ecosystem

user enables data

release

publ ication of (data) service on

IoTBnB marketplace (including

micropayment)

AQMP.script.3

4. Req_SmartAQ_4 4. Create heat maps AQMP.4Data shown in heatmaps with zoom

functional i ty

data from sensors (macro

and micro sca le)

visual ize heatmap on mobi le and

web vers ion of AQMPAQMP.script.4

5. Req_SmartAQ_5 5. Ensure efficency AQMP.5

Monitor and log (or show) duty cycle

of external sensor and mobi le

device, communication properties

and accuracy

device s tatus

monitoring of devices ' s tatus

(mobi le device and external

sensors )

AQMP.script.5

6. Req_SmartAQ_66. Compute Air

Qual i tyAQMP.6

Stress -test computation of models to

estimate and predict a i r qual i ty and

enhancement in accuracy

various (heterogenous)

sensor data

enriched sensor data with better

accuracyAQMP.script.6



4.3.4. Preparation of the KPI Test Scenarios for ex-post Evaluation of the bIoTope Pilots

Regular evaluation of a complex international research project is of great importance for the success of the project itself and the research and innovation programme as a whole. Based on the evaluation results, the responsible people can adjust or trigger actions to achieve the planned objectives. Of great importance for the evaluation of projects are well defined KPIS. Especially for research projects, considering the IoT, like bIoTope, demonstrating the success of the developed hardware and software solutions is of great importance. Therefore, Part B of bIoTopes’ proposal already defined the following KPIs:

30% increased length of battery operation of electric car on a daily basis – Smart Mobility Pilot

20% increased electric car battery life time – Smart Mobility Pilot

20% energy-reduction in smart building pilot – Smart Building and Equipment Pilot

25% enhanced predicted failure rate regarding HVAC equipment – Smart Building Pilot

50% enhanced early pollution detection – Smart Air Quality Pilot

80% time reduction for creating new services based on available information sources coming from different application domains – Cross-Domain Smart City Pilots

≥50% acceptance of services by citizens (collecting user feedback with a specific UIaaS widget)

Based on these superordinate KPIs the bIoTopes’ Pilots defined individual KPIs for each Use Case. Chapter 5 of deliverable D2.2 presents all KPIs being of importance for the bIoTope Use Cases. For evaluating the KPIs the project partners being responsible for each Pilot defined Test Scenarios. Those Test Scenarios are also described within chapter 5 of deliverable D2.2 along with the KPIs.

Due to the fact, that deliverable D2.2 was already prepared at M8 of the project, some necessary adjustments concerning the whole project and some Pilots have become necessary, since. Such adjustments are obviously when executing complex international projects such as bIoTope. Based on those adjustments it is necessary to update KPIs and the Test Scenarios as well. In the majority of cases, the reason for necessary updates are findings made during the progress of the project, which resulted in changed requirements.

Also within bIoTope the KPIs and the associated Test Scenarios need regular updates. This needs to be done by each project partner being responsible for a Pilot. The responsible project partners have to ensure, that the Test Scenario and the KPIs for the ex-post Evaluation are still in line with the Use Case’s requirements as well as the findings from software and hardware developments. In order to enable the evaluation team and the project management office to monitor the preparation of the ex-post Evaluation of bIoTopes’ KPIs, Table 38 summarizes the confirmations from the Pilots regarding the updates of the KPIs and the Test Scenarios during the Intermediate Evaluation.

Table 38: Progress of the preparation for the KPI Test Scenarios for ex-post Evaluation

Pilot KPIs confirmed [Y/N] Test Scenarios confirmed [Y/N]


Brussels – Capital Region N Y

FVH – Forum Virum Helsinki N Y

Greater Lyon N Y


Smart Mobility N Y

Smart Building and Equipment Y - updated Y - updated



Smart Air Quality N Y (to be updated)

Relevant updates of the KPIs and the Test Scenarios will be send to the evaluation team and the project management office during adaptations on use cases and demonstrators caused by on-going development and final documented in the evaluation deliverable D2.9.

In order to ensure that the projects partners being responsible for the evaluation of the project can execute the assessment in an efficient way and to have meaningful evaluation results, the Evaluation needs to be prepared. Therefore, the Evaluation hast to be planned and coordinated across the whole project. Considering the KPI Test Scenarios, being part of the project’s evaluation, Table 39 shows the start dates and the end dates of the execution of the tests. As already expressed in chapter 4, the User Acceptance Tests and the KPI-based ex-post Evaluation of the bIoTope Pilots can be executed at the same time. Hence, the project partners being responsible for the execution of the evaluation should align the start and end dates of the User Acceptance Tests, shown in Table 25, and the KPI Tests Scenarios shown inTable 39.

Table 39: Start dates and end dates for executing the KPI Test Scenarios for ex-post Evaluation within bIoTopes’ Pilots

Pilot Start of KPI Tests Scenarios End of KPI Tests Scenarios


Brussels – Capital Region M27 M31

FVH – Forum Virum Helsinki

Greater Lyon – Bottle Banks M27 M 31

Greater Lyon – Heat Wave Mitig. M29 M33


Smart Mobility

Smart Building and Equipment

Smart Air Quality

Smart Waste Management

In order to ensure an efficient evaluation of all bIoTope Use Cases, the assessment team developed a template for the KPI-based ex-post Evaluation of each Use Case within the Pilots. When executing the Test Scenarios the project partners should use the EXCEL-based template for documentation and reporting. The developers will also use the template to adapt the Business Services, if the objectives are not fully met during the first test run. Based on the template the evaluation results will be included in deliverable D2.9 in M36 of bIoTope.



Table 40: Template for the KPI-based ex-post Evaluation of the bIoTope Pilots

Use

Case no.Use Case KPI Tester [name]

Date raised

[dd.mm.yy]

KPI Test

Scenario ID

[A,B,C]

KPI Objective archieved [Y, N, %] Issue resolution status Assigned to [name]

1 Safe Roads for ChildrenLowering the amount of accidents in school zones and on

the i tinerary of the chi ldren

Shortening the average dai ly commute time of chi ldren and

their parents

Lowering traffic jams in around schools

Increase the effective securi ty of school chi ldren, to be

measured via perceived sense of securi ty

Increase usage of other transport methods than the

personal car

Lowering the amount of accidents in school zones

Bundl ing di fferent sources (sensors ) via O-DF/O-MI

Route planning of chi ldren keeps into account overa l l

pol lution a long poss ible i tineraries

KPI Test

Scenario

ID

A

B

C

Test Scenario Description

A survey with chi ldren and their parents i s foreseen to determine the s i tuation prior to the project and 3 months after the project has been deployed. This survey wi l l evaluate chi ldren ́s and their parents ’

perception on the fol lowing:

Sense of (in)securi ty of the chi ldren and their parents

Commute time of chi ldren separately

The need of parents to accompany chi ldren on chi ldren 's commute

Mode of transport (including chi ldren separately and/or together with the parents )

Shi fts in any of the above

KPI Test Log



5. Conclusion

Initially the development progress within bIoTope was made by assessing the Technology Readiness Level (TRL) and the Integration Readiness Level (IRL), of the intended technologies in the ecosystem, namely the core components (services), the cross-domain city pilots and the domain-specific pilots.

Summarized, current cross-pilot TRL assessment turned out legitimately high, while IRL provides more innovation potential. This is actually the representation of a problem statement, bIoTope addresses in the current landscape of IoT, whereas already existing smart objects (products with high TRL) need to be orchestrated in a way that seamless information flow between vertical silos can be achieved for seamless data exchange and creation of new services. The problem representation consequently is directly characterized in the IRL indications of the pilots and core components (XaaS-connected technologies).

Previously recorded KPIs and requirements are mapped to each other, which shows the alignment of planned use cases and evaluation strategy. This results in a stable preparation of user acceptance testing for a ex-post evaluation later in the project. Anyhow minor adaptations are necessarily not excluded due to changes in the pilots. The evaluation methodology is used to fundamentally reason such decisions and re-arrangements to ensure a positive project result in a holistic manner.

Future work will cover the evaluation of existing prototypes, which are initially provided by the domain-specific pilots and later in M25 by the cross-domain city pilots, as well as finalized demonstrators in M33 and M36. Introduced test schedules will be applied and further test scripts will created and executed.



6. References

Air Force Research Laboratory (AFRL). AFRL Hardware and Software Transition Readiness Level Calculator,

Version 2.2. [Online] [Cited: 16 01 2017] https://acc.dau.mil/CommunityBrowser.aspx?id=741047.

Hambling, Brian and von Goethem, Pauline. User Acceptance Testing: A Setp-By-Step Guide. Swindon: BCS,

2013.

Sauser, Brian; Gove, Ryan; Forbes, Eric and Ramirez-Marquez, Jose Emmanuel. Integration maturity metrics:

Development of an integration readiness level. In: Information Knowledge Systems Management, vol.9, pp 17-

46, 2010.

U.S. Government Accountability Office. Technology Readiness Assessment Guide: Best Practices for Evaluating

the Readiness of Technology for Use in Acquisition Programs and Projects. [Online] [Cited: 16 01 2017]

http://www.gao.gov/assets/680/679006.pdf.

https://acc.dau.mil/CommunityBrowser.aspx?id=741047

http://www.gao.gov/assets/680/679006.pdf



7. Annex 1

Template - IRL (Integration Readiness Level) Questionnaire (p. 69)

Template - TRL (Technology Readiness Level) Questionnaire (p. 70)

Template - UAT (User Acceptance Test) Worksheet (several sheets for different steps, pp.71)





IRL Questionnaire

IRL Definiton Description (Start at the top and first Definition answered with yes leads to potential rigth IRL)Checkbox

(y/n)potential IRL

IRL 9Integration is proven through successful

operations in the system environment.

IRL 9 represents the integrated technologies being used in the system environment successfully. In order for

a technology to move to TRL 9 it must first be integrated into the system, and then proven in the relevant

environment, so attempting to move to IRL 9 also implies maturing the component technology to TRL 9.

IRL 9

IRL 8

Actual integration completed and qualified through

test and demonstration, in the system

environment.

IRL 8 represents not only the integration meeting requirements, but also a system-level demonstration in the

relevant environment. This will reveal any unknown bugs/defect that could not be discovered until the

interaction of the two integrating technologies was observed in the system environment.

IRL 8

IRL 7

The integration of technologies has been verified

and validated and an acquisition/insertion

decision can be made.

IRL 7 represents a significant step beyond IRL 6; the integration has to work from a technical perspective,

but also from a requirements perspective. IRL 7 represents the integration meeting requirements such as

performance, throughput, and reliability.

IRL 7

IRL 6

The integrating technologies can accept,

translate, and structure information for its

intended application.

IRL 6 is the highest technical level to be achieved, it includes the ability to not only control integration, but

specify what information to exchange, unit labels to specify what the information is, and the ability to translate

from a foreign data structure to a local one.

IRL 6

IRL 5

There is sufficient control between technologies

necessary to establish, manage, and terminate

the integration.

IRL 5 simply denotes the ability of one or more of the integrating technologies to control the integration itself;

this includes establishing, maintaining, and terminating.IRL 5

IRL 4

There is sufficient detail in the quality and

assurance of the integration between

technologies.

Many technology integration failures never progress past IRL 3, due to the assumption that if two

technologies can exchange information successfully, then they are fully integrated. IRL 4 goes beyond

simple data exchange and requires that the data sent is the data received and there exists a mechanism for

IRL 4

IRL 3

There is compatibility (i.e. common language)

between technologies to orderly and efficiently

integrate and interact.

IRL 3 represents the minimum required level to provide successful integration. This means that the two

technologies are able to not only influence each other, but also communicate interpretable data. IRL 3

represents the first tangible step in the maturity process.

IRL 3

IRL 2

There is some level of specificity to characterize

the interaction (i.e. ability to influence) between

technologies through their interface.

Once a medium has been defined, a “signaling” method must be selected such that two integrating

technologies are able to influence each other over that medium. Since IRL 2 represents the ability of two

technologies to influence each other over a given medium, this represents integration proof-of-concept.

IRL 2

IRL 1

An interface between technologies has been

identified with sufficient detail to allow

characterization of the relationship.

This is the lowest level of integration readiness and describes the selection of a medium for integration. IRL 1

In the following Questionnaire, a framework is provided for calculating IRLs within bIoTope.

How to use: To calculate a IRL for a relevant technology, the answers and definitions within the following TOP LEVEL VIEW Table have to be answered, starting from the top of the table. The first question or

definition answered with YES leads to the potentially right IRL.





TRL Questionnaire

Nr. TOP LEVEL VIEW (Start at the top and first Question answered with yes leads to potential rigth TRL)Checkbox

(y/n)Continue with

1Has an identical technology component been successfully used in an operational scenario or in an identical

configuration?TRL 9

2Has an identical technology component been demonstrated in an operational scenario, but in a different

configuration/system architecture?TRL 8

3Has an identical technology component been qualified for an operational scenario but not operationally

demonstrated?TRL 8

4 Has a prototype technology component been demonstrated in an operational scenario? TRL 7

5 Has a prototype been demonstrated in a relevant scenario, on the target or surrogate platform? TRL 6

6Has a breadboard technology component been demonstrated in a relevant (typical; not necessarily

stressing) scenario?TRL 5

7 Has a breadboard technology component been demonstrated in a laboratory (controlled) scenario? TRL 4

8 Has analytical and experimental proof-of-concept been demonstrated? TRL 3

9 Has a concept or application been formulated? TRL 2

10 Have basic principles been observed and reported? TRL 1

11 None of the above TRL 0

In the following Questionnaire, a framework is provided for calculating TRLs within bIoTope. The framework is based on the AFRL Hardware and Software

Transition Readiness Level Calculator, Version 2.2 developed by the Air Force Research Laboratory (AFRL) [Air Force Research Laboratory]. This tool is

aligned with the TRL elements defined by the U.S. DoD.

How to use: To calculate a TRL for a relevant technology, the answers within the following TOP LEVEL VIEW Table have to be answered, starting from the top

of the table. The first question answered with YES leads to the potentially right TRL. In order to check whether this first estimation is right, more detailed

facts are going to be analysed. Those checks are provided in Sheet TRL 9 to TRL 0. All facts listed within each of the tables have to be answered with YES to

achieve the corresponding TRL.

DELIVERABLE D2.6 Evaluation Report of the bIoTope...

Documents

Transcript of DELIVERABLE D2.6 Evaluation Report of the bIoTope...