Stratumseind FIWARE Implementation and Configuration...

FIWARE Implementation and Configuration Guide

Rick Cookbook 1 of the Smart City StarterK!t

DITSS: Dutch Institute for Technology, Safety & Security Version 0.1 Draft October 2017

October 2017 Stratumseind FIWARE Implementation and configuration guide -

Versie 0.1 Cookbook 1 of the Smart City StarterK!t

Draft

Colophon Edition DITSS Dutch Institute for Technology, Safety & Security

Version control

Revision date Version Author(s) Changes Distribution

15-Jun-2017 0.1 Fourthcode Initial delivery DITSS, Municipality of Eindhoven

26-Jul- 2011 0.2 Fourthcode After rebuild DITSS DITSS, Municipality of Eindhoven, Venturespring

26-Jul- 2011 0.3 Fourthcode After review DITSS/Venturespring

18-Okt-2017 0.8 DITSS For distribution to Atos DITSS Atos

11-Nov-2017 0.92 DITSS After review Atos DITSS Atos

Reference documents in the Smart City StarterK!t Ref. Title Author(s) Version Date

A] Spotlight on Smart City Eindhoven Venturespring B.V. Final Oct 2015

B] Open Data Principles Eindhoven Final Mar 2016

C] Smart Society Charter, IoT Architecture principles & guidelines

Eindhoven 1.0 WIP

June 2016

D] Letter with principles Digital City (Dutch national level)

Ollongren, Depla Jan 2017

E] Cookbook 1: Stratumseind FIWARE Implementation and Configuration guide

DITSS WIP Oct 2017

F] Cookbook 2: IoT-specific Privacy Impact Assessment

DITSS Under development

G] Cookbook 3: Platform Branching Management’

Venturespring B.V 0.3 WIP

H] Cookbook 4: Urban Platform guidelines to create added-value information, services and/or ‘killer’ applications

DITSS Under development

..

Z] Conditions for using the Smart City StarterK!t

DITSS WIP Oct 2017

Terms and conditions This cookbook is part of the Smart City StarterK!it and are subject to the terms and conditions as defined in reference document Z].



Draft

4 DITSS Dutch Institute for Technology, Safety & Security

Contents

Colophon 2

Contents 4

1 Introduction 6 1.1 Position of this cookbook in the starter kit 6 2 Solution Architecture 9 2.1 Overview components 9 2.2 Architecture 10 3 FIWARE Components and Configuration 12 3.1 The Orion Context Broker 12 3.1.1 What is the Orion Context Broker? 12 3.1.2 Installation of Context Broker from source 12 3.1.3 Configuring the Context Broker 13 3.1.4 Testing the Context Broker 13 3.2 The Proton Complex Event Processor (CEP) 15 3.2.1 What is Complex Event Processor? 15 3.2.2 Installing the Complex Event Processor 15 3.2.3 Configuring the Complex Event Processor 17 3.2.4 Testing the Complex Event Processor 22 3.3 CKAN 26 3.3.1 What is CKAN? 26 3.3.2 Installing CKAN 26 3.3.3 Testing CKAN 29 3.4 CYGNUS 33 3.4.1 What is CYGNUS? 33 3.4.2 Installing CYGNUS 33 3.4.3 Configuring CYGNUS 35 3.4.4 Testing CYGNUS 38 3.5 IOT AGENT 40 3.5.1 What is the IOT Agent? 40 3.5.2 Installation of IoT Agent 40 3.5.3 IOT Agent tester 43 4 Custom Modules 46 4.1 Smart City Application in NodeJS 46 4.1.1 Installation 46 4.2 Web Interface 47 4.2.1 Dashboard 47 4.2.2 Munisense Sound Sensor Integration 48 4.2.3 ViNotion people counting Integration 48 4.2.4 Twitter Feeds Integration 50 4.2.5 Configuration Module 51 4.2.6 Charts 53 4.2.7 User and Session Management 53 4.3 Mobile Interface 54 4.3.1 Developing Mobile Interface Based on CEP 54 4.3.2 Developing Mobile Interface Based on Firebase 54 5 Integration 58 5.1 IOT Agent and Orion 58



Draft

DITSS Dutch Institute for Technology, Safety & Security 5

5.2 Orion and CEP 60 5.3 Orion and Cygnus 62 5.4 Cygnus and CKAN 64 5.4.1 Mapping NGSI events to NGSIEvent objects 65 5.4.2 Mapping NGSIEvents to CKAN data structures 65 5.4.3 Organizations naming conventions 65 5.4.4 Packages/datasets naming conventions 65 5.4.5 Resources naming conventions 65 5.4.6 Row-like storing 65 5.4.7 Column-like storing 66 5.4.8 Organization, dataset and resource names 66 5.4.9 Row-like storing 67 5.4.10 Column-like storing 68 6 Quality Assurance 70 6.1 Sensor Data Consistency 70 6.2 Integration Testing 70 6.2.1 Fiware Components 70 6.2.2 System and Environment 70 6.3 Mobile Responsiveness 70 6.4 Server Resource Utilization 70 6.4.1 vmstat 70 6.4.2 CPUUtilization 71 7 References 72 8 Environment 73



Draft


1 Introduction FIWARE is a middleware platform for cost-effective development and deployment of Future Internet applications and services in a variety of areas such as Smart Cities, Sustainable Transport and Mobility, Logistics, Renewable Energy, and Environmental Sustainability. FIWARE is actively promoted by the European Union. The API specification of FIWARE is open and royalty-free. The implementation consists of a set of common components (referred to as Generic Enablers) and a set of domain specific components (referred to as Specific Enablers). Sharing between FIWARE users is encouraged and the involvement of users and developers is essential to be a successful standard and reusable platform. The Stratumseind FIWARE Implementation (SEFI) aims to develop a smart context-aware application as a toolkit for other Smart City projects and consequently utilizes various FIWARE Generic Enablers (GEs). The main functionality offered by this toolkit is the smart management of a public café area by analysing sensor data from various installed sensors (sound, video, pollution, parking etc.). In Smart City parlance, this could be referred to as ‘Smart Clubbing’. We believe that SEFI can provide other cities and municipalities with a FIWARE implementation that they can use to also create a Smart Clubbing environment or can extend into other ‘smart domains’.

1.1 Position of this cookbook in the Smart City StarterK!t We have developed the Smart City StarterK!t to cover the following goals and principles:

1. To cater for the Municipality’s role in the current developments regarding Smart City, namely to:

• Safeguard public interest

• Stimulate economic development & eco-system

• Provide a framework to support the existing organic approach

• Create a futureproof environment, prepared to change 2. To have ‘a single clear … vision and strategy .. .to create a development

process for Eindhoven to become a Smart City faster’ and therefore follow as much as possible the main recommendations as described in the report ‘A Spotlight on Smart City Eindhoven’. See further reference A]:

• Set up an Eindhoven Smart City Architecture team to oversee all smart city projects from an architectural integrity and continuity perspective.

• Set up the governance and leadership team to initiate, approve, foster and terminate Smart City projects.

• Engage with stakeholders from industry, academic/research and Eindhoven Municipality

• Co-create a vision with the leaders and keep on iteratively updating that vision collaboratively.

• Set up an independent Security and Risk Assessment audit team for all operational smart city projects.

• Base operational programs on common architectures and platforms. Consider the desired solution as a City Platform. Partner with others to co-create a better, interoperable, jointly tested City Platform faster and to ensure future support commercially.

• Select a vendor independent, non-proprietary reference architecture based on open standards and specifications

• Ensure people see and experience the (financial and non-financial) results as meaningful

• Ensure that steps along the way contribute directly, give space and energy to grow.

3. To cater for the Open Data Principles, as described in reference B]. Most important:

• Data that are collected, generated or measured (for example by sensors that are placed in the public space) should be opened up in such way that everyone can make use of it for commercial and non-

https://www.fiware.org/



Draft


commercial purposes. While doing so, privacy and security aspects should be taken into consideration.

• Data that do not contain personal information (anymore) should be placed in such way that everyone can access these data in an equal manner (for example, through an Open Data portal). We call this “opening up” the data. There should be no technical or juridical obstacles that limit, discriminate or block access to data.

• Data are always opened up free of charge, without unnecessary processing (as much as possible in a raw form) and according to the functional and technical requirements that are yet to be defined.

• The city authorities keep an ongoing dialog with the parties that contribute to the development of data infrastructure in the city and strive to create earning opportunities and a fruitful economic climate

4. To comply with the IoT Charter principles:

See further reference C]

5. To further contribute to the Digital City Principles based on input by vice-mayors Ollongren and Depla to Dutch national level, see reference D].

6. To foster the IoT principles 2, 3, 4 and 5, this cookbook 1 consist of:

• A description of all FIWARE Generic Enablers used

• The installation sources of all components, with dependency packages and/or underlying (system) software

• The configuration of all components and dependencies, port numbers, APIs etc., including the parameterisation of the underlying (system) software.

• The start/stop procedures and tests of all components and dependencies

We think that this cookbook: 1. Provides a basic FIWARE platform, ready for use by all

parties/partners in a Triple or Quadruple Helix eco-system. 2. Reduces the time-to-market for such a basic Smart City platform

from weeks to days 3. Implicitly establishes principles for an Urban Data Platform as

envisaged by ATOS 4. Shares hands-on experience with the working and complexity of

FIWARE 5. Allows for partners to stretch their imagination beyond current

Smart City implementations and pursue innovations that will help to realize a vision of ‘Smart Society’ (in which technology and infrastructure is not the main scope, but its benefits for people)

7. To comply with (EU) legislation, of which the GDPR currently is most prominent, we are in the process of creating cookbook 2. With the aim to provide (among others) an IoT-specific version of a Privacy Impact Assessment, we will contribute to the IoT principles 1, 4 and 6.



Draft


8. To address the complexities involved with (Smart City toolkit or urban data platform) product branching over a lifecycle spanning many years and involving multiple partners with constant evolution of technologies, we are in the process of creating cookbook 3. With ‘Platform Branching Management’ we will contribute to the IoT principle #4.

9. To capture the road towards a single EU digital market, expanding the current building blocks to a full urban data or a Smart Society platform, we are in the process of creating cookbook 4. With the aim to provide (among others) guidelines to create value-adding information, services and/or ‘killer’ applications we will (again) contribute to the IoT principles 4 and 7.



Draft


2 Solution Architecture

2.1 Overview components Web Application Front End – Angular JS

• Main Page with Google Maps locating different sensors

• Various Pages showing Noise Data, People and Bicycle Count, Tweet Information an Configuration page

Web Application Back End – Node JS, MongoDB

• Various Node modules

• MongoDB – version 3.0

Fiware Components

• IOT Agent

• CEP Proton

• Context Broker

• Cygnus

• CKAN Active Sensors from Munisense and ViNotion Mobile Frontend for Notifications



Draft


2.2 Architecture

2.2.1 Base design, functionality and usage

The base design had as goal to connect the available IoT sensors and Actuators (in the

pink bottom layer) via the different existing ‘smart data grid’ connections (in the green

layer) to a (basic) Smart City Platform, consisting of the 4 FIWARE Generic Enablers (in

the blue layer) and providing (again basic) dashboard functionality for the different Fields

Labs undertaken by DITSS on the Stratumseind.

2.2.2 As build design, functionality and usage Note : CKAN is installed in a separate instance (Ubuntu) and its datasets can be called using APIs defined in the app.



Draft


During the implementation/rebuild of the as build we discovered that the sensors are

directly connected to the NodeJS front-end webservice for the dashboards, without the

‘proper’/separated instance and/or use of the IoT agent GE.

This must be fixed a.s.a.p.

2.2.3 ‘Dream’ design, functionality and usage

The dream design had as goal to expand the (basic) Smart City Platform with additional

functionality to:

• facilitate added value of data

This enhanced functionality (possibly delivered by additional IWARE Generic Enablers

as shown in the blue layer) can be used by:

• Again all the different Fields Labs undertaken by DITSS on the Stratumseind.

• All other living labs and/or projects in the Municipality of Eindhoven (Strijp-S, Eckart-

Vaartbroek, City Challenge, Real Estate,….)

• Brainport/B5

• Netherlands NEN

• EU syncity

Smart City Platform

Dataportalplatform

IntegratedData ManagementCKANOpenDataSoft

Smart Data Grid

Living Labs => Field Labs => projects

IoT Sensors + Actuators

REV. DESCRIPTION DATE BYIoT platform, stage 3, New Business Models 1.0 IoT platform, stage 1-3 08-07-2017 RJS

People CountVinotion

FibreopticsEHV Events

IoT Device ManagementIDAS

Sound Imaging

Sorama

Sound +Cell CountersMunisense

Social MediaCoosto

ParkingplacesQ-Park

Weather

Munisense

Light Level

Philips

MicroWaveEHV Events

Zigbee

EHV SH tower

WiFi

EHV Events

PublicInternet

IoT Broker

Context Publish Subscribe Broker

Orion

Big Data AnalyticsCosmos

RT Media Stream processingKurento

Complex EventProcessingProton

Data Opslag Mongo dB

CatalogSystem

DashboardApp(s)

ContextadaptersOpen Remote

GeoDataAdapterstbc

IDM

KeyRock

PEP Proxy

Wilma

Authorization + Authentication

E2Esecurity???

Pricing Models(recur, PpU)

Payments

PayPal

Accountingcallbacks

BusinessModel ManagementBusiness API Ecosystem

Billing &Charging

RevenueSharing

Service Orchestration???

Accountingbackoffice system

Hosting

1

2

2

1

3

3

4

5

LoRatbc

G5 ...GSMtbc

Other IoTsensorsCitizins, tbc

Other IoTactuatorsSMEs, tbc

6

7

7

Authorization PDPAuthZForce



Draft


3 FIWARE Components and Configuration First, you should disable selinux on your CentOS 6.9 server by editing the file

etc/sysconfig/selinux so it looks like this:

SELINUX=disabled

SELINUXTYPE=targeted

If you forget to disable selinux, you cannot access the Context Broker.

Please note you need to add applicable ports to your iptables, otherwise CKAN cannot

connect to the FIWARE CORE server or vice versa! E.g.

iptables -I INPUT -p tcp -m tcp --port 8080 -j ACCEPT

iptables -I INPUT -p tcp -m tcp --port 1026 -j ACCEPT

service iptables save

3.1 The Orion Context Broker

3.1.1 What is the Orion Context Broker? The Orion Context Broker is used to manage the whole lifecycle of context information including updates, queries, registrations and subscriptions. Use the Orion Context Broker to register context elements and manage them through updates and queries. In addition, one can subscribe to context information so when some condition occurs (e.g. a context element has changed) a notification is generated. These usage scenarios and the Orion Context Broker features are described in this document. Context information may come from many different sources:

1. Already existing systems 2. Users, through mobile apps 3. Sensor networks

3.1.2 Installation of Context Broker from source This section describes how to install the Context Broker on Centos 6.9 (6.X). This document uses yum for installing the packages. Alternatively (if yum fails), packages can be installed using rpm by downloading the corresponding rpm file

1. Configure epel repository for Centos: sudo yum install epel-release

2. Install packages dependencies: sudo yum install boost-filesystem boost-thread libmicrohttpd logrotate libcurl boost-regex

3. Install mongodb 2.6 Create a new file /etc/yum.repos.d/mongodb-org-2.6.repo and add the following [mongodb-org-2.6] name=MongoDB 2.6 Repository baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/ gpgcheck=0 enabled=1 sudo yum install mongodb-org

4. Install the Context Broker Create new file /etc/yum.repos.d/testbed-fi-ware.repo and paste the following



Draft


[testbed-fi-ware] name=Fiware Repository baseurl=http://repositories.testbed.fi-ware.org/repo/rpm/6/ gpgcheck=0 enabled=1 sudo yum install contextBroker

3.1.3 Configuring the Context Broker

The Context Broker runs as per the configurations in the file /etc/sysconfig/contextBroker. The default configuration is fine unless mongodb has authentication enabled or the default ports are not available (port 1026 for contextBroker, port 27017 for mongodb). Modify the config file as needed.

1. Start and stop the Context Broker The Context Broker is installed as a service. To start the service: sudo service contextBroker start OR sudo /etc/init.d/contextBroker start To stop the service: sudo service contextBroker stop OR sudo /etc/init.d/contextBroker stop

2. Accessing the Context Broker Context Broker will be running on port 1026 by default configuration Try accessing http://localhost:1026/version

API Documentation : http://fiware-Orion.readthedocs.io/en/master/user/walkthrough_apiv1/index.html

3.1.4 Testing the Context Broker

End to End Testing • Start Context Broker in default port (1026)

• Run the following command: curl --header 'Accept: application/json' localhost:1026/version

• Check that you get the version number as output (along with uptime information and compilation environment information):

List of Running Processes

A process named "contextBroker" should be up and running, Check with a command such as:

ps ax | grep contextBroker

http://fiware-orion.readthedocs.io/en/master/user/walkthrough_apiv1/index.html




Draft


8517 ? Ssl 8:58 /usr/bin/contextBroker -port 1026 -logDir /var/log/contextBroker -pidpath /var/log/contextBroker/contextBroker.pid -dbhost localhost -db Orion

Network Interfaces Up and Open Orion Context Broker uses TCP 1026 as default port, although it can be changed using the -port command line option. Databases The Orion Context Broker uses a MongoDB database, whose parameters are provided using the command line options dbhost, -dbuser, -dbpwd and -db. Note that -dbuser and -dbpwd are only used if MongoDB runs using authentication, i.e. with --auth. Check that the database is working using the mongo console:

mongo <dbhost>/<db> Check the different collections used by the broker with the following commands in the mongo console. However, note that the broker creates a collection the first time a document is to be inserted in it, so if it is the first time the Context Broker is run (or if database was cleaned) and the Context Broker has not received any request yet, no collection exists. Use show collections to get the actual collections list at a given moment.

> db.registrations.count() > db.entities.count() > db.csubs.count() > db.casubs.count()



Draft


3.2 The Proton Complex Event Processor (CEP) 3.2.1 What is Complex Event Processor? The Complex Event Processor Generic Enabler analyses event data in real-time and provides insight and enables response to changing conditions. While standard reactive applications are based on reactions to single events, the Complex Event Processor reacts to situations rather than to single events. A situation is a condition that is based on a series of events that have occurred within a dynamic time window called processing context. Situations include composite events (e.g., sequence), counting operators on events (e.g., aggregation) and absence operators. The Proactive Technology Online is an implementation of the Complex Event Processor Generic Enabler. The Proactive Technology Online is a scalable integrated platform to support the development, deployment, and maintenance of event-driven applications. The associated authoring tool allows the definition of Complex Event Processor applications using a web user interface. The engine is a runtime tool that receives information on the occurrence of events from event producers, detects situations, and reports the detected situations to external consumers. 3.2.2 Installing the Complex Event Processor

1. Prerequisites

• Proton CEP requires Java 7 or later and tomcat 7

2. Install Java 7

• sudo yum install java-1.7.0-openjdk

3. Install tomcat7

• Download https://archive.apache.org/dist/tomcat/tomcat-7/v7.0.64/bin/apache-tomcat-7.0.64.tar.gz

• Extract tomcat into a directory of choice in a different path than your local account, e.g. in /opt/apache-tomcat-7.0.64

• Follow instructions as described in Setup Apache Tomcat for Management in https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/CEP_GE_-_IBM_Proactive_Technology_Online_Installation_and_Administration_Guide for setting the CATALINA_HOME environment variable, configuring manager access (the suggested reference can be copy/pasted) and Enabling JMX access (the suggested reference can be copy/pasted)

• start tomcat Enter the /opt/apache-tomcat-7.0.64/bin/ directory ./startup.sh

• tomcat will be running on port 8080.

• Try accessing tomcat http://localhost:8080

4. Enable Manager Application Access in tomcat7 Manager application access can be enabled in tomcat7 by adding the following to conf/tomcat-users.xml

<tomcat-users>

<role rolename="manager" />

<role rolename="admin" />

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/CEP_GE_-_IBM_Proactive_Technology_Online_Installation_and_Administration_Guide


http://localhost:8080/



Draft


<role rolename="manager-gui" />

<role rolename="manager-status" />

<role rolename="manager-script" />

<role rolename="manager-jmx" />

<user username="{uname}" password{pswd} " roles="admin,manager,manager-gui,manager-status,manager-script,manager-jmx" />

</tomcat-users>

set appropriate {uname} and {pswd}

5. Restart tomcat

6. Download and Configure the Complex Event Processor components

• Download AuthoringTool.war, AuthoringToolWebServer.war, ProtonOnWebServerAdmin.war ProtonOnWebServer.war from

https://github.com/ishkin/Proton/tree/master/artifacts

• Put the war files under tomcat7/webapps. Once tomcat is running, the war files get automatically deployed and corresponding folders get created. If the folders are not created, restart tomcat and check again. As an alternative, use the manager webapp (GUI) on http://localhost:8080/manager to deploy each webapp manually

• Modify the following configurations in: ProtonOnWebServerAdmin/ProtonAdmin.properties definitions-repository={a directory_of_choice with read write permission for tomcat,eg: /home/projects/CEP/definitions }

manager-username={tomcat manager username configured in 2.4 } manager-password={tomcat manager password configured in 2.4 } tomcat-server-port={port} #8080 default tomcat-jmx-port={port} #8686 default Modify following configurations in ProtonOnWebServer/Proton.properties MetadataFileName = {path to the Proton definition file which contains the CEP rules,eg: /home/projects/CEP/definitions/sample.json} This file can be created using the AuthoringTool of CEP } metadataFilePathType = absolute inputPortNumber = {port} #3002 default outputPortNumber = {port} #3302 default

7. Run the Complex Event Processor

• export CATALINA_HOME=/tomcat-path/tomcat7

• export CATALINA_OPTS="-Dcom.sun.management.jmxremot -Dcom.sun.management.jmxremote.port=8686 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

• Restart Tomcat • Try to access the Complex Event Processor

http://localhost:8080/ProtonOnWebServerAdmin/resources/instances/ProtonOnWebServer Sample response:

{ state: "started", definitions-url: /ProtonOnWebServerAdmin/resources/definitions/sample"

https://github.com/ishkin/Proton/tree/master/artifacts

http://localhost:8080/

http://localhost:8080/ProtonOnWebServerAdmin/resources/instances/ProtonOnWebServer

http://localhost:8080/ProtonOnWebServerAdmin/resources/instances/ProtonOnWebServer



Draft


} The Complex Event Processor web interface for managing CEP definitions can be accessed at http://localhost:8080/AuthoringTool/

3.2.3 Configuring the Complex Event Processor

You can do this later after installation. Create a rule file (fiware.json) similar to the sample file attached and put it in definitions. Specify the access rights Provide all the attributes of event as key value pairs under events key Sample File { "epn": { "events": [ { "name": "NoiseReport", "attributes": [ { "dimension": 0, "name": "noise", "type": "Integer" } ] }, { "name": "NoiseIssue", "attributes": [ { "dimension": 0, "name": "Cost", "type": "Integer" }, { "dimension": 0, "name": "noise1", "type": "Integer" }, { "dimension": 0, "name": "noise2", "type": "Integer" } ] }, { "name": "PersonReport", "attributes": [ { "dimension": 0, "description": "The certainty that this event happen (value between 0 to 1)", "name": "Certainty", "defaultValue": "1", "type": "Double" }, { "dimension": 0, "description": "No value means it equals the event detection time, other option is to use one of the defined distribution functions with parameters", "name": "OccurrenceTime", "type": "Date" }, { "dimension": 0, "name": "ExpirationTime", "type": "Date" },

http://localhost:8080/AuthoringTool/



Draft


{ "dimension": 0, "description": "The cost of this event occurrence. Negative if this is an opportunity", "name": "Cost", "type": "Double" }, { "dimension": 0, "description": "Used in case the this event occur within an interval", "name": "Duration", "defaultValue": "0", "type": "Double" }, { "dimension": "0", "name": "entityId", "type": "String" }, { "dimension": "0", "name": "entityType", "type": "String" }, { "dimension": "0", "name": "personCount", "type": "Integer" }, { "dimension": "0", "name": "chaud", "type": "Boolean" } ], "createdDate": "Wed Feb 22 2017" }, { "name": "Out_PersonUpdate", "attributes": [ { "dimension": 0, "description": "The certainty that this event happen (value between 0 to 1)", "name": "Certainty", "defaultValue": "1", "type": "Double" }, { "dimension": 0, "description": "No value means it equals the event detection time, other option is to use one of the defined distribution functions with parameters", "name": "OccurrenceTime", "type": "Date" }, { "dimension": 0, "name": "ExpirationTime", "type": "Date" }, { "dimension": 0, "description": "The cost of this event occurrence. Negative if this is an opportunity", "name": "Cost", "type": "Double" }, { "dimension": 0, "description": "Used in case the this event occur within an interval", "name": "Duration", "defaultValue": "0",



Draft


"type": "Double" }, { "dimension": "0", "name": "entityId", "type": "String" }, { "dimension": "0", "name": "entityType", "type": "String" }, { "dimension": "0", "name": "personCount", "type": "Integer" }, { "dimension": "0", "name": "chaud", "type": "Boolean" } ], "createdDate": "Wed Feb 22 2017" } ], "name": "fiware", "consumers": [ { "events": [ { "name": "NoiseReport" } ], "name": "NoiseReportConsumer", "properties": [ { "name": "filename", "value": "\/usr\/share\/tomcat\/CEP\/sample\/Noise_NoiseReport.txt" }, { "name": "formatter", "value": "json" }, { "name": "delimiter", "value": ";" }, { "name": "tagDataSeparator", "value": "=" }, { "name": "SendingDelay", "value": "1000" } ], "type": "File" }, { "events": [ { "name": "NoiseIssue" } ], "name": "NoiseIssueConsumer", "properties": [ { "name": "URL", "value": "http:\/\/localhost:3000\/cep\/consume" },



Draft


{ "name": "contentType", "value": "application\/json" }, { "name": "formatter", "value": "json" }, { "name": "delimiter", "value": ";" }, { "name": "tagDataSeparator", "value": "=" }, { "name": "dateFormat", "value": "dd\/MM\/yyyy-HH:mm:ss" } ], "type": "Rest" }, { "events": [ { "name": "Out_PersonUpdate" } ], "name": "Out_PersonUpdateConsumer", "properties": [ { "name": "URL", "value": "http:\/\/localhost:3000\/cep\/consume" }, { "name": "contentType", "value": "application\/json" }, { "name": "formatter", "value": "json" }, { "name": "delimiter", "value": ";" }, { "name": "tagDataSeparator", "value": "=" }, { "name": "dateFormat", "value": "dd\/MM\/yyyy-HH:mm:ss" } ], "type": "Rest", "createdDate": "Wed Feb 22 2017" }, { "events": [ { "name": "PersonReport" } ], "name": "PersonReportConsumer", "properties": [ { "name": "filename", "value": "\/usr\/share\/tomcat\/CEP\/sample\/Person_PersonReport.txt" },



Draft


{ "name": "formatter", "value": "json" }, { "name": "delimiter", "value": ";" }, { "name": "tagDataSeparator", "value": "=" } ], "type": "File", "createdDate": "Wed Feb 22 2017" } ], "optimization": {}, "epas": [ { "inputEvents": [ { "instanceSelectionPolicy": "First", "alias": "NS1", "name": "NoiseReport", "consumptionPolicy": "Consume" }, { "instanceSelectionPolicy": "First", "alias": "NS2", "name": "NoiseReport", "consumptionPolicy": "Consume" } ], "derivedEvents": [ { "name": "NoiseIssue", "reportParticipants": false, "expressions": { "Cost": "NS2.noise-NS1.noise", "noise2": "NS2.noise", "noise1": "NS1.noise" } } ], "evaluationPolicy": "Immediate", "name": "IncreasingNoise", "internalSegmentation": [], "context": "3MinAfterStartUp", "epaType": "Sequence", "cardinalityPolicy": "unrestricted", "assertion": "NS2.noise >= NS1.noise + 10 OR NS2.noise <= NS1.noise - 10", "computedVariables": [] }, { "inputEvents": [ { "instanceSelectionPolicy": "First", "alias": "P1", "filterExpression": "P1.personCount>=65", "name": "PersonReport", "consumptionPolicy": "Consume" } ], "derivedEvents": [ { "name": "Out_PersonUpdate", "reportParticipants": false, "expressions": { "Cost": "PersonReport.Cost", "Certainty": "PersonReport.Certainty",



Draft


"personCount": "PersonReport.personCount", "OccurrenceTime": "PersonReport.OccurrenceTime", "Duration": "PersonReport.Duration", "entityId": "PersonReport.entityId", "ExpirationTime": "PersonReport.ExpirationTime", "chaud": "true", "entityType": "PersonReport.entityType" } } ], "evaluationPolicy": "Immediate", "name": "EPA_Change", "internalSegmentation": [], "context": "always", "epaType": "Basic", "cardinalityPolicy": "Single", "localPlacement": true, "createdDate": "Wed Feb 22 2017", "computedVariables": [] } ], "contexts": { "temporal": [ { "terminators": [ { "terminationType": "Terminate", "relativeTime": "25000", "terminatorType": "RelativeTime" } ], "neverEnding": false, "atStartup": false, "name": "3MinAfterStartUp", "type": "TemporalInterval", "createdDate": "Thu Jan 14 2016", "initiators": [ { "initiatorPolicy": "Add", "name": "NoiseReport", "initiatorType": "Event" } ] }, { "terminators": [], "neverEnding": true, "atStartup": true, "name": "always", "type": "TemporalInterval", "createdDate": "Wed Feb 22 2017", "initiators": [ { "initiatorPolicy": "Add", "name": "PersonReport", "initiatorType": "Event" } ] } ], "segmentation": [], "composite": [] }, "producers": [] } }

3.2.4 Testing the Complex Event Processor



Draft


End-to-End Testing To verify that the Complex Event Processor is running:

• Access the Apache Tomcat administrator tool via http://{host}:{port}/manager

• Log in with the user and password you configured in the setup Apache Tomcat for management

• Identify that all applications are installed and running (AuthoringTool, AuthoringToolWebServer, ProtonOnWebServerAdmin, and all Proton engine instances, e.g., ProtonOnWebServer).

• Access the authoring tool via http://{host}:{port}/AuthoringTool/Main.html (tested on Google Chrome).

• To test the engine on a sample definition project, run unit test 1.

• List of Running Processes o Apache Tomcat server running as a Java process.

• Network Interfaces Up and Open o Apache Tomcat server and JMX ports are in use (default 8080 and

8686 respectively) o Each Proton engine instance uses two ports, one for input and one for

output adapters, as configured in configuring an engine Instance. The single instance called ProtonOnWebServer provided with the base installation has the following ports as default: 3002, 3302.

• Databases o No database used in this release.

• Diagnostic Procedures o Proactive Technology Online uses Apache Tomcat logging. The log

files are located in the Apache Tomcat directory ./logs.

Unit Testing There are 10 Unit tests available. You can skip this for installation (only for testing oor learning purposes) These Unit Tests are used to check the correct functionality of the modules that implement the features of the Complex Event Processor instance. They are executed on the Complex Event Processor instance called the IBM Proactive Technology Online (Proton). These tests need an installed instance the engine and the authoring tool as described in the CEP GE - IBM Proactive Technology Online Installation and Administration Guide. Unit Test 1 This Unit Test is used to test that the IBM Proactive Technology Online engine processes incoming events according to defined patterns with defined context and correctly detect patterns based on the incoming events. The JSON definition file that is used as input to the engine includes event patterns with temporal context. The test also checks the adapter framework since the input events are read using an input file adapter and the derived events are stored in a file using an output file adapter.

Save the complete sample directory included in the install package to a sample directory on your machine.

Open the DoSAttack.json file from the sample directory. It contains 1 path for the input file DoSAttackScenarioJSON.txt and two paths to the output files DoSAttack_TrafficReport.txt, DoSAttack_PredictedCrash.txt. Change those paths to point to files with those names in your sample directory

Configure the engine to use the DoSAttack.json definition. One option to do so is to open the Proton.properties file located under the Apache Tomcat directory in ./webapps/ProtonOnWebServer and set the metadataFileName to the path of the DoSAttack.json file. Make sure you updated metadataFilePathType to absolute.






Draft


Activate the IBM Proactive Technology Online engine by starting the Apache Tomcat server. It will use the DoSAttack.json definition file with the input events from the DoSAttackScenarioJSON.txt scenario file

The input events in the input file are:

{"Name":"TrafficReport", "volume":"1000"} {"Name":"TrafficReport", "volume":"1600"} {"Name":"TrafficReport", "volume":"2500"}

According to the definition file the engine will identify patterns with context and conditions. It will read input events using the input file adapter and report detected patterns using the output file adapter (the selection of the adapters is defined in the definition file).

Expected Results : In ./sample directory you will see 2 output files:

1. DoSAttack_TrafficReport.txt -> a file representing a consumer for the TrafficReport events, the raw events.

{"DetectionTime":"21\/01\/2014-16:04:28","Duration":"0.0","EventId":"036bca79-36fe-47b3-9dbe-400e437acf0e", "EventSource":"","Certainty":"0.0","volume":"1000","Name":"TrafficReport","Cost":"0.0","Annotation":""} {"DetectionTime":"21\/01\/2014-16:04:33","Duration":"0.0","EventId":"66c3c961-69a8-45b6-98f5-3d53e14c61f3", "EventSource":"","Certainty":"0.0","volume":"1600","Name":"TrafficReport","Cost":"0.0","Annotation":""} {"DetectionTime":"21\/01\/2014-16:04:38","Duration":"0.0","EventId":"41a339cc-bc18-4a42-8061-6652cd2881f1", "EventSource":"","Certainty":"0.0","volume":"2500","Name":"TrafficReport","Cost":"0.0","Annotation":""}

2. DoSAttack_PredictedCrash.txt --> a file representing a consumer for the ExpectedCrash derived events that are generated during the test execution

{"DetectionTime":"21\/01\/2014-16:04:38","Duration":"0.0","EventId":"28adc76b-32e1-41e0-9eb8-93a35c84aad7", "EventSource":"","Certainty":"0.0","Name":"ExpectedCrash","Cost":"10.0","OccurrenceTime":"21\/01\/2014-16:04:38","Annotation":""}

Unit Test 2 This Unit Test is used for testing that the IBM Proactive Technology Online authoring tool is functioning correctly. It will check that the authoring tool is running, that a project definition can be read, verified, saved and exported. After the authoring tool applications are uploaded and are running on the Apache Tomcat server (see Running the authoring tool in the CEP installation and administration guide), open a browser (tested on Google Chrome) and point to the following link by completing the host and port values. http://{host}:{port}/AuthoringTool/Main.html

Unit Test 3 This Unit Test is used for testing that the IBM Proactive Technology Online authoring tool event editor is functioning correctly. It will check that an event editor can be opened and new attributes can be added to an event definition. After the authoring tool applications are uploaded and running on the Apache Tomcat server (see Running the authoring tool in the CEP installation and administration guide), open a browser (tested on Google Chrome) and point to the following link by setting the server IP address http://{host}:8080/AuthoringTool/Main.html. Click on Open Project button and select the DoSAttackproject

Unit Test 4 This Unit Test is used for testing that the Proactive Technology Online authoring tool EPA (event processing agent) editor and context editor are function correctly. It will check



Draft


that an EPA editor and a temporal context editor can be open and a different context can be set for the EPA definition. After the authoring tool applications are uploaded and running on the Apache Tomcat server (see Running the authoring tool in the CEP installation and administration guide), open a browser (tested on Google Chrome) and point to the following link by setting the server IP address http://{host}:8080/AuthoringTool/Main.html. Click on Open Project button and select the DoSAttack project

Unit Test 5 This Unit Test is used for testing that the Proactive Technology Online authoring tool Producer editor and Consumer editor are function correctly. It will check that an Producer editor and a Consumer editor can be open and a new consumer definition can be added. After the authoring tool applications are uploaded and running on the Apache Tomcat server (see Running the authoring tool in the CEP installation and administarion guide), open a browser (tested on Google Chrome) and point to the following link by setting the server IP address http://{host}:8080/AuthoringTool/Main.html. Click on Open Project button and select the DoSAttack project

Unit Test 6

Note: This feature is no longer supported in the latest versions.

This Unit Test is used for testing that the Proactive Technology Online authoring tool Event Processing Network (EPN) view. It will check that an EPN view can be open and it describes the actual EPN defined. After the authoring tool applications are uploaded and running on the Apache Tomcat server (see Running the authoring tool in the CEP installation and administration guide), open a browser (tested on Google Chrome) and point to the following link by setting the server IP address http://{host}:8080/AuthoringTool/Main.html. Click on Open Project button and select the DoSAttack project

Unit Test 7 This Unit Test is used for testing the IBM Proactive Technology Online multi-project and administration capabilities. It will check that a new instance can be brought up and configured to run with an Event Processing Network definition using the Admin REST API. After the configuration, the Admin REST API is tested for stopping and starting a particular instance.

Unit Test 8 This Unit Test is used for testing the push of events feature through a REST API to IBM Proactive Technology Online. It will check that an event can be pushed to the CEP engine and that the engine identifies the event and processes it for pattern detection.

Unit Test 9 This Unit Test is used for testing that the IBM Proactive Technology Online can process a message in the PubSub Broker GEa format as an input event, and that the IBM Proactive Technology Online can generate output event in the format expected by the PubSub Broker GE. Both the input message and the output message are in NGSI/XML format as used by the PubSub Broker GE. Unit Test 10 This Unit Test is used for testing that the IBM Proactive Technology Online can process the expressions added to improve expressability. In particular the expressions that can be used in an EPA expression https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/CEP_GE_-_IBM_Proactive_Technology_Online_Unit_Testing_Plan

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/CEP_GE_-_IBM_Proactive_Technology_Online_Unit_Testing_Plan

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/CEP_GE_-_IBM_Proactive_Technology_Online_Unit_Testing_Plan



Draft


3.3 CKAN 3.3.1 What is CKAN?

CKAN is a tool for open data websites. It helps manage and publish collections of data and is used by national and local governments, research institutions, and other organizations. Once data is published, users can use search features to browse and find the data needed, and preview it using maps, graphs and tables. Note! Ubuntu 12.04 has been used for the CKAN setup. This application collects data from CKAN datasets using a node module called ckan. Install using

npm install ckan --save

3.3.2 Installing CKAN

1. Install the CKAN package

CentOS: https://github.com/ckan/ckan/wiki/How-to-install-CKAN-2.0-on-CentOS-6.3-(new) pip install –upgrade pip On an Ubuntu 14.04 or 12.04 system, open a terminal and run these commands to install CKAN:

• Update Ubuntu’s package index: sudo apt-get update

• Install the Ubuntu packages that CKAN requires (and ‘git’, to enable you to install CKAN extensions): sudo apt-get install -y nginx apache2 libapache2-mod-wsgi libpq5 redis-server git-core

• Download the CKAN package: On Ubuntu 14.04: wget http://packaging.ckan.org/python-ckan_2.6-trusty_amd64.deb On Ubuntu 12.04: wget http://packaging.ckan.org/python-ckan_2.6-precise_amd64.deb Note: If wget is not present, install it via: sudo apt-get install wget

• Install the CKAN package: On Ubuntu 14.04: sudo dpkg -i python-ckan_2.6-trusty_amd64.deb On Ubuntu 12.04: sudo dpkg -i python-ckan_2.6-precise_amd64.deb Note: If the Apache WSGI module is not enabled the following error will be issued: Syntax error on line 1 of /etc/apache2/sites-enabled/ckan_default: Invalid command 'WSGISocketPrefix', perhaps misspelled or defined by a module not included in the server configuration Action 'configtest' failed. The Apache error log may have more information. ...fail! Enable the Apache WSGI module with these commands: sudo a2enmod wsgi sudo service apache2 restart

2. Install and configure PostgreSQL

https://github.com/ckan/ckan/wiki/How-to-install-CKAN-2.0-on-CentOS-6.3-(new)


http://packaging.ckan.org/python-ckan_2.6-trusty_amd64.deb

http://packaging.ckan.org/python-ckan_2.6-precise_amd64.deb



Draft


Tip: To install PostgreSQL and CKAN on different servers change the sqlalchemy.url setting in /etc/ckan/default/production.ini file to reference the PostgreSQL server.

• Install PostgreSQL with this command: sudo apt-get install -y postgresql orphan:

• Postgresql installation and initialization

• https://github.com/ckan/ckan/wiki/How-to-install-CKAN-2.0-on-CentOS-6.3-(new)

easy_install python-pip login to environment pip install PasteScript then paster

•

•

• Check that PostgreSQL was installed correctly by listing the existing databases: sudo -u postgres psql –l

• Check that the encoding of databases is UTF8, if not internationalisation may be a problem. Since changing the encoding of PostgreSQL may require deleting existing databases, it is suggested that this is fixed before continuing with the CKAN install.

• Next you’ll need to create a database user if one does not exist. Create a new PostgreSQL database user called ckan_default, and enter a password for the user when prompted. This password will be needed later: sudo -u postgres createuser -S -D -R -P ckan_default

• Create a new PostgreSQL database, called ckan_default, owned by the database user just created: sudo -u postgres createdb -O ckan_default ckan_default -E utf-8 Note: If PostgreSQL is run on a separate server, edit postgresql.conf and pg_hba.conf. For PostgreSQL 9.1 on Ubuntu, these files are located in etc/postgresql/9.1/main. o Uncomment the listen_addresses parameter and specify a comma-

separated list of IP addresses of the network interfaces PostgreSQL should listen on or ‘*’ to listen on all interfaces. For example,isten_addresses = 'localhost,192.168.1.21'

o Add a line similar to the line below to the bottom of pg_hba.conf to allow the machine running Apache to connect to PostgreSQL. Change the IP address as desired according to your network settings. host all all 192.168.1.22/32 md5

• Edit the sqlalchemy.url option in your CKAN configuration file (/etc/ckan/default/production.ini) file and set the correct password, database and database user.

3. Install and configure Solr

Tip: Solr and CKAN can be installed on different servers. Change the solr_url setting in /etc/ckan/default/production.ini file to reference the Solr server. Centos adduser solr

wget https://archive.apache.org/dist/lucene/solr/5.4.0/solr-5.4.0.zip

unzip solr-5.4.0.zip mv solr-5.4.0 /opt/solr

• Install Solr with this command:



https://archive.apache.org/dist/lucene/solr/5.4.0/solr-5.4.0.zip



Draft


sudo apt-get install -y solr-jetty The install will start then towards the end display: * Not starting jetty - edit /etc/default/jetty and change NO_START to be 0 (or comment it out). CKAN uses Solr as search platform and uses a customized Solr schema file that takes into account CKAN’s specific search needs. Now that we have CKAN installed, we need to install and configure Solr. Note: These instructions explain how to deploy Solr using the Jetty web server, but CKAN does notrequire Jetty - you can deploy Solr to another web server, such as Tomcat if that is convenient

• Edit the Jetty configuration file (/etc/default/jetty8 or /etc/default/jetty) and change the following variables: NO_START=0 JETTY_HOST=127.0.0.1 JETTY_PORT=8983 Note: This JETTY_HOST setting will only allow connections from the same machine. If CKAN is not installed on the same machine as Jetty/Solr you will need to change it to the relevant host or to 0.0.0.0 (and set up your firewall accordingly).

• Start or restart the Jetty server. For Ubuntu 16.04: sudo service jetty8 restart Or for Ubuntu 14.04: sudo service jetty restart Note: Ignore any warning that it Jetty was not running - some Ubuntu distributions choose not to start Jetty on install A welcome page from Solr is displayed on http://localhost:8983/solr/ (replace localhost with your server address if needed) Note If the message is displayed “Could not start Jetty servlet engine because no Java Development Kit (JDK) was found”, then edit the JAVA_HOME setting in /etc/default/jetty to point to the JDK install location. For example: JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.151.x86_64/jre or: JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.151.x86_64/jre

• Replace the default schema.xml file with a symlink to the CKAN schema file included in the sources. sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.xml.bak sudo ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml /etc/solr/conf/schema.xml

• Now restart Solr: For Ubuntu 16.04: sudo service jetty8 restart or for Ubuntu 14.04: sudo service jetty restart

• Check that Solr is running by opening http://localhost:8983/solr/

• Finally, change the solr_url setting in the CKAN configuration file (/etc/ckan/default/production.ini) to point to the Solr server, for example: solr_url=http://127.0.0.1:8983/solr

4. Update the configuration and initialize the database

• Edit the CKAN configuration file (/etc/ckan/default/production.ini) to set up the

following options:

• site_id: Each CKAN site should have a unique site_id, for example: ckan.site_id = default

http://localhost:8983/solr/

http://localhost:8983/solr/



Draft


site_url: Provide the site’s URL. For example: ckan.site_url = http://demo.ckan.org

• Initialize the CKAN database by with this command: sudo ckan db init

• Optionally, setup the DataStore and DataPusher by following the instructions in DataStore extension

• Also optional, enable file uploads by following the instructions in FileStore and file uploads.

5. Restart Apache and Nginx Restart Apache and Nginx by running this command in a terminal:

sudo service apache2 restart sudo service nginx restart

6. Installation done Open http://localhost in a web browser.

Functions defined in the system to call CKAN datasets ckanPublic This fetches dataset for Air pollution data stored in CKAN databases. A resource_id is required to fetch the data.

3.3.3 Testing CKAN For CKAN developing a CKAN extension or to install CKAN from source, ensure that CKAN’s tests pass This section explains how to run CKAN’s tests. CKAN’s testsuite contains automated tests for both the back-end (Python) and the front-end (JavaScript). In addition, the correct functionality of the complete front-end (HTML, CSS, JavaScript) on all supported browsers should be tested manually.

Back-end tests Most of CKAN’s test suite is for the backend Python code.

Install additional dependencies

Some additional dependencies are needed to run the tests. Ensure you have created a

config file at /etc/ckan/default/development.ini, then activate the virtual environment:

/usr/lib/ckan/default/bin/activate

Install nose and test-specific CKAN dependencies into the virtual environment:

Changed in version 2.1: In CKAN 2.0 and earlier the requirements file was called pip-

requirements-test.txt, not dev-requirements.txt as below:

pip install -r /usr/lib/ckan/default/src/ckan/dev-requirements.txt

Set up the test databases

Changed in version 2.1: Previously PostgreSQL tests used the databases defined in yourdevelopment.ini file, instead of using their own test databases.

Create test databases:

sudo -u postgres createdb -O ckan_default ckan_test -E utf-8

sudo -u postgres createdb -O ckan_default datastore_test -E utf-8

paster datastore set-permissions -c test-core.ini | sudo -u postgres psql

This database connection is specified in the test-core.ini file by the sqlalchemy.url parameter. Ensure that the Redis database configured in test-core.ini is different from the production database.

Configure Solr Multi-core

http://demo.ckan.org/



Draft


The tests assume that Solr is configured ‘multi-core’, whereas the default Solr set-up is often ‘single-core’. You can check how many cores Solr has been configured for:

curl -s 'http://127.0.0.1:8983/solr/admin/cores?action=STATUS' |python -c 'import sys;import xml.dom.minidom;s=sys.stdin.read();print xml.dom.minidom.parseString(s).toprettyxml()'

You can also tell from the ckan config: grep solr_url /etc/ckan/default/production.ini

# single-core: solr_url = http://127.0.0.1:8983/solr

# multi-core: solr_url = http://127.0.0.1:8983/solr/ckan

To enable multi-core:

Find the instanceDir of the existing Solr core. It is found in the output of the curl command above.

e.g. /usr/share/solr/or/opt/solr/example/solr/collection1

Make a copy of that core’s directory e.g.:

sudo cp -r /usr/share/solr/ /etc/solr/ckan

Configure Solr with the new core:

curl 'http://localhost:8983/solr/admin/cores?action=CREATE&name=ckan&instance

Dir=/etc/solr/ckan'

If successful, the status should be 0 - some XML containing: <int name="status">0</int>

Edit the main ckan config file (e.g. /etc/ckan/default/development.ini) and adjust the solr_url to match:

solr_url = http://127.0.0.1:8983/solr/ckan Run the tests

To run CKAN’s tests using PostgreSQL as the database, specify the —with-pylons=test-core.ini option on the command line. This command will run the tests for CKAN core and for the core extensions:

nosetests --ckan --with-pylons=test-core.ini ckan ckanext

The speed of the PostgreSQL tests can be improved by running PostgreSQL in memory and turning off durability, as described in the PostgreSQL documentation

By default the tests will keep the database open between test runs. If you wish to drop and reinitialize the database before the run you can use the reset-db option:

nosetests –ckan –reset-db –with-pylons=test-core.ini ckan

Migration testing

For a CKAN developer or extension developer and new code requires a change to CKAN’s model, you will need to prepare a migration script. To ensure that the migration

script itself gets tested, run the tests with the --ckan-migration option, for example:

nosetests --ckan --ckan-migration --with-pylons=test-core.ini ckan ckanext

By default tests are run using the model defined in ckan/model. With the –ckan-migration option the tests will run using a database that has been created by running the migration scripts in ckan/migration, which is how the database is created and upgraded in production.

Warning

A common error when running tests against a particular database is to change sqlalchemy.url in test.ini or test-core.ini. The problem is that these are versioned files and others may have checked in these by mistake, creating problems for other developers.

Common error messages

http://127.0.0.1:8983/solr/ckan

http://127.0.0.1:8983/solr/ckan



Draft


ConfigError

nose.config.ConfigError: Error reading config file 'setup.cfg': no such option 'with-pylons'

This error could be when running nosetests for two reasons:

1. Pylons nose plugin failed to run. If this is the case, then within a couple of lines of running nosetests you’ll see this warning: Unable to load plugin pylons followed by an error message

Fix the error here first

2. The Python module ‘Pylons’ is not installed into you Python environment. Confirm this with:

python -c "import pylons"

OperationalError

OperationalError: (OperationalError) no such function: plainto_tsquery ...

This error usually results from running a test which involves search functionality, which requires using a PostgreSQL database, but another (such as SQLite) is configured. The particular test is either missing a @search_related declaration or there is a mixup with

the test configuration files leading to the wrong database being used.

Nosetests

nosetests: error: no such option: --ckan

Nose is either unable to find ckan/ckan_nose_plugin.py in the python environment it is running in, or there is an error loading it. If there is an error, this will surface it:

nosetests --version

There are a few things to remedy this:

Commonly this is because the nosetests is not running in the python environment. Nose

should be installed in the python environment. To check which is running:

which nosetests

If you have activated the environment and this still reports /usr/bin/nosetests then you need to:

pip install --ignore-installed nose

If nose –version still fails, ensure that CKAN is installed in your environment:

cd /usr/lib/ckan/default/src/ckan

python setup.py develop

One final check - the version of nose should be at least 1.0. Check with:

pip freeze | grep -i nose

SolrError

SolrError: Solr responded with an error (HTTP 404): [Reason: None] <html><head><meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type" /><title>Error 404 NOT_FOUND</title></head><body><h2>HTTP ERROR 404</h2><p>Problem accessing /solr/ckan/select/. Reason:<pre> NOT_FOUND</pre></p><hr

/><i><small>Powered by Jetty://</small></i>

This means solr_url is not corresponding with SOLR. When running tests, it is usually easiest to change the set-up to match the default solr_url in test-core.ini. Often this means switching to multi-core – see Configure Solr Multi-core.

Front-end tests

Front-end testing consists of both automated tests (for the JavaScript code) and manual

tests (for the complete front-end consisting of HTML, CSS and JavaScript).



Draft


Automated JavaScript tests

The JS tests are written using the Mocha test framework and run via PhantomJS. First you need to install the necessary packages:

sudo apt-get install npm nodejs-legacy

sudo npm install -g [email protected] phantomjs@~1.9.1

To run the tests, make sure that a test server is running:

. /usr/lib/ckan/default/bin/activate

paster serve test-core.ini

Once the test server is running switch to another terminal and execute the tests:

mocha-phantomjs http://localhost:5000/base/test/index.html

Manual tests

All new CKAN features should be coded so that they work in the following browsers:

Internet Explorer: 11, 10, 9 & 8

Firefox: Latest + previous version

Chrome: Latest + previous version

These browsers are determined by whatever has >= 1% share with the latest months

data from: http://data.gov.uk/data/site-usage

http://data.gov.uk/data/site-usage



Draft


3.4 CYGNUS

3.4.1 What is CYGNUS? The Cygnus Generic Element is a connector in charge of persisting certain sources of

data in configured third-party storages to create a historical view of such data.

Internally, Cygnus is based on Apache Flume, a technology addressing the design and execution of data collection and persistence agents. An agent is basically composed of a listener or source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.

Cygnus is designed to run a specific Flume agent per source of data.

Current stable release is able to persist the following sources of data in the following third-party storages:

• NGSI-like context data

• HDFS, the Hadoop distributed file system

• MySQL, the well-known relational database manager

• CKAN, an Open Data platform

• MongoDB, the NoSQL document-oriented database

• STH Comet, a Short-Term Historic database built on top of MongoDB

• Kafka, the publish-subscribe messaging broker

• DynamoDB, a cloud-based NoSQL database by Amazon Web Services

• PostgreSQL, the well-known relational database manager

• Carto, the database specialized in geolocated data

• Twitter data

• HDFS, the Hadoop distributed file system.

Cygnus place in FIWARE architecture

Cygnus (more specifically, cygnus-ngsi agent) plays the role of a connector between Orion Context Broker (which is a NGSI source of data) and many FIWARE storages such as CKAN, Cosmos Big Data (Hadoop) and STH Comet. Other non-FIWARE storages such as MySQL, Kafka, Carto, etc can be used too.

3.4.2 Installing CYGNUS

1. CYGNUS NGSI COMMON SELF deplying:

cat > /etc/yum.repos.d/fiware.repo <<EOL [Fiware] name=FIWARE repository baseurl=http://repositories.lab.fiware.org/repo/rpm/x86_64/ gpgcheck=0 enabled=1 EOL And do this; https://flume.apache.org/FlumeUserGuide.html#setup

Installation from Source

• Prerequisites:

o yum install java-1.7.0-openjdk-devel

o export JAVA_HOME=/usr/lib/jvm/{your java version}

o wget http://www.eu.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz

o tar xzvf apache-maven-3.2.5-bin.tar.gz

o export APACHE_MAVEN_HOME=/opt/maven

https://flume.apache.org/FlumeUserGuide.html#setup



Draft


o mv apache-maven-3.2.5 /opt/maven

• CYGNUS user creation:

o (sudo) useradd Cygnus

o (sudo) passwd Cygnus

o su – cygnus

• log4j path: o mkdir -p /var/log/Cygnus

o chown -R Cygnus:Cygnus /var/log/Cygnus

• Installing Apache Flume:

o wget http://www.eu.apache.org/dist/flume/1.4.0/apache-flume-1.4.0-bin.tar.gz

o tar xvzf apache-flume-1.4.0-bin.tar.gz

o export APACHE_FLUME_HOME=/opt/flume

o mv apache-flume-1.4.0-bin APACHE_FLUME_HOME

o disable httpclient, httpcore and libthrift because they are duplicates:

o mv APACHE_FLUME_HOME/lib/httpclient-4.2.1.jar

APACHE_FLUME_HOME/lib/httpclient-4.2.1.jar.old

o mv APACHE_FLUME_HOME/lib/httpcore-4.2.1.jar APACHE_FLUME_HOME/lib/httpcore-4.2.1.jar.old

o mv APACHE_FLUME_HOME/lib/libthrift-0.7.0.jar APACHE_FLUME_HOME/lib/libthrift-0.7.0.jar.old

o mkdir -p APACHE_FLUME_HOME/plugins.d/Cygnus/

o mkdir APACHE_FLUME_HOME/plugins.d/Cygnus/lib

o mkdir APACHE_FLUME_HOME/plugins.d/Cygnus/libext

• Installing cygnus-common

• Cloning FIWARE Cygnus

o Install git if needed by using yum install git

o git clone https://github.com/telefonicaid/fiware-cygnus.git

o cd fiware-cygnus

• Installing cygnus-common

o cd cygnus-common

o APACHE_MAVEN_HOME/bin/mvn clean compile exec:exec assembly:single

o cp target/cygnus-common-<x.y.z>-jar-with-dependencies.jar

APACHE_FLUME_HOME/plugins.d/cygnus/libext

o cd cygnus-common

• APACHE_MAVEN_HOME/bin/mvn exec:exec package

o cp target/cygnus-<x.y.z>.jar

APACHE_FLUME_HOME/plugins.d/cygnus/libext

o mvn install:install-file -Dfile=APACHE_MAVEN_HOME/plugins.d/cygnus/libext/cygnus-common-<x.y.z>-jar-with-dependencies.jar -DgroupId=com.telefonica.iot -DartifactId=cygnus-common -Dversion=<x.y.z> -Dpackaging=jar -DgeneratePom=false

• Installing cygnus-flume-ng script

o cp target/classes/cygnus-flume-ng APACHE_FLUME_HOME/bin

o chmod a+x APACHE_FLUME_HOME/bin/cygnus-flume-ng

• Known issues

o export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=128m"

2. CYGNUS NGSI AGENT Installation From Source

• Prerequisites: cygnus-common must be installed. This includes Maven, cygnus user creation, log4j path creation, Apache Flume and cygnus-flume-ng script installation.

• Cloning fiware-cygnus: o git clone https://github.com/telefonicaid/fiware-cygnus.git o cd fiware-cygnus

https://github.com/telefonicaid/fiware-cygnus.git



Draft


o git checkout <branch>

• Installing cygnus-ngsi o cd cygnus-ngsi o APACHE_MAVEN_HOME/bin/mvn clean compile exec:exec

assembly:single o cp target/cygnus-ngsi-<x.y.z>-jar-with-dependencies.jar

APACHE_FLUME_HOME/plugins.d/cygnus/lib

• Known issues: Sometimes while compiling cygnus-ngsi the Maven JVM has insufficient memory.

3.4.3 Configuring CYGNUS

Installation and configuring cygnus-common:

https://github.com/telefonicaid/fiware-cygnus/tree/master/cygnus-common

installation and configuring cygnus-ngsi:

https://github.com/telefonicaid/fiware-cygnus/tree/master/cygnus-ngsi

1. CYGNUS NGSI COMMON

• Cygnus agent config cygnus_instance_<id>.conf Locate the file inside the clone of fiware-cygnus. For example in current instance /root/fiware-cygnus/cygnus-common/conf/cynus_instance_1.conf # The user that will be running Cygnus. Note this must be `root` if you want to run cygnus in a privileged # port (<1024), either the admin port or the port in which Cygnus receives Orion notifications

CYGNUS_USER=cygnus # What is the config folder CONFIG_FOLDER=/usr/cygnus/conf # What is the config file CONFIG_FILE=/usr/cygnus/conf/agent_<id>.conf # Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters # naming conventions, e.g. it appears in <AGENT_NAME>.sources.http-source.channels=... AGENT_NAME=cygnus-common # Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log # rotation works properly LOGFILE_NAME=cygnus.log # Administration port. Must be unique per instance ADMIN_PORT=8081 # Polling interval (seconds) for the configuration reloading POLLING_INTERVAL=30

b) agent_<id>.conf

Locate this file here:

/root/fiware-cygnus/cygnus-common/conf/agent_1.conf

The file agent_<id>.conf can be instantiated from a template given in the Cygnus repository

conf/agent.conf.template

While no specific Cygnus agent is used, this template is just the Apache Flume template.

https://github.com/telefonicaid/fiware-cygnus/tree/master/cygnus-common

https://github.com/telefonicaid/fiware-cygnus/tree/master/cygnus-ngsi



Draft


2. Flume env config:

The file flume-env.sh can be instantiated from a template given in the Cygnus repository

conf/flume- env.sh.template

To include in APACHE_FLUME_HOME/conf/flume-env.sh JAVA_HOME=/usr/lib/jvm/java-6-sun

Give Flume more memory and pre-allocate, enable remote monitoring via JMX

JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

Note that the Flume conf directory is always included in the classpath. FLUME_CLASSPATH="/path/to/the/flume/classpath"

flume-env.sh file has been inherited from Apache Flume, and it is used in order to configure certain Flume parameters such as an alternative classpath, some Java options etc.

3. log4j Configuration:

Define some default values. These can be overridden by system properties, e.g. the following logs in the standard output, which is very useful for testing purposes:

Dflume.root.logger=DEBUG,console

flume.root.logger=INFO,LOGFILE

#flume.root.logger=DEBUG,console

flume.log.dir=/var/log/cygnus/

.log.file=cygnus.log

# Logging levels for certain components. log4j.logger.org.apache.flume.lifecycle = INFO

log4j.logger.org.jboss = WARN

log4j.logger.org.mortbay = INFO

log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN

log4j.logger.org.apache.hadoop = INFO

# Define the root logger to the system property "flume.root.logger". log4j.rootLogger=${flume.root.logger}

# Stock log4j rolling file appender.

# Default log rotation configuration. log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender

log4j.appender.LOGFILE.MaxFileSize=100MB

log4j.appender.LOGFILE.MaxBackupIndex=10

log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}

log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout

log4j.appender.LOGFILE.layout.ConversionPattern=time=%d{yyyy-MM-dd}T%d{HH:mm:ss.SSSzzz} | lvl=%p | corr= %X{correlatorId} | trans=%X{transactionId} | svc=%X{service} | subsvc=%X{subservice} | comp=%X{agent} | op=%M | msg=%C[%L] : %m%n

Warning: If you enable the following appender it will fill up your disk if you do not have a cleanup job.

Cleanup job example: find /var/log/cygnus -type f -mtime +30 -exec rm -f {} \;



Draft


This uses the updated rolling file appender from log4j-extras that supports a reliable time-based rolling policy. See http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/rolling/TimeBasedRollingPolicy.html

Add "DAILY" to flume.root.logger above if you want to use this. log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender

log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy

log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}

log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}

log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout

log4j.appender.DAILY.layout.ConversionPattern=time=%d{yyyy-MM-dd}T%d{HH:mm:ss.SSSzzz} | lvl=%p | corr=%X{correlatorId} | trans=%X{transactionId} | svc=%X{service} | subsvc=%X{subservice} | comp=%X{agent} | op=%M | msg=%C[%L] : %m%n

# Console appender, i.e. printing logs in the standard output.

# Add "console" to flume.root.logger above if you want to use this. log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

log4j.appender.console.layout=org.apache.log4j.PatternLayout

log4j.appender.console.layout.ConversionPattern=time=%d{yyyy\]MM-dd}T%d{HH:mm:ss.SSSzzz} | lvl=%p | corr= %X{correlatorId} | trans=%X{transactionId} | svc=%X{service} | subsvc=%X{subservice} | comp=%X{agent} | op=%M | msg=%C[%L] : %m%n

http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/rolling/TimeBasedRollingPolicy.html

http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/rolling/TimeBasedRollingPolicy.html



Draft


CYGNUS NGSI AGENT

1. Cygnus configuration examples

+-----------------------------------+

| JVM |

| +-----+ +----+ +------+ |

| | src |-----| ch |-----| sink | |

| +-----+ +----+ +------+ |

+-----------------------------------+

2. Running cygnus-ngsi as a service sudo service cygnus status

sudo service cygnus start

sudo service cygnus restart

sudo service cygnus stop

sudo service cygnus status <id>

sudo service cygnus start <id>

sudo service cygnus restart <id>

sudo service cygnus stop <id>

3. Name Mappings :

Creating your own Name Mappings cat /path/to/conf/agent.conf

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource

cygnus-ngsi.sources.http-source.channels = hdfs-channel

cygnus-ngsi.sources.http-source ...

cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler

cygnus-ngsi.sources.http-source.handler ...

cygnus-ngsi.sources.http-source.interceptors = ts nmi

cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp

cygnus-ngsi.sources.http-source.interceptors.nmi.type = com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$Builder

cygnus-ngsi.sources.http-source.interceptors.nmi.name_mappings_conf_file = /path/to/conf/name_mappings.conf

3.4.4 Testing CYGNUS

Unit testing Running the tests require Apache Maven installed and Cygnus sources downloaded.

git clone https://github.com/telefonicaid/fiware-cygnus.git

cd fiware-cygnus/cygnus-ngsi

mvn test

End to End testing Cygnus can be tested form a e2e point of view by using any of the scripts emulating a NGSI-like notification. You can find both Json and XML examples of simple and compound notifications, with or without metadata, even model entities and loops of continuous notifiers.



Draft


For instance, if running the notification-json-simple.sh:

./notification-json-simple.sh http://localhost:5050/notify myservice myservicepath

* Trying ::1...

* Connected to localhost (::1) port 5050 (#0)

> POST /notify HTTP/1.1

> Host: localhost:5050

> Content-Type: application/json

> Accept: application/json

> User-Agent: Orion/0.10.0

> Fiware-Service: myservice

> Fiware-ServicePath: myservicepath

> Content-Length: 460

>

* upload completely sent off: 460 out of 460 bytes

< HTTP/1.1 200 OK

< Transfer-Encoding: chunked

< Server: Jetty(6.1.26)

* Connection #0 to host localhost left intact

The server (Cygnus) is sending back a 200 OK response.



Draft


3.5 IOT AGENT

3.5.1 What is the IOT Agent? The IOT Agent is a Node.js module to enable IoT Agent developers to build custom agents for their devices that can easily connect to NGSI Context Brokers (such as ORION). This is a component that lets groups of devices send their data to and be managed from a FIWARE NGSI Context Broker using their own native protocols. IoT Agents should also be able to deal with security aspects of the Fiware platform (authentication and authorization of the channel) and provide other common services to

the device programmer.

IMPORTANT

An IOT Agent is to manage several groups of devices. We have not used it as such because of several issues came across in configuration of MQTT and running it as standalone process/service in Centos. And MQTT broker was stopping unexpectedly at times and behaving strangely.

More on MQTT http://mosquitto.org/

Right now the project has IOTagent node module which can be used to register devices

and update context data.

Since it is not fully functional because of lack of MQTT agent, the workaround is simply streaming the sensor data from video and noise sensors and pushing directly to the Context Broker and CEP will analyse and return with alerts

Use package.json to add and update node modules

3.5.2 Installation of IoT Agent

The source is available at: https://github.com/telefonicaid/iotagent-node-lib

IoT agent is a node module which should be directly included in the backend.

Use npm install command for this

Configuring IoT Agent

var config = {

logLevel: 'DEBUG',

contextBroker: {

host: '192.168.56.101',

port: '1026'

},

server: {

port: 4041

},

deviceRegistry: {

type: 'memory'

},

types: {

},

service: 'tester',

subservice: '/test',

providerUrl: 'http://192.168.56.1:4041',

deviceRegistrationDuration: 'P1M',

defaultType: 'Thing'

};

module.exports = config;

http://mosquitto.org/

https://github.com/telefonicaid/iotagent-node-lib



Draft


Configurations are meant to simplifying the device provisioning for groups of similar devices. Considering

that different groups of devices may be created in the same subservice that may require different configurations, multiple configurations are allowed for each subservice. Considering the key in the association between Device and Configuration was the triplet (service, subservice, type), all of these elements are considered mandatory.

This statement does not hold true for older IoT Agents, though. In older versions of the IoT Agents, each device configuration was assigned to a particular subservice and just one configuration was allowed per subservice, so the relation between a Device and a Configuration did not need the type to discriminate between Configurations. That is why for those agents, type was not a mandatory parameter.

In order to allow backward-compatibility with those agents, the IoT Agent Library now implement a compatibility mode: the **Single Configuration Mode**, that makes the agent behave like the old agents. In this mode:

• Each Subservice can contain just one Configuration. If a second Configuration is created for a Subservice, an error is raised

• Each Device provisioned for a Subservice is automatically assigned to the Subservice one Configuration if there is any.

This compatibility has to be set for the whole IoT Agent, and there is no option of having both modes simultaneously running. Transitions from one mode to the other should be

made with care, and may involve data migration.

Registration

Whenever a device is registered, the IoT Agent reads the device's entity information from the request or, if that information is not in the request, from the default values for that type of device. Among this information, there should be the list of device attributes that will be considered lazy (or passive). With this information, the IoT Agent sends a new `registerContext` request to the Context Broker, registering itself as ContextProvider of all the lazy attributes for the device's entity. The `registrationId` is then stored along the other device information inside the IoT Agent device registry.

As NGSI9 does not allow the context registrations to be removed, when the device is removed from the IoT Agent, the registration is updated to an expiration date of 1s, so it is effectively disabled. Once it has been disabled, the device is removed from the IoT Agent's internal registry.

Lazy attributes

When a request for data from a lazy attribute arrives to the Context Broker, it forwards the request to the Context Provider of that entity, in this case the IoT Agent. The IoT Agent will in turn ask the device for the information needed, transform that information to a NSGI format and return it to the Context Broker. The latter will the forward the response to the caller, transparently.

Commands

Commands are modelled as updates over a lazy attribute. As in the case of the lazy attributes, updates over a command will be forwarded by the Context Broker to the IoT Agent, that will in turn interact with the device to perform the requested action. Parameters for the command will be passed inside the command value.

There are two differences with the lazy attributes:

* First of all, for every command defined in a device, two new attributes are created in the entity with the same name as the command but with a prefix:

• '_info': this attribute reflect the current execution status of the command. When a command request is issued by the Context Broker, the IoT Agent library generates this attribute with 'PENDING' value. The value of this attribute will be changed each time a command error or result is issued to the IoT Agent.



Draft


• '_result': this attribute reflect the result of the execution of the defined command.

* Commands can also be updated when new information about its execution arrives to the agent. This information will be mapped to the command's utility attributes `_info` and `_result` leaving alone the command attribute itself. The values for this attributes are stored locally in the Context Broker (instead of being redirected with the Context Provider operations).

There are two types of commands:

• Push commands

When a command of this type arrives to the IoTAgent, the IoTAgent will immediately forward the command request to the device, translating the request to the proper protocol (that will depend on the type of IoTAgent). The library implement this kind of commands by offering a set functions that can be used to set an IoTAgent-specific handler for incoming commands. In order for this type of commands to work properly, the devices must be preprovisioned with an endpoint of the proper protocol, where it can be accessed by the IoTAgent who pushes de commits.

• Poll commands

Polling commands are meant to be used on those cases where the device ca not be online the whole time waiting for commands. In this case, the IoTAgents must store the received commands, offering a way for the device to retrieve the pending commands upon connection. To enable this feature, the Library offers a set of functions to manage command storage, and a mechanism to automatically store incoming commands for those devices marked as 'polling

devices'.

The distinction between push and poll commands will be made based on the presence of a `polling` flag in the device provisioning data. The default option (with the flag with value `false` or not present) is to use push commands (as they were the only ones available until the latest versions).

Polling commands could be subjected to expiration: two configuration properties pollingExpiration` and `pollingDaemonFrequency` can be set to start a daemon that will remove expired commands from the DB if the device is taking too much to pick them up. See the configuration section for details.

The library does not deal with protocol transformation or South Bound communications for neither of the command types (that is the task for those specific IoTAgents using the library).

Active attributes

Whenever a device proactively sends a message to the IoT Agent, it should tranform its data to the appropriate NGSI format, and send it to the Context Broker as an `updateContext` request.

Features

These are the features an IoT Agent is supposed to expose (those not supported yet by this library are marked as PENDING):

Device registration

Multiple devices will be connected to each IoT Agent, each one of those mapped to a

CB entity. The IoT Agent will register itself as a Context Provider for each device,

answering to requests and updates on any lazy attribute of the device.

Device information update

Whenever a device haves new measures to publish, it should send the information to

the IoT Agent in its own native language. This message should , in turn, should be sent

as an `updateContext` request to

the Context Broker, were the measures will be updated in the device entity.



Draft


Device command execution and value updates

As a Context Provider, the IoT Agent should receive update operations

from the Context Broker subscriptions, and relay them to the corresponding device (decoding it using its ID and Type, and other possible metadata). This commands will arrive as `updateContext` operations redirected from the Context Broker to the IoT Agent

(Command execution PENDING; value updates available).

Device management

The IoT Agent should offer a device repository where the devices can be registered,

holding data

needed for the connection to the Context Broker as the following: service and subservice for the device, API Key the device will be using to connect to the IoT Agent, Trust token

the device will be using to retrieve the Keystone token to connect to the Context Broker.

Device provisioning

The IoT Agent should offer an external API to make a preprovision of any devices. This

preprovision should enable the user to customize the device`s entity name and type as well as their service information.

Type configuration

If a device is registered without a preregistration, only its `id` and `type` attributes are

mandatory. The IoT Agent should provide a mechanism to provide default values to the

device attributes based on its

type.

Almost all of these features are common for every agent, so they can be abstracted into

a library or external module.

The objective is to provide that abstraction. As all this common tasks are abstracted, the main task of

the concrete IoT Agent implementations will be to map between the native device protocol and the library API.

3.5.3 IOT Agent tester

Command line testing The library also offers a Context Broker and IoT Agent client that can be used to:

• Simulate operations to the Context Broker used by the IoT Agent, triggering Context Provider forwardings for lazy attributes and checking the appropriate

values for active ones.

• Simulate operations to the Device Provisioning API and Configuration API of the IoT Agent.

The tester can be started with the following command, from the root folder of the project:

bin/iotAgentTester.js

From the command line, the help command can be used to show a description of the

currently supported features. These are the following:

• stressInit Start recording a stress batch.

• stressCommit <delay> <times> <threads> <initTime> Executes the recorded batch as many times as requested, with delay (ms) between commands. The "threads" parameter indicates how many agents will repeat that same sequence. The "initTime" (ms) parameter indicates the mean of the random initial waiting times for each agent.

• exit Exit from the command line.

• update <entity> <type> <attributes>



Draft


Update the values of the defined set of attributes, using the following format: name#type=value(|name#type=value)*

• append <entity> <type> <attributes> Append a new Entity with the defined set of attributes, using the following format: name:type=value(,name:type=value)*

• query <entity> <type> Get all the information on the selected object.

• queryAttr <entity> <type> <attributes> Get information on the selected object for the selected attributes. discover <entity> <type> Get all the context providers for a entity and type.

• configCb <host> <port> <service> <subservice> Config a new host and port for the remote Context Broker.

• showConfigCb Show the current configuration of the client for the Context Broker.

• configIot <host> <port> <service> <subservice> Config a new host and port for the remote IoT Agent.

• showConfigIot Show the current configuration of the client for the IoT Agent.

• provision <filename> Provision a new device using the Device Provisioning API. The device configuration is read from the file specified in the "filename" parameter.

• provisionGroup <template> <data> <type> Provision a group of devices with the selected template, taking the information needed to fill the template from a CSV with two columns, DEVICE_ID and DEVICE_NAME. The third parameter, type will be used to replace the DEVICE_TYPE field in the template. All the devices will be provisioned to the same IoT Agent, once the templates have been fulfilled.

• listProvisioned List all the provisioned devices in an IoT Agent.

• removeProvisioned <deviceId> Remove the selected provisioned device from the IoT Agent, specified by its Device ID.

• addGroup <filename> Add a new device group to the specified IoT Agent through the Configuration API. The body is taken from the file specified in the "filename" parameter.

• listGroups List all the device groups created in the selected IoT Agent for the configured service

• removeGroup <apiKey> <resource> Remove the device group corresponding to the current configured subservice.

• authenticate <host> <port> <user> <password> <service> Authenticates to the given authentication server, and use the token in subsequent requests.

• setProtocol <protocol> Sets the protocol to use in the requests (http or https). Defaults to http.

• configMigration <host> <port> <originDb> Sets the configuration for a migration between a C++ IoTA and a Node.js one.

• showConfigMigration Shows the current migration configuration.

• addProtocols <protocols> Add a protocol translation table, in the following format: protocolOrigin1=protocolTarget1;protocolOrigin2=protocolTarget2...

• migrate <targetDb> <service> <subservice> Migrate all the devices and services for the selected service and subservice into the specified Mongo database. To perform the migration for all the services or all the subservices, use the "*" value.



Draft


The agent session stores transient configuration data about the target Context Broker and the target IoT Agent. This configuration is independent, and can be checked with the showConfigCb and showConfigIot commands, respectively. Their values can be changed with the configCb and configIot commands respectively. The new configurations will be

deleted upon startup.

Creating specialized testers The command line testing tools make use of the command-node Node.js library for command line utils. In order to help creating testing tools for IoTAgents of specific protocols, all the commands of the library tester are offered as a array that can be directly imported into other Command Line tools, using the following steps:

Require the ´iotagent-node-lib´ command line module in your command line tool:

var iotaCommands = require('iotagent-node-lib').commandLine;

Initialize the command line utils (the initialization function takes two arguments, that will be explained in detail below:

iotaCommands.init(configCb, configIot);

Add the IOTA Lib commands to your array of commands

commands = commands.concat(commands, iotaCommands.commands);

Execute the command line interpreter as usual:

clUtils.initialize(commandLine.commands, 'IoT Agent tester> ');

The command line module makes use of two configuration objects. Both can be shown and edited in the command line using the provided commands, but a default value must be present.

The Context Broker configuration object holds all the information about the Context Broker where the IoT Agent to be tested is connected. It MUST contain the following attributes:

host: host where the Context Broker instance is located.

port: port where the Context Broker instance is listening.

service: service that will be used in all the NGSI operations.

subservice: service that will be used in all the NGSI operations.

The IoT Agent configuration object holds information about the IoT Agent that is being tested. It MUST contain the following attributes:

host: host where the IoT Agent instance is located.

port: port where the IoT Agent instance is listening.

service: service that will be used to group devices and device information.

subservice: subservice that will be used to group devices and device

information.



Draft


4 Custom Modules

4.1 Smart City Application in NodeJS 4.1.1 Installation

1. Install node.js

sudo yum install nodejs npm

2. Setup SmartCity server Download SmartCity source code from https://bitbucket.org/fcsmartcity extract the code to a folder of choice. eg: SmartCity

cd SmartCity npm install export NODE_ENV=dev node app.js

3. SmartCity server will be running on port 3000 by default. To change the port

export SC_PORT= {port} node app.js

Access SmartCity dashboard at http://localhost:3000/



Draft


4.2 Web Interface 4.2.1 Dashboard

1. Connect to the Stratumseind website - change URL according to the setup 2. Log onto the website with username and password 3. This brings you to the home page with these icons in top left corner

1. Home



Draft


• This page includes the Google map of the area with sensor markers for light, noise, video, parking, and pollution on click of each provides the user with latest information.

• Data Widgets for Wifi, Breweries, Visitors, Parking, Law and Order, Weather, Waste etc.

2. Social Media

• Count of Incoming and Outgoing number of persons

• Latest Tweets and count the number of words in tweets based on certain keywords configured in configuration page

3. Data

• Latest counts for Bicycles and Persons with filters 4. Alerts

• Alerts from CEP is shown here 5. Configuration – options

• Input Keywords for Social Media data fetching

• Noise Analytics – Fetch the location of noise sensors

• Video Analytics – Fetch the location of video sensors

• Light Sensors – Fetch the location of light sensors

• Lighting Rules – Provide light color rules for lights located at various locations

• Parking Sensors – Fetch the location of parking sensors

• Widgets – Turn on and off various widgets 4.2.2 Munisense Sound Sensor Integration

1. Connect to the Stratumseind website 2. Log onto the website with username and password 3. When the noise sensors are fetched in configuration page it is reflected in

the map as markers 4. These markers on click will provide latest sensor data 5. In the background Noise fetched via the APIs given by Munisense

1. Find the devices that are accessible https://eindhoven2-geluid.munisense.net/webservices/v2/groups/38/soundmeasurementpoints?order_field=description&order_dir=asc&rowcount=100&fields=object_id%2Cdescription 2. Get the required values for a particular sound measure point https://eindhoven2-geluid.munisense.net/webservices/v2/soundmeasurementpoints/96?fields=object_id%2Cdescription%2Claeq&type=value 3. These data are taken every 2 minutes and stored in mongodb 4.2.3 ViNotion people counting Integration

https://eindhoven2-geluid.munisense.net/webservices/v2/groups/38/soundmeasurementpoints?order_field=description&order_dir=asc&rowcount=100&fields=object_id%2Cdescription




https://eindhoven2-geluid.munisense.net/webservices/v2/soundmeasurementpoints/96?fields=object_id%2Cdescription%2Claeq&type=value





Draft


1. When the video sensors are fetched in configuration page it is reflected in the map as markers

2. These markers on click will provide latest sensor data 3. In the background Noise fetched via the APIs given by ViSense

1. Start by getting token

http://<IP>/api/v1/authenticate Authentication API. Acquire and release client session tokens var xhr = new XMLHttpRequest(); xhr.open('POST', 'http://' + ip_address + '/api/v1/authenticate', false); xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded'); xhr.send( 'command=signIn'

+ '&user=' + user_name + '&password=' + password);

2. Parse the client token

var client_token = JSON.parse(xhr.responseText).data.clientToken; 3. Counting data is available from the counting-preset channels:

ws://<IP>/api/v1/data/main/count/preset0 ws://<IP>/api/v1/data/main/count/preset1 And the cumulative counting-preset channels: ws://<IP>/api/v1/data/main/count/cumulative/preset0 ws://<IP>/api/v1/data/main/count/cumulative/preset1

4. An example query setting the counting-preset channel data streaming time

range from 19:00h UTC to 20:00h UTC on the 24th of March 2016 would look like:

JSON {



Draft


"reference": "ExampleQuery", "command": { "set" : { "select" : { "Channel/Recorder/Storage/TimeStart": "2016-03-24T19:00:00.000", "Channel/Recorder/Storage/TimeStop": "2016-03-24T20:00:00.000" }, "filter" : "parameter" } } } Response { "reference": "ExampleQuery", "packet": { "id": "", "type": "RulePacket", "ts": "2016-04-07T21:47:36.953", "meta": 3, "payload": { "CountInterval": 30, "ThresholdNumberLocations": 13, "Direction": "North", "Label": "person" } } } 4.2.4 Twitter Feeds Integration

1. [ntwitter](http://github.com/AvianFlu/ntwitter) is a node module used for

twitter feeds. Install using npm install ntwitter

2. The keys listed below can be obtained from [dev.twitter.com](http://dev.twitter.com) after [setting up a new App] (https://dev.twitter.com/apps/new) javascript

var twitter = require('ntwitter');

var twit = new twitter({ consumer_key: 'Twitter', consumer_secret: 'API', access_token_key: 'keys', access_token_secret: 'go here' });

3. Stream data using

twit.stream('statuses/filter', {track: 'stratumseind, netherlands, eindhoven, smartcity, amsterdam, IOT'}, function (stream) {

stream.on('data', function (data) { data.created_at = new Date(data.created_at).getTime(); collection.insert(data, function (err, result) { if (err) { console.log(err);

https://dev.twitter.com/apps/new



Draft


return; }}); }); });

4.2.5 Configuration Module

Configuration Module is Setup as different sections for different sensors and keyword input for twitter as well as the widget options. Once a change is made in the configuration parameters it is reflected back in the dashboard



Draft




Draft


4.2.6 Charts

This module provides with a graphical representation of data stored from different sensors

4.2.7 User and Session Management

Login and Session is managed though node modules such as passport and angular routing npm install passport npm install passport-local



Draft


4.3 Mobile Interface

4.3.1 Developing Mobile Interface Based on CEP

Set up the Android development environment 1. Download and Install Android Studio

https://developer.android.com/studio/index.html#win-bundle 2. In the process of installing Android studio, download and install SDK for Android. 3. In Android SDK, download and install the version of Android SDK platform your app

is going to build (Version 25). 4. Create new project Give Project Name, Select Build Target, Application Name, Package Name, Activity Name, Min SDK version and click Finish. Create the Design : Material Design provides set of properties to customize the Material Design Color theme. But we use five primary attributes to customize overall theme.

1. colorPrimaryDark This is the darkest primary colour of the app and mainly applies to notification bar background.

2. colorPrimary This is the primary color of the app. This color will be applied to toolbar background.

3. textColorPrimary This is the primary color of text. This applies to toolbar title.

4. windowBackground This is the default background color of the app.

5. navigationBarColor This color defines the background color of footer navigation bar.

6. Open res ⇒ values ⇒ strings.xml and add string values required for the app.

7. Open res ⇒ values ⇒ dimens.xml and add dimensions like (padding, margin and text size).

1. Open styles.xml under res ⇒ values and define the styles which are to all android versions like (windowNoTitle, windowActionBar).

8. Open AndroidManifest.xml and add the permissions required for the app. 9. Open res folder => Layout folder => Create .xml layout file to display the

content on the screen. 10. Open App=> Java=>application Path=> to create Java files with

functionality like (MainActivity.java).MainActivity.java Contains 1. Mapping of xml Button, Images, Textview, Edittext Etc. 2. The functionality is written on Button Click to handle the UI to

respond. 3. The Adapter Functionality is used to display the list of alerts. 4. Once the button of alerts is Clicked the API is called using Http

service, result is obtained and displayed. 11. Open Gradle (Module:app) Add Dependencies and Sync Project. 12. On Click of Alert Button in toolbar the CEP is Triggered and Alerts are

received. Maximum of 10 alerts are fetched from the CEP. 4.3.2 Developing Mobile Interface Based on Firebase

Set up the Android development environment 1. Download and Install Android Studio

https://developer.android.com/studio/index.html#win-bundle

https://developer.android.com/studio/index.html#win-bundle



Draft


2. In the process of installing Android studio Download, Install SDK for Android. 3. In Android SDK, download and install the version of Android SDK platform your app

is going to build (Version 25) Create new project Give Project Name, Select Build Target, Application Name, Package Name, Activity Name, Min SDK version and click Finish. Create the Design : 1. Material Design provides a set of properties to customize the Material Design Color

theme. But we use five primary attributes to customize overall theme.

2. colorPrimaryDark – This is the darkest primary color of the app and mainly applies to notification bar background.

3. colorPrimary – This is the primary color of the app. This color will be applied to toolbar background.

4. textColorPrimary – This is the primary color of text. This applies to toolbar title. 5. windowBackground – This is the default background color of the app. 6. navigationBarColor – This color defines the background color of footer navigation

bar. 7. Open res ⇒ values ⇒ strings.xml and add string values required for the app.

8. Open res ⇒ values ⇒ dimens.xml and add dimensions like (padding, margin and text size).

9. Open styles.xml under res ⇒ values and define the styles which are to all Android versions like (windowNoTitle, windowActionBar).

10. Open AndroidManifest.xml and add the permissions required for the app. 11. Open res folder => Layout folder => Create .xml layout file to display the content

on the screen. 12. Open App=> Java=>application Path=> to create Java files with functionality like

(MainActivity.java). 13. MainActivity.java Contains

1. Mapping of xml Button, Images, Textview, Edittext Etc. 2. The functionality is written on Button Click to handle the UI to respond. 3. The Adapter Functionality is used to display the list of alerts. 4. Once the button of alerts is Clicked the API is called using Http service,

result is obtained and displayed. 14. Open Gradle (Module:app) Add Dependencies and Sync Project.



Draft


The Code below will display Alerts using handler with use of adapters.



Draft


This below code below accepts the list of alerts to display and returns the display to the View.



Draft


5 Integration

5.1 IOT Agent and Orion

Check the installation section of IOTagent for more info

The Orion Context Broker interacts with context producer applications (which provide sensor information) and a context consumer application (which processes that information, e.g. to show it in a graphical user interface). We will play the role of both kinds of applications in the tutorials. While configuring the iotagent the Context Broker running host and port are mentioned, along with the device service paths and subscriptions For example var config = { logLevel: 'DEBUG', contextBroker: { host: 'localhost', port: '1026' }, server: { port: 4041 }, authentication: { host: 'localhost', port: '5000', user: 'iotagent', password: 'iotagent' }, deviceRegistry: { type: 'memory'



Draft


}, types: { 'Light': { url: '/', apikey: '', type: 'Light', // service: '', // subservice: '', // trust: '' // cbHost: '', commands: [], lazy: [ { name: 'luminescence', type: 'Lumens' } ], active: [ { name: 'status', type: 'Boolean' } ] } }, service: 'smartGondor', subservice: '/gardens', providerUrl: 'http://localhost:4041', deviceRegistrationDuration: 'P1M', defaultType: 'Thing' };



Draft


5.2 Orion and CEP

A CEP application is created using the CEP web based authoring tool. Using this authoring tool, the application developer creates the building blocks of the application definitions. This is done by filling up forms without the need to write any code. There are REST services that allow you to manage the CEP definition repository that holds the CEP application definitions available to the CEP engine at run time. These services allow putting a new application definition to the repository, getting a specific application definition from the repository, updating a repository application definition or deleting a definition from the repository. In addition, there are REST services that allow controlling the CEP engine at run time. These services allow starting and stopping a CEP engine, updating CEP engine definitions and reading the state of the CEP engine (started/stopped and its definition url). The CEP provides a REST service for receiving input events. Event has a list of attributes, each attribute as a name and a value, and the name of the event is given in the "Name" attribute. The CEP supports three formats for input events: tag-delimited, JSON and NGSI-XML. To send event in the JSON format use the following query:

POST <cep_host>:<port>/{instance_name}/rest/events //(Example: POST: http://130.206.81.23:8080/ProtonOnWebServer/rest/events) Headers: {'Content-Type’: 'application/json’; 'X-Auth-Token' : <Oauth2.0 TOKEN>} { "Name":"TrafficReport", "volume":"1000"

} To send the same event in the tag-delimited format use the same query but with a different content type and a different data format:

POST <cep_host>:<port>/{instance_name}/rest/events //(Example: POST: http://130.206.81.23:8080/ProtonOnWebServer/rest/events) Headers: {'Content-Type’: ‘text/plain’; 'X-Auth-Token' : <Oauth2.0 TOKEN>} Name=TrafficReport;volume=1000;



Draft


The NGSI-XML format allows the CEP to get input events from the Context Broker GE. If you want the CEP to get input events from the Context Broker GE, you should add a subscription to the Context Broker with the CEP URL given above.



Draft


5.3 Orion and Cygnus

Cygnus takes advantage of the subscription-notification mechanism of Orion Context Broker. Specifically, Cygnus needs to be notified each time certain entity's attributes change, and in order to do that, Cygnus must subscribe to those entity's attribute changes. You can make a subscription to Orion on behalf of Cygnus by using a curl command or any other REST client. In the following example, we assume both Orion and Cygnus run in localhost, and Cygnus is listening in the TCP/5050 port: (curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Fiware-Service: vehicles' --header 'Fiware-ServicePath: /4wheels' -d @- | python -mjson.tool) <<EOF { "entities": [ { "type": "car", "isPattern": "false", "id": "car1" } ], "attributes": [ "speed", "oil_level" ], "reference": "http://localhost:5050/notify", "duration": "P1M", "notifyConditions": [ { "type": "ONCHANGE", "condValues": [ "speed" ] }



Draft


], "throttling": "PT1S" } EOF Which means: Each time the the 'car1' entity, of type 'car', which is registered under the service/tenant 'vehicles', subservice '/4wheels', changes its value of 'speed' then send a notification to http://localhost:5050/notify (where Cygnus will be listening) with the 'speed' and 'oil_level' values. This subscription will have a duration of one month, and please, do not send me notifications more than once per second.



Draft


5.4 Cygnus and CKAN

This would be a configuration for Cygnus CKAN Sink

cygnus-ngsi.sinks = ckan-sink cygnus-ngsi.channels = ckan-channel ... cygnus-ngsi.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.NGSICKANSink cygnus-ngsi.sinks.ckan-sink.channel = ckan-channel cygnus-ngsi.sinks.ckan-sink.enable_encoding = false cygnus-ngsi.sinks.ckan-sink.enable_grouping = false cygnus-ngsi.sinks.ckan-sink.enable_name_mappings = false cygnus-ngsi.sinks.ckan-sink.data_model = dm-by-entity cygnus-ngsi.sinks.ckan-sink.attr_persistence = column cygnus-ngsi.sinks.ckan-sink.ckan_host = 192.168.80.34 cygnus-ngsi.sinks.ckan-sink.ckan_port = 80 cygnus-ngsi.sinks.ckan-sink.ckan_viewer = recline_grid_view cygnus-ngsi.sinks.ckan-sink.ssl = false cygnus-ngsi.sinks.ckan-sink.api_key = myapikey cygnus-ngsi.sinks.ckan-sink.Orion_url = http://localhost:1026 cygnus-ngsi.sinks.ckan-sink.batch_size = 100 cygnus-ngsi.sinks.ckan-sink.batch_timeout = 30 cygnus-ngsi.sinks.ckan-sink.batch_ttl = 10 cygnus-ngsi.sinks.ckan-sink.batch_retry_intervals = 5000 cygnus-ngsi.sinks.ckan-sink.backend.max_conns = 500 cygnus-ngsi.sinks.ckan-sink.backend.max_conns_per_route = 100 cygnus-ngsi.sinks.ckan-sink.persistence_policy.max_records = 5 cygnus-ngsi.sinks.ckan-sink.persistence_policy.expiration_time = 86400 cygnus-ngsi.sinks.ckan-sink.persistence_policy.checking_time = 600

com.iot.telefonica.cygnus.sinks.NGSICKANSink, or simply NGSICKANSink is a sink designed to persist NGSI-like context data events within a CKAN server. Usually, such a context data is notified by a Orion Context Broker instance, but could be any other system speaking the NGSI language. Independently of the data generator, NGSI context data is always transformed into internal NGSIEvent objects at Cygnus sources. In the end, the information within these events must be mapped into specific CKAN data structures.



Draft


5.4.1 Mapping NGSI events to NGSIEvent objects Notified NGSI events (containing context data) are transformed into NGSIEvent objects (for each context element a NGSIEvent is created; such an event is a mix of certain headers and a ContextElement object), independently of the NGSI data generator or the final backend where it is persisted. This is done at the cygnus-ngsi Http listeners (in Flume jergon, sources) thanks to NGSIRestHandler. Once translated, the data (now, as NGSIEvent objects) is put into the internal channels for future consumption (see next section).

5.4.2 Mapping NGSIEvents to CKAN data structures CKAN organizes the data in organizations containing packages or datasets; each one of these packages/datasets contains several resources whose data is finally stored in a PostgreSQL database (CKAN Datastore) or plain files (CKAN Filestore). Such organization is exploited by NGSICKANSink each time a NGSIEvent is going to be persisted.

5.4.3 Organizations naming conventions An organization named as the notified fiware-service header value (or, in absence of such a header, the defaulted value for the FIWARE service) is created (if not existing yet). Since based in PostgreSQL only accepts, it must be said only alphanumeric characters and the underscore (_) are accepted. The hyphen ('-') is also accepted. This leads to certain encoding is applied depending on the enable_encoding configuration parameter. Nevertheless, different than PostgreSQL, organization lengths may be up to 100 characters (minimum, 2 characters).

5.4.4 Packages/datasets naming conventions A package/dataset named as the concatenation of the notified fiware-service and fiware-servicePath header values (or, in absence of such headers, the defaulted value for the FIWARE service and service path) is created (if not existing yet) in the above organization. Since based in PostgreSQL only accepts, it must be said only alphanumeric characters and the underscore (_) are accepted. The hyphen ('-') is also accepted. This leads to certain encoding is applied depending on the enable_encoding configuration parameter. Nevertheless, different than PostgreSQL, dataset lengths may be up to 100 characters (minimum, 2 characters).

5.4.5 Resources naming conventions CKAN resources follow a single data model (see the Configuration section for more details), i.e. per entity. Thus, a resource name always take the concatenation of the entity ID and type. Such a name is already given in the notified_entities/grouped_entities header values (depending on using or not the grouping rules, see the Configuration section for more details) within the NGSIEvent. It must be noticed a CKAN Datastore (and a viewer) is also created and associated to the resource above. This datastore, which in the end is a PostgreSQL table, will hold the persisted data. Since based in PostgreSQL, it must be said only alphanumeric characters and the underscore (_) are accepted. The hyphen ('-') is also accepted. This leads to certain encoding is applied depending on the enable_encoding configuration parameter. Despite there is no real limit on the resource names, Cygnus will keep limiting their lengths up to 100 characters (minimum, 2 characters), accordingly to what's done with organization and package names.

5.4.6 Row-like storing Regarding the specific data stored within the datastore associated to the resource, if attr_persistence parameter is set to row (default storing mode) then the notified data is stored attribute by attribute, composing an insert for each one of them. Each insert contains the following fields:

recvTimeTs: UTC timestamp expressed in miliseconds. recvTime: UTC timestamp in human-readable format (ISO 8601). fiwareServicePath: Notified fiware-servicePath, or the default configured one if not notified. entityId: Notified entity identifier.



Draft


entityType: Notified entity type. attrName: Notified attribute name. attrType: Notified attribute type. attrValue: In its simplest form, this value is just a string, but since Orion 0.11.0 it can be Json object or Json array. attrMd: It contains a string serialization of the metadata array for the attribute in Json (if the attribute has not metadata, an empty array [] is inserted).

5.4.7 Column-like storing Regarding the specific data stored within the datastore associated to the resource, if attr_persistence parameter is set to column then a single line is composed for the whole notified entity, containing the following fields:

recvTime: UTC timestamp in human-redable format (ISO 8601). fiwareServicePath: The notified one or the default one. entityId: Notified entity identifier. entityType: Notified entity type.

For each notified attribute, a field named as the attribute is considered. This field will store the attribute values along the time. For each notified attribute, a field named as the concatenation of the attribute name and _md is considered. This field will store the attribute's metadata values along the time. Example NGSIEvent Assuming the following NGSIEvent is created from a notified NGSI context data (the code below is an object representation, not any real data format): ngsi-event={ headers={ content-type=application/json, timestamp=1429535775, transactionId=1429535775-308-0000000000, correlationId=1429535775-308-0000000000, fiware-service=vehicles, fiware-servicepath=/4wheels, <grouping_rules_interceptor_headers>, <name_mappings_interceptor_headers> }, body={ entityId=car1, entityType=car, attributes=[ { attrName=speed, attrType=float, attrValue=112.9 }, { attrName=oil_level, attrType=float, attrValue=74.6 } ] } }

5.4.8 Organization, dataset and resource names Given the above example and using the old encoding, these are the CKAN elements created Orgnaization: vehicles. Package: vehicles_4wheels. Resource: car1_car. Using the new encdoing:



Draft


Orgnaization: vehicles. Package: vehiclesxffffx002f4wheels. Resource: car1xffffcar.

5.4.9 Row-like storing Assuming attr_persistence=row as configuration parameter, then NGSICKANSink will persist the data within the body as: $ curl -s -S -H "Authorization: myapikey" "http://192.168.80.34:80/api/3/action/datastore_search?resource_id=3254b3b4-6ffe-4f3f-8eef-c5c98bfff7a7" { "help": "Search a DataStore resource...", "success": true, "result": { "resource_id": "3254b3b4-6ffe-4f3f-8eef-c5c98bfff7a7", "fields": [ { "type": "int4", "id": "_id" }, { "type": "int4", "id": "recvTimeTs" }, { "type": "timestamp", "id": "recvTime" }, { "type": "text", "id": "fiwareServicePath" }, { "id": "entityId", "type": "text" }, { "id": "entityType", "type": "text" }, { "type": "text", "id": "attrName" }, { "type": "text", "id": "attrType" }, { "type": "json", "id": "attrValue" }, { "type": "json", "id": "attrMd" } ], "records": [ { "entityId": "car1", "entityType": "car", "fiwareServicePath": "4wheels",



Draft


"attrType": "float", "recvTime": "2015-04-20T12:13:22.41.124Z", "recvTimeTs": 1429535775, "attrMd": null, "attrValue": "112.9", "attrName": "speed", "_id": 1 }, { "entityId": "car1", "entityType": "car", "fiwareServicePath": "4wheels", "attrType": "float", "recvTime": "2015-04-20T12:13:22.41.124Z", "recvTimeTs": 1429535775, "attrMd": null, "attrValue": "74.6", "attrName": "oil_level", "_id": 2 } ], "_links": { "start": "/api/3/action/datastore_search?resource_id=3254b3b4-6ffe-4f3f-8eef-c5c98bfff7a7", "next": "/api/3/action/datastore_search?offset=100&resource_id=3254b3b4-6ffe-4f3f-8eef-c5c98bfff7a7" }, "total": 2 } }

5.4.10 Column-like storing If attr_persistence=colum then NGSICKANSink will persist the data within the body as: $ curl -s -S -H "Authorization: myapikey" "http://130.206.83.8:80/api/3/action/datastore_search?resource_id=611417a4-8196-4faf-83bc-663c173f6986" { "help": "Search a DataStore resource...", "success": true, "result": { "resource_id": "611417a4-8196-4faf-83bc-663c173f6986", "fields": [ { "type": "int4", "id": "_id" }, { "type": "timestamp", "id": "recvTime" }, { "type": "text", "fiwareServicePath": "4wheels" }, { "type": "text", "entityId": "car1" }, { "type": "text", "entityType": "car"



Draft


}, { "type": "json", "id": "speed" }, { "type": "json", "id": "speed_md" }, { "type": "json", "id": "oil_level" }, { "type": "json", "id": "oil_level_md" } ], "records": [ { "recvTime": "2015-04-20T12:13:22.41.124Z", "fiwareServicePath": "4wheels", "entityId": "car1", "entityType": "car", "speed": "112.9", "speed_md": null, "oil_level": "74.6", "oil_level_md": null, "_id": 1 } ], "_links": { "start": "/api/3/action/datastore_search?resource_id=611417a4-8196-4faf-83bc-663c173f6986", "next": "/api/3/action/datastore_search?offset=100&resource_id=611417a4-8196-4faf-83bc-663c173f6986" }, "total": 1 } } NOTE: curl is a Unix command allowing for interacting with REST APIs such as the exposed by CKAN.



Draft


6 Quality Assurance

6.1 Sensor Data Consistency Since the data is coming from a number of sensors in different presets the chances of data missing at a certain point is there is not taken care of it programmatically. Whenever there is a failure to get data from particular sensor application checks for the last stored date and time and retriever the missing data.

6.2 Integration Testing

6.2.1 Fiware Components

1. CEP 2. IOT Agent 0.8.4 3. Context Broker 1.7.0 4. CKAN 2.7.0 5. CYGNUS

6.2.2 System and Environment

1. Operating System: Centos 6.8 Final

2. Processor: GenuineIntell(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz

3. Database System: MongoDB - MongoDB shell version: 3.0.15

4. Node v7.8.0 5. Webserver System version:

Apache Tomcat/7.0.75 6. Java virtual machine version1.7.0_131-mockbuild_2017_02_13_15_19-b00

by Oracle Corporation 7. Python : 2.6.6

6.3 Mobile Responsiveness Tested Device List 8. Blackberry Z30 9. Samsung Galaxy S III 10. Samsung Galaxy Note 3 11. Samsung Galaxy S 5 12. LG Optimus L70 13. Nexus 5 14. Nexus 6 15. Nexus 7 16. Asus Zenfone Z00AD 17. iPhone 4 18. iPhone 5 19. iPhone 6 Plus 20. And various virtual devices via Chrome Device Tool

6.4 Server Resource Utilization 6.4.1 vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 13272552 182984 1373920 0 0 0 4 0 1 0 0 100 0 0 uptime



Draft


09:17:53 up 51 days, 21:44, 1 user, load average: 0.00, 0.00, 0.00 grep processor /proc/cpuinfo | wc -l 4

6.4.2 CPUUtilization From AWS Console for Last 6 weeks

CPU Utilization is The percentage of allocated EC2 compute units that are currently in use on the instance. This metric identifies the processing power required to run an application upon a selected instance. To use the percentiles statistic, you must enable detailed monitoring. Depending on the instance type, tools in your operating system can show a lower percentage than CloudWatch when the instance is not allocated a full processor core. Units: Percent



Draft


7 References 1. www.fiware.org/devguides/fiware-tour-guide-application-a-tutorial-on-how-to-integrate-the-main-fiware-ges/ 2. https://www.fiware.org/smart-cities/ 3. https://catalogue.fiware.org/ 4. http://fiware-cygnus.readthedocs.io/en/latest/ 5. http://fiware-iot-stack.readthedocs.io/en/latest/ 6. https://github.com/telefonicaid/iotagent-ul/blob/master/docs/installationguide.md 7. https://fiware-Orion.readthedocs.io/en/master/ 8. http://fiware-Orion.readthedocs.io/en/master/user/walkthrough_apiv1/index.html 9. http://docs.ckan.org/en/latest/ 10. https://github.com/telefonicaid/iotagent-ul 11. http://proactive-technology-online.readthedocs.io/en/latest/index.html 12. https://github.com/ishkin/Proton/blob/master/documentation/apiary/CEP-apiary-blueprint.html

http://www.fiware.org/devguides/fiware-tour-guide-application-a-tutorial-on-how-to-integrate-the-main-fiware-ges/

http://www.fiware.org/devguides/fiware-tour-guide-application-a-tutorial-on-how-to-integrate-the-main-fiware-ges/

https://www.fiware.org/smart-cities/

https://catalogue.fiware.org/

http://fiware-cygnus.readthedocs.io/en/latest/

http://fiware-iot-stack.readthedocs.io/en/latest/

https://github.com/telefonicaid/iotagent-ul/blob/master/docs/installationguide.md

https://fiware-orion.readthedocs.io/en/master/


http://docs.ckan.org/en/latest/

https://github.com/telefonicaid/iotagent-ul

http://proactive-technology-online.readthedocs.io/en/latest/index.html

https://github.com/ishkin/Proton/blob/master/documentation/apiary/CEP-apiary-blueprint.html

https://github.com/ishkin/Proton/blob/master/documentation/apiary/CEP-apiary-blueprint.html



Draft


8 Environment • Centos – 6.8

• Mongo – 2.6 or above

• Node – 7.8 or above

• Npm – 4.0 or above Context Broker

• Version – 1.7.0

• Requires mongodb CEP Proton Requires;

• Java – 7 or later

• Tomcat – 7 or later CYGNUS Requires

• Java – 7 or later

• Maven – 3.2.5 or later

• Flume – 1.4.0 Read the Doc below for elaborate descriptions http://fiware-cygnus.readthedocs.io/en/latest/ CKAN (2.0) Requires:

• Python – 2.6.6 or later

• SOLR – 1.4.1

Nodejs Files are in /opt/node-js/smart-city-streamer Put the smart city streamer anywhere Turn on node app.js old pass = Gheap7 controllers.js is where the devices connections are

http://fiware-cygnus.readthedocs.io/en/latest/

Stratumseind FIWARE Implementation and Configuration...

Documents

Transcript of Stratumseind FIWARE Implementation and Configuration...