ORION CONTEXT BROKER INSTALLATION & ADMINISTRATION MANUAL

INTRODUCTION

Welcome to the Orion Context Broker Installation & Administration Manual!

Any feedback on this document is highly welcome, including bug reports, typos or stuff you think should be included but is not. Please send feedback through Github. Thanks in advance!

INSTALLING ORION

INTRODUCTION The recommended procedure is to install using RPM packages in CentOS 7.x. If you are interested in building from sources, check this document.

REQUIREMENTS

• System resources: see these recommendations • Operating system: CentOS/RedHat. The reference operating system is CentOS

7.4.1708 but it should work also in any later CentOS/RedHat 7.x version. • Database: MongoDB is required to run either in the same host where Orion Context

Broker is to be installed or in a different host accessible through the network. The recommended MongoDB version is 3.6 (Orion may work with older versions but we do not recommend it at all!).

• RPM dependencies (some of these packages could not be in the official CentOS/RedHat repository but in EPEL, in which case you have to configure EPEL repositories, see http://fedoraproject.org/wiki/EPEL): o The contextBroker package (mandatory) depends on the following packages:

libstdc++, boost-thread, boost-filesystem, gnutls, libgcrypt, libcurl, openssl, logrotate and libuuid

INSTALLATION There are two available packages:

• Nightly, which are built from master branch every night. • Release, "official" release RPMs. The usual release period is 1-2 months.

Depending on their type (nightly or release), packages have different versioning:

• Release packages: contextBroker-X.Y.Z-1 • Nightly packages: contextBroker-X.Y.Z-yyyy.mm.dd

Keep in mind, the version of nightly packages will always be ahead of release, and we advise to use nightly packages only for testing purposes.

From this point, if you want to use the nightly builds, remember that it has a different system of versions.

USING YUM (RECOMMENDED) Configure the FIWARE yum repository as described here. Then you can install doing (being root):

Sometimes the above command fails due to yum cache. In that case, run yum clean all and try again.

USING RPM FILE Download the package directly from the FIWARE Yum repository (both types of packages are provided).

Next, install the package using the rpm command (as root):

UPGRADING FROM A PREVIOUS VERSION Upgrade procedure depends on whether the upgrade path (i.e. from the installed Orion version to the target one to upgrade) crosses a version number that requires:

• Upgrading MongoDB version • Migrating the data stored in DB (due to a change in the DB data model).

MIGRATING THE DATA STORED IN DB You only need to pay attention to this section of the Manual if your upgrade path crosses 0.14.1, 0.19.0, 0.21.0 or 1.3.0. Otherwise, you can skip this section. You can also skip this section if your DB are not valuable (e.g. debug/testing environments) and you can flush your DB before upgrading.

• Upgrading to 0.14.1 and beyond from a pre-0.14.1 version • Upgrading to 0.19.0 and beyond from a pre-0.19.0 version • Upgrading to 0.21.0 and beyond from a pre-0.21.0 version • Upgrading to 1.3.0 and beyond from a pre-1.3.0 version • Upgrading to 1.5.0 and beyond from a pre-1.5.0 version

If your upgrade cover several segments (e.g. you are using 0.13.0 and want to upgrade to 0.19.0, so both "upgrading to 0.14.1 and beyond from a pre-0.14.1 version" and "upgrading to 0.19.0 and beyond from a pre-0.19.0 version" applies to the case) you need to execute the segments in sequence (the common part are done only one time, i.e. stop CB, remove package, install package, start CB). In the case of doubt, please ask using StackOverflow (remember to include the "fiware-orion" tag in your questions).

STANDARD UPGRADE If you are using yum, then you can upgrade doing (as root):

Sometimes the above command fails due to yum cache. In that case, run yum clean all and try again.

If you are upgrading using the RPM file, then first download the new package from the FIWARE yum repository (both types of packages are provided).

Then upgrade the package using the rpm command (as root):

USING YUM REPOSITORIES

This document describes the guidelines of using FIWARE Yum repository to install Orion Context Broker. Provided configuration corresponds to x86_64 architecture and CentOS/RHEL 7 OS.

There are two available repositories:

• Nightly, for nightly packages. • Release, for release packages.

You can read about differences between packages here

You can manually add a config for repositories, or download it from the FIWARE public repository. Keep in mind, if you use both repositories together on the same server, nightly packages will always be ahead of release.

Use this configuration for release repository:

or download it from FIWARE public repository via

Use this configuration for nightly repository:

or download it from FIWARE public repository via

Next step you can simply install ContextBroker via

BUILDING FROM SOURCES

Orion Context Broker reference distribution is CentOS 7.x. This does not mean that the Orion cannot be built in other distributions (actually, it can). This section also includes indications on how to build in other distributions, just in the case it may help people that do not use CentOS. However, note that the only "officially supported" procedure is the one for CentOS 7.x; the others are provided "as is" and can get obsolete from time to time.

CENTOS 7.X (OFFICIALLY SUPPORTED) The Orion Context Broker uses the following libraries as build dependencies:

• boost: 1.53 • libmicrohttpd: 0.9.48 (from source) • libcurl: 7.29.0 • openssl: 1.0.2k • libuuid: 2.23.2 • Mongo Driver: legacy-1.1.2 (from source) • rapidjson: 1.0.2 (from source) • gtest (only for make unit_test building target): 1.5 (from sources) • gmock (only for make unit_test building target): 1.5 (from sources)

The basic procedure is as follows (assuming you don't run commands as root, we use sudo for those commands that require root privilege):

• Install the needed building tools (compiler, etc.).

• Install the required libraries (except what needs to be taken from source, described in following steps).

• Install the Mongo Driver from source.

• Install rapidjson from sources:

• Install libmicrohttpd from sources (the ./configure command below shows the recommended build configuration to get minimum library footprint, but if you are an advanced user, you can configure as you prefer)

• Install Google Test/Mock from sources (there are RPM packages for this, but they do not work with the current CMakeLists.txt configuration). Previously the URL was http://googlemock.googlecode.com/files/gmock-1.5.0.tar.bz2, but Google removed that package in late August 2016 and it is no longer working.

• Get the code (alternatively you can download it using a zipped version or a different URL pattern, e.g git clone git@github.com:telefonicaid/fiware-orion.git):

• Build the source:

• (Optional but highly recommended) run unit test. Firstly, you have to install MongoDB as the unit and functional tests rely on mongod running in localhost. Check the official MongoDB documentation for details. Recommended version is 3.6 (although 3.2 and 3.4 should also work fine).

• Install the binary. You can use INSTALL_DIR to set the installation prefix path (default is /usr), thus the broker is installed in $INSTALL_DIR/bin directory.

• Check that everything is ok, invoking the broker version message:

The Orion Context Broker comes with a suite of functional, valgrind and end-to-end tests that you can also run, following the following procedure (optional):

• Install the required tools:

• Prepare the environment for test harness. Basically, you have to install the accumulator-server.py script and in a path under your control, ~/bin is the recommended one. Alternatively, you can install them in a system directory such as /usr/bin, but it could collide with an RPM installation, thus it is not recommended. In addition, you have to set several environment variables used by the harness script (see scripts/testEnv.sh file).

• Run test harness (it takes some time, please be patient).

• Once passed all the functional tests, you can run the valgrind tests (this will take longer than the functional tests, arm yourself with a lot of patience):

You can generate coverage reports for the Orion Context Broker using the following procedure (optional):

• Install the lcov tool

• Do first a successful pass for unit-test and functional-test, to check that everything is ok (see above)

• Run coverage

You can generate the RPM for the source code (optional):

• Install the required tools

Generate the RPM

• The generated RPMs are placed in directory ~/rpmbuild/RPMS/x86_64. RUNNING ORION AS SYSTEM SERVICE

Once installed, there are two ways of running Orion Context Broker: manually from the command line or as a system service (the later only available if Orion was installed as RPM package). It is not recommended to mix both ways (e.g. start the context broker from the command line, then use /etc/init.d/contextBroker status to check its status). This section assumes you are running Orion as system service. From command line alternative, check this document.

You will typically need superuser privileges to use Orion Context Broker as a system service, so the following commands need to be run as root or using the sudo command.

In order to start the broker service, run:

Then, to stop the context broker, run:

To restart, run:

You can use chkconfig command to make contextBroker automatically start/stop when your system boots/shutdowns (see chkconfig documentation for details).

CONFIGURATION FILE The configuration used by the contextBroker service is stored in the /etc/sysconfig/contextBroker file, which typical content is:

All the fields except BROKER_USER and BROKER_EXTRA_OPS map to one of the options described in command line options, as follows:

• BROKER_USER doesn't map to CLI option but is used to the init.d script to set the owner of the contextBroker process

• BROKER_PORT maps to -port

• BROKER_LOG_DIR maps to -logDir

• BROKER_LOG_LEVEL maps to -logLevel

• BROKER_PID_FILE maps to -pidpath

• BROKER_DATABASE_HOST maps to -dbhost

• BROKER_DATABASE_NAME maps to -db

• BROKER_DATABASE_RPLSET maps to -rplSet

• BROKER_DATABASE_USER maps to -dbuser

• BROKER_DATABASE_PASSWORD maps to -dbpwd

Regarding BROKER_EXTRA_OPS, it is used to specify other options not covered by the fields above, as a string that is appended to the broker command line at starting time. Note that this string is "blindly" appended, i.e. the service script doesn't do any check so be careful using this, ensuring that you are providing valid options here and you are not duplicating any option in other BROKER_* field (e.g. not set BROKER_EXTRA_OPS="-port 1026" as BROKER_PORT is used for that).

Regarding BROKER_USER, it is the user that will own the contextBroker process upon launching it. By default, the RPM installation creates a user named 'orion'. Note that if you want to run the broker in a privileged port (i.e. 1024 or below) you will need to use 'root' as BROKER_USER.

CHECKING STATUS In order to check the status of the broker, use the following command with superuser privileges (using the root user or the sudo command):

If broker is running you will get:

If broker is not running you will get:

RUNNING ORION FROM COMMAND LINE

You can run the broker by typing the following command:

The broker runs in the background by default, so you will need to stop it using signals.

You can use command line arguments, e.g. to specify the port in which Orion Context Broker listens, using the -port option:

To know all the possible options, have a look at the next section.

COMMAND LINE OPTIONS Command line options can be used directly (in the case of running from the command line) or indirectly through the different fields in /etc/sysconfig/contextBroker (in the case of running as a system service). To obtain a list of available options, use:

To get more information on the options (including default values and limits), use:

The list of available options is the following:

• -u and -U. Shows usage in brief or long format, respectively.

• --help. Shows help (very similar to previous).

• --version. Shows version number

• -port. Specifies the port that the broker listens to. Default port is 1026.

• -ipv4. Runs broker in IPv4 only mode (by default, the broker runs in both IPv4 and IPv6). Cannot be used at the same time as -ipv6.

• -ipv6. Runs broker in IPv6 only mode (by default, the broker runs in both IPv4 and IPv6). Cannot be used at the same time as -ipv4.

• -rush. Use rush in host and port. Default behavior is to not use Rush. See section on using Rush relayer.

• -multiservice. Enables multiservice/multitenant mode (see multi service tenant section).

• -db. The MongoDB database to use or (if -multiservice is in use) the prefix to per-service/tenant databases (see section on service/tenant database separation. This field is restricted to 10 characters max length.

• -dbhost. The MongoDB host and port to use, e.g. -dbhost localhost:12345.

• -rplSet. If used, Orion CB connnects to a MongoDB replica set (instead of a stand-alone MongoDB instance). The name of the replica set to use is the value of the parameter. In this case, the -dbhost parameter can be a list of hosts (separated by ",") which are used as seed for the replica set.

• -dbTimeout. Only used in the case of using replica set (-rplSet), ignored otherwise. It specifies the timeout in milliseconds for connections to the replica set.

• -dbuser. The MongoDB user to use. If your MongoDB doesn't use authorization then this option must be avoided. See database authorization section.

• -dbpwd. The MongoDB password to use. If your MongoDB doesn't use authorization then this option must be avoided. See database authorization section.

• -dbPoolSize. Database connection pool. Default size of the pool is 10 connections.

• -writeConcern <0|1>. Write concern for MongoDB write operations: acknowledged (1) or unacknowledged (0). Default is 1.

• -https. Work in secure HTTP mode (See also -cert and -key).

• -cert. Certificate file for https. Use an absolute file path. Have a look at this script for an example on how to generate this file.

• -key. Private server key file for https. Use an absolute file path. Have a look at this script for an example on how to generate this file.

• -logDir <dir>. Specifies the directory to use for the contextBroker log file.

• -logAppend. If used, the log lines are appended to the existing contextBroker log file, instead of starting with an empty log file.

• -logLevel. Select initial logging level, supported levels:

• NONE (suppress ALL log output, including fatal error messages),

• FATAL (show only fatal error messages),

• ERROR (show only error messages),

• WARN (show error and warning messages - this is the default setting),

• INFO (show error, warning and informational messages),

• DEBUG (show ALL messages). Note that the log level can be modified in run-time, using the admin API.

• -t. Specifies the initial trace levels for logging. You can use a single value (e.g. "-t 70"), a range (e.g. "-t 20-80"), a comma-separated list (e.g. "-t 70,90") or a combination of them (e.g. "-t 60,80-90"). If you want to use all trace levels for logging, use "-t 0-255". Note that trace levels can be changed dynamically using the management REST interface. Details of the available trace levels and their values can be found here (as a C struct).

• -fg. Runs broker in foreground (useful for debugging). Log output is printed on standard output (in addition to the log file, but using a simplified format).

• -localIp. Specifies on which IP interface the broker listens to. By default it listens to all the interfaces.

• -pidpath. Specifies the file to store the PID of the broker process.

• -httpTimeout. Specifies the timeout in milliseconds for forwarding messages and for notifications.

• -reqTimeout. Specifies the timeout in seconds for REST connections. Note that the default value is zero, i.e., no timeout (wait forever).

• -cprForwardLimit. Maximum number of forwarded requests to Context Providers for a single client request (default is no limit). Use 0 to disable Context Providers forwarding completely.

• -corsOrigin. Enables Cross-Origin Resource Sharing, specifing the allowed origin (use __ALL for *). More information about CORS support in Orion can be found in the programmers manual.

• -corsMaxAge. Specifies the maximum time (in seconds) preflight requests are allowed to be cached. Defaults to 86400 if not set. More information about CORS support in Orion can be found in the programmers manual.

• -reqMutexPolicy. Specifies the internal mutex policy. See performance tuning documentation for details.

• -subCacheIval. Interval in seconds between calls to subscription cache refresh. A zero value means "no refresh". Default value is 60 seconds, apt for mono-CB deployments (see more details on the subscriptions cache in this document).

• -noCache. Disables the context subscription cache, so subscriptions searches are always done in DB (not recommended but useful for debugging).

• -notificationMode (Experimental option). Allows to select notification mode, either: transient, permanent or threadpool:q:n. Default mode is transient.

• In transient mode, connections are closed by the CB right after sending the notification.

• In permanent connection mode, a permanent connection is created the first time a notification is sent to a given URL path (if the receiver supports permanent connections). Following notifications to the same URL path will reuse the connection, saving HTTP connection time.

• In threadpool mode, notifications are enqueued into a queue of size q and n threads take the notifications from the queue and perform the outgoing requests asynchronously. Please have a look at the thread model section if you want to use this mode.

• -simulatedNotification. Notifications are not sent, but recorded internally and shown in the statistics operation (simulatedNotifications counter). This is not aimed for production usage, but it is useful for debugging to calculate a maximum upper limit in notification rate from a CB internal logic point of view.

• -connectionMemory. Sets the size of the connection memory buffer (in kB) per connection used internally by the HTTP server library. Default value is 64 kB.

• -maxConnections. Maximum number of simultaneous connections. Default value is 1020, for legacy reasons, while the lower limit is 1 and there is no upper limit (limited by max file descriptors of the operating system).

• -reqPoolSize. Size of thread pool for incoming connections. Default value is 0, meaning no thread pool.

• -statCounters, -statSemWait, -statTiming and -statNotifQueue. Enable statistics generation. See statistics documentation.

• -logSummary. Log summary period in seconds. Defaults to 0, meaning Log Summary is off. Min value: 0. Max value: one month (3600 * 24 * 31 == 2678400 seconds). See logs documentation for more detail.

• -relogAlarms. To see every possible alarm-provoking failure in the log-file, even when an alarm is already active, use this option. See logs documentation for more detail.

• -disableCustomNotifications. Disabled NGSIv2 custom notifications. In particular:

• httpCustom is interpreted as http, i.e. all sub-fields except url are ignored

• No ${...} macro substitution is performed.

• -logForHumans. To make the traces to standard out formatted for humans (note that the traces in the log file are not affected)

• -disableMetrics. To turn off the 'metrics' feature. Gathering of metrics is a bit costly, as system calls and semaphores are involved. Use this parameter to start the broker without metrics overhead.

• -insecureNotif. Allow HTTPS notifications to peers which certificate cannot be authenticated with known CA certificates. This is similar to the -k or --insecureparameteres of the curl command.

DATABASE ADMINISTRATION

INTRODUCTION We assume that the system administrator has knowledge of MongoDB (there are very good and free courses at MongoDB education site). Otherwise, we recommend to be very careful with the procedures described in this section.

BACKUP The usual procedure for MongoDB databases is used.

Use mongobackup command to get a backup of the Orion Context Broker database. It is strongly recommended that you stop the broker before doing a backup.

This will create the backup in the dump/ directory.

Note that if you are using multitenant/multiservice you need to apply the procedures to each per-tenant/service database.

RESTORE The usual procedure for MongoDB databases is used.

Use the mongorestore command to restore a previous backup of the Orion Context Broker database. It is strongly recommended that you stop the broker before doing a backup and to remove (drop) the database used by the broker.

Let's assume that the backup is in the dump/ directory. To restore it:

Note that if you are using multitenant/multiservice you need to apply the procedures to each per-tenant/service database.

DATABASE AUTHORIZATION MongoDB authorization is configured with the -db, -dbuser and -dbpwd options (see section on command line options). There are a few different cases to take into account:

• If your MongoDB instance/cluster doesn't use authorization, then do not use the -dbuser and -dbpwd options.

• If your MongoDB instance/cluster uses authorization, then: o If you run Orion in single service/tenant mode (i.e. without -multiservice) then

you are using only one database (the one specified by the -db option) and the authorization is done with -dbuser and -dbpwd in that database.

o If you run Orion in multi service/tenant mode (i.e. with -multiservice) then the authorization is done at admin database using -dbuser and -dbpwd. As described later in this document, in multi service/tenant mode, Orion uses several databases (which in addition can potentially be created on the fly), thus authorizing on admin DB ensures permissions in all of them.

MULTISERVICE/MULTITENANT DATABASE SEPARATION Normally, Orion Context Broker uses just one database at MongoDB level (the one specified with the -db command line option, typically "orion"). However, when multitenant/multiservice is used the behaviour is different and the following databases are used (let <db> be the value of the -db command line option):

• The database <db> for the default tenant (typically, orion) • The database <db>-<tenant> or service/tenant <tenant> (e.g. if the tenant is named

tenantA and default -db is used, then the database would be orion-tenantA.

Per-service/tenant databases are created "on the fly" as the first request involving tenant data is processed by Orion.

Finally, in the case of per-service/tenant databases, all collections and administrative procedures (backup, restore, etc.) are associated to each particular service/tenant database.

DELETE COMPLETE DATABASE This operation is done using the MongoDB shell:

SETTING INDEXES Check database indexes section in the performance tuning documentation.

DATABASE MANAGEMENT SCRIPTS Orion Context Broker comes with a few scripts that can be used for browsing and administrative activities in the database, installed in the /usr/share/contextBroker directory.

In order to use these scripts, you need to install the pymongo driver (version 2.5 or above), typically using (run it as root or using the sudo command):

pip-python install pymongo

DELETING EXPIRED DOCUMENTS NGSI specifies an expiration time for registrations and subcriptions (both context and context availability subscriptions). Orion Context Broker does not delete the expired documents (they are just ignored) as expired registrations/subscription can be "re-activated" using a subscription update request, modifying their duration.

However, expired registrations/subscriptions consume space in the database, so they can be "purged" from time to time. In order to help you in that task, the garbage-collector.py script is provided along with the Orion Context Broker (in /usr/share/contextBroker/garbage-collector.py after installing the RPM).

The garbage-collector.py looks for expired documents in registrations, csubs and casubs collection, "marking" them with the following field:

The garbage-collector.py program takes as arguments the collection to be analyzed. E.g. to analyze csubs and casubs, run:

After running garbage-collector.py you can easily remove the expired documents using the following commands in the mongo console:

LATEST UPDATED DOCUMENT You can take a snapshot of the latest updated entities and attributes in the database using the latest-updates.py script. It takes up to four arguments:

• Either "entities" or "attributes", to set the granularity level in the updates. • The database to use (same as the -db parameter and BROKER_DATABASE_NAME

used by the broker). Note that the mongod instance has to run in the same machine where the script runs.

• The maximum number of lines to print • (Optional) A filter for entity IDs, interpreted as a regular expression in the database

query. I.e.

ORION ERRORS DUE TO DATABASE If you are retrieving entities using a large offset value and get this error:

then the DB has raised an error related to sorting operation failure due to lack of resources. You can check that the Orion log file contains an ERROR trace similar to this one:

The typical solution to this is to create an index in the field used for sorting. In particular, if you are using the default entities ordering (based on creation date) you can create the index with the following command at mongo shell:

DATA MODEL

INTRODUCTION Normally you do not need to access MongoDB directly as Orion Contex Broker uses it transparently. However, for some operations (e.g. backup, fault recovery, etc.) it is useful to know how the database is structured. This section provides that information.

In the case you need to access database directly, be very careful when manipulation it, as some actions could be irreversible (doing a backup at the beginning it's a good idea).

Orion Context Broker uses four collections in the database, described in the following subsections.

ENTITIES COLLECTION The entities collection stores information about NGSI entities. Each document in the collection corresponds to an entity.

Fields:

• _id stores the EntityID, including the ID itself and type. Given that we use _id for this, we ensure that EntityIDs are unique. The JSON document for this field includes: o id: entity NGSI ID o type: entity NGSI type o servicePath: related with the service path functionality.

• attrs is a keymap of the different attributes that have been created for that entity. The key is generated with the attribute name (changing "." for "=", as "." is not a valid character in MongoDB document keys). Each element in the map has the following information: o type: the attribute type o value: the attribute value (for those attribute that has received at least one update).

Up to version 0.10.1, this value is always a string, but in 0.11.0 this value can be

also a JSON object or JSON vector to represent an structured value (see section about structured attribute values in user manual).

o md (optional): custom metadata. This is a keymap of metadata objects. The key is generated with the metadata name (changing "." for "=", as "." is not a valid character in MongoDB document keys), e.g. a metadata with name "m.x" will use the key "m=x". The object value of each key has two fields: type and value (of the metadata).

o mdNames: an array of strings. Its elements are the names of the metadata of the attribute. Here the "." to "=" replacement is not done.

o creDate: the timestamp (as integer number) corresponding to attribute creation (as a consequence of append).

o modDate: the timestamp (as integer number) corresponding to last attribute update. It matches creDate if the attribute has not been modified after creation.

• attrNames: an array of strings. Its elements are the names of the attributes of the entity (without IDs). In this case, the "." to "=" replacement is not done.

• creDate: the timestamp (as integer number) corresponding to entity creation date (as a consequence of append).

• modDate: the timestamp (as integer number) corresponding to last entity update. Note that it uses to be the same that a modDate corresponding to at least one of the attributes (not always: it will not be the same if the last update was a DELETE operation). It matches creDate if the entity has not been modified after creation.

• location (optional): geographic location of the entity, composed of the following fields: o attrName: the attribute name that identifies the geographic location in the attrs

array o coords: a GeoJSON representing the location of the entity. See below for more

details. • lastCorrelator: value of the Fiware-Correlator header in the last update request on

the entity. Used by the self-notification loop protection logic. • expDate (optional): expiration timestamp (as a Date object) for the entity. Have a look

to the transient entities functionality for more detail.

Regarding location.coords in can use several formats:

• Representing a point (the one used by geo:point):

• Representing a line (the one used by geo:line):

• Representing a polygon (the one used by geo:box and geo:polygon):

• Finally, location.coords could hold an arbitrary JSON object, representing a location in GeoJSON format. Arbitrary GeoJSON can be used with the geo:json attribute type and it is up to the user to introduce a valid object. Note that the three above cases are actually GeoJSON representation for "fixed" cases.

Note that coordinate pairs use the longitude-latitude order, which is opposite to the order used in the geo-location API. This is due to the internal MongoDB geolocation implementation, (which is based in GeoJSON) uses longitude-latitude order. However, other systems closer to users (e.g. GoogleMaps) use latitude-longitude format, so we have used the latter for the API.

Example document:

REGISTRATIONS COLLECTION The registrations collection stores information about registrations. Each document in the collection corresponds to a registration.

Fields:

• _id is the registration ID (the value that is provided to the user to update the registration). Given that we use _id for this, we ensure that registration IDs are unique

and that queries by registration IDs will be very fast (as there is an automatic default index in _id).

• format: the format to use to send forwarded requests. The only accepted value for now is JSON (meaning NGSIv1 format), although this may change in the future (see issue about NGSIv2-based forwarding).

• servicePath: related with the service path functionality. • status (optional): either active (for active registrations) or inactive (for inactive

registrations). The default status (i.e. if the document omits this field) is "active". • description (optional): a free text string describing the registration. Maximum length is

1024. • expiration: this is the timestamp for which the registration expires. • contextRegistration: is an array whose elements contain the following information:

o entities: an array containing a list of entities (mandatory). The JSON for each entity contains id, type, isPattern and isTypePattern. Note that, due to legacy reasons, isPattern may be "true" or "false" (text) while isTypePattern may be true or false (boolean).

o attrs: an array containing a list of attributes (optional). The JSON for each attribute contains name and type.

o providingApplication: the URL of the providing application for this registration (mandatory)

Example document:

CSUBS COLLECTION The csubs collection stores information about context subscriptions. Each document in the collection corresponds to a subscription.

Fields:

• _id is the subscription ID (the value that is provided to the user to update and cancel the subscription). Given that we use _id for this, we ensure that subscription IDs are unique and that queries by subscription IDs are very fast (as there is an automatic default index in _id).

• servicePath: related with the service path functionality. This is the service path associated to the query "encapsulated" by the subscription. Default is /#.

• expiration: this is the timestamp on which the subscription expires. For permanent subscriptions an absurdly high value is used (see PERMANENT_SUBS_DATETIME in the source code).

• lastNotification: the time when last notification was sent. This is updated each time a notification is sent, to avoid violating throttling.

• throttling: minimum interval between notifications. 0 or -1 means no throttling. • reference: the URL for notifications • entities: an array of entities (mandatory). The JSON for each entity contains id, type,

isPattern and isTypePattern. Note that, due to legacy reasons, isPattern may be "true" or "false" (text) while isTypePattern may be true or false (boolean).

• attrs: an array of attribute names (strings) (optional). • blacklist: a boolean field that specifies if attrs has to be interpreted as a whitelist (if

blacklist is equal to false or doesn't exist) or a blacklist (if blacklist is equal to true).

• metadata: an array of metadata names (strings) (optional). • conditions: a list of attributes that trigger notifications. • expression: an expression used to evaluate if notifications has to be sent or not when

updates come. It may be composed of the following fields: q, mq, georel, geometry and/or coords (optional)

• count: the number of notifications sent associated to the subscription. • format: the format to use to send notification, possible values are JSON (meaning

JSON notifications in NGSIv1 legacy format), normalized, keyValues and values (the last three used in NGSIv2 format).

• status: either active (for active subscriptions) or inactive (for inactive subscriptions).

• description (optional field): a free text string describing the subscription. Maximum length is 1024.

• custom: a boolean field to specify if this subscription uses customized notifications (a functionality in the NGSIv2 API). If this field exist and its value is "true" then customized notifications are used and the headers, qs, method and payload fields are taken into account.

• headers: optional field to store the HTTP headers keymap for notification customization functionality in NGSIv2.

• qs: optional field to store the query parameters keymap for notification customization functionality in NGSIv2.

• method: optional field to store the HTTP method for notification customization functionality in NGSIv2.

• payload: optional field to store the payload for notification customization functionality in NGSIv2.

• lastFailure: the time when last notification failure occurred. Not present if the subscription has never failed.

• lastSuccess: the time when last successful notification occurred. Not present if the subscription has never provoked a successful notification.

Example document:

CASUBS COLLECTION The casubs collection stores information about context availability subscriptions. Each document in the collection corresponds to a subscription.

Fields:

• _id is the subscription ID (the value that is provided to the user to update and cancel the subscription). Given that we use _id for this, we ensure that subscription IDs are unique and that queries by subscription IDs are very fast (as there is an automatic default index in _id).

• expiration: this is the timestamp on which the subscription will expire. • reference: the URL to send notifications • entities: an array of entities (mandatory). The JSON for each entity

contains id, type and isPattern. • attrs: an array of attribute names (strings) (optional).

• lastNotification: timestamp corresponding to the last notification sent associated to a given subscription.

• count: the number of notifications sent associated to the subscription. • format: the format to use to send notification, currently "JSON" meaning JSON

notifications in NGSIv1 format.

Example document:

LOGS LOG FILE The default log file is /tmp/contextBroker.log. Remember that the directory where the log file is stored (/tmp by default) can be changed using the -logDir command line option.

When starting the Orion context broker, if a previous log file exists:

• If -logAppend is used, then the log is appended to the existing file. • If -logAppend is not used, then the existing file is renamed, appending the text ".old" to

its name.

The -logLevel option allows to choose which error messages are printed in the log:

• NONE: no log at all • FATAL: only FATAL ERROR messages are logged • ERROR: only ERROR messages are logged • WARN (default): WARN and ERROR messages are logged • INFO: INFO, WARN and ERROR messages are logged • DEBUG: DEBUG, INFO, WARN and ERROR messages are logged

When Orion runs in foreground (i.e. with the -fg CLI argument), it also prints the same log traces (but in a simplified way) on the standard output.

The log level can be changed (and retrieved) in run-time, using the admin API exposed by Orion.

LOG FORMAT The log format is designed to be processed by tools like Splunk or Fluentd.

Each line in the log file is composed by several key-value fields, separed by the pipe character (|). Example:

The different fields in each line are as follows:

• time. A timestamp corresponding to the moment in which the log line was generated in ISO8601 format. Orion prints timestamps in UTC format.

• lvl (level). There are six levels: o FATAL: This level designates severe error events that lead the application to exit.

The process can no longer work.

o ERROR: This level designates error events. There is a severe problem that must be fixed.

o WARN: This level designates potentially harmful situations. There is a minor problem that should be fixed.

o INFO: This level designates informational messages that highlight the progress of Orion.

o DEBUG: This level designates fine-grained informational events that are most useful to debug an application. Only shown when tracelevels are in use (set with the -t command line option.

o SUMMARY: This is a special level used by log summary traces, enabled with the -logSummary CLI option. Have a look at the section on summary traces for details.

• corr (correlator id). Can be either "N/A" (for log messages "out of transaction", e.g. log lines corresponding to Orion Context Broker startup), or it is a string in the UUID format. An example: "550e8400-e29b-41d4-a716-446655440000". This 'correlator id' is either transferred from an incoming request, or, if the incoming request doesn't carry any HTTP header "Fiware-Correlator", the correlator is generated by the Orion context broker and then used in the log file (as well as sent as HTTP header in forwarding messages, notifications and responses). The correlator id is a common identifier among all applications involved in the 'message chain' for one specific request.

• trans (transaction id). Can be either "N/A" (for log messages "out of transaction", as the ones corresponding to Orion Context Broker startup) or a string in the format "1405598120-337-00000000001". The transaction id generation logic ensures that every transaction id is unique, also for Orion instances running in different VMs (which is useful in the case you are aggregating logs from different sources), except if they are started in the exact same millisecond. Note that transaction ID is independent of correlator ID. Transaction ID has a local nature while correlator ID is meaningful end-to-end, involving other software components apart from Context Broker itself. There are two types of transactions in Orion: o The ones initiated by an external client invoking the REST API exposed by Orion.

The first message on these transactions use the pattern "Starting transaction from url", where url includes the IP and port of the client invoking the operation and the path is the actual operation invoked at Orion. The last message on such transaction is "Transaction ended".

o The ones that Orion initiates when it sends a notification. The first message on these transactions use the pattern "Starting transaction to url", where url is the URL used in the reference element of the subscription, i.e. the URL of the callback to send the notification. The last message of both transaction types is "Transaction ended".

• from. Source IP of the HTTP request associated to the transaction, except if the request includes X-Forwarded-For header (which overrides the former) or X-Real-IP (which overrides X-Forwarded-For and source IP).

• srv. Service associated to the transaction, or "pending" if the transaction has started but the service has not been yet obtained.

• subsrv. Subservice associated to the transaction, or "pending" if the transaction has started but the subservice has not been yet obtained.

• comp (component). Current version always uses "Orion" in this field. • op. The function in the source code that generated the log message. This information

is useful for developers only. • msg (message). The actual log message. The text of the message includes the name

of the file and line number generating the trace (this information is useful mainly for Orion developers).

ALARMS Alarm conditions:

Alarm ID Severity Detection strategy Stop condition Description Action

1 CRITICAL A FATAL trace is found N/A A problem has occurred at Orion Context Broker startup. The FATAL 'msg' field details the particular problem.

Solving the issue that is precluding Orion Context Broker startup, e.g. if the problem was due to the listening port is being used, the solution would be either changing Orion listening port or ending the process that is already using the port.

2 CRITICAL The following ERROR text appears in the 'msg' field: "Runtime Error (<detail>)"

N/A Runtime Error. The <detail> text containts the detailed information.

Restart Orion Context Broker. If it persists (e.g. new Runtime Errors appear within the next hour), scale up the problem to development team.

3 CRITICAL The following ERROR text appears in the 'msg' field: "Raising alarm DatabaseError: <detail>"

The following ERROR text appears in the 'msg' field: "Releasing alarm DatabaseError". Orion prints this trace when it detects that DB is ok again."

Database Error. The <detail> text contains the detailed information.

Orion is unable to access MongoDB database and/or MongoDB database is not working properly. Check database connection and database status. Once the database is repaired and/or its connection to Orion, the problem should disappear (Orion service restart is not needed). No specific action has to be performed at Orion Context Broker service.

4 WARNING The following WARN text appear in the 'msg' field: "Raising alarm BadInput <ip>: <detail>".

The following WARN text appears in the 'msg' field: "Releasing alarm BadInput <ip>", where is the same one that triggered the alarm. Orion prints this trace when it receives a correct request from that client.

Bad Input. The <detail> text contains the detailed information.

The client has sent a request to Orion that doesn't conform to the API specification, e.g. bad URL, bad payload, syntax/semantic error in the request, etc. Depending on the IP, it could correspond to a platform client or to an external third-party client. In any case, the client owner should be reported in order to know and fix the issue. No specific action has to be performed at Orion Context Broker service.

5 WARNING The following WARN text appears in the 'msg' field: "Raising alarm NotificationError <url>: <detail>".

The following WARN text appears in the 'msg' field: "Releasing alarm NotificationError ", where is the same one that triggered the alarm. Orion prints this trace when it successfully sent a notification to that URL.

Notification Failure. The <detail>text contains the detailed information.

Orion is trying to send the notification to a given receiver and some problem has occurred. It could be due to a problem with the network connectivity or on the receiver, e.g. the receiver is down. In the second case, the owner of the receiver of the notification should be reported. No specific action has to be performed at Orion Context Broker service.

By default, Orion only traces the origin (i.e. raising) and end (i.e. releasing) of an alarm, e.g:

This means that if the condition that triggered the alarm (e.g. a new invalid request from the 10.0.0.1 client) occurs again between the raising and releasing alarm messages, it wouldn't be traced again. However, this behaviour can be changed using the -relogAlarms CLI parameter. When -relogAlarms is used, a log trace is printed every time a triggering condition happens, e.g:

Log traces between "Raising" and "Releasing" messages use "Repeated" in the message text. Note that the details part of the message is not necessarily the same in all traces, so re-logging alarms could be a means to get extra information when debugging problems. In the example above, it could correspond to a client that after fixing the problem with the JSON payload now has a new problem with the URL of the Orion API operation.

SUMMARY TRACES You can enable log summary traces with the -logSummary CLI parameter which value is the summary reporting period in seconds. For example -logSummary 5 involves that summary traces will be print each 5 seconds (no matter which log level has been set with -logLevel).

Four traces are printed each time, as follows (the lines have been abbreviated, omitting some fields, for the sake of clarity):

• First line (Transactions) shows the current number of transactions and the new ones in the last summary reporting period.

• Second line is about DB alarms. It shows the current DB status (either "ok" or "erroneous"), the number of raised DB alarms (both total since Orion started and in the

last summary reporting period) and the number of released DB alarms (both total since Orion started and in the last summary reporting period).

• Third line is about notification failure alarms. It shows the current number of active notification failure alarms, the number of raised notification failure alarms (both total since Orion started and in the last summary reporting period) and the number of released notification failure alarms (both total since Orion started and in the last summary reporting period).

• Fourth line is about bad input alarms. It shows the current number of bad input alarms, the number of raised bad input alarms (both total since Orion started and in the last summary reporting period) and the number of released bad input alarms (both total since Orion started and in the last summary reporting period).

LOG ROTATION Logrotate is installed as an RPM dependency along with the Context Broker. The system is configured to rotate once a day, or more, in case the log file size exceeds 100MB (checked very 30 minutes by default):

• For daily rotation: /etc/logrotate.d/logrotate-contextBroker-daily: which enables daily log rotation

• For size-based rotation: o /etc/sysconfig/logrotate-contextBroker-size: in addition to the previous

rotation, this file ensures log rotation if the log file grows beyond a given threshold (100 MB by default)

o /etc/cron.d/cron-logrotate-contextBroker-size: which ensures the execution of etc/sysconfig/logrotate-contextBroker-size at a regular frequency (default is 30 minutes)

Depending on your expected load you would need to adjust the default settings. In this sense, take into account that in INFO log level every transaction can consume around 1-2 KB (measured with Orion 0.14.1), e.g. if your expected load is around 200 TPS, then your log file will grow 200-400 KB per second.

WATCHDOG

Although Orion Context Broker is highly stable, it may fail (see the section on diagnosis procedures for more information about detecting problems with the broker). Thus, it is recommendable to use a watchdog process to detect if the broker process has stopped running, so it can be re-started automatically and/or you get a notification of the problem.

You can write the watchdog program yourself (e.g. a script invoked by cron in a regularly basic that checks /etc/init.d/contextBroker status and starts it again if is not working and/or send you a notification email) or use existing tools. This section includes a procedure using Monit.

First of all, install the RPMs available at http://rpmfind.net/linux/rpm2html/search.php?query=monit. The following procedure has been prepared considering monit-5.1.1-4.el6.x86_64.rpm, although it also should work with other versions of the RPM.

Create a directory for monit stuff, eg:

Create monitBROKER.conf file in that directory. In this example, we configure monit to restart contextBroker if CPU load is greater than 80% for two cycles or if allocated memory is greater than 200MB for five cycles (that would be a symptom of memory leaking). In addition to resource checking, monit will restart the process if it is down. The duration of a cycle is defined using a monit command line parameter (described below).

Make root the owner of that file and set permissions only for owner:

Create monit start script start_monit_BROKER.sh. The "-d" command line parameter is used to specify the checking cycle duration (in the example we are setting 10 seconds).

Make root the owner of that file and set execution permissions:

To run monit do:

To check that monit is working properly, check that the process exist, e.g.:

Then, kill Context Broker, e.g.:

and check with ps that after a while (less than 30 seconds) Context Broker is up again.

RUSH RELAYER

Apart from running Orion Context Broker in "stand alone" mode, you can also take advanage of Rush as notification relayer. Thus, instead of managing the notifications itself (including waiting for the HTTP timeout while the notification receives responses), Orion passes the notification to Rush, which in turn deals with it. Thus, Orion can implement a

"fire and forget" policy for notification sending, realying in Rush (a piece of sofware hihgly specialized in that task) for that.

In addition, you can send notifications using HTTPS using Rush ,see security section in Users and Programmers manual.

In order to use Rush you need:

• A running Rush instance network-reachable from Orion, e.g. in the same host and reachable using "localhost". The installation of Rush is out the scope of this manual, please check the Rush documentation for that.

• Run Orion using the -rush command line interface, which value has to be the Rush host and port, eg. -rush localhost:1234 means that Rush is listening in port 1234 in localhost.

HTTP Rush relayer is used only for notifications. Other cases in which Orion acts as HTTP client (eg. forwarding query/updates to Context Providers) doesn't use Rush and Orion always sends the HTTP request itself.

MANAGEMENT REST INTERFACE

LOG AND TRACE LEVELS Apart from the NGSI interface, Orion Context Broker exposes a REST API for management that allows to change the log level and the trace levels (whose initial value is set using -t and -logLevel command line options).

To change the log level:

To retrieve the log level:

which response follows the following pattern:

To manage the trace level:

'PUT-requests' overwrite the previous log settings. So, in order to ADD a trace level, a GET /log/trace must be issued first and after that the complete trace string to be sent in the PUT request can be assembled.

TRACELEVEL RELATED WITH INPUT/OUTPUT PAYLOADS The following traceleves are particularly useful in order to make Orion print input/output payload in traces:

Thus, you can enable all them using:

SEMAPHORES Another useful (especially if the broker stops responding correctly) REST API is the semaphore listing offered:

The response is a listing of information of all the broker's semaphores:

Short explanation of the semaphores: alarmMgr, protects the data of the Alarm Manager connectionContext, protects the curl context for sending HTTP

notifications/forwarded messages connectionEndpoints, protects the curl endpoints when sending HTTP notifications/forwarded messages. dbConnectionPool, protects mongo connection pool dbConnection, protects the set of connections of the mongo connection pool logMsg, makes sure that not two messages are written simultaneously to the log-file metrics, protects internal data of the Metrics Manager request, makes sure there are not two simultaneous requests to mongodb subCache, protects the Subscription Cache timeStat, protects the data for timing statistics *transaction, protects the Transaction ID (for the log-file)

The information supplied for each of the semaphores is: * status: free or taken.

[At the moment, only one item per semaphore but the idea is to add more information in the future]

METRICS API

INTRODUCTION Orion implements a REST-based API that can be used to get relevant operational metrics. This API is a complement to the statistics API which is more low-level and aimed at debugging.

Note that this API can be switched off to avoid overhead (gathering of metrics is a bit costly, as system calls and semaphores are involved) using the -disableMetrics CLI parameter.

OPERATIONS Get metrics

The response payload is a multi-level JSON tree storing the information in a structured way. It is based on the service and subservice (sometimes referred to as "service path"). At any point of the tree, the value of a key could be {} to mean that there isn't actual information associated to that key.

At the first level there are two keys: services and sum. In sequence, services value is an object whose keys are service names and whose values are objects with information about the corresponding service. The sum value is an object with information for the aggregated information for all services.

Regarding service information objects, they use two keys: subservs and sum. In sequence, subservs value is an object whose keys are subservice names and whose values are objects with information about the corresponding subservice. The sum value is an object with information for the aggregated information for all subservices in the given services.

Subservice names in the above structure are shown without the initial slash. E.g. if the subservice name is (as used in the Fiware-ServicePath header) /gardens then the key used for it would be gardens (without /). Others slashes, apart the initial, one are not removed, e.g. /gardens/north uses the key gardens/north.

Regarding subservice information object, keys are the name of the different metrics.

The list of metrics is provided in metrics section.

Some additional remarks:

• Requests corresponding to invalid services or subservices (i.e. the ones that don't follow the syntax rules described here and here) are not included in the payload (i.e. their associated metrics are just ignored).

• The default service uses default-service as service key. Note that it cannot collide with regular services as the - character is not allowed in regular services.

• The root subservice (/) uses root-subserv as subservice key. Note that it cannot collide with regular subservices as the - character is not allowed in regular subservices.

• Requests using an enumeration of subservices (e.g. Fiware-ServicePath: /A, /B) are associated to the first element of the list, i.e. A.

• Requests using "recursive subservice" (e.g. Fiware-ServicePath: /A/#) are associated to the subservice without considering recursivity, i.e. A.

RESET METRICS

This operation resets all metrics, as if Orion would had just been started.

GET AND RESET

This operation (in fact, a variant of get metrics) get results and, at the same time in an atomic way, resets metrics.

METRICS

• incomingTransactions: number of requests consumed by Orion. All kind of transactions (no matter if they are ok transactions or error transactions) count for this metric.

• incomingTransactionRequestSize: total size (bytes) in requests associated to incoming transactions ("in" from the point of view of Orion). All kind of transactions (no matter if they are ok transactions or error transactions) count for this metric.

• incomingTransactionResponseSize: total size (bytes) in responses associated to incoming transactions ("out" from the point of view of Orion). All kind of transactions (no matter if they are ok transactions or error transactions) count for this metric.

• incomingTransactionErrors: number of incoming transactions resulting in error. • serviceTime: average time to serve a transaction. All kind of transactions (no matter if

they are ok transactions or error transactions) count for this metric.

• outgoingTransactions: number of requests sent by Orion (both notifications and forward requests to CPrs). All kind of transactions (no matter if they are ok transactions or error transactions) count for this metric.

• outgoingTransactionRequestSize: total size (bytes) in requests associated to outgoing transactions ("out" from the point of view of Orion). All kind of transactions (no matter if they are ok transactions or error transactions) count for this metric.

• outgoingTransactionResponseSize: total size (bytes) in responses associated to outgoing transactions ("in" from the point of view of Orion). All kind of transactions (no matter if they are ok transactions or error transactions) count for this metric.

• outgoingTransactionErrors: number of outgoing transactions resulting in error.

STATISTICS

Warning: This section of the manual is yet work in progress, so the current Orion implementation may not be fully aligned with it. In addition, take into account that statistics API is by the moment in beta status, so changes in the name of the different counters or the JSON structure could take place in the future.

Orion Context Broker provides a set of statistics (mainly for testing and debugging) through the GET /statistics and GET /cache/statistics operations. These operations support JSON encoding only.

For now, statistics accuracy is not 100% guaranteed under high load conditions. Atomic counters are not used in all places, so race conditions on requests running at the same time could (theoretically) lead to losing some data. However, under low load conditions, statistics should be accurate. There is an open issue about this.

GET /STATISTICS The statistics JSON is composed of four conditional blocks and two unconditional fields:

Conditional blocks are enabled using -statXXX flags at command line (in the future, we may implement the possibility to activate/deactivate them using the management API. This is due to two reasons. First, to avoid too verbose statistics output for users that are not interested in some parts. Second, measuring some of the information might involve a performance penalty (in general, to measure time always needs additional system calls) so explicit activation is desirable.

• "counters" (enabled with the -statCounters) • "semWait" (enabled with the -statSemWait) • "timing" (enabled with the -statTiming) • "notifQueue" (enabled with the -statNotifQueue)

Unconditional fields are:

• uptime_in_secs, Orion uptime in seconds. • measuring_interval_in_secs, statistics measuring time in seconds. It is set to 0

each time statistics are reset. If statistics have not been reset since Orion start, this field matches uptime_in_secs.

COUNTER BLOCK The counter block provides information about counters for the times a particular request type has been received (also for notifications being sent), e.g:

If a particular request type has not been received, its corresponding counter is not shown.

SEMWAIT BLOCK The SemWait block provides accumulates waiting time for the main internal semaphores. It can be useful to detect bottlenecks, e.g. if your DB is too slow or your DB pool is undersized, then dbConnectionPool time would be abnormally high (see section on performance tunning for more details).

TIMING BLOCK Provides timing information, i.e. the time that CB passes executing in different internal modules.

The block includes two main sections:

• last: times corresponding to the last request processed. If the last request does not use a particular module (e.g. a GET request doesn't use parsing), then that counter is 0 and it is not shown.

• accumulated: accumulated time corresponding to all requests since the broker was started.

The particular counters are as follows:

• total: processing time for the whole request, excluding the time that the HTTP library takes for request/response dispatching (pseudo end-to-end time)

• jsonV1Parse: time passed in NGSIv1 JSON parsing module (pseudo self-time) • jsonV2Parse: time passed in NGSIv2 JSON parsing module (pseudo self-time) • mongoBackend: time passed in mongoBackend module (pseduo self-time) • render: time passed in rendering module (pseudo self-time) • mongo*Wait: time passed waiting for MongoDB for Read, Write or Cmd operations.

Note that if a given request involves several read/write/cmd calls to MongoDB, the time shown in mongo*Wait under last includes the accumulation for all of them. In the case of mongoReadWait, only the time used to get the results cursor is taken into account,

but not the time to process cursors results (which is time that belongs to mongoBackend counters).

Times are measured from the point in time in which a particular thread request starts using a module until it finishes using it. Thus, if the thread is stopped for some reason (e.g. the kernel decides to give priority to another thread based on its scheculing policy) the time that the thread was sleeping, waiting to execute again is included in the measurement and thus, the measurement is not accurate. That is why we say pseudo selt/end-to-end time. However, under low load conditions this situation is not expected to have a significant impact.

NOTIFQUEUE BLOCK Provides information related to the notification queue used in the thread pool notification mode. Thus, it is only shown if -notificationMode is set to threadpool.

The particular counters are as follows:

• avgTimeInQueue: average time that each notification waits in the queue (equal to timeInQueue divided by out)

• in: number of notifications that get into the queue • out: numbers of notifications that get out of the queue • reject: number of notifications that get rejected, due to queue full. In other words,

notifications that are not even enqueued. • sentOk: number of successfully sent notifications • sentError: number of unsuccessful notification-attempts • timeInQueue: accumulated time of notifications waiting in queue • size: current size of the queue

GET /CACHE/STATISTICS Provides counters for the context subscription cache operations (refresh, insert, remove and update), along with the current number of cached items.

Note that the "ids" field could get extremely long. To avoid a too long response, the broker sets a limit of the size of the 'ids' field. If the length is longer than that limit, instead of presenting the complete list of subscription-identifiers, the text "too many subscriptions" is presented instead.

RESETING STATISTICS To reset the statistics counters (note that the fields that come from the state of the system are not reset, e.g. subs cache items or notification queue size) just invoke the DELETE operation on the statistics URL:

• DELETE /statistics • DELETE /cache/statistics

Resetting statistics yields the following response:

PERFORMANCE TUNING

MONGODB CONFIGURATION From a performance point of view, it is recommended to use MongoDB 3.6 with WireTiger, especially in update-intensive scenarios.

In addition, take into account the following information from the official MongoDB documentation, as it may have impact on performance:

• Check that ulimit settings in your system are ok. MongoDB provides the following recomendations As described in that document, in RHEL/CentOS you have to create a /etc/security/limits.d/99-mongodb-nproc.conf file, in order to set soft/hard process limit to at least 32000 (check details in the cited document).

• You should also disable Transparent Huge Pages (HTP) to increment the performance as explain in this document.

DATABASE INDEXES Orion Context Broker doesn't create any index in any database collection (with one exception, described at the end of this section), to give flexibility to database administrators. Take into account that index usage involves a tradeoff between read efficiency (usage of indexes generally speeds up reads) and write efficiency (the usage of indexes slows down writes) and storage (indexes consume space in database and mapped RAM memory) and it is the administrator (not Orion) who has to decide what to prioritize.

However, in order to help administrators in this task, the following indexes are recommended:

• Collection entities o _id.id

o _id.type

o _id.servicePath

o attrNames

o creDate

In the case of using orderBy queries (i.e. GET /v2/entities?orderBy=A), it is also recommended to create indexes for them. In particular, if you are ordering by a given attribute 'A' in ascending order (i.e. orderBy=A) you should create an index {attrs.A.value: 1}. In the case of ordering by a given attribute 'A' in descending order (i.e. orderBy=!A) you should create an index {attrs.A.value: -1}.

The only index that Orion Context Broker actually ensures is the "2dsphere" in the location.coords field in the entities collection, due to functional needs geo-location functionality. The index is ensured on Orion startup or when entities are created.

You can find an analysis about the effect of indexes in this document, although it is based on an old Orion version, so it is probably outdated.

WRITE CONCERN Write concern is a parameter for MongoDB write operations. By default, Orion uses "acknowledged" write concern which means that Orion waits until MongoDB confirms it has applied the operation in memory. You can change this behavior with the -writeConcern CLI option. When "unacknowledged" write concern is configured, Orion doesn't wait to get confirmation, so it can execute write-operations much faster.

Note however that there is a tradeoff between performance and reliability. Using "unacknowledged" write concern you get better performance, but the risk to lose information is higher (as Orion doesn't get any confirmation that the write operation was successful).

NOTIFICATION MODES AND PERFORMANCE Orion can use different notification modes, depending on the value of -notificationMode.

Default mode is 'transient'. In this mode, each time a notification is sent, a new thread is created to deal with the notification. Once the notification is sent and the response is received, the thread with its connection context is destroyed. This is the recommended mode for low load scenarios. In high level cases, it might lead to a thread exhaustion problem.

Permanent mode is similar, except that the connection context is not destroyed at the end. Thus, new notifications associated to the same connection context (i.e. the same destination URL) can reuse the connection and save the HTTP connection time (i.e. TCP handshake, etc.). Of course, this requires that the server (i.e. the component that receives the notification) also maintains the connection open. Note that while in some cases the permanent mode could improve performance (as it saves the time required to create and destroy HTTP connections), in others it may cause notifications associated to the same connection context to have to wait (as only one of them may use the connection at a time). In other words, only one notification thread can use the connection at a time, so if the notification request/response transmission time exceeds the notification inter-triggering time then threads will block. You can detect this situation when the connectionContext value in statistics is abnormally high.

Finally, threadpool mode is based on a queue for notifications and a pool of worker threads that take notifications from the queue and actually send them on the wire, as shown in the figure below. This is the recommended mode for high load scenarios, after a careful tuning on the queue length and the number of workers. A good starting point is to

set the number of workers to the number of expected concurrent clients that send updates, and the queue-limit as N times the number of workers (e.g. N equal to 10, although it could be more or less depending on the expected update burst length). The statistics on the notifQueue block may help you to tune.

HTTP SERVER TUNING The REST API that Orion implements is provided by an HTTP server listening on port 1026 by default (this can be overridden by the -port CLI parameter). You can tune its behavior using the following CLI parameters (see details in the corresponding document):

• connectionMemory. Sets the size of the connection memory buffer (in kB) per connection used internally by the HTTP server library. Default value is 64 kB.

• maxConnections. Maximum number of simultaneous connections. Default value is 1020, for legacy reasons, while the lower limit is 1 and there is no upper limit (limited by max number of file descriptors of the operating system).

• reqPoolSize. Size of thread pool for incoming connections. Default value is 0, meaning no thread pool at all, i.e., a new thread is created to manage each new incoming HTTP request and destroyed after its use. Thread pool mode uses internally the epoll() system call, which is more efficient than the one used when no thread pool is used (poll()). Some performance information regarding this can be found in the documentation of the HTTP server library itself.

• reqTimeout. The inactivity timeout in seconds before a connection is closed. Default value is 0 seconds, which means infinity. This is the recommended behaviour and setting it to a non-infinite timeout could cause Orion to close the connection before

completing the request (e.g. a query request involving several CPr forwards can take a long time). This could be considered an "HTTP-unpolite" behaviour from the point of view of the server (Orion) given that, in this case, it should be the client who decides to close the connection. However, this parameter may be used to limit resource consumption at server side (Orion). Use it with care.

Given that thread creation and destruction are costly operations, it is recommended to use -reqPoolSize in high load scenarios. In particular, according to MHD feedback, the pool should be sized with a value equal or close to number of available CPU cores. If you set -reqPoolSize to a value higher than number of CPU cores then you'll most probably experience performance decrease.

The other three parameters (-reqTimeout, -maxConnections and -connectionMemory) usually work well with their default values.

ORION THREAD MODEL AND ITS IMPLICATIONS Orion is a multithread process. With default starting parameters and in idle state (i.e. no load), Orion consumes 4 threads:

• Main thread (the one that starts the broker, then sleeps forever) • Subscription cache synchronization thread (if -noCache is used then this thread is not

created) • Listening thread for the IPv4 server (if -ipv6 is used then this thread is not created)

• Listening thread for the IPv6 server (if -ipv4 is used then this thread is not created)

In busy state, the number of threads will be higher. With default configuration, Orion creates a new thread for each incoming request and for each outgoing notification. These threads are destroyed once their work finalizes.

The default configuration is fine for low to medium load scenarios. In high load scenarios you may have a large number of simultaneous requests and notifications so the number of threads may reach the per process operating system level. This is known as the thread exhaustion problem and will cause Orion to not work properly, being unable to deal with new incoming request and outgoing notifications. You can detect that situation by two symptoms.

• First, a number of threads associated to the process very close to the per process operating system limit.

• Second, error messages like this appearing in the logs:

Runtime Error (error creating thread: ...)

In order to avoid this problem, Orion supports thread pools. Using thread pools you can statically set the number of threads that the Orion process uses, removing the dynamics of thread creation/destruction the thread exhaustion problem is avoided. In other words, pools make the behavior of Orion more predictable, as a way of guaranteeing that the Orion process doesn't go beyond the per process operating system thread limit.

There are two pools that can be configured independently:

• Incoming requests pool. Set by the -reqPoolSize c parameter, being c the number of threads in this pool. See HTTP server tuning section in this page for more information.

• Notifications pool. Set by -notificationMode threadpool:q:n, being n the number of threads in this pool. See notification modes and performance section in this page.

Using both parameters, in any situation (either idle or busy) Orion consumes a fixed number of threads:

• Main thread (the one that starts the broker, then sleeps forever) • Subscription cache synchronization thread (if -noCache is used then this thread is not

created)

• c listening threads for the IPv4 server (if -ipv6 is used then these threads are not created)

• c listening threads for the IPv6 server (if -ipv4 is used then these threads are not created)

• n threads corresponding to the workers in the notification thread pool.

Apart from avoiding the thread exhaustion problem, there is a trade-off between using thread pools and not. On the one side, using thread pools is beneficial as it saves thread creation/destruction time. On the other hand, setting thread pools is a way of "capping" throughput. If the thread workers are busy all the time, at the end the queue saturates and you will end up losing ongoing notifications.

FILE DESCRIPTORS SIZING The following inequity ensures the number of file descriptors used by Orion is below the operating system limit:

where

• max fds is the per process file descriptors limit, i.e. the output of the ulimit -n command. It can be changed with ulimit -n <new limit>.

• n, number of threads in the notification thread pool. The factor 5 is due to that each thread can hold up to 5 connections (libcurl pool).

• max cons is the size of the thread pool for incoming connections, configured with -reqPoolSize CLI parameter. Note that if you don't use this parameter, default is not using any pool for incoming connections. Thus, a burst of incoming connections large enough could exhaust in theory all available file descriptors.

• db pool size is the size of the DB connection pool, configured with -dbPoolSize CLI parameter, which default value is 10.

• extra an amount of file descriptors used by log files, listening sockets and file descriptors used by libraries. There is not a general rule for this value, but one in the range of 100 to 200 must suffice most of the cases.

If the above inequity doesn't hold, you may have file descriptors exhaustion problem and Orion Context Broker will not work properly. In particular, it may happen that Orion is unable to accept new incoming connections and/or send notifications due to lack of file descriptors.

Note that having a large number of client connections at Orion in CLOSE_WAIT status is not a problem. This is part of the libcurl connection cache strategy, in order to save time by reusing connections. From libcurl email discussion about this topic:

The CLOSE_WAIT sockets are probably the ones that libcurl has in its connection cache but that have been "closed" (a FIN was sent) by the server already but not yet by libcurl. They are not there "indefinitely" (and they really can't be) since the connection cache has a limited size so eventually the old connections should get closed.

IDENTIFYING BOTTLENECKS LOOKING AT SEMWAIT STATISTICS The semWait section in the statistics operation output includes valuable information that can be used to detect potential bottlenecks.

• connectionContext. An abnormally high value in this metric may be due to that many notifications want to use the same permanent connection. In that case, stop using permanent notification mode and use transient or threadpool instead (note that the value of this metric is always 0 if permanent notification mode is not used).

• dbConnectionPool. Orion keeps a DB connection pool (which size is established with -dbPoolSize). An abnormally high value of this metric means that Orion threads wait too much to get a connection from the pool. This could be due to the size of the pool is insufficient (in that case, increase the value of -dbPoolSize) or that there is some other bottleneck with the DB (in that case, review your DB setup and configuration).

• request. An abnormally high value in this metric means that threads wait too much before entering the internal logic module that processes the request. In that case, consider to use the "none" policy (note that the value of this metric is always 0 if "none" policy is used). Have a look at the section on mutex policy.

Other metrics (timeStat, transaction and subCache) are for internal low-level semaphores. These metrics are mainly for Orion developers, to help to identify bugs in the code. Their values shouldn't be too high.

LOG IMPACT ON PERFORMANCE Logs can have a severe impact on performance. Thus, in high level scenarios, it is recommended to use -logLevel ERROR or WARN. We have found in some situations that the saving between -logLevel WARN and -logLevel INFO can be around 50% in performance.

METRICS IMPACT ON PERFORMANCE Metrics measurement may have an impact on performance, as system calls and semaphores are involved. You can disable this feature (thus improving performance) using the -disableMetrics CLI parameter.

MUTEX POLICY IMPACT ON PERFORMANCE Orion supports four different policies (configurable with -reqMutexPolicy):

• "all", which ensures that not more than one request is being processed by the internal logic module at the same time

• "read", which ensures that in a given CB node not more than one read request is being processed by the internal logic module at the same time - write requests can execute concurrently.

• "write", which ensures that in a given CB node not more than one write request is being processed by the internal logic module at the same time, read requests can execute concurrently

• "none", which allows all the requests to be executed concurrently.

Default value is "all", mainly due to legacy reasons (a leftover of the times in which some race condition issues may occur). However, for the time being, "none" can safely be used, leading to a better performance (as no thread is blocked waiting for others at the internal logic module entry). In fact, in Active-Active Orion configuration, using something different than "none" doesn't provide any advantage (as the mutex policy is local to the Orion process).

OUTGOING HTTP CONNECTION TIMEOUT It may happen that a given notification receiver or a context provider (to which a query/update has been forwarded) takes too long to respond to an HTTP request. In some cases, the receiver is not even listening, so a long timeout (the default one established by the operating system) has to pass before the request can be considered a failure and the sending thread unblocks. This may have a significant impact.

In the case of notifications, it causes that the thread (either transients, persistent or in the thread pool) is blocked. In transient or persistent mode, it involves an idle thread inside the process, counting toward the maximum per-process thread limit but doing no effective work (this can be especially severe in the case of persistent mode, as it will block other notifications trying to send to the same URL). In the second case, it means there are workers in the pool that cannot take on new work while waiting.

In the case of queries/updates forwarded to context providers, the effect is that the original client will take a long time to get the answer. In fact, some clients may give up and close the connection.

In this kind of situations, the -httpTimeout CLI parameter may help to control how long Orion should wait for outgoing HTTP connections, overriding the default operating system timeout.

SUBSCRIPTION CACHE Orion implements a context subscription cache in order to speed up notification triggering. In the current version (this may change in the future), context availability subscriptions do not use any cache.

The cache synchronization period is controlled by the -subCacheIval (by default it is 60 seconds). Synchronization involves two different tasks:

• Reading for changes in the context subscription collection in the database and update the local cache based on it. Note that in a multi-CB configuration, one node may modify the context subscription collection, so this is the way other nodes get aware of the modification.

• Writing some transient information associated to each subscription into the database. This means that even in mono-CB configurations, you should use a -subCacheIval different from 0 (-subCacheIval 0 is allowed, but not recommended).

Note that in multi-CB configurations with load balancing, it may pass sometime between (whose upper limit is the cache refresh interval) a given client sends a notification and all CB nodes get aware of it. During this period, only one CB (the one which processed the subscription and have it in its cache) will trigger notifications based on it. Thus, CB implements "eventual consistency" regarding this.

Note also that there is a tradeoff with the cache refresh interval. Short intervals mean that changes regarding context subscriptions are propagated faster between CB nodes (i.e. it has to pass less time to pass from "eventual consistency" to full consistency) but there is more stress on CB and DB. Large intervals mean that changes take more time to propagate, but the stress on CB and DB is lower.

As a final note, you can disable cache completely using the -noCache CLI option, but that is not a recommended configuration.

GEO-SUBSCRIPTION PERFORMANCE CONSIDERATIONS Current support of georel, geometry and coords expression fields in NGSIv2 subscriptions (aka geo-subscriptions) relies on MongoDB geo-query capabilities. While all other conditions associated to subscriptions (e.g. query filter, etc.) are evaluated on a memory image of the updated entity, the ones related with the georel, geometry and coords of a given subscription need a query in the DB.

However, note that the impact on performance shouldn't be too heavy (the operation invoked in MongoDB is count() which is relatively light).

Our future plan is to implement geo-subscription matching in memory (as the rest of the conditions), but this is not a priority at the moment.

CREATE AN ORION CONTEXT BROKER CLUSTER IN HIGH AVAILABILITY

This documentation describe how to set-up on a generic infrastructure an Orion Context Broker cluster in High Availability.

The architecture is made by three logical layers: A Load Balancer (implemented using HA Proxy) The Orion Context Broker * The MongoDB ReplicaSet backend

The ideal solution for providing Active-Active access to the HA Proxy cluster (and consequently to the Context Broker) is the adoption of a DNS service that supports multiple IPs for a single name. This will allow clients to use a round-robin solution to pick the actual host to connect with, in case of failure, the second will be used and so on. The alternative is the usage of a VIP mechanism to provide Active-Passive access to the HA Proxy cluster. In this case, the active instance of HA proxy will be only one, in case of a failure of the active HA Proxy, one of the other HA Proxy will take on the Virtual IP.

The picture below present the network architecture of the deployed cluster.

In the following we describe the different steps to configure such a cluster on a set of virtual machines. In docker related documentation, we shortly discuss how to create the same architecture leveraging on a Docker Swarm cluster.

SERVICES CONFIGURATION The configuration discussed below assume you have one server per each box represented in the architecture. Of course, different layers, e.g. HA Proxy and Context Broker can be combined together on a single server.

Requirements: 3 VMs running Centos for the Context Broker layer. 6 VMs running Ubuntu 16.04 LTS for the HA Proxy and MongoDB layers.

MONGODB CONFIGURATION For each server:

1. Configure the Firewall to block access from outside the local network except for SSH and on local network for 27017

```bash $ nano /etc/iptables.rules

*filter :INPUT DROP [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :LOGGING - [0:0]

COMMIT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -s 10.0.64.0/25 -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 27017 -s 10.0.64.0/25 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT COMMIT

$ iptables-restore < /etc/iptables.rules ```

2. Configure the hosts file to include local network hostname resolution

```bash $ nano /etc/hosts

10.0.64.32 mongo1 10.0.64.33 mongo2 10.0.64.34 mongo3

10.0.64.35 contextbroker1 10.0.64.36 contextbroker2 10.0.64.37 contextbroker3

10.0.64.38 proxy1 10.0.64.39 proxy2 10.0.64.40 proxy3 ``` 3. Install MongoDB 3.2 using the following commands. WARNING: this test was done

when Mongo DB 3.2 was supported in Orion, test should be redone with the current version.

```bash $ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927 $ echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

$ sudo apt-get update

$ sudo apt-get install -y mongodb-org ``` 4. Define the ReplicaSet configuration in the MongoDb configuration file /etc/mongodb.conf,

comment the bindIp directive to allow connection from localhost and 10.0.64.0/25 network:

```bash $ nano /etc/mongodb.conf

MONGOD.CONF

FOR DOCUMENTATION OF ALL OPTIONS, SEE:

HTTP://DOCS.MONGODB.ORG/MANUAL/REFERENCE/CONFIGURATION-OPTIONS/

WHERE AND HOW TO STORE DATA. storage: dbPath: /var/lib/mongodb journal: enabled: true

ENGINE:

MMAPV1:

WIREDTIGER:

WHERE TO WRITE LOGGING DATA. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log

NETWORK INTERFACES net: port: 27017

BINDIP: 10.0.64.32 replication: replSetName: contextBroker ```

5. Start all the MongoDB

bash service mongod start On the primary node (you choose, during the set-up it was used mongo1):

1. Set the export lang

bash EXPORT LC_ALL=C 2. Connect to mongodb

bash mongo 3. Provide the configuration of the Primary node

config = { _id: "contextBroker", members: [ { _id: 0, host: "mongo1:27017" } ] 4. Initialise the replica set

rs.initiate(config);

5. Add the other nodes

rs.add("mongo2:27017"); rs.add("mongo3:27017"); 6. Check that the status of the cluster is correct

``` rs.status();

{ "set" : "contextBroker", "date" : ISODate("2017-02-02T17:25:57.605Z"), "myState" : 2, "term" : NumberLong(5), "syncingTo" : "mongo2:27017", "heartbeatIntervalMillis" : NumberLong(2000), "members" : [ { "_id" : 0, "name" : "mongo1:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3967, "optime" : { "ts" : Timestamp(1486052399, 1), "t" : NumberLong(5) }, "optimeDate" : ISODate("2017-02-02T16:19:59Z"), "syncingTo" : "mongo2:27017", "configVersion" : 3, "self" : true }, { "_id" : 1, "name" : "mongo3:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3929, "optime" : { "ts" : Timestamp(1486052399, 1), "t" : NumberLong(5) }, "optimeDate" : ISODate("2017-02-02T16:19:59Z"), "lastHeartbeat" : ISODate("2017-02-02T17:25:55.622Z"), "lastHeartbeatRecv" : ISODate("2017-02-02T17:25:55.622Z"), "pingMs" : NumberLong(0), "syncingTo" : "mongo1:27017", "configVersion" : 3 }, { "_id" : 2, "name" : "mongo2:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 3967, "optime" : { "ts" : Timestamp(1486052399, 1), "t" : NumberLong(5) }, "optimeDate" : ISODate("2017-02-02T16:19:59Z"), "lastHeartbeat" : ISODate("2017-02-02T17:25:55.623Z"), "lastHeartbeatRecv" : ISODate("2017-02-02T17:25:55.623Z"), "pingMs" : NumberLong(0), "electionTime" : Timestamp(1486052398, 1), "electionDate" : ISODate("2017-02-02T16:19:58Z"), "configVersion" : 3 } ], "ok" : 1 } ```

CONTEXT BROKER CONFIGURATION For each server:

1. Configure the Firewall to block access from outside the local network except for SSH and on local network for 1026

```bash $ nano /etc/sysconfig/iptables

FIREWALL CONFIGURATION WRITTEN BY SYSTEM-CONFIG-FIREWALL

MANUAL CUSTOMIZATION OF THIS FILE IS NOT RECOMMENDED. *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j

ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 1026 -s 10.0.64.0/25 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT

$ iptables-restore < /etc/sysconfig/iptables ``` 2. Configure the hosts file to include local network hostname resolution

```bash $ nano /etc/hosts

10.0.64.32 mongo1 10.0.64.33 mongo2 10.0.64.34 mongo3

10.0.64.35 contextbroker1 10.0.64.36 contextbroker2 10.0.64.37 contextbroker3

10.0.64.38 proxy1 10.0.64.39 proxy2 10.0.64.40 proxy3 ``` 3. Install the Context Broker with the following commands

```bash $ nano /etc/yum.repos.d/fiware.repo

[fiware] name=Fiware Repository baseurl=http://repositories.lab.fiware.org/repo/rpm/$releasever gpgcheck=0 enabled=1

$ yum install contextBroker ``` 4. Configure the Context Broker by listing all the mongodb hosts and the name of the

replica set

```bash $ nano /etc/sysconfig/contextBroker

BROKER_DATABASE_HOST=mongo1,mongo2,mongo3 BROKER_DATABASE_RPLSET=contextBroker ```

5. Start the Context Broker

bash $ /etc/init.d/contextBroker start 6. Test the Context Broker

bash $ curl localhost:1026/version -s -S HA PROXY CONFIGURATION 1. Configure the Firewall to block access from outside the local network except for SSH

and for the 1026 that will be used to load balance Context Broker traffic

```bash $ nano /etc/iptables.rules

*filter :INPUT DROP [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :LOGGING - [0:0]

COMMIT

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -s 10.0.64.0/25 -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 1026 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT COMMIT

$ iptables-restore < /etc/iptables.rules ``` 2. Configure the hosts file to include local network hostname resolution

```bash $ nano /etc/hosts

10.0.64.32 mongo1 10.0.64.33 mongo2 10.0.64.34 mongo3

10.0.64.35 contextbroker1 10.0.64.36 contextbroker2 10.0.64.37 contextbroker3

10.0.64.38 proxy1 10.0.64.39 proxy2 10.0.64.40 proxy3 ``` 3. Install the HA Proxy

bash $ sudo apt-get install haproxy 4. Configure the HA Proxy

```bash $ nano /etc/haproxy/haproxy.cfg

frontend www bind *:1026 default_backend ctx_pool

backend ctx_pool balance roundrobin mode http server ctx1 contextbroker1:1026 check server ctx2 contextbroker2:1026 check server ctx3 contextbroker3:1026 check ```

5. Enable the automatic restart of HA Proxy on reboot

```bash $ nano /etc/default/haproxy

ENABLED=1 ``` 6. Start the HA Proxy

bash $ service haproxy reload 7. Test the HA Proxy

bash $ curl localhost:1026/version -s -S