Mongo db as a universal data store for process data

MongoDB as a universal data store for Process Data

Munich2013

Munich2013

The talk in briefIt is all about two technology standards (coming together)…

Munich2013

Timo KlingenmeierCo-founder, GM and Technical Lead inmation.com

[email protected]

89-93 EMP (Engineering Company)Software Developer

93-03 GM, Founder iDV/best solutions AGIndustrial System Integration, Custom SW Development

03-10 GM, Matrikon Deutschland AGIndustrial Software Specialist, Real-Time Connectivity,

Alarm Management, Performance Monitoring

11/12 ConsultantOffshore Wind Projects

13 inmation.comNextGen Information Management Software

Munich2013

Munich2013

1ms – 1sec

Realtime

Munich2013

MES and PIMS

Control & Automation

ERP / B2B / B2C

Munich2013

MESand

PIMS

ACRONYM ALERT!

MES = Manufacturing Execution System

PIMS = Process Information Management System

Munich2013

15

Sites

2000

Applications

1

Refining Company

CAPEX OPEX (Maintenance Contracts, etc.)

Munich2013

ERP

Control

Advanced Process Control

Laboratory Information Management

Quality Management

Mass Balancing

Production Scheduling

Typical Production Process Applications

Performance, Reliability, Safety Management

Continuous Improvement(Generic Data-Mining)

Alarm Management

Operational

Excellence

Munich2013

The evolution of industrial system integration

(a brief history of time)

Munich2013

Industry 3.0

Emptying the parking lots

Source: flickr.com/acarlos1000

Munich2013

Munich2013

DistributedControl Systems

Entire SitesCountless Items

and Actors

Today

Munich2013

The Spinal Cord in Production: Contextualized Time-Series

historization of numerical data and process events.

Munich2013

The easiest way to improve a prediction is to add data. You can’t infer without data. So, store the data now and

analyze it later,…[]Dwight Merriman, MongoDB evangelist

Munich2013

TheBabylonian

Era

Munich2013

Information Contextualization

Bandwidth and Throughput

Implementation Complexity

Robustness

Protocol Differences

Munich2013

Munich2013

Birth of a standard

Munich2013

Munich2013

You can talk DCOM to me.

I will present you a qualified and more or

less structured namespace, consisting of

symbolically named items (tags).

Each item may have additional properties.

I can give you the actual value, the

milisecond accuracy and the value quality.

Munich2013


On request, I will constantly deliver new

Alarms & Events of any kind. Depending on

the subordinated control systems I am

connected to, the detail content of a single

event record may vary.

I can either supply all or filtered events.

Munich2013


Similar to my DA colleague on the left, I

maintain a structured namespace of tags.

Unlike this guy, I know about the entire

history of their values.

On request I will return raw values and

statistical aggregates for any period of

time.

Munich2013

You can talk SOAP / XML to me.

I can do what the DA guy does, but a little

simplified.

Munich2013

You can talk either binary TCP or XML to me.

I offer various options for secure communications.

I have many different profiles and facets.

(Some people are confused about me)

My smallest incarnation can work in a single chip

solution, while I’m still qualifying for an enterprise-wide

service. Obviously – Unified Architecture – I can supply

all services of those little guys in one.

Munich2013

Munich2013

You can talk to me.

(Everybody knows that)

Very similar to the SQL language, I’m not

young of age but definitely not willing to

retire!

Munich2013

One standard – many faces

Munich2013

Globally standardizedUnknown to broader

audienceUniversal Real-time

Data Access

Munich2013

Industrial IT Data Storage

Strategies Today

Proprietary High-Frequency Time-Series Data Formats

SQLDatabases

Munich2013

Application Differences

Supported Interfaces

Data ScopeExtraction

Data Format Storage

Interfaces to next level

Munich2013

Toomany

data silos

Munich2013

Data Reduction & Loss

Munich2013

How can industries create affordable, maintainable, open data stores which allow for the “Merriman paradigm” in the

specific context of industrial data mining requirements ?

Munich2013

the

approach

Munich2013

The problem to solve

Munich2013

OS Platform? Scalability? Make your choice!...or simply grow as required

Munich2013

How we use MongoDB in our product

12 months ago, there were SQL parts

There was homegrown data serialization

Today, we only use MongoDB for any kind of data storage

All network transports use inner BSON chunks, extended for efficient real-time object communication

Munich2013

Core

Connector

MongoDB

cmd_errors

cmd

cmd_processedinSys

Command ProcessorCommand ProcessorCommand Processor

Working ModelWorking Model

Core Object

Working Model

Connector Object

Working Model

Connector ObjectConnector Object

Core Object


Core Object

Connector Object

Working Model

Core Object

Connector Object

Working ModelWorking ModelWorking ModelWorking Model

Core Object


Connector ObjectConnector ObjectConnector Object


Connector Object

Working Model

Connector Object


Endpoint Object

Core Object

Connector Object

Endpoint Object


Core Object

Connector Object

Endpoint Object

BSON BSON

BSON

BSON


Endpoint ObjectEndpoint Object

Connector Object


Endpoint Object

Connector Object

Working Model

Core Object

Connector Object

Endpoint Object

Endpoint

Working Model

Endpoint Object

Connector Object

Munich2013

Remote/LocalOPC A&E Server

Endpoint

Remote/LocalOPC DA Server

Endpoint

Remote XML-DA Server

Endpoint

Remote OPC UA Server

Endpoint

Remote /Local OPC HDA Server

Endpoint

Connector Service

Munich2013


Endpoint


Endpoint


Endpoint


Endpoint


Endpoint

SPROX Protocol (Single Port TCP)

Secure Prioritized Realtime Object Xchange

Connector Service Core Service

Munich2013


Endpoint


Endpoint


Endpoint


Endpoint


Endpoint

Mongo BSON Object

Bulk Inserts

Schema Design:

• Multiple Databases

• Multiple Collections

Database Design:

• Replication (Redundancy)

• Sharding (Horizontal Scaling)SPROX Protocol (Single Port TCP)

Secure Prioritized Realtime Object Xchange

Different OPC Servers

Core ServiceConnector Service MongoDB

Munich2013

Command Stack (API)

Any MongoDB Driver

C++, C#, Java, …

MongoDB

Third-Party Application

• Realtime Data Access• Component Registration• Historical Data Access

• Alarms and Events

• System Health Monitoring

Any MongoDB Driver

C++, C#, Java, …

• Component Configuration

Core ServiceConnector Service

Munich2013

BOX Protocol (Single Port TCP)

BSON Object Xchange

DataStudio

Connector Service Core Service MongoDB

• Component Registration

• Component Configuration

• System Health Monitoring

• Log Analysis

• Item Monitoring

Munich2013

Core Service MongoDB

DataProxy

Connector Service

OPC UA Server

BOX Protocol (Single Port TCP)

BSON Object Xchange

MongoDB

BSON

Munich2013

Munich2013

Using MongoDB-basedReal-time Data

(and a little bit of Google, too)

Munich2013

Munich2013

Summary – MongoDB qualifies for process data storage

It has the performance and the scalability options required to store hi-freq data in huge amounts

Timestamp accuracy is sufficient

Schema (-less) flexibility fits the variant data structures, originated in (OPC) source systems

Unbeatable value offer (both software and hardware utilization)

It exposes ‘natural’ data structures which make any kind of analysis fun

It satisfies IT people and engineers

Munich2013

Thank you very much for your attention

Mongo db as a universal data store for process data

Technology

Transcript of Mongo db as a universal data store for process data