Post on 21-Jan-2015
description
MongoDB as a universal data store for Process Data
Munich2013
Munich2013
The talk in briefIt is all about two technology standards (coming together)…
Munich2013
Timo KlingenmeierCo-founder, GM and Technical Lead inmation.com
timo.klingenmeier@inmation.com
89-93 EMP (Engineering Company)Software Developer
93-03 GM, Founder iDV/best solutions AGIndustrial System Integration, Custom SW Development
03-10 GM, Matrikon Deutschland AGIndustrial Software Specialist, Real-Time Connectivity,
Alarm Management, Performance Monitoring
11/12 ConsultantOffshore Wind Projects
13 inmation.comNextGen Information Management Software
Munich2013
Munich2013
Munich2013
1ms – 1sec
Realtime
Munich2013
MES and PIMS
Control & Automation
ERP / B2B / B2C
Munich2013
MESand
PIMS
ACRONYM ALERT!
MES = Manufacturing Execution System
PIMS = Process Information Management System
Munich2013
15
Sites
2000
Applications
1
Refining Company
CAPEX OPEX (Maintenance Contracts, etc.)
Munich2013
ERP
Control
Advanced Process Control
Laboratory Information Management
Quality Management
Mass Balancing
Production Scheduling
Typical Production Process Applications
Performance, Reliability, Safety Management
Continuous Improvement(Generic Data-Mining)
Alarm Management
Operational
Excellence
Munich2013
The evolution of industrial system integration
(a brief history of time)
Munich2013
Industry 3.0
Emptying the parking lots
Source: flickr.com/acarlos1000
Munich2013
Munich2013
DistributedControl Systems
Entire SitesCountless Items
and Actors
Today
Munich2013
The Spinal Cord in Production: Contextualized Time-Series
historization of numerical data and process events.
Munich2013
The easiest way to improve a prediction is to add data. You can’t infer without data. So, store the data now and
analyze it later,…[]Dwight Merriman, MongoDB evangelist
Munich2013
TheBabylonian
Era
Munich2013
Information Contextualization
Bandwidth and Throughput
Implementation Complexity
Robustness
Protocol Differences
Munich2013
Munich2013
Birth of a standard
Munich2013
Munich2013
You can talk DCOM to me.
I will present you a qualified and more or
less structured namespace, consisting of
symbolically named items (tags).
Each item may have additional properties.
I can give you the actual value, the
milisecond accuracy and the value quality.
Munich2013
You can talk DCOM to me.
On request, I will constantly deliver new
Alarms & Events of any kind. Depending on
the subordinated control systems I am
connected to, the detail content of a single
event record may vary.
I can either supply all or filtered events.
Munich2013
You can talk DCOM to me.
Similar to my DA colleague on the left, I
maintain a structured namespace of tags.
Unlike this guy, I know about the entire
history of their values.
On request I will return raw values and
statistical aggregates for any period of
time.
Munich2013
You can talk SOAP / XML to me.
I can do what the DA guy does, but a little
simplified.
Munich2013
You can talk either binary TCP or XML to me.
I offer various options for secure communications.
I have many different profiles and facets.
(Some people are confused about me)
My smallest incarnation can work in a single chip
solution, while I’m still qualifying for an enterprise-wide
service. Obviously – Unified Architecture – I can supply
all services of those little guys in one.
Munich2013
Munich2013
You can talk to me.
(Everybody knows that)
Very similar to the SQL language, I’m not
young of age but definitely not willing to
retire!
Munich2013
One standard – many faces
Munich2013
Globally standardizedUnknown to broader
audienceUniversal Real-time
Data Access
Munich2013
Industrial IT Data Storage
Strategies Today
Proprietary High-Frequency Time-Series Data Formats
SQLDatabases
Munich2013
Application Differences
Supported Interfaces
Data ScopeExtraction
Data Format Storage
Interfaces to next level
Munich2013
Toomany
data silos
Munich2013
Data Reduction & Loss
Munich2013
How can industries create affordable, maintainable, open data stores which allow for the “Merriman paradigm” in the
specific context of industrial data mining requirements ?
Munich2013
the
approach
Munich2013
The problem to solve
Munich2013
OS Platform? Scalability? Make your choice!...or simply grow as required
Munich2013
How we use MongoDB in our product
12 months ago, there were SQL parts
There was homegrown data serialization
Today, we only use MongoDB for any kind of data storage
All network transports use inner BSON chunks, extended for efficient real-time object communication
Munich2013
Core
Connector
MongoDB
cmd_errors
cmd
cmd_processedinSys
Command ProcessorCommand ProcessorCommand Processor
Working ModelWorking Model
Core Object
Working Model
Connector Object
Working Model
Connector ObjectConnector Object
Core Object
Working ModelWorking Model
Core Object
Connector Object
Working Model
Core Object
Connector Object
Working ModelWorking ModelWorking ModelWorking Model
Core Object
Working ModelWorking Model
Connector ObjectConnector ObjectConnector Object
Working ModelWorking Model
Connector Object
Working Model
Connector Object
Working ModelWorking Model
Endpoint Object
Core Object
Connector Object
Endpoint Object
Working ModelWorking Model
Core Object
Connector Object
Endpoint Object
BSON BSON
BSON
BSON
Working ModelWorking Model
Endpoint ObjectEndpoint Object
Connector Object
Working ModelWorking Model
Endpoint Object
Connector Object
Working Model
Core Object
Connector Object
Endpoint Object
Endpoint
Working Model
Endpoint Object
Connector Object
Munich2013
Remote/LocalOPC A&E Server
Endpoint
Remote/LocalOPC DA Server
Endpoint
Remote XML-DA Server
Endpoint
Remote OPC UA Server
Endpoint
Remote /Local OPC HDA Server
Endpoint
Connector Service
Munich2013
Remote/LocalOPC A&E Server
Endpoint
Remote/LocalOPC DA Server
Endpoint
Remote XML-DA Server
Endpoint
Remote OPC UA Server
Endpoint
Remote /Local OPC HDA Server
Endpoint
SPROX Protocol (Single Port TCP)
Secure Prioritized Realtime Object Xchange
Connector Service Core Service
Munich2013
Remote/LocalOPC A&E Server
Endpoint
Remote/LocalOPC DA Server
Endpoint
Remote XML-DA Server
Endpoint
Remote OPC UA Server
Endpoint
Remote /Local OPC HDA Server
Endpoint
Mongo BSON Object
Bulk Inserts
Schema Design:
• Multiple Databases
• Multiple Collections
Database Design:
• Replication (Redundancy)
• Sharding (Horizontal Scaling)SPROX Protocol (Single Port TCP)
Secure Prioritized Realtime Object Xchange
Different OPC Servers
Core ServiceConnector Service MongoDB
Munich2013
Command Stack (API)
Any MongoDB Driver
C++, C#, Java, …
MongoDB
Third-Party Application
• Realtime Data Access• Component Registration• Historical Data Access
• Alarms and Events
• System Health Monitoring
Any MongoDB Driver
C++, C#, Java, …
• Component Configuration
Core ServiceConnector Service
Munich2013
BOX Protocol (Single Port TCP)
BSON Object Xchange
DataStudio
Connector Service Core Service MongoDB
• Component Registration
• Component Configuration
• System Health Monitoring
• Log Analysis
• Item Monitoring
Munich2013
Core Service MongoDB
DataProxy
Connector Service
OPC UA Server
BOX Protocol (Single Port TCP)
BSON Object Xchange
MongoDB
BSON
Munich2013
Munich2013
Munich2013
Using MongoDB-basedReal-time Data
(and a little bit of Google, too)
Munich2013
Munich2013
Summary – MongoDB qualifies for process data storage
It has the performance and the scalability options required to store hi-freq data in huge amounts
Timestamp accuracy is sufficient
Schema (-less) flexibility fits the variant data structures, originated in (OPC) source systems
Unbeatable value offer (both software and hardware utilization)
It exposes ‘natural’ data structures which make any kind of analysis fun
It satisfies IT people and engineers
Munich2013
Thank you very much for your attention