Presentation mongo db munich
-
Upload
mongodb -
Category
Technology
-
view
334 -
download
1
Transcript of Presentation mongo db munich
![Page 1: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/1.jpg)
BRÜCKNER MASCHINENBAU
STRETCHING THE LIMITShttp://www.brueckner.com/http://tiny.cc/brueckner
![Page 2: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/2.jpg)
Team Leader Software Development at Brückner Maschinenbau
@mkappeller
About
Matthias Kappeller
![Page 3: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/3.jpg)
Chef de Cuisine at MongoSoup
@loomit
About
Johannes Brandstetter
![Page 4: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/4.jpg)
We Build the Largest Lines in the Industry
BOPP 10.4m (35ft) – winder
© Brückner Maschinenbau 4
![Page 5: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/5.jpg)
We Build the Fastest Lines in the Industry
TDO of a 8.7 m BOPP line – 525 m/min production speed
© Brückner Maschinenbau 5
![Page 6: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/6.jpg)
Are You in Packaging Films?
© Brückner Maschinenbau 6
![Page 7: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/7.jpg)
Are You in Technical Films?
© Brückner Maschinenbau 7
![Page 8: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/8.jpg)
How we deploy our products
© Brückner Maschinenbau 8
![Page 9: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/9.jpg)
010010100111 010101101101 10110101 1011
Sensor Data
![Page 10: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/10.jpg)
Temperature
Speed
Pressure
Thickness
Density
Alarms
Sensor-Status
~100'000 Datapoints per Line
1-8 lines per customer
![Page 11: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/11.jpg)
~100000 data points in total ~4000 variables are high frequency
Sensor Data
![Page 12: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/12.jpg)
Sensor Data
Frequency of incoming data at 5-10Hz (100-200ms)
http://www.fantom-xp.com/en_19__Intel_Core_i7_high_speed.html
![Page 13: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/13.jpg)
Sensor Data
Time Series Graph Track Single Parameter
© Brückner Maschinenbau 13
![Page 14: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/14.jpg)
Document for every productof a campaign
Quality Profile Thickness Scan Scan speed: 300-500 m/min Analyzed via Heatmap
Scanner Data
![Page 15: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/15.jpg)
Analytics
© Brückner Maschinenbau 15
![Page 16: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/16.jpg)
Difficult and time-consuming configuration and setup Typical ETL (information loss) Many Stored-Procedures Low performance Low availability Outdated UI-Technology (Delphi) Proprietary PLC-Driver (to read sensor data)
Our current approach and its problems
© Brückner Maschinenbau 16
![Page 17: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/17.jpg)
Write> 3'000 updates / sec
Scalability
Low System-Complexity
Near Future Goals
![Page 18: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/18.jpg)
Intelligent Line
Management Cockpits
Smart Recipes
Far Future Goals
![Page 19: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/19.jpg)
OEM Customer has no IT-Administration or low IT-KnowHow Low-Cost Server Infrastructure Highly scalable Server infrastructure Bad network connection Many power shutdowns Production 24/7
• High availability
• Complex system update
Challenges
© Brückner Maschinenbau 19
![Page 20: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/20.jpg)
PoC
http://dilbert.com/strips/comic/2009-11-18/
![Page 21: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/21.jpg)
MongoDB
![Page 22: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/22.jpg)
MongoDB Hosting from Germany
Bring your own data center
Customer-focused solutions
![Page 23: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/23.jpg)
Web based management toolsReal-time visualizationReal-time loggingSSL connector
![Page 24: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/24.jpg)
![Page 25: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/25.jpg)
Schema free Ease and speed of development Ease of operations Realtime analysis of streaming data Possibility to add Hadoop for heavier Analytics later
Reasons for choosing MongoDB
![Page 26: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/26.jpg)
Architecture Overview (simplified)
OPC
REST
Data Input
Streaming
MongoDB
MetaData
Service
• Scan / Lab Data
• Machine Configuration
• Recipe
Streaming Data
Document Data
Streaming Data
Hadoop
Analytics
![Page 27: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/27.jpg)
Schema Design
Time Series Schema vs. Bucket Schema vs. “Simple Schema”
Comparison
![Page 28: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/28.jpg)
Time Series Schema
http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb
![Page 29: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/29.jpg)
{ _id: {timestamp_hour: ISODate("2013-10-10T23:00:00.000Z"), type: “velocity_m1”},
values: { 0: { 0: 93, 1: 91, …, 59: 95 }, 1: { 0: 95, 1: 89, …, 59: 87 }, …, 59: { 0: 91, 1: 90, …, 59: 92 } }
}db.metrics.update(
{ _id.timestamp_hour: ISODate("2013-10-10T23:00:00.000Z"), _id.type: “velocity_m1” }, {$set: {“values.59.59”: 92 } })
Time Series Schema
![Page 30: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/30.jpg)
Schema design for time series data well understood Sensor Data is not time series data Sensors have own thresholds
Time Series Schema
![Page 31: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/31.jpg)
Time Series Schema
http://www.culatools.com/wp-content/uploads/2011/09/sparse_matrix.png
![Page 32: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/32.jpg)
Preallocate space with empty documents for upcoming time periods -> in-place updates
Primary Index: timestamp Good fit for streaming data But lots of sparse documents
Time Series Schema
Performance
![Page 33: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/33.jpg)
Bucket Schema
http://flickr.com/photos/99255685@N00/2063575447
![Page 34: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/34.jpg)
“A bucket is most commonly a type of data buffer or a type of document in which data is divided into regions.“http://en.wikipedia.org/wiki/Bucket_(computing)
Bucket Schema
![Page 35: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/35.jpg)
Bucket Schema
Have one bucket per sensor
Fill up with values till it‘s full Use next bucket
![Page 36: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/36.jpg)
{ _id: {timestamp_start:ISODate("2013-10-10T23:00:00.000Z"), type:“velocity_m1”
}, state: “OPEN”,
values: [ { t: 456, v: 93 }, { t: 572, v: 89 }, ..., { t: 2344, v: 92 }
..., ...] }db.metrics.update( {_id.timestamp_start: ISODate("2013-10-10T23:00:00.000Z"), _id.type: “velocity_m1” }, {$push: {“values”: {t: 2500, v: 90}})
Bucket Schema
![Page 37: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/37.jpg)
Pre-allocate space with empty documents for upcoming data count -> in-place updates
Different bucket sizes for different sensors Still good for range based queries Have to keep counter in app to determine bucket state Good fit for sparse data
Bucket Schema
Performance
![Page 38: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/38.jpg)
Simple Schema
One document per update Fragmentation
•disk and memory will get fragmented over time
•even more, the smaller the documents are
![Page 39: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/39.jpg)
Index Size
• Bucket decreases index size by factor of number of documents it holds
• Index should fit in RAM + working set
• Index updates cause locks and I/O
Bucket Schema vs. Simple Schema
Indexing
![Page 40: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/40.jpg)
Update driven vs. insert driven
• individual writes are smaller
• performance and concurrency benefits
Simple Schema
Performance benefits
![Page 41: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/41.jpg)
Optimal with tuned block sizes and read ahead
• Fewer disk seeks
• Fewer random I/O
Bucket Schema vs. Simple Schema
Performance
Write Performance Query PerformanceSimple Schmea ~ 22k/sec 78'611 millisBucket Schema ~13k/sec 42'057 millis
![Page 42: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/42.jpg)
SSD vs. HDD
SSD
HDD
![Page 43: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/43.jpg)
Schema design matters Increase performance:
• In-Memory caching
• Concurrency
• Queue
• Core-Sharding
Flexible and scalable system
• We can build a system with low complexity for simple use cases
• We can provide a system for „bigger“ use cases by increasing complexity
Lessons learned
![Page 44: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/44.jpg)
Development of appliance like systems can be challenging Ease of use Resilience
Conclusion
![Page 45: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/45.jpg)
PoC and its results are approved by the management Workout / Design the UI/UX Develop the software system
Ship prototypes to some sites Explore and develop analytic-algorithms Fieldtest for our software system with MongoDB First machine with this solution to be deployed in 2016
Where we are now
![Page 46: Presentation mongo db munich](https://reader036.fdocuments.in/reader036/viewer/2022062313/55851670d8b42aff298b4ef5/html5/thumbnails/46.jpg)
@mongosoup
www.mongosoup.de