Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv
Big Data and Spark Streaming. Oil production sensors data monitoring
-
Upload
softelegance -
Category
Data & Analytics
-
view
624 -
download
1
Transcript of Big Data and Spark Streaming. Oil production sensors data monitoring
![Page 1: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/1.jpg)
Big Data, Spark Streaming, Oil and Gas
Kyiv, Ukraine. 07 June 2016
Oil production sensors data monitoring
Yaroslav Nedashkovsky, System Architect
![Page 2: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/2.jpg)
Shell data project
![Page 3: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/3.jpg)
![Page 4: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/4.jpg)
![Page 5: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/5.jpg)
“Next year, BP will connect 650 wells to the Industrial Internet. If all goes according to plan, the companies will expand the scope to 4,000 BP subsea wells around the world”
![Page 6: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/6.jpg)
Digital Oil Field
![Page 7: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/7.jpg)
![Page 8: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/8.jpg)
How we could handle such huge data flow ?
What kind of streaming technology could we use ?
![Page 9: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/9.jpg)
Stream processing system- Apache Spark Streaming
- Apache Storm- Apache Samza- Azure Stream Analytics- Google Dataflow- Heron
- Apache Flink
![Page 10: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/10.jpg)
What we need from streaming system?- scalable- fault-tolerant- low latency- data distribution- distributed computations- good API- “exactly-once” guarantees or maybe “at most once” or “at least once ” will be enough ?
![Page 11: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/11.jpg)
(near real time, but not real time)
![Page 12: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/12.jpg)
![Page 13: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/13.jpg)
- high-level api (windows, joins, etc.)
- exactly-one semantics (?!), fault tolerant, scalable
- integration with SQL, DataFrames, Mllib, GraphX
![Page 14: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/14.jpg)
How this work ?
![Page 15: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/15.jpg)
![Page 16: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/16.jpg)
![Page 17: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/17.jpg)
![Page 18: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/18.jpg)
![Page 19: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/19.jpg)
![Page 20: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/20.jpg)
![Page 21: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/21.jpg)
![Page 22: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/22.jpg)
![Page 23: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/23.jpg)
![Page 24: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/24.jpg)
![Page 25: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/25.jpg)
Spark 2.0: Structured Streaming
![Page 26: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/26.jpg)
Oil location data flow monitor
![Page 27: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/27.jpg)
![Page 28: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/28.jpg)
“Christmas tree”
![Page 29: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/29.jpg)
![Page 30: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/30.jpg)
IoT (MQTT) + Spark Streaming + Vizualization
![Page 31: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/31.jpg)
Let look at “monitor” implementation and see how it works
![Page 32: Big Data and Spark Streaming. Oil production sensors data monitoring](https://reader035.fdocuments.in/reader035/viewer/2022062900/58f0e74e1a28ab4b388b45cb/html5/thumbnails/32.jpg)