PLA1703C – Splunk as IoT Analytics Platform
Transcript of PLA1703C – Splunk as IoT Analytics Platform
© 2020 SPLUNK INC.
© 2020 SPLUNK INC.
PLA1703C –Splunk as IoT Analytics PlatformLeading technology group establishes Splunk as data platform for manufacturing process improvements
Dr. Martin DöringData Scientist | Heraeus
Ronald PerzulStaff Consulting Sales Engineer |
Splunk
During the course of this presentation, we may make forward‐looking statements regarding future events or plans of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results may differ materially. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, it may not contain current or accurate information. We do not assume any obligation to update any forward‐looking statements made herein.
In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionalities described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Data-to-Everything, D2E and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names or trademarks belong to their respective owners. © 2020 Splunk Inc. All rights reserved
Forward-LookingStatements
© 2020 SPLUNK INC.
Data Scientist | Heraeus
Dr. Martin Döring
© 2020 SPLUNK INC.
Staff Consulting Sales Engineer | Splunk
Ronald Perzul
© 2020 SPLUNK INC.
Agenda 1) About Heraeus
2) Digital Transformation
3) Benefits and Selected Use Cases
4) Architecture
5) Data Onboarding
6) Summary and Next Steps
7) Technical Appendix
© 2020 SPLUNK INC.
About Heraeus
© 2020 SPLUNK INC.
Facts & Figures
© 2020 SPLUNK INC.
Portfolio
© 2020 SPLUNK INC.
Fields of Activity
© 2020 SPLUNK INC.
Digital Transformation
© 2020 SPLUNK INC.
Starting Point:Digital Transformation Initiatives
End of 2016: first data analysis initiatives“We have a lot of data, where can we put them?”“We have a system for that, let us have a look!”
Mid of 2017: first large use case“Let’s analyze a whole year’s data of our production line!”
Start of 2018Founding of the “Heraeus Digital Factory”Department specialized on digital transformation
Start of 2019“We need an enterprise-grade IoT platform!”Vendor selection based on defined assessment criteria
4
3
2
1
© 2020 SPLUNK INC.
Today: Impact-Driven Approach
Identify use casesApply Six Sigma
methodologyProvide consulting
services
Evaluate impactBusiness unit rates
impact of use cases
Get commitmentExpected impact is
underwritten by relevant stakeholders• Business unit
management• Business unit controlling• Corporate controlling
Conduct projectStandardized project
runbook• Connectivity• Data onboarding• Data visualization• User training
© 2020 SPLUNK INC.
The Splunk Journey
Aug 2016Introduced
Splunk for Security and IT Operations
Aug 2017First license
extension
May 2019Additional
Splunk instance for use in SOCLicense Volume:
Splunk Enterprise 300 GBSplunk ES250 GB
Oct 2019Decision Splunk
for IoT
Oct 202020 IoT Use
Cases deployedLicense Volume
IoT: 50 GB
© 2020 SPLUNK INC.
Benefits and Selected Use Cases
© 2020 SPLUNK INC.
Impact and Benefits
Number of use cases: 4Savings: ~5M
Number of use cases: 12Savings: ~2M
Number of use cases: 4Savings: ~0.5M
Process optimization
Machine status & capacity
utilizationDefect
statistics
Of the 20 use cases implemented so far
© 2020 SPLUNK INC.
Process Data OverviewAudience
• Process engineers• Quality engineers• R&D
Benefits • Analyze machine sensor data as
basis for production process optimization
• Establish real-time alerting in case of anomalies
© 2020 SPLUNK INC.
Machine StatusAudience
• Shop floor• Lead workers
Benefits • Achieve fine-granular
transparency about machine status
• Use machine status data for OEE visualization
• Decrease downtimes and increase yield and throughput
© 2020 SPLUNK INC.
Capacity UtilizationAudience
• Shop floor• Production planning
Benefits • Achieve real-time transparency
about and gain insights into capacity utilization using machine sensor data
• Increase yield and throughput by optimizing capacity utilization
© 2020 SPLUNK INC.
Defect StatisticsAudience
• Process engineers• Quality engineers
Benefits • Gain insights into product quality
using data from quality systems• Correlate with machine sensor
data to optimize production processes
© 2020 SPLUNK INC.
Architecture
© 2020 SPLUNK INC.
MQTT:• Lightweight, publish-subscribe protocol to
transport messages between devices
IoT Platform and Protocols Primer
IoT platform:• Multi-layer technology that enables straightforward
provisioning, management, and automation of devices and industrial assets
IoT analytics platform:• Sub-component of an IoT platform with the following
capabilities: reporting, visualization, analytics, alerting, rules, machine learning, and augmented reality
OPC Unified Architecture (OPC-UA):• Machine-to-machine communication protocol for
industrial automation
MQTT Broker
Sensor
Server
Mobile Device
publish 15W
Publish to topic ”Power Consumption”
Subscribe to topic ”Power Consumption”
© 2020 SPLUNK INC.
End-to-End Architecture (High-Level)From PLC to analytical layer
• Real-time availability of data• Strong dashboarding capabilities• Ingestion of massive amounts of data• Interface to data science platform
Machine
Gateway
MessageBroker
IoT Platform
SAP PCo
• HiveMQ distributes machine data in the Heraeus network
• Ensures that all data points reach the IoT platform
• Dell gateways are available globally in all Heraeus locations
• Using SAP PCo ensures compatibility with SAP MES in the future
© 2020 SPLUNK INC.
HF
Indexers SHHF
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
Indexers SH
PLC
HF
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
Indexers SH
Gateway
PLC
HF
OPC-UA
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
Indexers SH
Gateway
PLC
HF
OPC-UA
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
Gateway
PLC
HF
MQTT
OPC-UA
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
Gateway
PLC
HF
MQTT
OPC-UA
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
Gateway
PLC
HF
MQTT
OPC-UA
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
Gateway
PLC SAP ERP
HF
MQTT
OPC-UA
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP ERP
HF
MQTT
OPC-UA
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
HF
MQTT
OPC-UA
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
HF
MQTT
OPC-UA
REST
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
HF
MQTT
OPC-UA
REST
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
LB
HF
MQTT
OPC-UA
REST
HEC
HF w/ MQTT
HF w/ MQTT
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
LB
HF
MQTT
OPC-UA
REST
HEC
HF w/ MQTT
HF w/ MQTT
File system
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
LB
HF
MQTT
OPC-UA
REST
HEC
HF w/ MQTT
HF w/ MQTT
File system
UF
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
LB
HF
MQTT
OPC-UA
REST
HEC
HF w/ MQTT
HF w/ MQTT
File system
UF
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
HF
HiveMQ
HiveMQ
LB Indexers SH
DataikuGateway
PLC SAP HANA BWSAP ERP
LB
HF
MQTT
OPC-UA
REST
HEC
HF w/ MQTT
HF w/ MQTT
File system
UF
Custom code(if required)
End-to-End Architecture (Detailed)
© 2020 SPLUNK INC.
Data Onboarding
© 2020 SPLUNK INC.
Data Onboarding
Some design guidelines:• Decide whether to use events or metrics to
ingest MQTT payloads (data interval, data types)
• Always use index time field extractions.• Establish proper transformations rules.• If possible, standardize on message payload to
reduce amount of transformations needed.• Use log2metrics to convert MQTT messages to
Splunk metrics.– See Appendix for detailed props.conf and transforms.conf
example
Events/Metrics
Yes
No
• Index time field extractions• Convert boolean
(true/false) to 1/0• Create additional
descriptive fields (batch ID, asset)
• Index time field extractions• Convert boolean (true/false)
to 1/0• Convert machine state (e.g.
running) to numerical values• Create dimensions for
descriptive fields (batch ID, asset)
Data interval <= 10 sec
Metrics index Event index
© 2020 SPLUNK INC.
Data Onboarding
SAP ERP data• Order information, batch information, quality inspections
Data sets are frequently updated• Only the newest data is required for analysis and visualization• Events/metrics are not suited well for that purpose• Use KV stores instead
KV Stores
REST API Saved searches
SAP ERP
SAP HANA BW
Dataiku Splunk DB Connect
KV store
JDBC JDBC
© 2020 SPLUNK INC.
Summary and Next Steps
© 2020 SPLUNK INC.
Why Splunk?
Evaluation based on defined assessment criteria.Based on this, Splunk was
selected.
Criteria Rating CommentData onboarding via MQTT (and OPC-UA) ++
Time-series visualization of key process parameters
++
Visualization of OEE +
Self-service dashboarding capabilities 0 Requires user training
IoT asset management -- No management
Integration with existing machine learning platform
+ Possible, but custom dev
Scalability ++
Pricing +
Rating of Key Requirements
© 2020 SPLUNK INC.
1. Conduct advanced trainings for business users
2. Onboarding of additional machines (approx. 500 until end of 2021)
3. Establish internal Splunk community with forum functionality
4. Investigate live dashboards with real-time refresh
Next Steps
© 2020 SPLUNK INC.
Outlook
Additional self-service capabilities
Analytics Workspace
Extend SOC for OT security
OT Security
Provide access to data when
information is needed
Augmented Reality
Leverage new Splunk
Dashboard App
Dashboards
Direct integration with HiveMQ (partner SVA)
HiveMQ
Additional uses of Splunk for IoT
© 2020 SPLUNK INC.
1. Splunk expert can help avoid complex proof events
2. Leveraging existing Splunk environment and know-how reduced implementation time
3. Using Splunk can help avoid more complex IoT platform architectures
4. Splunk can compete with and beat legacy IoT platform vendors regarding costs, benefits, and capabilities
Key Takeaways
SESSION SURVEYPlease provide feedback via the
© 2020 SPLUNK INC.
Thank You
© 2020 SPLUNK INC.
Appendix
© 2020 SPLUNK INC.
Convert MQTT Messages to MetricsUse log2metrics feature to convert MQTT messages to metric data points
High-level transformations to create metric data points:1. Identify timestamp prefix, format, and time zone2. Strip quote characters, replace “true” / “false” by 1 / 03. Extract key-value pairs (regex with repeated matching)4. Identify metrics dimensions (e.g. “oven_no”)
See next slide for detailed props.conf and transforms.conf example
© 2020 SPLUNK INC.
Input Transformation ExampleTransform events (as received from MQTT add-on) to metrics
Example payloadTue Jul 21 16:30:33 CEST 2020 name=mqtt_msg_received event_id= topic=<production_area>/<mqtt_client>/<production_process>/machinedata msg={ "oven_no": "oven14", "timestamp": "2020-07-21T14:30:28.153", "performance": 34.41088, "volume_flow": 24.89, "temperature": 2033, "running": true }
props.confTIME_PREFIX = timestamp": "TIME_FORMAT = %FT%H:%M:%S.%3QTZ = UTCSEDCMD-modifypayload = s/"//g s/true/1/g s/false/0/gTRANSFORMS-extractpayload = extract_dimensions,extract_fieldsMETRIC-SCHEMA-TRANSFORMS=metric-schema:metrics_for_iot_data
transforms.conf[extract_dimensions] [extract_fields]REGEX = topic=(.*/(.*)/machinedata) msg= REGEX = \s([\S]+):\s([^,]*)[,\s]FORMAT = topic::$1 production_process=$2 FORMAT = $1::$2WRITE_META = true REPEAT_MATCH = true
WRITE_META = true[metric-schema:metrics_for_iot_data]METRIC-SCHEMA-MEASURES = _ALLNUMS_METRIC-SCHEMA-WHITELIST-DIMS = topic,production_process,oven_no
© 2020 SPLUNK INC.
Top Search CommandsAs used in all IoT dashboards so fareval 4919 54.95%
stats 783 8.75%
table 561 6.27%
timechart 393 4.39%
sort 257 2.87%
streamstats 197 2.20%
search 193 2.16%
where 178 1.99%
fields 157 1.75%
delta 141 1.58%
mstats 124 1.39%
lookup 118 1.32%
appendcols 111 1.24%
autoregress 101 1.13%
rename 94 1.05%
eventstats 79 0.88%
chart 63 0.70%
fillnull 62 0.69%
transaction 58 0.65%
addinfo 45 0.50%
addtotals 43 0.48%
append 40 0.45%
bin 34 0.38%
dedup 27 0.30%
rex 23 0.26%
gauge 22 0.25%
transpose 22 0.25%
filldown 20 0.22%
fieldformat 17 0.19%
reverse 16 0.18%
makemv 10 0.11%
replace 9 0.10%
foreach 6 0.07%
join 5 0.06%
union 5 0.06%
mvcombine 4 0.04%
mvexpand 4 0.04%
nomv 4 0.04%
rangemap 2 0.02%
top 2 0.02%
regex 1 0.01%
untable 1 0.01%
xyseries 1 0.01%
totaling 93%