DataOps Toolchain for Continuous Control Monitoring · #PIWorld ©2018 OSIsoft, LLC DataOps...

#PIWorld ©2018 OSIsoft, LLC

DataOps Toolchain for Continuous Control Monitoring

Mahyar SEPEHR

1


Who is apoIO?

2

We work with our clients to optimize their Digital Journey

• Cloud Architecture

• Internet Of Things

• Industrial Big Data: PI System


A Typical Client and her Use-Cases

3

Laura is a P&U Project Manager

and Business Analyst

Water

Network

issues

Water leak

detection

Energy

consumption

optimization

Smart Meters

Average

Consumption

Water Network

Quality Control

Behaviour of the

consumers, etc.


Sensors from the Water Network send by GSM values in a non chronological way : Invalidates all rules based on events

Optimisation of substances use on a supply chain :

Many timeranges contain exceptions because of

Data Quality

4

Business Challenges on Data Quality


5

Data has imperfections : Welcome to Reality!

03:00 03:10 03:20 03:30 03:40 03:50

Speed

Oil

Temp

Oil

Temp

Speed

Torque

03:00

03:10

03:20

03:30

03:40

03:50

Oil

Te

mp

60

60

60

120

60

Oil

Te

mp

50

50

50

50

45

Sp

ee

d

10.0

10.2

9.8

10.0

10.1

Sp

ee

d

9.8

10

10

9.9

9.9

To

rqu

e

25

24

25

25

25

Comm Failure

Different

start times

Additional

Measure

Uneven Spacing

Spike /

Out of Range


Analysis of the problem

6

Lack of direct

communication :

Business & the Team

More direct and frequent

communication between

actors

Cycle from

conception to

execution too long

Organize some short

iteration to avoid

loosing time on failure

Issues with Data

quality discovered far

after conception &

implementation

Integrate the data

quality monitoring

Organisation

Process

Toolchain

Wh

at

wen

t w

ron

g?

Ho

w c

an

we f

ix it?

?

?

?


DevOps Approach : Continuous Improvement

7

© https://en.wikipedia.org/wiki/DevOps_toolchain

https://en.wikipedia.org/wiki/DevOps_toolchain


Process : DevOps toolchain

8

Source Code

Management

Source

Code

Test / Build

Continuous

Integration

Code

Quality

Review

Deploy

To

Dev. Env.

Deploy

To

Staging Env.

Acceptance

Testing

Deploy

To

Production

Env.

Deployment Environment

Configuration Deployment tools


PI System Project life-cycle: Data Driven Processes

9

10101010101010110011010 0101010101010101010010101010101010101010101010

10101010101010110011010 0101010101010101010010101010101010101010101010

DATA


Organization Chart

Laura has a new role:

Product Owner

10

Security

System

Administrator

Data

Architect

Product

Owner Tech

Lead

Ops

Data

Engineer

Data

Scientist

End-Users

Enterprise

Architect


Agile Management : Rethink the Journey

11

z Hotfix

z z z Iteration1

3w

Iteration2

3w

Iteration3

3w D

em

o

Fe

atu

res

De

mo

Fe

atu

res

Writes User Stories

Core Features

including Data Quality

Launch Launch Launch


1 1

1

2

3

4

5

1 6

Simple Excel DataLink

Design with Asset Analytics

Design an AF SDK based Custom Program

Design a PI WEB API Custom Program

STAR : Self Test And Repair by Code

STAR : Self Test And Repair with PI Integrator

Continuous Control Monitoring : MVP Toolchains

Minimum Viable

Product (MVP)

12


MVP : STAR with Machine Learning

13

PULL

CLEANSE

Data quality

AUGMENT Data

aggregation

SHAPE

Model

normalization

TRANSMIT Data

compatibility

PUSH

PI Integrator for

Business Analytics

2 3 4 5 6

Analyse Transform Visualise Write

in to

PI System

Report


14

Risks of DataOps Approach

Count on a clear Data Governance & Support?

Data Exploration managed by Business need

Tasks in iterations depending on other Teams

Does DataOps Team have Testing Background?


15

CCM bringing Business values From Data Quality to Data Exploration

Data Quality Statistics helping optimize Business rules

Threshold Optimization

Data Exploration helping in Prediction

Data Exploration cleaning useless Monitoring Alerts


16

Call for Participation!

Let’s contribute to

a Common Goal:

Answer to

some sharp

questions about

PI System Data Exploration

pi-data-exploration

PI WEB API

Get Data

Data

Preparation

Analysis &

Transform

Test

Store

into

PI System

Report

1

2

3

4

5

6

https://github.com/msepehr/pi-data-exploration







pi-data-exploration Description

Sharp Questions:

• Classification Modeling as Leak Detection (no threshhold)

• Autofill on Missed Data

• Statistics over the data as Missed Data

Python Dependencies:

• PI-Web-API-Client-Python : PI Client for Python

• pandas : Data structure manipulation

• numpy : perform calculations over the data


17







Key Takeaways Key Insights Lessons learned from our journey

01 Blueprint for success Do not let your transformation initiative

fail because of a lack of anticipation on

the project due to the data

02 Data Exploration, Data Discovery more & more vital in Businesses in any

sector

03 Agility with a clear Roadmap Iteration on core features is completed

with a real vision in a long term

04 Empowering End-Users Better and faster communication,

interactions with DataOps Team

05 New Roles to Integrate PO, Data preparation, Data Scientists,

Business Intelligence, Data Visualization

06 Automate, automate, automate! From Data Exploration to Business

Rules, Industrialize as much as you can

18


19

Data Exploration makes us grow up…

Let’s grow up together!

Mahyar SEPEHR

IT Architect

[email protected]

mailto:[email protected]

mailto:[email protected]


Questions?

Please wait for

the microphone

State your

name & company

Please rate this session

in the mobile app!

Search

“OSIsoft” in

your app store

20


21

DataOps Toolchain for Continuous Control Monitoring · #PIWorld ©2018 OSIsoft, LLC DataOps...

Documents

Transcript of DataOps Toolchain for Continuous Control Monitoring · #PIWorld ©2018 OSIsoft, LLC DataOps...