ODSC - Neural Networks on AWS Lambda
-
Upload
anoop-vasant-kumar -
Category
Data & Analytics
-
view
378 -
download
0
Transcript of ODSC - Neural Networks on AWS Lambda
![Page 1: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/1.jpg)
OPEN DATA SCIENCE CONFERENCE
Real Time Training and Forecasting using LSTM/RNNs & AWS Lambda
Anoop Vasant KumarData Scientist
Hitachi Consulting, UK
![Page 2: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/2.jpg)
Understanding the problem Nuclear Power Phase Out
Switch to low carbon,
environment friendly and
affordable energy.
Forecast of Energy Price
Trading Optimization and
Energy Price Forecast
Generation Optimization
Power Generation Optimization
![Page 3: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/3.jpg)
Project objective #1: Cost effective method to provide real time market price forecasting service.
![Page 4: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/4.jpg)
Project objective #2: A scalable and distributed solution, that forecasts prices for each of the 100+ energy products based on live market input feeds.
![Page 5: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/5.jpg)
High Price 48 Euro
Generation Cost 30 Euro
Low Price 10 Euro
T-5 T-4 T-3 T-3 T-2 T-1 T
Time Series of a typical energy product
![Page 6: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/6.jpg)
Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product
![Page 7: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/7.jpg)
Model
A model that could learn complex nonlinear relationships
Multiple features with each input being a time series on it own
Ability to remember the past trend and sequence
Ability to consider long term dependencies in data
Ability to improve forecasts and update model parameters real time based on new data inputs
![Page 8: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/8.jpg)
Correlation of Products
![Page 9: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/9.jpg)
Recurrent Neural Networks are networks with loops in them, allowing information to persist.
It makes use of sequential information unlike traditional neural networks.
RNN - A network that remembers
![Page 10: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/10.jpg)
Long Short Term Memory - Recurrent Neural NetworksRemembering long term
dependencies is their default nature.
Think of LSTMs as exactly same as RNNs except the method is which the hidden state is calculated, is slightly different!.
![Page 11: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/11.jpg)
Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product
![Page 12: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/12.jpg)
LSTM - Unrolled
![Page 13: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/13.jpg)
X
y
![Page 14: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/14.jpg)
Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product
![Page 16: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/16.jpg)
Compute as-a service
Provisioning and managing of the servers to run
code.
Event-driven compute service - runs code in response
to events, such as changes to data in an Amazon
S3 bucket.
Runs code on a high-availability compute
infrastructure and performs all of the
administration.
Compute Infrastructure - AWS Lambda
![Page 17: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/17.jpg)
High Level Architecture /Neural Networks on the Cloud
High Speed Data Connection
Scheduled Daily Lambda Trigger
Pull
Dat
a Fe
eds
and
Dat
a Pr
epro
cess
Pandas DataFrame
Prd 1Prd 2
Prd 3
Object Upload
S3 Object Upload Event Notification
Prediction Lambda Triggered Per Object Upload
Prd 120
Prd 5Prd 4
Prd 3
Prd 1Prd 2
Prd 120
Prd 1
Prediction Date2016-08-03Key ValuePrd1 36Prd2 29…Prd 120 41
Amazon API Gateway
Time Series Product 2 in Market abc
Time Series - Product a
Time Series Product b
Preprocessed Data
Predicted Prices
![Page 18: ODSC - Neural Networks on AWS Lambda](https://reader035.fdocuments.in/reader035/viewer/2022070602/5873b03b1a28aba3548b7367/html5/thumbnails/18.jpg)
X
y