Smart Grids: the Big Data challenge
JOURNÉE ORES DU 19 NOVEMBRE 2015
Zacharie DE GREVE, Lazaros EXIZIDIS, Martin HUPEZ, Vasiliki KLONARI, Benjamin PICART, Jean-François TOUBEAU, François VALLEE
Electrical Power Engineering Unit, Faculty of Engineering, University of Mons
GREDOR Project
Université de Mons
One selected message from CIRED 2015
2Z. De Grève | Electrical Power Engineering Unit
« … What are the [new] roles of DSOs in the transformation of the energy system ?• DSOs will need to actively manage and operate smarter grids• DSOs will implement the roll out of smart metering• DSOs will become data managers […and enter the world of Big Data…]• DSOs will play a key role in the design and implementation of local energy
policies and the development of smart cities…»
Philippe Monloubou, CEO at ERDF, « Power distribution at the heart of the energy transition »,
CIRED 2015, 15th of June 2015, Lyon
(23rd International Conference and Exhibition on
Electricity Distribution, June 2015, Lyon, France)
Université de Mons
Massive roll out of Smart Metering devices
3Z. De Grève | Electrical Power Engineering Unit
ERDF (principal DSO in France) Linky project: up to 35 million of metering devices until 2021
ORES: Smart Metering (SM) devices for all the customers in 15 years(starting from 2019) [O. Durieux, Journée ORES du 20/11/2014,
Faculty of Engineering, UMONS]
[P. Monloubou, CIRED2015, Lyon, France]
… (Enexis in The Netherlands, Enel in Italy, etc.)
Université de Mons
Outline
Why do we need data in Smart Grids?
Some data related challenges
Focus on Big Data Analytics in Smart Grids
A. Data characteristics
B. Two fundamental problems and illustrations
Conclusion and perspectives
4Z. De Grève | Electrical Power Engineering Unit
Université de Mons
Outline
Why do we need data in Smart Grids?
Some data related challenges
Focus on Big Data Analytics in Smart Grids
A. Data characteristics
B. Two fundamental problems and illustrations
Conclusion and perspectives
5Z. De Grève | Electrical Power Engineering Unit
Université de Mons
A great wise man once said…
6Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics Conclusion and perspectives
A close monitoring of the electrical consumption of households will help improving the energy efficiency
« The <to be completed> energy is the one we do not use. »
• <cheapest> if you are an economist,• <greenest> if you are an ecologist,• <most efficient> if you are an engineer,• …
Université de Mons
From classical networks to Smart Grids
Massive integration of Renewable Energy Sources (RES), typically windor solar, in electricity distribution networks
7Z. De Grève | Electrical Power Engineering Unit
Towards a coordinated « smart » management of the network to avoidtechnical problems (e.g. overvoltages, congestions)
• A guideline: consume the energy when it is produced, locally if possible (flexibility of demand using financial incentives, storage, etc.)
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics Conclusion and perspectives
• Recourse to advanced optimization algorithms (dynamic optimization, in an uncertain environment, with nonlinear equations, and a mix between continuous and integer variables)…
Uncertainty of electrical quantities
• Wind and solar production
• Electrical consumption (or load)
Strongly relies on the observability of the distribution network !
Université de Mons 8Z. De Grève | Electrical Power Enigineering Unit
A simple scenario-oriented Monte Carlo approach
8
RES
PriceLoad
StochasticModels
Trajectories
(or scenarios)
• Probabilities of congestion
Indicators
• Probabilities of over/undervoltage
• Reliability indexes (LOLE, etc.)
Sampling
I.
Power flow
computation
II.
Networkmodel
0. Build stochasticmodels
Historical data
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
Conclusion and perspectives
Data for analyzing modern distribution grids
Université de Mons
For building stochastic models of electricalquantities, we need data…
9F. Vallée & Z. De Grève | Electrical Power Engineering Unit
RES TTF/TTR
PriceLoad
SequentialModels
Trajectories
(or scenarios)
• Probabilities of congestion
Indicators
• Probabilities of over/undervoltage
• Reliability indexes (LOLP, etc.)
Sampling
I.
Power flow
computation
II.
Networkmodel
0. Build
Historical data
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
Conclusion and perspectives
stochasticmodels
Data for analyzing modern distribution grids
Université de Mons
Outline
Why do we need data in Smart Grids?
Some data related challenges
Focus on Big Data Analytics in Smart Grids
A. Data characteristics
B. Two fundamental problems and illustrations
Conclusion and perspectives
10Z. De Grève | Electrical Power Engineering Unit
Université de Mons
The technical challenges related to data are numerous…
Communication between devices
11Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics Conclusion and perspectives
Data storage
Privacy and security
• Power Line Communication – or PLC – and GPRS (e.g. ERDF in France, ORES, Enel in Italy, etc.)
• Radio transmission (e.g. Enexis in The Netherlands)
• …
Metering technology
[D. Lonneke, Journée ORES du 20/11/2014,
Faculty of Engineering, UMONS]
Université de Mons
The technical challenges related to data are numerous…
12Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics Conclusion and perspectives
Investment strategies for IT infrastructures
Analysis of the recorded data (Big Data Analytics)
• For DSOs: better coordination of the production/consumptionin a Smart Grid context
• For consumers: improve energy efficiency by closelymonitoring consumption
• For energy suppliers: establish client typical profiles
• …
Focus of the restof this talk
Université de Mons
Outline
Why do we need data in Smart Grids?
Some data related challenges
Focus on Big Data Analytics in Smart Grids
A. Data characteristics
B. Two fundamental problems and illustrations
Conclusion and perspectives
13Z. De Grève | Electrical Power Engineering Unit
Université de Mons 14Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
A. Data characteristicsConclusion and
perspectives
RES and load data are intrisicallysequential
Example of wind speed
Autocorrelation function:
A (biased) estimator:
with:
A “non sequential” signal
A “sequential” signal
Autocorrelation functions of typical wind days:
Day/night cycles= seasonality
KNMI database
http://www.knmi.nl/samenw/hydra/
Université de Mons 15Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
A. Data characteristicsConclusion and
perspectives
RES and load data are intrisicallysequential
0,0E+00
1,0E-05
2,0E-05
3,0E-05
4,0E-05
5,0E-05
1 8 15 22 29 36 43 50 57 64 71 78 85 92
Synthetic Load Profiles (SLPs)
Source: Synergrid (residential customer)http://www.synergrid.be/index.cfm?PageID=16896&language_code=FRA
(To be multiplicatedby annual indexes)
Market quantities (prices, etc.)
Université de Mons 16Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
A. Data characteristicsConclusion and
perspectives
The sequentiality needs to bemodeled !
To evaluate techno-economic strategies which imply a time coupling (e.g. flexibility)
• Storage
Need to store an history, in order to:
• remain within capacity limits,• control the number of cycles (for a
battery), etc.
• Load shifting
with LS
without LSLoad rebound !
Université de Mons 17Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
A. Data characteristicsConclusion and
perspectives
Modeling the sequentialityAbundant literature on the topic of stochastic modeling of RES, at various time scales
• Statistical/data mining: Artificial Neural Networks (ANNs), KalmannFilters, Markov chains (hidden or not), etc.
• Physical (e.g. meteo): computational fluid dynamics for wind, etc.
• Hybrid approaches, etc.
Time Series models: a simple and efficient way for generating future RES trajectories/scenarios• Able to mimic the statistical properties of real data
• Stored in a compact form. Ex:
Ex: wind speed
AutoRegressive Moving Average (ARMA)
MA(q) process(innovation)
AR(p) process
Université de Mons
Outline
Why do we need data in Smart Grids?
Some data related challenges
Focus on Big Data Analytics in Smart Grids
A. Data characteristics
B. Two fundamental problems and illustrations
Conclusion and perspectives
18Z. De Grève | Electrical Power Engineering Unit
Université de Mons 19Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
B. Two fund. problemsand illustrations
Conclusion and perspectives
Two fundamental problemsI. High dimensionnality of the underlying optimization problems
… ……
……
Scenario tree (e.g. for wind/PV production, load, etc.)
Probability of occurrence of scenario
Computational burden !
1. Clustering techniques to limit the number of scenarii
Ex. 1: a clustering example on wind data
…
2. Orient the sampling by modeling dependencies inherent to data
Ex. 2: wind geographical
correlation
Wind speed for site 2
[m/s]
Wind speed for site 1 [m/s]
Université de Mons 20Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 1: clusteringon wind speed
Conclusion and perspectives
Clustering for merging scenariiExample 1: clustering on wind speed
Wind speed
Wind speed
Wind speed
Wind speed
Wind speed
Wind speed
t
t
t
t
t
t
Clustering
X N
…
Wind speed
Wind speed
Wind speed
t
t…
• Generation of a reduced number K of typical days of wind, starting from N days of historical data (N >> K)
• Use of K-means, K-medoids algorithms
[B. Picart et al, to be submitted]
Université de Mons 21Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 1: clusteringon wind speed
Conclusion and perspectives
Clustering for merging scenariiKNMI database
• Hourly values of the wind speed and direction
• Since 1950
• 65 stations in Holland
• Open access at:
http://www.knmi.nl/samenw/hydra/
Université de Mons 22Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 1: clusteringon wind speed
Conclusion and perspectives
Clustering for merging scenariiExample 1: clustering on wind speed
Power Spectral Density (PSD): describes how the power of a signal is distributed over the different frequencies
Estimated here using the periodogram method
• Why typical days ?
Zoom
Analysis of Schiphol station (1981-1990) shows a peak at a frequency f ≈ 11,56*10-6 Hz, which corresponds to a period T = 1/f ≈ 24h
[B. Picart et al, to be submitted]
Université de Mons 23Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 1: clusteringon wind speed
Conclusion and perspectives
Clustering for merging scenariiExample 1: clustering on wind speed
• Methodology1. Feature selection using Principal Component Analysis (PCA)2. Clustering using K-means/K-medoids3. Associate a typical day to each cluster (centroids ?)
3 dimensional vectors instead of 24 !
[B. Picart et al, to be submitted]
Université de Mons 24Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 1: clusteringon wind speed
Conclusion and perspectives
Clustering for merging scenariiExample 1: clustering on wind speed
• Methodology1. Feature selection using Principal Component Analysis (PCA)2. Clustering using K-means/K-medoids3. Associate a typical day to each cluster (centroids ?)
Step 1 Step 2 Step 3 Step 4
K-means algorithm:
Centroid
Initialize centroids Assign object to nearest centroid
Compute new centroids
Re-assign
[B. Picart et al, to be submitted]
Université de Mons 25Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 1: clusteringon wind speed
Conclusion and perspectives
Clustering for merging scenariiExample 1: clustering on wind speed
• Methodology1. Feature selection using Principal Component Analysis (PCA)2. Clustering using K-means/K-medoids3. Associate a typical day to each cluster (centroids here)
Similar performance than classical k-means but:
+ 3D instead of 24D vectors+ Physical interpretation of the 3 dominant directions (modeling ramps)
To be addressed:
• “Smoothness” of typical days (centroids…)
• “Extreme” scenario
Ongoing: power network reliability analysis(See B. Picart poster for more details)
[B. Picart et al, to be submitted]
Université de Mons 26Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
B. Two fund. problemsand illustrations
Conclusion and perspectives
Two fundamental problemsI. High dimensionnality of the underlying optimization problems
… ……
……
Scenaro tree (e.g. for wind/PV production, load, etc.)
Probability of occurrence of scenario
Computationnalburden !
1. Clustering techniques to limit the number of scenarii
2. Orient the sampling by modeling dependencies inherent to data
Ex. 1: a clustering example on wind data
Ex. 2: wind geographical
correlation
…
Wind speed for site 2
[m/s]
Wind speed for site 1 [m/s]
Université de Mons 27Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 2: modelingwind correlation
Conclusion and perspectives
« Smart » model sampling strategiesExample 2: modeling wind correlation
• Using the Cholesky decomposition [Villanueva et al, IEEE Trans. on Sus. Energy, 2012]
ARMA model 1
ARMA model 3
ARMA model 2
1. Identify n ARMA models separately, based on historical data
2. Compute the (n x n) correlation matrix R, from historical data
Pearson cofficient
3. Compute the Cholesky decomposition of R
Université de Mons 28Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 2: modelingwind correlation
Conclusion and perspectives
« Smart » model sampling strategiesExample 2: modeling wind correlation
• Using the Cholesky decomposition• Using the Cholesky decomposition [Villanueva et al, IEEE Trans. on Sus. Energy, 2012]
ARMA model 1
ARMA model 3
ARMA model 2
4. Generate correlated wind speed time series
ARMA 1 ARMA 2 ARMA 3
uncorrelated
Ex: KNMI database, Schiphol and Ijmuijgen sites:
Université de Mons 29Z. De Grève | Electrical Power Engineering Unit
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 2: modelingwind correlation
Conclusion and perspectives
« Smart » model sampling strategiesExample 2: modeling wind correlation
• Impact of the correlation on power system reliability indices
[Z De Grève et al, EnergyCon2016, Submitted]
• Ongoing work
Non linear correlation (copula based methods) Complete review of the SoA and comparison on the same test
case Time varying correlation
(Test network with two generation units subject to failures, two wind farms, two loads. Collaboration with Tractebel Engineering)
LOLP: Loss of Load Probability EENS: Expected Energy Not Supplied
Université de Mons 30Z. De Grève | Electrical Power Engineering Unit
Why do we need data in Smart Grids ?
Some data relatedchallenges
Focus on Big Data analytics
B. Two fund. problemsand illustrations
Conclusion and perspectives
Two fundamental problemsII. The case of missing or incomplete data
• SM devices are not installed everywhere,
• sensor failures may generate “holes” in the historical database, etc.
Strategy: extrapolate the missing data based on the available
Ex. 3: a low voltage example using reference Cumulative Distribution Functions (CDFs)
Ex. 4: an approach for filling “holes” in electrical consumption
Université de Mons
And if no SM data is available ?
31F. Vallée & Z. De Grève | Electrical Power Engineering Unit
Example 3: a Low Voltage example using reference CDFs
• (Smart) Metering devices are currently not installed everywhere
SM
HV/MV
MV/LV
SM
MV/LV
Strategy: take advantage of what we already have…
Lack of data more particularly at MV/LV substations (and in LV networks)
• Cluster power of a given area into c= cL + cG categories: cL Demand Components (DCs): residential load, tertiary, industrial,
etc. cG Dispersed Generation Components (DGCs): photovoltaïc, wind,
etc.
LVnetwork
LVnetwork
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 3: building reference CDFs
Conclusion and perspectives
Université de Mons
And if no SM data is available ?
32F. Vallée & Z. De Grève | Electrical Power Engineering Unit
Example 3: a Low Voltage example using reference CDFs
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 3: building reference CDFs
Conclusion and perspectives
• Build c reference Cumulative Distribution Functions (CDFs)
1. normalize energy recordings for nodes withSMs, based on annual produced/consumedenergies,
2. compute c ref CDFs,3. assign the CDFs to nodes without SMs, and
perform analysis (denormalize !).
[Toubeau et al, EnergyCon2016, Submitted]
Available
Data
Université de Mons
And if no SM data is available ?
33Z. De Grève | Electrical Power Engineering Unit
Example 3: a Low Voltage example using reference CDFs
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 3: building reference CDFs
Conclusion and perspectives
• A Low Voltage network with PV (ORES, Flobecq, Belgium)
MV/LVtransformer
1D clustering on annual indexes
[Toubeau et al, EnergyCon2016, Submitted]
Université de Mons
And if no SM data is available ?
34Z. De Grève | Electrical Power Engineering Unit
Example 3: a Low Voltage example using reference CDFs
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 3: building reference CDFs
Conclusion and perspectives
• Comparison with measured data and SLPs on the power exchanged at the MV/LV substation during the month of July
Box plot
CDFs
[Toubeau et al, EnergyCon2016, Submitted]
Université de Mons
Sensors may have failures
35
Client 1 X X ? X
Client 2 X X X X
… X ? X X
? X X X
X X X ?
≈
Why do we needdata in Smart Grids ?
Some data relatedchallenges
Focus on BigData analytics
B. Two fund. problemsand illustrations
Ex. 4: fillingholes in data
Conclusion and perspectives
Example 4: matrix factorization to fill the holesOngoing work with F. Lecron, Management
and Computer Science Group, FPMs
.
• Real data is projected on a space of dimension f < m and f < n• Compute matrixes W and H from the incomplete version of X• W and H yield the missing elements of X thereafter• First tests are ongoing…
Z. De Grève | Electrical Power Engineering Unit
…
Université de Mons
Outline
Why do we need data in Smart Grids?
Some data related challenges
Focus on Big Data Analytics in Smart Grids
A. Data characteristics
B. Two fundamental problems and illustrations
Conclusions and perspectives
36Z. De Grève | Electrical Power Engineering Unit
Université de Mons
Conclusions and perspectives
An improved observability of distribution grids is needed to implementSmart strategies: towards the world of Big Data
37Z. De Grève | Electrical Power Engineering Unit
• Reducing dimensionality (and avoid non realistic states) by using clustering techniques and by orienting the sampling
• Missing (or incomplete) data
• Other issues will appear… (profiling clients ?)
Analysis of metering data (Big data analytics)
New competences for analyzing data in Smart Grids
Signal Processing
Probabilities and Statistics
Machine Learning
Time Series Modeling
Université de Mons
Thank you for your attention
38Z. De Grève | Electrical Power Engineering Unit
Special thanks to:Lazaros Exizidis(3), Martin Hupez(1), Vasiliki Klonari(1), Fabian Lecron(2), Benjamin
Picart(3), Jean-François Toubeau(4), François Vallée(2)
GREDOR Project
(1) (2) (3) (4)