Multimedia on the Mountaintop: Using Public Snow Images to Improve Water
Systems Operation
A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani Politecnico di Milano, Italy
ACM MM 2016, AmsterdamBNI session
The (hopefully brave new) idea
• There is a lot of multimedia content out there, produced by– People– Ground sensors
• There are many environmental problems that lack affordable and accessible input data
• Question: is public web visual content good enough to help in such environmental problems?
Observing the earth
• Not everything can be done from above• There is not a single satellite product good for all• (Useful) satellite products are costly• Clouds may be a problem
The grand challenge: water scarcity• Climate change, urban concentration and agriculture
put water resources under stress• Predicting future availability is key• When you have mountains, water is stored as snow
UK_WATER SUPPLY UTILITY15 million customers2.6 Gl/day drinking water3 billion $ revenue (2013-14)
The contentInput• User generated
– 700.000 Flickr images crawled so far within 300x160 km
• Sensor generated– 2000 webcams queried every
minute (10 – to 1500 images per web cam per day)
– More than 10M images crawled so far
Output• Virtual Snow Indexes:
numerical time series that are a proxy of the quantity of water stored in the snow pack (Snow Water Equivalent – SWE)
The multimedia pipelines
• Differences– Web cam images have high temporal density, UG images
have broader spatial coverage– UG photos searched by keywords may be irrelevant,
webcam images always portrait mountains– UG photo mountain classifier already discards bad
weather images
UG Image relevance
• 7000 images randomly sampled and used for a crowdsourcing experiment: “Do you see a mountain in this picture?”
• Classifier trained (94% precision, 96.3% recall)
Webcam image enhancement
Remove/attenuate:• Variability of illumination• Shadows• People & irrelevant objects
Daily median image
Mountain peak identification
orginal image edge maps
skyline estimationDEM generated virtual panoram
VCC best matching
Snow mask extractionSnow classification at the pixel level
Snow mask extraction
Snow Virtual Indexes
The case study• Regulation of mountain inflow dependent lakes
Lake Como Catchment area Lake Como 4500 km2
Reservoirs Lake Como 247 Mm3 Alpine HP 545 Mm3
StakeholdersFarmers:
irrigated area 1400 km2
Floods:lake and downstream
….
Local folklore
Formalization: 2 objectives optimization
• Decide the daily lake outflow ( lake level)
• So to– Maximize water for
downstream irrigation– Minimize # of flood days
• Respecting– Minimum outflow
requirement for ecological preservation of effluents
• Based on– Policy input (X)
• Regulator's policies– Baseline: regulator only considers lake
level and day of year– Upper bound: regulator knows the
water that will be available (lake inflow) in the future
– P_x: regulator knows partial information (x) on the water that will be available (lake inflow) in the future
• What is X?– P1: Official snow water equivalent
data estimated from Region Lombardy– P2: virtual snow indexes from nearby
mountain images– P3: official SWE data + virtual snow
indexes
PS: Upper bound policy can be calculated retrospectively for the past, where you know how much water you actually got day by day
Assessment method
Select information based on its
expected value(Iterative
Input Selection)
Design control policy based on selected input
information
Quantify performance of policy + selected
information
Quantify value of perfect
information Expected Value of Perfect Information (EVPI)
Inflow data series Outflow data series
Baseline policy Upper
boundpolicy
Input data
series(exogenous
variables)
Most Valuable Information
(X)
X_informed control policy(P_x)
J(P_x)Performance of
P_x
Performance metricsHyper Volume Indicator
(HV)
Performance improvement over baseline(ΔHV)
Assessment results
Thank you & … see you soon in the PlayStore
Content processing pipeline• Photo contains/does not contain mountain landscape
binary classifier– SVM with Dense SIFT, Spatial Histograms. 7k annotated
images (majority of 3 votes). 95.1% Accuracy on balanced dataset.
• Peak identification / Photo orientation estimation– Ad-hoc algorithm with edge extraction and vector cross-
correlation. 160 images manually aligned w.r.t. Digital Elevation Model. 75-81% of images correctly aligned (depending on weather conditions).
• Pixel-wise snow/non snow classifierRandom Forest, trained/evaluated on 60 manually segmented images (single annotator) for a total of 7M of labeled pixels. 91% accuracy.
Iterative input selection
Select information based on its
expected value(Iterative
Input Selection)
Design control policy based on selected input
information
Quantify performance of policy + selected
information
Quantify value of perfect
information Expected Value of Perfect Information (EVPI)
Inflow data series Outflow data series
Baseline policy Upper
boundpolicy
Input data
series(exogenous
variables)
Most Valuable Information
(X)
X_informed control policy(P_x)
J(P_x)Performance of
P_x
Performance metricsHyper Volume Indicator
(HV)
Performance improvement over baseline(ΔHV)
D=distance metric
Policy search
Select information based on its
expected value(Iterative
Input Selection)
Design control policy based on selected input
information
Quantify performance of policy + selected
information
Quantify value of perfect
information Expected Value of Perfect Information (EVPI)
Inflow data series Outflow data series
Baseline policy Upper
boundpolicy
Input data
series(exogenous
variables)
Most Valuable Information
(X)
X_informed control policy(P_x)
J(P_x)Performance of
P_x
Performance metricsHyper Volume Indicator
(HV)
Performance improvement over baseline(ΔHV)
Good decisions matter
WATER DEFICIT
FLOOD THRESHOLD
EFFECT OF REGULATION
For more info• A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani:
[email protected]• http://snowwatch.polimi.it/