Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving...
-
Upload
lawrence-dalton -
Category
Documents
-
view
216 -
download
0
description
Transcript of Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving...
![Page 1: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/1.jpg)
Locations
![Page 2: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/2.jpg)
Soil Temperature Dataset
![Page 3: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/3.jpg)
Observations
• Data is – Correlated in time and space– Evolving over time (seasons)– Gappy (Due to failures)– Faulty (Noise, Jumps in values)
![Page 4: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/4.jpg)
Temporal Gap Filling
• Data is correlated in the temporal domain
• Data shows– Diurnal patterns– Trends
• Trends are modeled using slope and intercept
• Correlations captured using functional PCA
![Page 5: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/5.jpg)
Gap Filling (Connolly & Szalay)+
Original signal representation as a linear combination of orthogonal vectors
Optimization function- Wλ = 0 if data is absent- Find coefficients, ai’s, that minimize function.
+ Connolly et al., A Robust Classification of Galaxy Spectra: Dealing with Noisy and Incomplete Data, http://arxiv.org/PS_cache/astro-ph/pdf/9901/9901300v1.pdf
![Page 6: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/6.jpg)
Soil Temperature Dataset (same as slide 2)
![Page 7: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/7.jpg)
Gap-filling using Daily Basis
Only 2% of the data could be filled using daily model
Good Period
![Page 8: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/8.jpg)
Observations
• Hardware failures causes entire “days” of data to be missing– Temporal method (only using temporal
correlation) is less effective
• If we look in the spatial domain,– For a given time, at least > 50% of the sensors are
active– Use the spatial basis instead.– Initialize spatial basis using the period marked
“good period” in previous slide
![Page 9: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/9.jpg)
Gap-filling using sensor correlations
Faulty data
Too many missing readings for this period (Snow storm!)
![Page 10: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/10.jpg)
Accuracy of gap-filling
High Errors for some locations are actually due to faults Not necessarily due to the gap-filling method
![Page 11: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/11.jpg)
Examples of Faults
![Page 12: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/12.jpg)
Observations & Incremental Robust PCA *• Data are faulty– Pollutes the basis– Impacts the quality of gap-filling
• Basis is time-dependent– Correlations are changing over time
• Simultaneous fault-detection & gap-filling– Initialize basis using reliable data– Using basis at time t,
• Detect and remove faulty readings• Weight new vectors depending on residuals• Update thresholds for declaring faults
– Iterate over data to improve estimates+ Budavári, T., Wild, V., Szalay, A. S., Dobos, L., Yip, C.-W. 2009 Monthly Notices of the Royal Astronomical Society, 394, 1496, http://adsabs.harvard.edu/abs/2009MNRAS.394.1496B
![Page 13: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/13.jpg)
Gap-filling using sensor correlations (same as slide 8)
![Page 14: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/14.jpg)
Iterative and Robust gap-filling
-Fault removal leads to more gaps and less data to work with -However, data looks much cleaner.
![Page 15: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/15.jpg)
Impact of number of missing values
Even when > 50% of the values are missingReconstruction is fairly good
![Page 16: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/16.jpg)
Iteration - I
![Page 17: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/17.jpg)
Iteration - II
![Page 18: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/18.jpg)
Accuracy using spatiotemporal gap-fillingLocations
![Page 19: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/19.jpg)
Example of running algorithm
![Page 20: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/20.jpg)
ETC
![Page 21: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/21.jpg)
Daily Basis
![Page 22: Locations. Soil Temperature Dataset Observations Data is Correlated in time and space Evolving over time (seasons) Gappy (Due to failures) Faulty.](https://reader036.fdocuments.in/reader036/viewer/2022081404/5a4d1b947f8b9ab0599c2e6c/html5/thumbnails/22.jpg)
Collected measurements