National Ecological Observatory NetworkA project sponsored by the National Science Foundation and operated under cooperative agreement by Battelle.

1

NEON: from scientific strategy to long-term operation

Stefan Metzger1, Ankur R. Desai2, David Durden1, Jörg Hartmann3, Jiahong Li4, Hongyan Luo1, Natchaya Pingintha-Durden1, Torsten Sachs5, Andrei Serafimovich5, Cove Sturtevant1, Ke Xu2

[1]: Battelle Ecology, National Ecological Observatory Network Project, Boulder, CO, USA[2]: University of Wisconsin-Madison, Dept. of Atmospheric and Oceanic Sciences, Madison, WI, USA[3]: Alfred Wegener Institute - Helmholtz Centre for Polar and Marine Research, Bremerhaven, Germany[4]: LI-COR Biosciences, Lincoln, NE, USA[5]: GFZ German Research Centre for Geosciences, Potsdam, Germany

• incorporate lessons-learned through collaborations with bottom-up networks like AmeriFlux, ICOS, LTER, TERN…

• NEON's centralized approach lends itself to explore novel systemic solutions

• starting to give back:

• boots on the ground

• software

• data

• educational resources

2

the spirit

3

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook

4

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook

5

How does spatial structure influence ecosystem function and how do we integrate within and between spatial scales to assess function?

“…data still remain too sparse spatially to test mechanisms of change using models…”

“…provide the required data at high spatial and temporal resolution with the necessary continuity…”

6

7

“... scaling down from global and continental measurements and scaling up from stand-level measurements both are critical for an integrative program of C research.”

“Each approach has different spatial and temporal domains ..., and they have the potential to constrain the next level up or down”

“One of the main barriers to rapid improvement of models is the lack of ground data for validation purposes.”

8

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook

9

The National Ecological Observatory Network

10

NEON multi-scale observing system

11

state of the NEON “fairytale”

12

state of the NEON “fairytale”

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

instrumented sites

25

in-situ sampling, proximal and remote sensing

26

in-situ sampling, proximal and remote sensing

27

in-situ sampling, proximal and remote sensing

28

in-situ sampling, proximal and remote sensing

29

in-situ sampling, proximal and remote sensing

30

in-situ sampling, proximal and remote sensing

31

32

the National Ecological Observatory Network

33

0 10 20 30Exp

lain

ed

eco

-

clim

ati

c va

ria

bil

ity

Number of Domains

Insufficient Explanatory Power Unbounded Expense

why 20 domains?

2/3 detection rate

34

data: science implications

Mahecha et al., 2017

detecting extreme events

128 AF sites

39 NEON sitesAmeriFlux

sites across eco-climatic regions

Hargrove and Hoffman, 1999 & 2004

NEON

35

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook

36

data operations architecture

37

science operations management

• problem tracking and resolution along the entire chain

• training

• sensor preventive maintenance

• sensor calibration

• sensor health status monitoring, incident tracking and resolution

• data processing

• continuous data quality monitoring

• data revisioning

38

• problem tracking and resolution along the entire chain

science operations management

39

• problem tracking and resolution along the entire chain

science operations management

40

data products and access

41

data products and access

42

data products and access

43

data products and access

44

data products and access

45

data: interoperability

46

data: interoperability

47

data: interoperability

48

data: interoperability

NEON airborne remote sensing

49

data: interoperability

collocated1. Bartlett Experimental Forest (US-Bar) / BART2. Harvard Forest (US-Ha1) / HOPB3. Konza Prairie (US-Kon) / KONZ-KING-KONA cluster4. Kansas Field Station (US-KSF) / KFS5. Niwot Ridge (US-NR1) / NIWO-COMO6. Santa Rita Creosote (US-SRC) / SRER7. Eight Mile Lake Permafrost (US-EML) / HEAL8. Barrow (US-NGB & US-Bes) / BARR

adjacent9. Poker Flat Research Range (US-Prr) / CARI-BONA

in vicinity (assignable assets)10. Slashpine Austin Cary (US-SP1) / BARC-OSBS-SUGG cluster11. Sylvania (US-Syv) / UNDE- CRAM cluster

NEON airborne remote sensing

50

educational resources

51

educational resources

52

educational resources

53

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook

54

data operations architecture

• eddy4R family of R-packages (raw data → 30 min)

55

software: eddy4R eddy-covariance R-packages

• eddy4R family of R-packages (raw data → 30 min)

• NEON's eddy4R, nneo, metscanner + MPI's REddyProc R-packages: end-to-end, modularly adjustable and extensible workflows in single R-environment

56

software: eddy4R eddy-covariance R-packages

R workflow

tur ulen e

processing

parameters

L1 – L4

HDF5 files

loop around planar-fit/Wavelet period

collect around planar-fit/Wavelet period

ingest remote-sensing data

collect: turbulence data

transform SONIC planar-fit

field-validate IRGA

data-parallel: turbulence data read L0p

transform AMRS

calculate Wavelet

data-parallel: processing core correct frequency response

calculate L1-L4 DPs, Wavelet stats

collect: processing core

model footprint

perform, summarize QA/QC

quantify, summarize uncertainty

lag-correction

• Docker shipping container system for code

• eddy4R-Docker: turn-key, reproducible, extensible and portable data processing + analysis environment

• DevOps community development framework

57

software: community access and extensibility

Computer

58

data: eddy-covariance

processing pipeline characteristics

• near-real-time (1 week → 1 month)

• extensible: e.g. operational data fusion

59

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook

60

scope of the taskTe

mpo

ral s

cale

Spatial scale [km2]

Second

Minute

Hour

DayWeekMonth

Year

Decade

Century

Airborne

Space-borne

10−6 100 106 1012

Ground-based

Airborne

Space-borne

Ground-based

Resolution

Coverage

Model-data fusion

Earth systemmodel

61

environmental response function virtual control volume

Footprint modeling over coordinated observations of drivers and responses

Metzger (2017)

62

environmental response function virtual control volumegrid-projected turbulent heat flux

r CpdT / dt

[W /m3]

2011-08-13 CST

volume-projected heat storage change

Xu et al. (2017)

63

environmental response function virtual control volumestorage flux turbulent flux grids surface-atmosphere

exchange grids

[W/m2]

r CpdT / dt

[W /m3]

aaadfweferggdgfa

Xu et al. (2017)

application examples

• near-real-time capability (1 week → 1 month, e.g. AmeriFlux)

• operational data fusion (e.g., NASA ECOSTRESS)

64

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook

• bridging scales one of the fundamental challenges in ecology

• NEON’s observation hierarchy is designed for scaling (continuity)

• centralized data operations efficiently manage data quality and standardization

• coordinated, interoperable NEON flux, in-situ, proximal and remote sensing data

• public NEON “eddy4R” + GitHub + Docker flux software and usability tools

• combining data across scales with environmental response functions

• joint proposal writing for operational data fusion

65

conclusions and outlook

66

• NEON data products span large spatio-temporal scales, e.g. airborne remote sensing and automated tower measurements

• we aim to bridge these spatio-temporal gaps in NEON data products using a unmanned aerial vehicle remote-sensing platform

airborne remote-sensing observation node (ARGON)

• objectives

develop less expensive and more agile remotely sensed data products.

enabling target-of-opportunity measurement campaigns for extreme ecological events

67

overview

• why bridging observational scales?

• NEON: designed for scaling

• centralized data operations and monitoring

• scalable scientific computing

• combining ground-based, airborne and space-borne data

• summary and outlook