Integrated Air Quality Information System: Challenges Posed by Key Community Members

EPA• Rich Scheffe, EPA OAQPS• Steve Young, EPA OEI• Terry Keating, EPA ORD

NASA

• L. Friedl, K. Fontaine, NASA HQ• Frank Lindsey, NASA HQ

WMO/IGOS• Len Barrie, WMO

GEOSS

Scheffe Challenge

GEOSS

Eco-informatics

Accountability/indicatorsSIPs, nat.rules

designations

PHASE

PM research

Risk/exposureassessments

AQ forecastingPrograms

NAAQS setting

EPANOAA

NASA

NPS

USDA

DOE

PrivateSector

States/TribesRPO’s/Interstate

Academia

NARSTO

NAS, CAAACCASAC, OMB

Enviros

Organizations

CDC

Supersites

IMPROVE, NCorePM monit, PAMS

CASTNET

Lidarsystems

NADP Satellite data

Intensive studies

PM centers

Other networks:SEARCH, IADN..

Data sources

CMAQGEOS-CHEM

EmissionsMeteorology

Health/mort.records

The Scheffe Challenge: ‘Organizations - Programs – Data Mess’

Info System Challenges:

What’s the overall dependencyInformation FlowForces and Controls on Data FlowCooperation, Competition, Co-OpetitionInformation System of Systems

Air Quality Management System: Components and Functions

Public

AnalyzingInterpreting Evaluating Separating Synthesizing

OrganizingQuality controlFormattingDocumentingDisplaying

DecidingEvaluate optionsMatching goals CompromisingChoosing

Data Manager, Organizer

Technical Analysts, Program Manager

Policy Analysts, Decision Maker

Valu

e Ad

ding

Pr

oces

ses

Human Agents

Decision Support System (DSS)

The primary purpose of data systems is to serve programs/projectsPrograms perform analysis for Orgs., the DSS is within programsThe big decisions of societal importance are Organizations (This needs more wisdom from the practioners)

Relationship BetweenOrganizations - Programs – Data

Version 0.1

Goals $$Info needs, $$

Data need, $$

Judge, Decide, ActAnalyze, Report

Actionable Knowledge

Decision, Action

Public

Measure, Organize

Organized Data

Flow of InformationData systems organize the measurements and models and provide them to programs. Programs analyze the data and provide actionable knowledge to organizations.Organizations evaluate multiple information sources, make decisions and act.

Flow of ControlPublic and special interest groups set up organizations and provides them with funding Organizations develop programs, define their scope, governance and fundingPrograms satisfy their information needs by monitoring or by using other’s dataData sources acquire the data for their parent programs and also expose them for reuse

Information System Components for AQ Programs

Integrated DataDatasets

Std.

In

terf

ace

Data Views

Std.

In

terf

ace

Data

Control

Reports

Reports

Public

Human Information SystemMachine Information System

Obs. & Models Decision Support System

Flow of Data and Usage Control

Flow of ControlReports are commissioned by programs Analysts select, explore, and process data for a report

Flow of DataProviders expose data to analysts who extract the needed subsetThe data is pulled into data exploration or processing software

Integrated DataDatasets

Std.

In

terf

ace

Data Views

Std.

In

terf

ace

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Flow of Data and Usage Control

Data

ControlRequesting Information

Providing Information

Sensors

Acquisition processing User

Agencies

User Progra

msNAAQS SIPs Forecast GEOSS …

Info System

hhh

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Steve Young Challenge

The Steve Young Challenges

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

• Real-time monitoring & watch for surprises• Inform decision-makers & assess outcomes• Support for adaptive management

S. Young Challenge #1: Real-time monitoring & watching for surprises

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Choose Data & Tools Browse, Explore Trigger Response

S. Young Challenge #2:Inform decision-makers & assess outcomes

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

• This needs more work• Ideas? How could the info-system help here?

AirQuality

AssessmentCompare to GoalsPlan ReductionsTrack Progress

Controls (Actions)

Monitoring(Sensing)

Close the sensory-motor loop!

S. Young Challenge #3. Support for Adoptive AQ Management

From ‘Stovepipe’ to Workflow Software

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Flow of DataFlow of Control

Air Quality Data

Meteorology Data

Emissions Data

Informing Public

AQ Compliance

Status and Trends

Network Assess.

Tracking Progress

Data to Knowledge Transformation

Loosely Coupled Workflow

Terry Keating Challenge

The Terry Keating Challenges

Facilitate data-model comparison, assimilation

Domain ProcessingData Sharing

Std.

In

terf

ace

Gen. ProcessingSt

d.

Inte

rfac

e

Data

Control

Reports

Reporting

Obs. & Models Decision Support System

Include global perspective (observation, model, analysis)

Architechture Challenge

Friedl Fontaine Barrie Lindsey

GEOSS Architecture Mapped to Air QualityThe L. Friedl, K. Fontaine Challenge

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Not part of DataFedPolicy, Management, Personal Decisions

Special Architectural Features:Implied data assimilation into modelsDecision Support black boxFocus on model predictions, not ‘simulation’ Pollutant ‘Characterization’ is implicit

IGOS Architecture Mapped to DataFedThe Len Barrie Challenge

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Not part of DataFedQuality assurance, CalVal?Integrated Data Archives? -

Special Architectural Features:Real-time data assimilation into AQ modelsCharacterization of global pattern

F. Lindsey, NASA:

Air Quality Collaborative Consortium

Goal: Cross-leverage the shared partner resources and activities, while

Maintaining partner’s autonomy in capabilities and activities

Approach:Exchange the resources at the web ‘interfaces’, like portals.

Infuse cutting-edge technologies and resources into Agencies as needed. Use ESIP portal as the linker and the Air Quality cluster as the mediator.

Outcome:Sharing and integration could form next-generation capabilities.

Savings by re-use and better information by broader perspectivesBetter air quality through more informed management

NOAAEPA

PrivDoI

Air Quality ConsortiumData, Tools, Methods

SharedPrivate

NASA

Other FedS

Applications

AQ Policy

Regulation

Research

DataFed Challenge

Value-Adding Processes

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

AnalyzingFilter/IntegrateAggregate/FuseAssimilate

OrganizingDocumentStructureFormat

ExploringDisplay BrowseCompare

Valu

e-Ad

ding

Pr

oces

ses

Data Manager Technical Analysts

DB System Analysis System Reporting System

Loosely Coupled Data Access through Standard ProtocolsOGC Web Coverage Service (WCS)

Client request Capabilities

Server returns Capabilities and data ‘Profile’

Client requests data by ‘where, when, what’ query

Server returns data ‘cube’ in requested format

GetCapabilities

GetData

Capabilities, ‘Profile’

Data

Where? When? What? Which Format?

Server

Back End St

d.

Inte

rfac

e

Client

Front EndSt

d.

Inte

rfac

e

Query GetData Standards

Where? BBOX OGC, ISO

When? Time OGC, ISO

What? Temperature CFFormat netCDF, HDF.. CF, EOS, OGC

T2T1

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Web Services and Workflow for Loose Coupling

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Service Broker

Service Provider

PublishFind

BindServiceUser

Web Service Interaction Service Chaining & Workflow

Collaborative Reporting and Dynamic Delivery

Integrated DataDatasets

Std.

In

terf

ace

Data ViewsSt

d.

Inte

rfac

e

Data

Control

Reports

Reports

Obs. & Models Decision Support System

Co Writing - Wiki

Screencast

Analysis Reports: Information supplied by manyNeeds continuous program feedbackReport needs many authorsWiki technologies are for collaborative writing

Dynamic Delivery: Much of the content is dynamicAnimated presentations are compellingMovies and screencasts are for dynamic delivery

The Network Effect:Less Cost, More Benefits through Data Reuse

ProgramPublic

Data Organization

DataData Program

ProgramOrganizatio

nDataData

ProgramData

Orgs Develop Programs

Programs ask/get Data Public sets

up Orgs

Pay only once Richer content

Less Prog. Cost More Knowledge

Less Soc. Cost More Soc. Benefit

Data Re-Use Network Effect

Data are costly resource – should be reused (recycled) for multiple applicationsData reuse saves $$ to programs and allows richer knowledge creationData reuse, like recycling takes some effort: labeling, organizing, distributing

GEOSS Challenge 1: Jose Achache

GEOSS/AMI Domain of ActionsBased on ideas of P. Senge: Architecture of Learning Organizations

Possible DataFed Roles:

1. Guiding Idea: Refine, solidify, evangelize the System of Systems idea

2. Methods and Tools: Develop, promote, implement standards; Coordinate SoS use cases

3. Infrastructure: Maintain the DataFed middleware for distributed data access; Supply Agency champions with SoS ‘sales’ material

P. Senge et. al, 1994: Fifth Discipline Fieldbook (Link)

Guiding Idea: System of Systems

Methods, Tools: Standards, Use Cases

Infrastructure: Web 2.0, Local Champs

Domain of ActionOrganizational Architecture

Key Agencies for Air Quality Management and Science

Flexible NAAMS

Advanced Monitoring InitiativeProgrammatic Path to GEOSS

Advanced Monitoring InitiativeProgrammatic Path to GEOSS

• Goal: Advance air quality model-observation complex to level of meteorological FDDA systems

• Build Organizational frameworks : IGACO-EMEP efforts• Data base, IT standards: data unification center? Practical LRTAP task?• Standardization/QAQC Reference material, or method (aerosols)• Adopt model evaluation/fusion as a design principle• Support linkage of ground based and satellite observation platforms through

development of a sustainable vertical profiling system (aircraft and ground based lidar)

• Address integration of disparate data bases– QA/QC: provide requirements for data standards/metadata descriptions– Data base unification

• Harness the communities around major tools, platforms and programs. – Air quality modeling platforms (GEOS-chem, MOZART, CMAQ,….more)– Satellite Instruments (MODIS, OMI)– Existing routine surface (AIRNow) and aircraft programs– Integration efforts (AEROCOM)

The Architecture of SoS Networks

• SoS Architecture – Form and Function Different aspects of the network

•

Right Level of Networking?

• One of the key factors dictating the achievement of system-of-systems configurations that support network centric operations is the availability of mechanisms that promote information sharing among systems. Direct system-to-system linkages presuppose that the systems know a priori about each system that might benefit from any other system. Many combat situations have disproved this assumption.

• Emergent behavior in a network that is only created by the combination, not by the individual members.

• Emergent behavior: novel and coherent (logically connected, purposeful, meaningful) structures, patterns and properties arising from self-organization (mashers are self-organizers) in complex systems Wikipedia

• Emergent behavior is created by mashing • Mashing occurs when open on both ends of network.- diffusion

– Input side – grow by assimilation– Output side – Harvest (i.e. DataFed harvests datasets)

Stages of Self-Organization

• Stage 1: Expose goods. Sharing was doable, but lots of work. Need glue, duct tape, shop-based approach. Value Shop.

• Burden: user pays • Stage 2: Mediated sharing. DataFed,

where wrappers/adapters used. Loose coupling automate the wrapper and mediator

• Burden: Mediator pays. Users gain. • Stage 3: Creating new things together,

service chaining, mashing, collaboration – Emergent behavior where network produces value, non-linearly.

• Burden: creators• Tim Burners-Lee: The web allows

creating new things together. • Howard Rheingold: The computer as

mind amplifiers. • The ultimate goal of all this is to both

amplify the mind of individuals and also connect the minds.