Dynamic Data-Driven Application Systems

(DDDAS)What is DDDAS?

Dynamic Data Driven Application Systems

DDDAS is a new paradigm.

Old ways: Application simulations use static data input and start up.

DDDAS (continued)

Application simulations are able to

RECEIVE AND RESPOND to ONLINE

physical data and measurements and/or

control the measurements.

Why DDDAS? Computer models were not designed to deal

with dynamic conditions as the simulations already started.

Example: simulating wild fire. We need to know which community needs to be evacuated. The wild fire simulation needs to adjust based on the wind direction and other factors.

Why DDDAS (continued)

Also fueled by advances in: Applications and algorithms for parallel and

distributed platforms. Computational steering and visualization. Computing. Networking. Sensors and data collection. System software technologies.

What DDDAS NeedsThree research areas: Application: data driven technology.

Algorithm: dynamic data injection and data perturbation tolerance.

System software: support for dynamic environments.

Static applications can be changed into a more usefuldynamic data driven applications.

DDDAS General PropertiesWhat’s new in DDDAS?Feedback and control interactions between computations and the physical measurement systems.

There are three ways to interact with DDDAScomputation: Human interaction. Physical system interaction. Computational infrastructure interaction (machines

and their connections).

DDDAS Interactions

DDDAS Key Characteristics

Key characteristics that need to be addressed:

Time dependency and/or real time aspect.

Data streams in addition to data sets.

Interactive visualization and steering.

DDDAS ExampleTsunami simulation.(http://www.pgc.nrcan.gc.ca/geoscapevictoria)

Where we get the data from?Tsunami can be caused by tectonic plate movement, or sea floor quakes.

Data sources: GPS stations, sea floor sensors.

DDDAS Example 1 (continued)

DDDAS Example 1 (continued)

DDDAS Example 1 (continued)

DDDAS Example 2

Air traffic simulation. (http://www.simlabs.arc.nasa.gov/cvsrf/atcs.html)

Where we get the data from?We need the world location for all the aircrafts, as well as their schedules. Outside data: weather simulations.

Data source: Aircraft transponders.

DDDAS Example 2 (continued)

DDDAS Example 3Medical imaging and simulation.(http://www.bitc.gatech.edu/bitcprojects/eye_sim/eye_surg_sim.html)The Georgia Institute of Technology and the Medical College of Georgia.

We need the real life model of the object of interest. We can do this by using Intensity Modulated Radiation Therapy (IMRT).

Simulating eye surgery. To blind or not blind.

DDDAS Example 3 (continued)

DDDAS Example 3 (continued)

DDDAS Researches: Past, Present, and Future.

Reference taken from DDDAS.org Home Page, NSF Official DDDAS Page, NSF 2000 Workshop on Dynamic Data-Driven Application Systems, and Performance Engineering Technology Notes.

DDDAS: Past and Present

Past ExperienceFebruary 17, 2000: Meteorologists missed

predicting the track and magnitude of a major storm in January 24-25, 2000, that blanketed major cities from South Carolina to New England.

May 7, 2000: The National Park Service started a controlled burn near Los Alamos National Laboratory. Within a day, the fire was labeled a wildfire.

What is being sought from DDDAS?

DDDAS Capability: simulation applications that can dynamically accept and respond to field data and measurements, and can control such measurements in a dynamic manner through the symbiotic measurement and simulation systems.

How DDDAS Affects Our Present and Future Examples of Applications benefiting from DDDAS:

Engineering Design and Control: aircraft design, oil exploration, semiconductor manufacturing, structural engineering, computing systems hardware and software design and runtime.

Medical: customized surgery, radiation treatment, BioMechanics/BioEngineering

How DDDAS Affects Our Present and Future (continued) Economy: Production planning and control,

financial trading (stock market, portfolio analysis).

Crisis Management and Environmental Systems: Weather, hurricane/tornado prediction system, floods, fire propagation, transportation systems (emergency planning, accident response)

Challenges in Enabling DDDAS Capabilities

Application Simulations Developments

Algorithms

Computing Systems Support

Challenges in Enabling DDDAS Capabilities (continued) Applications:

Ability of the application to interface with measurement systems.

The stream data might introduce new modalities to describe the system, like a different level of the physics involved, in cases where the analysis is about a physical system.

Ability to dynamically select the application components depending on the dynamically streamed data.

Challenges in Enabling DDDAS Capabilities (continued) Algorithms:

Stable to dynamically injected data/ tolerant to perturbations of dynamic input data.

Visualization with a human in the loop: feedback to the simulations.

Handling data uncertainties: continuous sensitivity analysis

Challenges in Enabling DDDAS Capabilities (continued) Computing Systems Support:

Support environments where the application requirements change during the execution depending on the streamed data/dynamic execution support on heterogeneous environments.

Extended spectrum of platforms: Grid Computing and beyond.

What is Grid Computing?

What is Grid Computing? (continued)

Grid computing enables the virtualization of distributed computing and data resources such as processing, network bandwidth and storage capacity to create a single system image, granting users and applications seamless access to vast IT capabilities.

Just as an Internet user views a unified instance of content via the Web, a grid user essentially sees a single, large virtual computer.

At its core, grid computing is based on an open set of standards and protocols that enable communication across heterogeneous, geographically dispersed environments. With grid computing, organizations can optimize computing and data resources, pool them for large capacity workloads, share them across networks and enable collaboration.

What is Grid Computing? (continued) Like the Web, grid computing keeps complexity hidden: multiple users enjoy a single,

unified experience.

Unlike the Web, which mainly enables communication, grid computing enables full collaboration toward common business goals.

Like peer-to-peer, grid computing allows users to share files.

Unlike peer-to-peer, grid computing allows many-to-many sharing — not only files but other resources as well.

Like clusters and distributed computing, grids bring computing resources together.

Unlike clusters and distributed computing, which need physical proximity and operating homogeneity, grids can be geographically distributed and heterogeneous.

Like virtualization technologies, grid computing enables the virtualization of IT resources.

Unlike virtualization technologies, which virtualizes a single system, grid computing enables the virtualization of vast and disparate IT resources.

Why DDDAS Now?DDDAS has the potential to revolutionize science, engineering, medicine, economy, management systems, and etc.

We have technological progresses that has advances the level of overcoming the challenges:

- computing speed (terascale, uni- and multi-processor systems, Grid Computing, Sensor Networks).

Why DDDAS Now? (continued)- System software

- Applications (parallel and grid computing)

- Algorithms (parallel and grid computing, numeric and non-numeric techniques, data

assimilation, and chaotic Monte-Carlo method).

Summary and What the Future Holds

NGS: Next Generation Software (1998 - …) Develops systems software supporting dynamic

resource execution

Summary and What the Future Holds (continued)

SES: Scalable Enterprise Systems Program (1999, 2000-2003) Geared toward commercial applications.

ITR: Information Technology Research (NSF-wide project) Has been used as an opportunity to support DDDAS-

related efforts.

Summary and What the Future Holds (continued) Simultaneous advances on the models, Simultaneous advances on the models,

methods, and algorithms that underpin the methods, and algorithms that underpin the components – and on their components – and on their systematic systematic integrationintegration to target strategic applications – to target strategic applications – are crucial for realizing the potential of are crucial for realizing the potential of DDDAS.DDDAS.

But hardware, software, and Cyber But hardware, software, and Cyber Infrastructure alone are insufficient to achieve Infrastructure alone are insufficient to achieve this goal.this goal.

Case Study: Architecture of the DDDAS Wildfire Model

Major Components of the Model Coupled atmosphere/fire model

Legacy NCAR code Combined with the latest techniques, such as OpenMP and Multigrid

Data acquisition From the Internet: GIS maps, past fire information, weather Field information: photos taken from aircraft, field sensors

Visualization and user interface runs on PDAs or cell phones in the field

Dynamic Data Assimilation control module Incorporates data from the field Bayesian data assimilation Data assimilation steers the data acquisition

Guaranteed secure communication infrastructure

The NCAR Coupled Atmosphere/Fire Model The interaction of fire and atmosphere is important

Heat flux from the fire to the atmosphere produces fire wind Wind facilitates the fire spread

Traditional fire models cannot represent this interaction Wildfires are difficult to model Limited computational resources

An meteorological model is coupled with an fire spread model The atmosphere model is based on the Clark-Hall Atmospheric Model Fire model can be an empirical model, or a more realistic Stochastic

Reaction-Diffusion Equation Fire Model Represents the important interaction between fire and atmosphere,

more accurate and closer to reality

Data Acquisition Geographical, weather, and fire information available from the

Internet Weather information: NOAAPORT broadcast, MesoWest weather stations,

and the Rapid Update Cycle (RUC) weather system by NOAA Past fire information: the GeoMAC project at USGS Fuel type: national database

Advancement of the fire front infrared pictures taken from aircraft GPS for aircraft position 3-axis inertial measurement to get the pointing direction of the camera

Field fire and weather information data Sensors arbitrarily placed in the vicinity of a fire for point measurements of fire

and weather information

Visualization and User Interface Simulation results (prediction) will be transferred to the

firefighters in the field

Need an easy and portable way to visualize the simulation result

Firefighters don’t want to carry a notebook when fighting fires

PDA (or cell phone) and Java is the natural choice

Limited computing power and memory

Needs Java-based graphic software

Why Dynamic Data Assimilation? Wildfire is a complex process with lots of uncertainties, such as

wind, humidity, temperature

Neither empirical model nor physical model cannot represent the wildfire very well

Lots of parameters cannot be measured accurately

All in all, the system is heavily non-linear and ill-posed

Sequential Bayesian data assimilation can be used to guide the simulation

How Dynamic Data Assimilation works? The state of the system at any time - physical variables and

parameters of interest at mesh points

Time-state vector x - snapshots of system states at different points in time

The knowledge of the time-state of the system - probability density function p(x)

p(x) is represented by a ensemble of time-state vectors x1, x2, …, xn

Number of system states maintained = size of the ensemble * number of snapshots

Thousands of simulation run simultaneously

Sequential Bayesian Filtering Current state of the model

prior probability density –

Incorporate data from the field Measurements – vector y How the data is derived from

x – p(y|x)

Posterior probability density –

The system advances in time from the posterior probability density until new data arrives; this process is called an analysis cycle

)(xp f

dpypxpxypxp

f

fa

)()|()()|()(

Standard Approach of Data Assimilation by Ensemble Filter

Initialization: generate initial ensemble by a random perturbation of initial conditions

Repeat the analysis cycle: Advance ensemble states

to a target time by solvingthe model PDEs in time

Inject data with time-stampsequal to the target time: modify ensemble states by a Bayesian update

Overall Pictures

References Geoscape Victoria http://www.pgc.nrcan.gc.ca/geoscapevictoria/ Air Traffic Controls Simulator

http://www.simlabs.arc.nasa.gov/cvsrf/atcs.html Eye Surgery Simulation

http://www.bitc.gatech.edu/bitcprojects/eye_sim/eye_surg_sim.html

DDDAS by Dr. Craig Douglas http://www.dddas.org/ DDDAS by NSF

http://www.nsf.gov/cise/cns/darema/dd_das/index.jsp Jan Mandel’s DDDAS Website

http://www-math.cudenver.edu/~jmandel/dddas03/