World Modelers - DARPA v2.pdfModeling at Sub-national Scales ... Step 1: Generate or ......

World Modelers

Paul R. Cohen

Information Innovation Office, DARPA

The Goal of World Modelers

Approved for Public Release, Distribution A 22017/01/23 Paul Cohen [email protected]

World Modelers technology will enable analysts to build models rapidly to analyzequestions relevant to national and global security. World Modelers analyses will becomprehensive, causal, probabilistic, and timely enough to recommend specificactions that could avert crises.

A Food Shortage Forecast Example


Task: For any subnational location (e.g., Southern Sudan) generate food shortage scenariostwo years out. Consider a comprehensive set of causes. Integrate current and historical dataand human expert analysis. Model the scenarios quantitatively and probabilistically. Makeforecasts, explain sources of uncertainty, and identify and explain specific potential solutions.Update models, forecasts and probabilities on receipt of new data.

Large organizations have spent years perfecting analytical methods that do some of theabove. World Modelers technology is expected to build models to solve this problem – andmany others like it – in a month.

What/Who FactorsConsidered

Scale Methodology Horizon Time toPrepare

ForecastAccuracy

FEWS-NET1 / USAID Comprehensive Subnational Qualitative 3-6mo. 3-6mo. ?

GIEWS2 / FAO U.N. Comprehensive Subnational Qualitative 3-6mo. 3-6mo. ?

Reports / OECD-FAO3 Agro./Econ. National/Regional Quantitative 10yr 1yr ?

IMPACT / IFPRI4 Climate/Agro./Econ. Regional/Global Quantitative 20yr. 6-24mo. ?

World Modelers Comprehensive Subnational Quantitative 1-3yr 1mo. continuousevaluation

Sources (click for web pages): 1: Famine Early Warning System, 2: Global Information and Early Warning System, 3: Organization for EconomicCooperation and Development/U.N. Food and Agriculture Organization, 4: International Food Policy Research Institute

http://bit.ly/1hqiMba

http://bit.ly/LP8Azl

http://bit.ly/1emFCnT

http://bit.ly/1emFCnT

http://bit.ly/1RVdffN

Modeling at Sub-national Scales


availability

utilization

access

prices

stock productionimportsaid

rural population

urban population

population

shortage/surplus

internaldisplacement

conflict

cross-borderdisplacement

security

rainfall

soil

laborgrowthconditions

shocks

floods drought

El Nino/SouthernOscillationclimate

refugee population

disease

irrigation

markets transport

inputs

profits

World Modelers will model atsubnational scales, preserving thevariability seen at smaller scales.

This variability matters becausefood and other resources aredistributed unequally, withconsequences at subnationalscales (districts or cities).

Modeling variability is harder thanmodeling averages or trends. Morefactors must be considered.

Will there be enough food to feedmost of the world next year? Yes.Will Southern Sudan have enoughfood? This is a much harderquestion.

A simplified, generic model of somefactors that affect food insecurity.Edges represent affects/affected-by and part-wholerelationships.

How World Modelers Will Work


1 2 3 4 5 6 7

Refine models basedon Uncertainty Report

Suppose the task is to forecast food insecurity in districts of Southern Sudan two years out.

◦ Step 1: Generate or retrieve a generic qualitative, causal food security model;

◦ Step 2: Modify the model for the specific analyses of Southern Sudan;

◦ Step 3: Build workflows of expert, quantitative models, where available;

◦ Step 4: Parameterize quantitative models and the qualitative, causal model;

◦ Step 5: Configure scenarios and run analyses, producing quantitative results for factorsof interest (e.g., food prices, calorie intake);

◦ Step 6: Produce an “uncertainty report” that documents sources of uncertainty, rununcertainty-reduction procedures and sensitivity analyses;

◦ Step 7: Identify possible actions to affect factors of interest (e.g., peacekeepers atmarkets).

Step 1: Get a Generic, Causal, Qualitative Model


By reading online material machines will buildgeneric, qualitative models of food insecurity.

If such models exist (perhaps from previousanalyses), machines will access them.

availability

utilization

access

prices


rural population

urban population

population

shortage/surplus


conflict


security

rainfall

soil


shocks

floods drought

ENSOclimate

refugee population

disease

irrigation

markets transport

inputs

profits

A generic model of food insecurity

Sources and Technologies:

◦ Machine reading of online resources such as U.N. Food andAgriculture Organization Country Briefs∗ and OECD long-termprojections∗∗ .

◦ Machine grounding of entities in text into ontologies.

◦ Machine assembly of qualitative, causal models from fragments,based on Big Mechanism technology).

*http://www.fao.org/giews/countrybrief/country.jsp?code=MAR, **10.1787/agr_outlook-2016-en

http://www.fao.org/giews/countrybrief/country.jsp?code=MAR

10.1787/agr_outlook-2016-en

Step 2: Tune Generic Model to Specific Analyses


availability

utilization

access

prices


rural population

urban population

population

shortage/surplus


conflict


security

rainfall

soil


shocks

floods drought

ENSOclimate

refugee population

disease

irrigation

markets transport

inputs

profits

currencydevaluation

uganda

sudan

ethiopia

sudan

uganda

kenya

nodes in the generic model considered relevant by the machine/analystnew nodes added to the generic model by reading about Southern Sudan

Decide which factors in the generic model are, orcould be, relevant specifically in Southern Sudan,adding factors as necessary.


◦ Machines can read recent reports about Southern Sudan tomake good guesses about relevant factors. For example

“The spike in food prices at the end of 2015 coincided withthe decision of the Central Bank to move from a fixed to afloating exchange rate regime that led to a devaluation of thelocal currency by about 84 percent.”**

Market activities have slightly improved in recent months insome conflict-affected areas but food supplies remain wellbelow the pre-crisis levels and food prices remainexceptionally high and volatile, largely influenced by thedistribution of food aid.

“Since the start of the conflict in mid-December 2013, over 2.3million people have fled their homes, including about 1.7million internally displaced and about 647 000 individualscurrently hosted in neighbouring countries (Ethiopia,Uganda, the Sudan and Kenya) as refugees.”

◦ Human analysts would edit the machine’s choices based onexpert or commonsense knowledge that machines won’t have.

**http://www.fao.org/giews/countrybrief/country.jsp?code=SSD

http://www.fao.org/giews/countrybrief/country.jsp?code=SSD

Step 3: Workflow Compiler


availability

utilization

access

prices


rural population

urban population

population

shortage/surplus


conflict


security

rainfall

soil


shocks

floods drought

ENSOclimate

refugee population

disease

irrigation

markets transport

inputs

profits

currencydevaluation

uganda

sudan

ethiopia

sudan

uganda

kenya

FAO and OECD AgLink-Cosimo Model of Agriculture Markets

NOAA Global TropicalSST Forecasts

AgMIP DSSATcrop yield model

Blue boxes are extant software models, e.g., rainfall, crop, and market models.

Where quantitative models are available, link themto the qualitative causal model. This produces acomputational workflow among models.


◦ Build and maintain a corpus of expert, quantitative models inareas such as climate, weather, hydrology, drought forecasting,cropping and yield forecasting, market clearing and price setting,migration, etc.

◦ Build on technologies such as grid computing and modelcomposition systems such as WINGS to suppport human-machineassembly of workflows of models.

◦ Build on the natural language dialog capabilities ofCommunicating with Computers, which support human-machineconstruction of complicated biological models.

Step 4: Parameterize Models


availability

utilization

access

prices


rural population

urban population

population

shortage/surplus


conflict


security

rainfall

soil


shocks

floods drought

ENSOclimate

refugee population

disease

irrigation

markets transport

inputs

profits

currencydevaluation

uganda

sudan

ethiopia

sudan

uganda

kenya




“devaluation of local currency by 84%”

“about 647000 individuals”

access = f(market,price,production)

“estimated at 920,000 tons”

Sources for numeric parameters (in green) include text, tables, and functions that combine outputs of model components

Find parameter values in online sources, setmodel parameters, and specify how outputs ofmodels are combined at nodes in causal network.

In effect, this transforms a qualitative causalmodel into a quantitative model.


◦ Develop technology to semi-automatically access onlinesources (e.g. rainfall maps), read data in various formats(e.g., Big Mechanism “table reading” of historic price data),and read parameter values in documents (e.g., “647,000individuals”).

◦ Develop technology to help analysts find or write functions tocombine numeric parameters when this isn’t already done bynumeric models. For example, while the FAO-OECD AgLinkmodel combines production and import data to forecastprices, no numerical code exists to combine market, price andproduction data to estimate the degree of access to food (incalories per person) in Southern Sudan. Such functions willoften be described in literature (and could be found bymachine reading) but might have to be implemented by hand.

Step 5: Specify Scenarios and Run Analyses


availability

utilization

access

prices


rural population

urban population

population

shortage/surplus


conflict


security

rainfall

soil


shocks

drought

ENSOclimate

refugee population

disease

irrigation

markets transport

inputs

profits

currencydevaluation

uganda

sudan

ethiopia

sudan

uganda

kenya




JFMAMJJASOND JFMAMJJASOND

JFMAMJJASOND JFMAMJJASOND

Analysts can “drive” scenarios by having the machine sample parameter values from distributions, in blue (e.g. rainfall)

Set up scenarios. Scenarios are specified byinput parameter values (blue arrows) thatpropagate through the model to yield results(red arrows).

Run multiple scenarios. Parameter values maybe sampled from distributions. This producesdistributions over output values, confidenceintervals and other representations ofsensitivity or uncertainty.


◦ The program will address some technical issues that continue todrive modeling research (e.g., running linked models that havevery different time scales). Some performers may havecapabilities to run very large simulations on supercomputers,clusters and grids.

◦ The program will develop technology to help analysts avoid“semantic errors” when they specify scenarios, not unlike somekinds of type checking or graph analysis in programming. Forexample, in the current model the analyst’s distribution overrainfall conditions might clash with NOAA rainfall forecasts. Theanalyst must be made aware of this and given options for, e.g.,specifying that his distributions take precedence.

Step 6: Analyze & Mitigate Sources of Uncertainty


availability

utilization

prices

stockimportsaid

rural population

urban population

population

shortage/surplus


conflict


security

rainfall

soil


shocks

floods drought

ENSOclimate

refugee population

disease

irrigation

markets transport

inputs

profits

currencydevaluation

uganda

sudan

ethiopia

sudan

uganda

kenya




Currently access is dominated by security concerns. Future status is uncertain and effects on prices and markets are estimated to be large. Recommend analysis of multiple security scenarios.

In Southern Sudan, growth conditions depend on rainfall. ENSO conditions two years out are uncertain. Recommend sensitivity analysis over ENSO conditions.

production

access

Blue boxes represent illustrative contents of uncertainty reports.

Generate an Uncertainty Report. Uncertaintyaccumulates along causal pathways, driven byinput distributions, combining functions, modeluncertainties, etc. ... A single, scalar “margin oferror” for analyses is not credible. Instead wewant a detailed “uncertainty report” for eachanalysis.


◦ The program will develop technology to A) keep track of whereuncertainty is introduced, B) trace its effects down causalpathways, C) apply uncertainty reduction methods at each pointwhere uncertainty is introduced (e.g., ensemble models, biasingwith or checking against historical data, asking an expert,optimum estimation methods, optimal sampling methods, etc.) D)run multiple scenarios to see the effects of uncertainty (sensitivityanalysis).

◦ Uncertainty analysis is a human-machine task, but the machinecan do much of the work of A,B,C and D under human guidance,and can do exhaustive diagnostic work (e.g., checking whetherintermediate results are within expected ranges) that humanswould find tiresome.

Step 7: Identify Possible Actions


production

availability

utilization

prices

stockimportsaid

rural population

urban population

population

shortage/surplus


conflict


security

rainfall

soil


shocks

floods drought

ENSOclimate

refugee population

disease

irrigation

markets transport

inputs

profits

currencydevaluation

uganda

sudan

ethiopia

sudan

uganda

kenya




Shortages are due primarily to access, which is due to prices and security at markets. We could increase security at markets. We could also increase food aid to internal refugee populations.

access

Green nodes are influences or effects. The green box represents one of perhaps many plans to influence food shortages

Which actions are likely to have positive effects onfactors? What can be done to reduce projectedfood insecurity in Southern Sudan?


◦ Adapt Big Mechanism algorithms that find drug combinations tocombat cancer. These algorithms trace back from desiredoutcomes through causal models to find “pressure points” thatprobably have big effects.

◦ Because it is difficult for humans to find combinations of actions,we expect to adapt Big Mechanism algorithms like MUTEX to find“cliques” of pressure points where multiple actions can beeffective.

◦ The program will build on human-machine, mixed-initiativeplanning algorithms that can take guidance from human users, asmachines lack general world knowledge (e.g., reversing currencydevaluation is harder than increasing food aid to refugee camps)

Task Areas


TA Description Supports Steps Builds On

1 Build Qualitative Modelsfrom Online Sources

1,2,4 Big Mechanism reading for causalfragments, grounding in ontologies,assembly algoithms

2 Workflow Compiler 3 Scientific workflow technology (e.g.,WINGS), grid computing, distributedsimulation, CwC collaborative dialogcapability

3 Parameterize Models 4 Big Mechanism table reading, remotesensing feeds,crowdsourcing and polling

4 From Scenarios to Actions 5,7 Big Mechanism tech for finding“pressure points” in causal networks,AI planning tech for multi-step plans

5 Uncertainty Reports 6 CwC tech for human-machine examinationof models will help. However, human-machine uncertainty analysis for verycomplicated models requires new science

Test Problems and Phasing


Phase 1

Mo.18 Mo.36

Phase 2 Phase 3

Mo.48

Develop Test Develop Test Develop Test

In successive phases, the test problems become less like the development problems:

◦ Phase 1: The test problem is to analyze the same question as seen in the development period,but in a new country (e.g., food insecurity has different causes in Venezuela and South Sudanso the models to be analyzed will be different).

◦ Phase 2: The test problem is to analyze a related question that uses some models and datasources from earlier problems (e.g., migration shares many factors with food security, so a testmight involve a migration problem when none had been seen during development);

◦ Phase 3: The test problem is to analyze a problem unlike any seen before, requiring theassembly of at least some data and models that haven’t been seen in any previousdevelopment problems (e.g., via an MOU with OCIA (part of DHS) we might analyze the effectsof continued drought on critical infrastructure in California).

Evaluation and Scoring


Evaluation will test the central claims of World Modelers:

◦ It provides technology to enable analysts to rapidly build models to analyze questionsrelevant to national and global security.

◦ Analyses will be comprehensive, causal, probabilistic, and timely enough torecommend specific actions that could avert crises.

Evaluations will address specific questions about the value of the technology to analysts:

◦ Utility and Accuracy – Can real analysts use the technology, and do they want tokeep on using it? Are the models’ projections accurate enough to be useful, i.e., tosupport recommendations?

◦ Plausibility – are analyses plausible, do they agree with concurrent, human-only,expert analyses (e.g., do we get roughly the same forecasts for Southern Sudan asexperts at FAO, and if not, are they plausible enough to warrant analysts’ attention)?

◦ Diagnosticity and robustness – do users understand the sources of uncertainty intheir analyses, can they make analyses more robust with the help of the machine?

◦ Latency – how long does it take to set up and run analyses?

World Modelers - DARPA v2.pdfModeling at Sub-national Scales ... Step 1: Generate or ......

Documents

Transcript of World Modelers - DARPA v2.pdfModeling at Sub-national Scales ... Step 1: Generate or ......