Comments from Last Night - to.isac.cnr.it · PDF file 20Sheets/U1a% . 1000 ft Ua1 shaft. 2...

15 June 2016 Termites and Penguins Alpine Summer School, Aosta © Leonard Smith

Comments from Last NightMore on Big Surprise? Are they real? How do we estimate them? Use of Black Box Models. What is structural model error? Parameter estimation…Uncertainty on types of uncertainty.

Incentive structures. (Why do people…)Downscaling (how do I get local information)

Isn’t it OK to lie for the greater good?What if being honest had bad consequences?

What good is knowing a 20 year time limit if we have to build something that will last 70 years?

How do you get PDFs from an ensemble?Why the threshold at one in 200? Do you think we can do that? (no, but alternative asks include 1 in 10,000)

Fiddle, tune and tweak.What are we looking for: Plausible Stories or Actionable Numbers (probabilities are numbers)?


Modelling Complicated Systems in Practice

LSE’s Bill Phillip’s MONIAC (Monetary National Income Analogue Computer)

My criticisms connect with real-world practice when models are (over) interpreted quantitatively.

They need not apply when used qualitatively for general insight and testable ideas (“fiddling” , sensitivity, of hunting hints of “Big Surprises”). (Manabe & Weatheral)

All models are wrong, some are dangerous.

Models don’t mislead people, people mislead people


Challenge: How long can we avoid using the word “Uncertainty” saying only which kind we mean?!?

Smith, L.A. and Stern, N. (2011) 'Uncertainty in science and its role in climate policy', Phil. Trans. R. Soc. A, 369, 1-24.

Imprecision

Ambiguity

Intractability

Indeterminacy

A well defined value that is considered imprecisely known(thermal expansion coefficient of H20, height of the pot as f(Θ),...)on which we put a probabilitydistribution given information I

A well defined value for which we lack sufficient information to pose a quantitative probability distribution.

A quantity that may be precisely defined, but which is beyond our (current) ability to estimate with quantified precision. (GMT2500, T2100(ORD))

A quantity which is in fact not uniquely (precisely) defined.(numerical viscosity, radius of an epicycle of Mars, ...)

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/86_SmithStern_Uncertainty_2011.pdf


What is Structural Model Error?

Smith, L.A. (1997) 'The maintenance of uncertainty,' in Proc International School of Physics "Enrico Fermi", Course CXXXIII, 177-246, Societ'a Italiana di Fisica, Bologna, Italy. Smith, L.A. (2000) 'Disentangling uncertainty and error: on the predictability of nonlinear systems', in Mees, A.I. (ed.) Nonlinear Dynamics and Statistics, Boston: Birkhauser, 31-64.

Answer: when the mathematical structure of your model difference from “that of the system” (such a thing exists).

f(x) = 1 – αx2 F(x) = ι – αx2 + βx3 + γx4

α is indeterminate F(x) = function(x) F(x) = function(c sin(x/c))

~ ~ ~ ~~ ~ ~ ~ ~

~ ~ ~ ~ ~

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/31_MaintOfUncert_Fermi_1997.pdf

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/40_Disentangling_2000.pdf




Answer: when the mathematical structure of your model difference from “that of the system” (such a thing exists). A structurally imperfect nonlinear model cannot shadow indefinitely.

Trajectory of Model A







A structurally imperfect nonlinear model cannot shadow indefinitely.

Trajectories of Model ATrajectories of Model B (Shadows 8+)






A structurally imperfect nonlinear model cannot shadow indefinitely.

Trajectories of Model ATrajectories of Model B (Shadows 8+)

Is there a better way to determine parameters than finding those that maximise shadowing times? (Achieving Nonlinear Consistency with Obs!)


Uncertain parameters in perfect modelsS

kill

1 -

0 -

-1 -

-2 -

-3 -

-4 -

-5 -

-6 -

Parameter Value

Given a Perfect Model Structure, make probability forecasts for the future:All Proper Scores agree on the best parameter, the correct value is uncertain (but it exists). IGN has the bonus of providing the Info Deficit, as well as an easy interpretation in decision theory.

True Value of the Parameter

Bröcker, J. and Smith, L.A. (2007) 'Scoring probabilistic forecasts: the importance of being proper', Weather and Forecasting, 22 (2): 382-388.

Du, H.& Smith, L. A. (2012) Parameter estimation using ignorance Physical Rev E

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/DRAFT_Scoring_beingproper_2006.pdf

http://www.lse.ac.uk/CATS/Publications/Publications%20Abstracts/Du&Smith-PhysicsReviewE-016213-2012.pdf



Ski

ll

1 -

0 -

-1 -

-2 -

-3 -

-4 -

-5 -

-6 -

Parameter Value

Given a Perfect Model Structure:All Proper Scores agree on the best parameter, the correct value is uncertain but exists.Given an Imperfect Model StructureThe particular Score matters, the forecast lead-time matters, the value you aretargeting is indeterminate (none exists), and the implied IGN reveals info deficit.

True Value of the Parameter

Indeterminate parameters in imperfect models.

Du, H.& Smith, L. A. (2012) Parameter estimation using ignorance Physical Rev E




http://www.metoffice.gov.uk/media/pdf/s/c/A3_plots-temp-MAM.pdfEnsemble Interpretation

http://www.metoffice.gov.uk/media/pdf/s/c/A3_plots-temp-MAM.pdf


Ensembles Members In - Predictive Distributions OutBlending: Ensemble Members to Model Distributions

. . ... . . … . . . ….. . . . .. . . Pclim=∑ K(x,oi)/nclim

nclim

i=1

P1(x)= ∑ K(x,si1)/neps

neps

i=1

K is the kernel, with parameters σ,δ (at least)

Forecast busts and lucky strikes remain a major problem when the archive is small.

J Bröcker, LA Smith (2008) From Ensemble Forecasts to Predictive Distribution Functions Tellus A 60(4): 663.

http://www.lse.ac.uk/collections/cats/papersPDFs/74_Broecker_PDFs_tellus_2007.pdf


α

Ensembles Members In - Predictive Distributions OutBlending: for a fixed ensemble size α decreases with time

M1 =α1 P1 + (1-α1)Pclim

1

Pclim

P1

Lead time

1 -

½ -

0 -

Even with a perfect model and perfect ensemble, we expect α to decrease with time for small neps

Small :: neps/ nclim

J Bröcker, LA Smith (2008) From Ensemble Forecasts to Predictive Distribution Functions Tellus A 60(4): 663.

Kernel & blend parameters are fit simultaneously to avoid adopting a wide kernel to account for a small ensemble.

http://www.lse.ac.uk/collections/cats/papersPDFs/74_Broecker_PDFs_tellus_2007.pdf


Seasonal Forecasting: Ensemble Interpretation

Bröcker, J. and Smith, L. A. (2008) From Ensemble Forecasts to Predictive Distribution Functions Tellus A 60(4): 663.

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/74_Broecker_PDFs_tellus_2007.pdf


Seasonal Forecasting: Ensemble Interpretation







A high fidelity model, with an extensive forecast-outcome archive, allows distributions which are:Useful : Better decisions are made with them than without, as long as no one bets on these predictive distributions as probabilities.

Actionable: Not as a Probability Forecast, due to structural error.

XInformative versus Actionable:To What extent is this a Probability Forecast?


Any Questions on the Answers to your Questions?



www.lsecats.ac.uk

From Termites and Penguins to Blocking and the Andes:

Interpreting Models of the Earth System

Leonard Smith

London School of Economics

& Pembroke College, Oxford

This Talk Would Not Be Possible without:

http://www.lsecats.ac.uk/

http://images.google.com/imgres?imgurl=http://www.classicalrecording.co.uk/images/pembroke_crest.gif&imgrefurl=http://www.classicalrecording.co.uk/catalogue/pages/pembroke.htm&h=126&w=107&sz=3&tbnid=F6NKgEvDB9cJ:&tbnh=84&tbnw=72&start=11&prev=/images?q=Pembroke++crest&hl=en&lr=

http://images.google.com/imgres?imgurl=http://www.classicalrecording.co.uk/images/pembroke_crest.gif&imgrefurl=http://www.classicalrecording.co.uk/catalogue/pages/pembroke.htm&h=126&w=107&sz=3&tbnid=F6NKgEvDB9cJ:&tbnh=84&tbnw=72&start=11&prev=/images?q=Pembroke++crest&hl=en&lr=

http://www.ensembles-eu.org/

http://www.ensembles-eu.org/


Before Discussing Types of Probability Further,How are Unknown Unknowns Different? So we knew termites might be important.and arguably we know the Andes are important.

“Known Unknowns” we can sample with ensembles (as long as the “unknown” is only a real number!)

Those we choose not to sample (or not to include at all) are “Known Neglecteds”, not really unknowns at all. ?max sim time?

increases P(BS)Before moving on to real climate model simulation (or talking about the Andes in detail) let’s consider “Unknown Unknowns” and how they limit our ability to make decision relevant statements of probability.

We must not have a mind shaft gap!

HE3 in a Changing Climate Bad Honnef 7 Dec 2014 Leonard Smith

Mature Probabilities P(x|I)A mature probability is not expected to change without additional observation or new theoretical insight. (An nontrivial change in I )

If the fidelity of a simulation model is constrained by technology (as when you know exactly what you would do with more compute power, and it is NOT to run massive ensembles/emulators), then probability distributions based on simulations from that model (or family of similar models) are not expected to be mature.Rational action is constrained only by mature probabilities.Model-based Probability can be used in creative ways (as data).

Over confidence (“belief”) leads to the mine shaft gap…(?How might one use im-mature distributions in decision support?)

(Generalised from IJ Good’s “Dynamic Probability”, as when output from a chess program must be used before the algorithm completes. )

IJ Good (~1979) Science


Mind Shaft Gap

We cannot simulate many things we can think of!didnot

Either no probabilities or two: Prob(x|Imodel) Prob(Big Surprise)

"The cost of solving the Comet mystery must be reckoned neither in money nor in manpower."Winston Churchill, 1954

Neptune Vulcan1846 1859

LeVerrier

https://en.wikipedia.org/wiki/Winston_Churchill


Mind Shaft Gap

We cannot simulate many things we can think of!didnot

How about estimating the time it takes a ball to reach the ground when dropped from a tower?

(thanks to Dave Higdon)


Chasing Model Inadequacy(by dropping balls off towers)

http://www2.nstec.com/Documents/Fact%20Sheets/U1a%20Facility.pdf

1000

ft

Ua1

sha

ft

2 bowling balls3 Basketballs2 golf balls3 Wiffle balls… (no rubber duck)

detectors

Ball(s)


How close was our median time for basketballs?

A. < 0.01 sec

B. <0.1 sec

C. < 1 sec

D. < 1 min

E. > 1 min

Basket ball.Initial velocity zero.1000 ft “tower”.

Laser sheet timing.

Q3.1


How close was our median time for basketballs?

A. < 0.01 sec

B. <0.1 sec

C. < 1 sec

D. < 1 min

E. > 1 min

< 0.01 sec

<0.1 sec

< 1 sec

< 1 min

> 1 min

0%

18%14%

41%

27%

Basket ball.Initial velocity zero.1000 ft “tower”.

Laser sheet timing.

Q3.1

My students and postdocs


So: How accurate were our drop

time estimates?

“The bowling ball was completely destroyed.”“One of the basket balls failed to make it to the bottom.”

A “Big Surprise” is when something your model doesn’t reflect is important: We thought surface roughness was the main Unknown.

Sometimes, scientists can estimate Prob(BS) (but not within the models, of course)

I only have preliminary results, but I think they make my point rather better than any data I could have made up:


An Oxford Bus Stop, Summer 2010:

Addressing Climate-like Questions ScientificallyRequires doing Science in the Dark

Scientists are forced to “violate” traditional best practice guidance if such violations are imposed by the nature of the question being addressed.

One cannot wait 50 years for out-of-sample observations.It is a brute fact that a climate model’s lifetime is less than it’s forecast lead-time!

The physics underlying CO2 induced warming remains as solid as science gets.

Other groups working in the dark sometimes embrace model inadequacy more than we do, and speak much more tentatively.

Geoscientists are not aloneClimate-like Questions are also found in:

Nuclear Stewardship, Novel Engineering DesignNational Intelligence, Reactor Safety, Americas Cup Design, Vehicle Crash-worthiness, Waste Disposal,

(Re)Insurance

Geoscientists are not aloneClimate-like Questions are also found in:

Nuclear Stewardship, Novel Engineering DesignNational Intelligence, Reactor Safety, Americas Cup Design, Vehicle Crash-worthiness, Waste Disposal,

(Re)Insurance


Things are NOT HOPELESS (nor Useless)!

http://www.edfenergy.com/html/index.jsp

http://www.edfenergy.com/html/index.jsp


Once you accept that the model is wrong and potentially dangerous, and that no actionable probabilities are going to magically appear, then you need to get close to the specific problem you are trying to solve.

One thing these projects have taught me is that real-world decision makers make decisions.

Given what ever they have in hand, they make a decision.They do not “require” anything (other than a deadline).

Getting very close to them, understanding their pain, is a key towards their making better decisions with you than without you.


Things are NOT HOPELESS (Useless)!Even When Predicting Pirates!

In Weather-like tasks one builds up a large archive of forecast-outcome pairs; the life-time of the model is much longer than the lead-time of the forecast.

In Climate-like tasks, the lifetime of a model (sometimes a professional) is much less than the lead-time of the forecast. Knowledge is gained with time, but the problem remains one of extrapolation.


Page 591

Where should decision makers draw the line?

Clear, plain spoken discussion of what today’s models cannotcapture quantitatively would be of great value.


Climate Modelling in Practice


In this

http://www.globalchange.gov/images/cir/pdf/20page-highlights-brochure.pdf

http://www.ipcc.ch/publications_and_data/ar4/wg1/en/figure-spm-4.html

Statistical post-processing: These are anomalies, not temperatures.How can we ignore an range of anomaly corrections wider than what would cause “dangerous climate change”?

?When are (well labelled) anomalies decision-relevant?

http://www.globalchange.gov/images/cir/pdf/20page-highlights-brochure.pdf

http://www.ipcc.ch/publications_and_data/ar4/wg1/en/figure-spm-4.html

Hanover How to Build Trust 10 June 2015 Leonard Smith

Models agree that a wide range of sorta-Earth-like planets warm about the same amount under the observed forcing.

Anomalies, Systematic Errors, Laws of Physics

IPCC AR4


As we refocus from climate-past to the climate-future, how do we cope with such systematic errors, even as we work to reduce them?

Obs

AR4 Simulations without 1900-1950 anomaly adjustment

Anomalies may be fine for mitigation.They are a nonsense for quantitative adaptation.

Systematic errors are larger than the observed effect


As we refocus from climate-past to the climate-future, how do we cope with such systematic errors, even as we work to reduce them?

Obs

AR4 Simulations without 1900-1950 anomaly adjustment

Moving to anomaly space requires letting go of the “Laws of Physics”.Note model anomalies are notexchangeable even after 100 years!

Anomalies may be fine for mitigation.They are a nonsense for quantitative adaptation.

(and the laws of physics.)(and biology.)

(Ice melts at zero C, plants die at ….)

While systematic errors are larger than the observed effect


AnomaliesThe AR5 is a bit more forthcoming

}


CMIP5


Anomalies

And why was the anomaly period shifted?

The AR5 is a bit more forthcoming

}


Limits to Transparency: Anomalies 2013 X


Model Fidelity

The detail you see above is what is missing in HadCM3: the large squares reflect model grid resolution, the detail reflects the difference between the observed surface height and the model surface height, “constant” “within” a grid point.Insurance Company with a snowfall question…

Climate Model Points(the squares)(What you see is NOT in the model)

A very schematic schematic reflecting phenomena the model “includes”.

“included” vs “realistically simulated”

Karl and Trenberth 2003

“in

clu

ded

” vs

“r

ealis

tica

lly s

imu

late

d”


AR4Projection/Prediction/Forecast?

The IPCC itself might say which space-time scales are realistic as a function of lead-time?

Smith, L.A. (2002) 'What might we learn from climate forecasts?', Proc. National Acad. Sci. USA, 4 (99): 2487-2492

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/LA-Smith,-(2002)-What-Might-We-Learn-from-Climate-Forecasts.pdf


AR5

Real-world GMT is “likely” (~66% chance) to be in “the range” of model-land GMT.

That suggests there is a significant chance the real-word will be outside the range of the models.

If your downscaling model was perfect, there remains a huge chance you could not catch the relevant pathway (as none of today’s models do).

I think it is fair say the IPCC implies that the Probability of a Big Surprise (GMT in 2100) is about one in ~ four to ~ten.

https://www.ipcc.ch/report/ar5/wg1/docs/WGIAR5_SPM_brochure_en.pdf

By law, we require banks and insurance companies hold costly reserves to cover one in 200 year events.

Implications for quantitative prediction?

https://www.ipcc.ch/report/ar5/wg1/docs/WGIAR5_SPM_brochure_en.pdf


Ensembles are not UselessOne does, however, need to be clear which question is being answered.And whether that question was poised in model-land or real-world.


Climateprediction.net



Relative Frequency, for a particular model.In a particular experimental design.

This is not the probability of anything in the world.



Testing statistical assumptions make to “answer the question” with another particular experimental design (small ensembles).

Is there anything to learn here beyond “we do not do that anymore.”?


>>

Sexton 2010 http://www.exeter.ac.uk/media/universityofexeter/research/inspiringresearch/sciencestrategy/ccsf/docs/Making_probabilistic_climate_projections_for_the_UK_presentation.pdf

For all we know, this piece of the pie is bigger than all the other pieces combined.

A lower bound with 2014 hardware is not an estimated value.

What to do? Say?

Model diversity is only a lower bound on structural uncertainty, which may well be by far the biggest piece of the pie.

We know today our model-based probabilities are not mature.

http://www.exeter.ac.uk/media/universityofexeter/research/inspiringresearch/sciencestrategy/ccsf/docs/Making_probabilistic_climate_projections_for_the_UK_presentation.pdf


http://www.ukcip.org.uk/

Is it plausible to provide a PDF of hottest or stormiest summer day in 2080’s Oxford???

http://www.ukcip.org.uk/


UKCP09 Worked Examples…An ensemble of “charitable reading” is possible, and leads to vastly different implications for intended interpretation and use, and implied adequacy.

The intention is made rather more clear by the “worked examples” in the main report, and the description of the designers said what would will allow.


Ensembles are not UselessOne needs to be clear which question is being answered.

And whether that question was poised in model-land or real-world.

Whether or not real-world conclusions have both a mathematical and a scientific basis


How did we get in this mess?


Two Different Approaches to Simulation

Mod

el F

idel

ity

Model Completeness

*

Complecatedness/Convolutedness


Mod

el F

idel

ity

Model Completeness

+

*

Least ComplexAdequateModel

+

Target V&VModellingApproach


Mod

el F

idel

ity

Model Completeness

*

+

*

+

CommonKitchen-sinkClimate ModelApproach

+


V&VModellingApproach

cpdn was a study of model relative frequencies,not probabilities of the real-world. Model Fidelity was too low.Should we have given a PDF?

X

I love ensembles!


Mod

el F

idel

ity

Model Completeness

+


*

+

Target V&VModellingApproach


Mod

el F

idel

ity

Model Completeness

*

+

*

+

CommonKitchen-sinkClimate ModelApproach



Enter the Subjective Bayesian

xi o

By “eliciting” “your” uncertaintyvia P(x), P(α) (not α) and considering a small set ofsimilar models {*, i, x, o}

He extracts a “Bayesian” PDFfor the hottest/wettest day in my Oxford College (5 km) in 2092. All from a low fidelity model (s).

What kind of entity is P(α) when F(x, α) is structurally inadequate?[For me, the question is meaningless, or at best P(α) = 0]

Indeterminacy is not Imprecision : A Bayesian Bait and Switch“It is better to include it than not: UQ is important.”

? Or is that a simulation it is better not to run?

~


What should we aim for:Weather-proof or Weather-wise?

“Increasingly, precise, short-term, extremely local forecasts can help...”

http://www.ibm.com/ibm/ideasfromibm/science/092506/

“Weatherproof?”, “Climateproof?”, “Futureproof?”or Just Enough Decisive Information (JEDI)?

Businesses can respond to build resilience and climate-proof their interests.

http://www.ibm.com/ibm/ideasfromibm/science/092506/


Clarity Before ConsensusBe clear as to what the science can and cannot deliver, and where differences of opinion here lie. Deprecate rank-order model beauty contests and anomalies..Better distinguish levels confidence in model outputs from that in their real world name-sakes.

Design the Experiments for Decision Support

What new insights will balance the cost of delay?What is the expected information-gain in CMIP6 given we

have CMIP5 (quantitative information of the world).How much will ProbBigSurprise(GMT2080-2100) change?

How many SRES variants are swamped in model noise?


• Empirically viable models (Resource reallocation).• Reduced lead-times, Reduced “included” phenomena.• Demonstrable skill

• Transparent tools for easy access to allow fiddling.• DECC Climate Calculator

• Tales of Future Weather• Embed well understood weather models in a climate future

• Providing “Just Enough Decisive Information” (JEDI)• Alter Regulations from “Survive” to “Avoid”

• Talk to those who use the information • DECC, Netherlands, Capital Hill…

• Recognised Vulnerability Approaches

Alternative Paths Forward


Dynamic Climatology: A Surrogate Model

Define “zero-skill” by constructing a physics free model from the observations.Forecasting GMT with a RAP Surrogate Model:• Take the time series of GMT over the last 51 years• Compute the 50 one-year differences 1Δi• To form an ensemble forecast add each 1Δi to this year’s GMT• Kernel dress this ensemble to form a 1-year ahead probability forecast

Repeat the procedure with two-year differences (2Δi ) to form a 2 year ahead probability forecast.

And so on.


Decadal Forecasting (ENSEMBLES)

Suckling, E.B. and Smith, L.A. (2013) 'An evaluation of decadal probability forecasts from state-of-the-art climate models', Journal of Climate, 26 (23): 9334-9347.

Less skill than Benchmark

More Skill than Benchmark

Bits

http://www.lse.ac.uk/CATS/Publications/Publications%20PDFs/95_Suckling&Smith-decadal-hindcasts-v9-final-JoC.pdf

15 June 2016 Termites and Penguins Alpine Summer School, Aosta © Leonard Smith Smith et al (2015) in review

CMIP5 Models: HadCM3

Of course the simulations in the AR5 know when past volcanoes went off.Information Deficits

Remain

23 Feb 2016 Flood & Coast 2016 Telford JEDI Insights Beyond Projection © LA Smith 2016

When the “best available” long-range forecast is thought neither adequate for purpose nor robust …

...consider building for a flexible response with reliable early warning, and being happy for a mitigation pathway.

Gauging the Limitations of Imperfect Model Prediction with Sculpted Ensembles

Approaches to Risk Management of the Large Scale in the Long Range Include:

Select and Build to a Probability Isopleth from the Best Available Prediction/Projection

orFocus on Your Vulnerabilities,

Base lines and Flexible Response


Build Now to Avoid VulnerabilityAct Then to Cover in Real Time

Turn off Coastal Infrastructure (rather than harden it)Deploy Timely Defences Well in Advance

Insure on the (Glimpse) Forecast

Reinsure on Early Warning

Regulate in Light of GLIMSE

Gauging the Limitations of Imperfect Model Prediction with Sculped Ensembles

Weather models in 2050 will be rather better than today’s...

For more information, please write [email protected]

mailto:[email protected]


The Dutch government officially requested greater clarity in “uncertainty” in the AR5 Summary for Policy Makers

SPM-39 More transparent and consistent uncertainty formulationThe SPM should include a clear distinction between process-based and model-based uncertainty formulation. Model uncertainty is not transparent enough.[Government of Netherlands]

Design the Experiments for Decision SupportCommunicate Insight and its Limits more Clearly


SPM-39

http://www.climatechange2013.org/images/report/WGIAR5_FGD_FinalDraftSPMComments.pdf



DECC Climate Calculator

Address Simpler Question under Simple Clear Assumptions

http://tool.globalcalculator.org/globcalc.html?levers=22rfoe2e13be1111c2c2c1n31h444444222hp233f211111fn2211111111/climate/en

http://tool.globalcalculator.org/globcalc.html?levers=22rfoe2e13be1111c2c2c1n31h444444222hp233f211111fn2211111111/climate/en


http://tool.globalcalculator.org

You decide: design a climate pathway…

http://tool.globalcalculator.org/

...regarding the climate information you’d like to haveThe utility of (known to be) unreliable quantitative information:

A. highB. zeroC. negative

...regarding the climate information you’d like to haveThe utility of (known to be) unreliable quantitative information:

A. highB. zeroC. negative

high zero

negat

ive

42% 42%

17%

???


If we were to start modelling climate for decision making today, how might we do it differently?

which initial phenomena?require clear empirical evaluation?

enhanced numerical awareness!On what timescales can probabilistic simulation based forecasting outperform benchmark empirical models? (< decade)

Might a minimalist approach:• reduce the “out of range” probability reduce the Prob(BS)• allow principled development of complexity Numerically efficient• yield empirically supported estimates of reliability? At least to 30 years!• suggest development/evaluation pathways for centennial modelling programs.

What is the decision relevant value-added of CMIP6 given CMIP5?How much will the probability of a big surprise decrease?

?What supports “return to skill” arguments in the climate case?

There is value in better probabilistic foresight on weeks to seasons+ in the current climate with which we familiar, and even more in a changed climate

A Modest Proposal: Minimalist Empirically-Adequate Climate Modelling


LA Smith, (2002) What Might We Learn from Climate Forecasts? Proc. National Acad. Sci. USA 4(99): 2487 -2492.Du, H. and Smith, L. A. (2012) Parameter estimation using ignorance Physical Review E 86, 016213Bröcker, J. and Smith, L. A. (2008) From Ensemble Forecasts to Predictive Distribution Functions Tellus A 60(4): 663Smith, LA and Stern, N (2011) Uncertainty in science and its role in climate policy Phil. Trans. R. Soc. A (2011), 369, 1-24

R Hagedorn and LA Smith (2009) Communicating the value of probabilistic forecasts with weather roulette. Meteori App 16 (2): 143K Judd, CA Reynolds, LA Smith & TE Rosmond (2008) The Geometry of Model Error. J of Atmos Sci 65 (6), 1749-1772.DA Stainforth, MR Allen, ER Tredger & LA Smith (2007) Confidence, uncertainty and decision-support relevance in climate predictions, Phil. Trans. R. Soc. A, 365, 2145-2161.

LA Smith (2006) Predictability past predictability present. Chapter 9 of Predictability of Weather and Climate (eds T. Palmer and R Hagedorn). Cambridge, UK. Cambridge University Press.

L A Smith and D A Stainforth (2012) Clarify the limits of climate models in Nature Vol. 489

D Orrell, LA Smith, T Palmer & J Barkmeijer (2001) Model Error in Weather Forecasting, Nonlinear Processes in Geophysics 8: 357

LA Smith (2000) 'Disentangling Uncertainty and Error: On the Predictability of Nonlinear Systems' in Nonlinear Dynamics and Statistics, ed. Alistair I Mees, Boston: Birkhauser, 31-64.

www2.lse.ac.uk/CATS/Publications/Leonard Smith Publications.aspx

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/LA-Smith,-(2002)-What-Might-We-Learn-from-Climate-Forecasts.pdf



http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/86_SmithStern_Uncertainty_2011.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/78_Weather_Roulette_t.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/77_Judd_GeomOfModelError_JAS.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/75_Stainforth_ConfidenceUncertaintyRelevance_2007.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/Stainforth&Smith_Nature_correspondence.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/45_ModelError_2001.pdf

http://www2.lse.ac.uk/CATS/Publications/Publications%20PDFs/40_Disentangling_2000.pdf

http://www2.lse.ac.uk/CATS/Publications/Leonard%20Smith%20Publications.aspx


Thank you


END


What do you say when “Best Available” is known not to be “Adequate for Purpose”

Refuse to interpret this to answer questions for which it is inadequate.

Apply state-of-the-art (baffling) statistical fixes (known not to be adequate).

Attempt physical fixes (testing adequacy to the extent possible)Tales of Future Weather (Nature, 2015)

Construct models optimised for adequacy (and thus, say, only addressing lead times on which they add information to “physics-free” empirical [surrogate] models)

(In Adaptation) Change regulatory/design to incorporate real time insight Just Enough Decisive Information (JEDI) Design


Improved Transparency: EquidismalityRank order beauty contests, without comparison to any absolute measure of quality, are misleading in several ways. How can we better communicate the fidelity of todays best models?

Median model has twice the error of this model.


Improved Transparency: EquidismalityRank order beauty contests, without comparison to any absolute measure of quality, are misleading in several ways. How can we better communicate the fidelity of todays best models?


Moving Forward:Plausible Planets or Implausible Earths?

How can we best develop our models as the available computational power increases?

A) Simulate potentially real planets that get more and more Earth-like while omitting any Earth-relevant process for which the model cannot provide coherent physical drivers on Earth-like scales. (no suggestion of linear superposition intended!)

Does water vapour come after mountains?Does vegetation come after water vapour?Do we avoid the penguin effect? (until it is simulated realistically)

B) Via an hodgepodge of unphysical/unbiological simulations resembling no planet that could possibly exist, but “including” every phenomena we can think of that might be important (including penguins), and hoping the simulated planets will suddenly become Earth-like at some resolution in an ill-defined higgledy-piggledy way.

One might argue physical intuition is more effective in evaluating plausible planets, as there is physics to intuit in that case. (and at least a few examples.)


http://riskfriends.wordpress.com/2009/03/10/solvency-ii-dealing-with-operational-risk/

Solvency II is a set of regulatory requirements for insurance firms that operate in the EU designed to prevent insurance company failures by unbundling “operational risk”.

The aim here is not to integrate over all risks and opportunities to estimate the PDF of expected annual income but simply to ensure that insurers have sufficient "regulatory capital" to survive any (every) adverse event which has more than a 1 in 200 chance of occurring.

Question: Can climate science ascertain whether the probability of an outcome is a) >> 1 in 200b) ~ 1 in 200c) << 1 in 200

Clearly identify risks without the investigative distraction posed by the whole shebang of a PDF.

The One in 200 Threshold and Risk Management

Liam

X

http://riskfriends.wordpress.com/2009/03/10/solvency-ii-dealing-with-operational-risk/


Types of Probability (Forecasts): P(x| data, I)

Rational Decisions I. J. Good (1952) Journal of the Royal Statistical Society. Series B (Methodological) Vol. 14, No. 1 , pp. 107-114Good Thinking I.J. Good (1983) Dover.

(o) Tautological Probability. A probability P(E|H) the value of which is specified in the definition of H. (“a fair coin”, H is called “a simple statistical hypothesis”)

(i) Physical Probability: P(x) “True probability” (Laplace’s Demon/Inf Rat Org)(ii) Psychological Probability: “Personal probability inferred from one’s behaviour.”(iii) Subjective Probability: P(x|G) probability of x given our information G is true

(Demon’s Apprentice/?semi-finite Rational Org?) (iv) Dynamic Probability: Pt(x| gt<G) when an algorithm encapsulating G has not

yet terminated (finite algorithm, merely still running).Dynamic in the sense that this probability is expected to change without any empirical information (by reflection only).

(v) Mature Probability: P(x| g<G) when G is known (not) to be encapsulated in g.Mature probability is not expected to change without some additional insight or additional empirical observation (even given vast computational power).

Groningen 16 March 2011 © Leonard Smith

The Modeler’s MantraThis is the best available information, so it must be of value. Everyone knows the limitations. Everyone understands the implications of these assumptions.This is better than nothing. No one has proven this is wrong. There is no systematic error, on average. The systematic errors don't matter. The systematic errors are accounted for in the post processing. Normality is always a good first approximation. In the limit, it has to be normally distributed, at least approximately.Everyone assumes it is normally distributed to start with.Everyone makes approximations like that.Everyone makes this approximation. We have more advanced techniques to account for that. The users demand this. The users will not listen to us unless we give them the level of detail they ask for.We must keep the users on-board.If we do not do this, the user will try and do it themselves.There is a commercial need for this information, and it is better supplied by us than some cowboy. Refusing to answer a question is answering the question.Refusing to use a model is still using a model. Even if you deny you have a subjective probability, you still have one. All probabilities are subjective.The model just translates your uncertainty in the inputs to your rational uncertainty in the future.Sure this model is not perfect, but it is not useless.No model is perfect. No model is useless if interpreted correctly. It is easy to criticise. This model is based on fundamental physics. The probabilities follow from the latest developments in Bayesian statistics. Think of the damage a decision maker might do without these numbers.Any rational user will agree. Things will get better with time, we are making real progress.You have to start somewhere. What else can we do? It might work, can you deny that?What damage will it do?


A Third Wave of ModellingRedeploy computational resources to increase model simulations with documented skill (short range: “years”)

“Include” fewer processes than centennial modelsSimplify the model by turning things off

Simulate with higher fidelity

Informative forecasts over days, months and years are of value in the current climate, and even more value in an altered climate.

Design the Experiments for Decision Support

Establish skill relative to simple empirical modelsEmpirical Models yield competitive decadal forecasts today!

Establish value added (relevant information from) and utility of GCMs compared to empirical models.


Was Stephen Foster ahead of his Time?

It rained all nightThe day I leftThe weather it was dry

The sun so hotI froze to deathSusanna don’t you cry

These words always seemed nonsense to me.And then I learned about “anomalies”.

Comments from Last Night - to.isac.cnr.it · PDF file 20Sheets/U1a% . 1000 ft Ua1 shaft. 2...

Documents

Transcript of Comments from Last Night - to.isac.cnr.it · PDF file 20Sheets/U1a% . 1000 ft Ua1 shaft. 2...