Reconstructing carbon dioxide for the last 2000 years: a ... · How to trade a paper clip for a...

33
Reconstructing carbon dioxide for the last 2000 years: a hierarchical success story. Douglas Nychka National Center for Atmospheric Research National Science Foundation UBC October 2014

Transcript of Reconstructing carbon dioxide for the last 2000 years: a ... · How to trade a paper clip for a...

  • Reconstructing carbon dioxidefor the last 2000 years:a hierarchical success story.

    Douglas NychkaNational Center for Atmospheric Research

    National Science Foundation UBC October 2014

  • Outline

    D. Nychka Tour of an inverse problem 2

    • A 19th Century Inverse Problem• CO2 reconstruction problem• Hierarchical models (Data, process, priors)• Variational problems• back to ice cores

    Main ideas:Solve an ”inverse problem” by

    • separating what is observed from what is to be estimated.• using Bayesian statistics as a strategy for inference.

  • 222b Baker Street

    D. Nychka Tour of an inverse problem 3

    From “The Adventure of the Speckled Band”

    ”Good-morning, madam,” said Holmes cheerily. ”My nameis Sherlock Holmes.

  • 222b Baker Street

    D. Nychka Tour of an inverse problem 4

    Her features and figure were those of a woman of thirty, buther hair was shot with premature grey, and her expressionwas weary and haggard.

  • 222b Baker Street

    D. Nychka Tour of an inverse problem 5

    ”We shall soon set matters right, I have no doubt. You havecome in by train this morning, I see. ...and yet you had a good drive in a dog-cart, along heavyroads, before you reached the station.... ”

  • D. Nychka Tour of an inverse problem 6

    ”The left arm of your jacket is spattered with mud in noless than seven places. The marks are perfectly fresh.

    There is no vehicle save a dog-cart which throws up mud inthat way, and then only when you sit on the left-hand sideof the driver.”

  • Holmes’ calculation

    D. Nychka Tour of an inverse problem 7

    Before meeting Ms. Helen Stoner:

    • A PRIOR probability of type of vehicle

    Knowledge of vehicles effects:• LIKELIHOOD of observation given type of vehicle

    Combine prior with observation:• POSTERIOR is a product:Likelihood of mud stains given type of vehicle

    × Probability of type of vehicle

    Maximize over vehicle

  • D. Nychka Tour of an inverse problem 8

    Holmes’ conclusion – the highest probability• vehicle = dog cart

  • Some differences

    D. Nychka Tour of an inverse problem 9

    Holmes’ genius:Uses observations without assuming much prior information!

    Weather and climate applications:

    • Our observations are not as decisive – we must rely more on the priorinformation.

    • Have to use a computer to do the computation!

    NCAR/U Wyoming, Yellowstone Supercomputer, 70,000+ processors.

  • Convenient notation

    D. Nychka Tour of an inverse problem 10

    X and Y are random variables

    • [X] probability density function for X

    • [X,Y ] joint probability density function for X and Y

    • [Y |X] conditional probability density function for Y given X

    Properties• By definition: [X,Y ] = [Y |X][X]

    • Iterating: [X,Y, Z] = [Y |X,Z][X|Z][Z]

  • D. Nychka Tour of an inverse problem 11

    Bayes Theorem (1764)

    [X|Y ] =[Y |X][X]

    [Y ]

    From “Y given X” to “X given Y” !

    Another view of Holmes’ deductionFrom knowledge of type of given the transport → kind of vehicle giventhe observations

    Our problemFrom knowledge of how CO2 is trapped in ice given the atmospheric

    concentration

    → past CO2 concentrations given ice core observations.

  • Law Dome, East Antarctica

    D. Nychka Tour of an inverse problem 12

    Ice core drill and glaciologists from the Australian Antarctic Division and

    Antarctic Climate and Ecosystems CRC, Law Dome, East Antarctica

    What is the past history of CO2 concentrationsin the atmosphere?

  • Observations

    D. Nychka Tour of an inverse problem 13

    CO2 concentrations as a function depth.

    ●●●●●

    ●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●

    ●●●●●●● ● ●●●●●●

    ●●●●●●

    ● ●●● ●●

    ●●●●●●●●●●●●●

    ●●●●●●●●●●●●●●●●●●● ●●●●●

    ●●

    280 300 320 340

    ppm CO2

    dept

    h (m

    )

    800

    600

    400

    200 Conceptually: Depth is related to the

    time air was trapped in the core.

    y(depth) = F(c(time))

    F developed by Trudinger et al. (2013)

    Inverse problem: Invert CO2 concentra-tions by depth to concentrations by time.

    c(time) = F−1(y(depth)

    Firn Ice Inverse problem harder for upper ice layers that have notcompletely consolidated.

  • How to trade a paper clip for a house

    D. Nychka Tour of an inverse problem 14

    Home The Book The One Red Paperclip Project About Kyle Speaking Contact Press Things I'm doing now

    Older PostsHome

    Subscribe to: Posts (Atom)

    Posted by Kyle MacDonald Links to this post

    one red paperclipShare 2618

    Like 2,618 people like this.

    This might not surprise you, but below is a picture of a paperclip. It is red.

    This red paperclip is currently sitting on my desk next to my computer. I wantto trade this paperclip with you for something bigger or better, maybe a pen, aspoon, or perhaps a boot.

    If you promise to make the trade I will come and visit you, wherever you are,to trade.

    So, if you have something bigger or better than a red paperclip to trade, emailme with the details at [email protected]

    Hope to trade with you soon!

    Kyle

    PSI'm going to make a continuous chain of 'up trades' until I get a house. Or anIsland. Or a house on an island. You get the idea.

    Like 2,618 people like this.

    *Click here to see what happenednext!

    Ads by Google Trade Paper Clip Paperclip AD Trade a Boat Barter House

    CLICK BELOW TO CHECK OUT THE SLIDESHOW!

    How to Trade a Red Paperclip for a House - Part 2

    View more presentations from Kyle MacDonald.

    Share 2618

    Like 2,618 people like this.

    There's a book about the red paperclip adventure!

    Share Report Abuse Next Blog» Create Blog Sign In

    The first trade:

    a fish pen...

    14th trade: Movie role→

  • Back to Hierarchical Models

    D. Nychka Tour of an inverse problem 15

    Y - data, c = c1, c2, ..., cn process at closely spaced times(yearly, θ parameters

    Data level: [Y |c, θ] What we observe if we knew the true process.

    Process level: [c|θ] The dynamics of the process (regardless of whatwe measure)

    Priors: [θ] Uncertainty in the statistical parameters used to describethe data and process levels.

    Joint distribution:

    [Y |c, θ][c|θ][θ]

  • Where we are going: Bayes theorem

    D. Nychka Tour of an inverse problem 16

    Inversion (also called posterior)

    [c, θ|Y ] proportional to [Y |c, θ][c|θ][θ]

    take logs

    log([c, θ|Y ]) = log([Y |c, θ]) + log([c|θ]) + log([θ]) + constants

    fit to data weak constraint on c parameters

    • This is a complete probability density that quantifies un-

    certainty in result.

    • Large values (high probability) indicate good estimates of

    the parameters and process.

  • The first part: Data model

    D. Nychka Tour of an inverse problem 17

    c: a vector of the n concentrations at an annual time scale for last2000+ years

    y = y1, y2, ..., ym: a vector of the concentrations at an annual timescale for last 2000 years

    W : a m × n matrix where each row maps the concentrations to themeasurements at the depths.

    Wc: Expected concentrations at observed depths– the forward model F

  • D. Nychka Tour of an inverse problem 18

    The measurements:

    y = Wc+ e

    e = e1, e2, ..., em : measurement errors – assumed to be N(0, σ2)

  • What does W look like?

    D. Nychka Tour of an inverse problem 19

    W = 101× 2026

    Rows are weights applied to concentrations over time.

    Note: W is not invertible!

  • Effect of indirectly observing CO2.

    D. Nychka Tour of an inverse problem 20

    Rows of W for most recent ice layers.

  • Second Part: Process level

    D. Nychka Tour of an inverse problem 21

    Annual concentrations based on a linear trend, human emissions covari-

    ate, and autoregressive “noise”.

    ct = α1 + α2t+ α3Emissions(t) + ut

    ut = βut−1 + vt

    vt are uncorrelated normals and the variance of ut is ρ

    This can be written as c is N(Xα, ρΣ(β))

    Recent carbon emissions: Emissions(t)

    1700 1750 1800 1850 1900 1950 2000

    020

    0050

    00

    Year CE

    Car

    bon

    emis

    sion

    s

  • more on the autoregressive noise

    D. Nychka Tour of an inverse problem 22

    AR 1 process:ut = βut−1 + white noise – same as corr(ut, us) = e−|t−s|/δ

    β = .8

    0 200 400 600 800 1000

    −3

    −1

    01

    23

    proc

    ess

    β = .99

    0 200 400 600 800 1000

    −2

    −1

    01

    2

    proc

    ess

  • Third part: Priors – parameters

    D. Nychka Tour of an inverse problem 23

    • σ2 measurement error variance• α1, α2, α3 regression parameters• β autoregressive parameter (correlation)• ρ variance of autoregressive process.• W

    Empirical BayesFor this application we will estimate these parameters by maximum

    likelihood and just use these values.

    Maximum likelihood ≡ uniform ( or flat) prior distribution

  • Uncertainty in W

    D. Nychka Tour of an inverse problem 24

    Rows of W : mean and 20 member ensemble

    1800 1820 1840 1860

    0.00

    0.04

    0.08

    0.12

    Years CE

    Wei

    ghts

    Contribution to an ice core layer

  • Turning the crank

    D. Nychka Tour of an inverse problem 25

  • Finding the concentrations

    D. Nychka Tour of an inverse problem 26

    Posterior probability density:

    [c(.), θ|Y ] ∼ [Y |c(.), θ][c(.)|θ][θ]

    log([c(.), θ|Y ]) = log([Y |c(.), θ]) + log([c(.)|θ]) + log([θ]) + constant

    Data + Process + Prior

    Using data and process level models:

    −(1/2)(y −Wc)T (y −Wc)/σ2−(1/2)ρ(c−Xα)TΣ(β)−1(c−Xα)+ stuff

    fit to data weak constraint on c parameters

  • D. Nychka Tour of an inverse problem 27

    The program:

    • Maximize over c and parameters to get “ best ” estimate.

    • Sample from the posterior density to get an ensemble.

  • Example of likelihood surface

    D. Nychka Tour of an inverse problem 28

    Likelihood can be maximized analytically over all parameters except for

    correlation and “ noise to signal ” ratio ( variance of measurement error

    to variance of concentrations ).

    Maximize these last two numerically.

    log Likelihood surface

    0.2 0.4 0.6 0.8

    0.90

    0.92

    0.94

    0.96

    0.98

    noise/signal variance

    Pro

    cess

    cor

    rela

    tion

    −285

    −280

    −275

    −270

    −265

    −260

    −255

  • D. Nychka Tour of an inverse problem 29

    Full reconstruction Just trend and emissions.

  • Inference for decrease during Little Age

    D. Nychka Tour of an inverse problem 30

    1580 1600 1620 1640 1660

    265

    270

    275

    280

    285

    290

    cTime

    outM

    LE$c

    Hat

    ●●●●●

    ●●

    ● ●●●●●●●●●

    ●●●●●

    ●●●●●●●●

    ● ●●●

    ●●

    ●●●●

    ● ●●●●

    ●●●●●● ●

    ●●●●●●●

    ●●●●

    ●●●●●●

    ●● ●●

    ●●● ●

    ●●●●●●●●●●●●●●●●●●●

    1580 1600 1620 1640 1660

    265

    270

    275

    280

    285

    290

    cTime

    outMLE$cHat

    ●●●●●

    ●●

    ● ●●●●●●●●●

    ●●●●●

    ●●●●●●●●

    ● ●●●

    ●●

    ●●●●

    ● ●●●●

    ●●●●●● ●

    ●●●●●●●

    ●●●●

    ●●●●●●

    ●● ●●

    ●●● ●

    ●●●●●●●●●●●●●●●●●●●

    Minima and locations of the posterior reconstructions help to pin down

    the timing and size of the decrease during this cooler period of Northern

    Hemisphere climate.

  • Onset of industrialization.

    D. Nychka Tour of an inverse problem 31

    1650 1670 1690 1710 1730 1750 1770 1790 1810 1830 1850 1870 1890 1910 1930 1950 1970 1990

    27

    02

    80

    29

    03

    00

    31

    03

    20

    33

    03

    40

    35

    03

    60

    Years CE

    CO

    2 C

    on

    ce

    ntr

    atio

    n (

    pp

    m)

    Reconstructed CO2 Concentration: 1650−2000

    ● ●

    ●●

    ●●●

    ● ●

    ●●

    ●●

    ●●

    ● ● ●

    ●●

    ●● ●

    ●●

    ●●●

    ●●

    ● ●

    ●● ●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ● ●●●

    ●●

    ●●

    ● ● ●●

    ●● ●

    ● ●

    ● ●

    ●●

    ● ●

    ● ●●

    ●● ●

    ● ● ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●●

    ● ●

    ● ●

    ● ●

    ●●

    ● ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ● ●●●

    ●●

    ●● ●●

    ●●

    ●● ● ●

    ● ●

    ● ●●

    ● ●●

    ●●

    ●●

    ●●●● ●

    ●●

    ●●

    ● ●● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●● ●

    ●● ●

    ●●

    ●●●

    ●●

    ● ●

    ●●

    ● ●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●●

    ●●

    ●●●

    ●●

    ●●●

    ●●

    ●●

    ● ●●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●● ●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ●●●

    ●●

    ● ●●

    ● ●

    ●●

    ●●

    ●● ●

    ●●

    ●●●

    ●● ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●●●

    ● ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ● ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ● ●

    ●●●

    ●●●

    ●●

    ● ●●●

    ●●

    ●●●

    ●●

    ●● ●

    ●●

    ●● ●

    ●●

    ●●●

    ● ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●●●

    ●●

    ●●

    ●●

    ● ●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ● ●

    ●●

    ● ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ● ●

    ●●●● ● ●

    ●●●

    ●●

    ● ●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●● ●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ● ●●

    ●●●

    ●●

    ●● ●●

    ●●

    ●● ●

    ● ●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ● ●●

    ●●

    ●●

    ● ●

    ●●

    ●●● ●

    ● ●

    ● ●●

    ●●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ● ●●

    ●● ●

    ●●● ●

    ●●

    ● ●

    ● ●

    ●● ●

    ●● ●

    ●●

    ● ●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ● ●

    ●●

    ● ●●

    ● ●●

    ● ●

    ●●●

    ● ●●

    ● ●

    ● ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ● ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ● ●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●● ●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●●

    ●●

    ●● ●

    ●● ●

    ● ●

    ●●● ●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ●●●

    ●●

    ● ●

    ●●

    ● ●

    ●● ●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ● ●

    ●●

    ●●

    ●●● ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●● ●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ● ●

    ● ●

    ●● ●

    ● ●●

    ●●●

    ●●

    ● ●●●

    ●●

    ●●

    ●●

    ●●●

    ●●●

    ●●

    ●●

    ●●

    ●●●

    ● ●

    ●●

    ● ●●

    ●●

    ●●

    ● ●●●

    ●●

    ●●●

    ●● ●●

    ●●

    ●●●

    ● ●

    ● ●

    ●●

    ●●

    ●●●

    ●●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●● ●

    ●●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●●●

    ●●

    ● ●●

    ●●●

    ●●

    ●●

    ●●

    ● ●

    ● ●● ● ●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●●●

    ●●

    ●●●

    ●●●

    ●●

    ● ●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●● ●

    ●● ●

    ●●

    ●●

    ●●●

    ●●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●●

    ●●●

    ● ●

    ● ●●

    ●●

    ● ●

    ●●

    ●● ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●●

    ●●

    ● ●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●● ●● ●

    ●●

    ●●

    ● ●●

    ●● ●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ● ●

    ●●

    ● ●

    ● ● ●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ●● ●

    ●●

    ● ●

    ●●

    ●●

    ●● ●●

    ●●

    ● ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●●

    ● ●● ●

    ●●

    ●●

    ●●●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ● ●

    ●●

    ●● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●● ●

    ● ●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●● ●●

    ●● ●

    ● ●

    ● ●● ● ●

    ●● ● ●

    ● ●

    ● ●

    ●●

    ●●● ●

    ●●

    ● ●●

    ●●●

    ● ●

    ●●

    ●●

    ●● ●

    ●●

    ● ●

    ●●

    ●● ● ●

    ●●

    ● ●

    ● ●

    ●● ●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ● ●●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ● ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ● ●

    ●●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●●

    ● ●

    ●● ●

    ●●

    ● ●

    ●●●

    ●●

    ● ●

    ●● ●

    ●●

    ● ●

    ●●●●

    ●● ●

    ● ●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●● ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●● ●

    ●●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●● ●

    ●●

    ● ●

    ● ● ●

    ●●

    ●●

    ●●●

    ●●

    ●●●

    ●● ● ●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●●

    ●● ●

    ●●

    ●●

    ●● ●

    ●●●

    ●● ●

    ● ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●●

    ●●

    ● ●●

    ●●

    ● ●●●

    ● ●

    ● ●●●

    ●●

    ●●● ● ●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ● ●●

    ● ●

    ●●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●● ●

    ●●

    ●●●

    ●●

    ●●

    ●●●

    ● ●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●●

    ● ●

    ● ●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ● ●

    ●●

    ● ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●● ●

    ●●

    ●●

    ●●

    ● ●●●

    ●● ● ●

    ●●

    ● ●●

    ●●

    ●●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ● ●●●

    ● ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ● ●

    ●● ●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●● ●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●● ●

    ●●

    ●●●●

    ●●●

    ● ●● ● ●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●●●

    ●●

    ●●

    ●●●

    ●● ●

    ●●●

    ●●

    ●●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●●● ●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●● ●

    ●●

    ●● ●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ● ●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●● ●●

    ●●

    ● ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●●

    ● ●● ●

    ● ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●●

    ●●

    ●●

    ●●●

    ●●●

    ●● ●

    ●●

    ● ●●

    ● ●

    ●●

    ●●

    ●● ● ●

    ●●

    ● ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●●●

    ●●

    ●● ●●

    ●● ●

    ●● ●●

    ●●

    ● ●

    ●●

    ● ●

    ● ●●

    ●●

    ● ●

    ● ●●●

    ● ●●

    ●●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●● ●●●

    ●●

    ● ●●

    ● ●

    ●●

    ●●● ●

    ●●●

    ● ●●

    ●●●

    ●● ●

    ●● ●

    ● ●●

    ● ●

    ● ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ● ●●●

    ● ●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ● ●● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ● ●

    ● ●●

    ● ●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ● ●

    ●● ●

    ●●

    ● ●●

    ● ●● ●

    ● ● ●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ●● ●

    ● ●

    ●● ●

    ●●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ● ●●

    ● ● ●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●● ●

    ●●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ● ● ●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ● ●●

    ● ●●

    ● ●

    ●●

    ●●●

    ●●

    ●● ●

    ● ●●

    ● ●

    ●●

    ● ●

    ●●●

    ●●

    ●●

    ●●●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ● ●

    ●●

    ●●●

    ●●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ● ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●● ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ● ● ●●●

    ●●●

    ● ●

    ●●●

    ●●●

    ●●

    ●●

    ● ●●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ● ● ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●●

    ● ●

    ●●●

    ● ●●

    ●●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●●

    ●●

    ●●

    ●●

    ● ●

    ● ● ●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●

    ●●

    ●● ● ●

    ●●

    ● ● ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ● ●

    ●●

    ●●

    ●● ●

    ●● ●

    ● ●●

    ●●

    ●●

    ●● ● ●

    ●●●

    ●●

    ●●

    ●● ●●

    ● ●

    ●●

    ●● ●

    ●●

    ●●

    ●●●

    ● ●

    ● ●

    ●●

    ● ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●● ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ●● ●

    ●●

    ●●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ● ●●

    ●● ●

    ● ●

    ●●

    ● ●●

    ●●

    ●●

    ●● ●●

    ●●

    ● ●● ●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●●

    ●● ●

    ●● ●

    ●●

    ●● ● ●

    ● ●

    ●● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ●●

    ● ●●●

    ●●

    ●●●

    ● ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ● ●

    ●●

    ●● ●

    ● ●●●

    ● ●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●●

    ●●

    ●●

    ● ●● ●

    ● ●

    ● ●

    ● ●●

    ● ●●

    ●● ●● ●

    ●●

    ● ●●

    ●●●

    ●●

    ● ●●

    ●●

    ●●

    ● ●●

    ●●

    ● ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●● ●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●● ● ●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ● ●●

    ●●●

    ●● ●

    ●●

    ●●

    ●●●

    ● ●

    ● ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●●

    ●●● ● ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●● ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ● ●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ● ●●

    ● ●● ●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ● ●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●● ●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●●

    ● ●●●●

    ● ●

    ● ●●

    ● ●

    ● ●

    ●●

    ●●

    ●● ●

    ●● ●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ●● ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●● ●

    ●●●●

    ●●

    ●●

    ●●

    ●●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ●●● ●

    ●●●

    ●●

    ●●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ●●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ● ●

    ●●

    ●●●●

    ●●

    ●●

    ● ●●

    ●●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●● ● ●●

    ●●

    ●● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●

    ● ●●

    ● ●

    ●●

    ●●

    ● ●

    ● ●

    ●●

    ● ●

    ● ●●

    ● ●●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●● ●

    ●●

    ● ●●

    ●●

    ●●

    ● ●●

    ●● ●

    ● ●●

    ● ●● ●

    ●●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●● ● ●●

    ●●●

    ●●

    ● ●●

    ●●

    ●●●

    ●●

    ●●

    ● ●

    ●●●

    ●●

    ●●

    ●● ●

    ●●

    ●●●

    ●●

    ●●

    ●● ●●

    ●●

    ●●●

    ● ●● ●

    ● ●

    ●●

    ●●

    ●● ●

    ●●●

    ●●

    ●●●

    ●●

    ●●

    ●●

    ● ●

    ● ●●

    ●●

    ●●

    ●● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●● ●

    ● ●●

    ●●

    ● ●

    ●● ●

    ●●

    ●●

    ●●

    ● ●●

    ● ●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●●

    ● ●

    ●●●

    ● ●

    ● ●

    ●●

    ●●

    ●●

    ●●●

    ● ●●

    ●●

    ● ●●

    ●●

    ● ●●

    ●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ● ●●

    ● ● ●●

    ●●●●

    ● ●●

    ●●

    ●●●●

    ●●

    ●●

    ●●

    ●●

    ● ●

    ●● ●

    ●●