Practical Dynamic Programming in Ljungqvist – Sargent (2004) Presented by Edson Silveira Sobrinho

Practical Dynamic Programmingin

Ljungqvist – Sargent (2004)

Presented byEdson Silveira Sobrinho

for Dynamic Macro class

University of HoustonEconomics Department

Chapter 3 presents some computational methods for Dynamic programs of the form:

Subject to:

given x.

The Euler Equation helps in a variety of cases.

One can also start with a Vo and iterate until convergence.

However, it may be impossible to solve analytically. Then we have to adopt some numerical approximations.

MotivationFirst approach

Second approachConclusions

The curse of dimensionality

2

V ( x )= max u {r ( x , u )+ β V ( x̃ ) }

x̃= g ( x ,u )

First approach for obtaining numerical approximations:

The first method replaces the original problem with another problem by forcing the state vector to live on a finite and discrete grid of points.

Then applies discrete-state dynamic programming to this problem.

Downside is that it requires a small number of discrete states.

Guess what? Markov chains!

3

Discretization of state spaceDiscrete-state dynamic programming

Application of Howard Improvement algorithmNumerical implementation



Set up: an infinitely lived household likes to consume one good, which it can acquire by using labor income (ws

t) or accumulated savings (a

t).

st is endowment of labor at time t. s

i is endowment of labor in state i.

m-state Markov chain with transition matrix P. Say m = 2, where si = 0 (if

unemployed) or si = 1 (if employed).

The wage w is fixed over time.

The household can choose to hold a single asset in discrete amount at

where at Є A is a grid [ a

1 < a

2 < … < a

n].

The asset bears a gross rate of return r fixed over time.

4





Household maximization problem (Eq. 4.2.1):

Subject to :

Assume that:

5

E∑t=0

∞βt u (c t)

c t+ a t+1=(1+ r )a t+ ws t

β (1+ r )< 1





Bellman equation:

Which is a matrix m x n (2 states of employment x n choices of a).

Let a' and s' indicate next period's asset and state.

v (a,s) is the optimal value of the objective function, starting from asset and employment state (a, s).

A solution of this problem is a value function v(a, s) that satisfies the Bellman Equation and an associated policy function a' = g(a, s) mapping this period’s (a, s) pair into an optimal choice of assets to carry into next period.

6

v (a , s )=Max a '∈ A {u [(1+ r )a + ws−a ' ]+ β E v (a ' , s ' | z )}





For discrete-state space of small size, it is easy to solve the Bellman equation numerically by manipulating matrices. Here is how to write a computer program

Define two n x 1 vectors vj, j=1,2 , i.e. one vector for each employment state.

Each row takes the value of v(i) for a given asset i.

Let 1 be the n x 1 vector consisting entirely of ones

7





Define two n x n matrices Rj whose (i, h) element is (Eq. 4.3.0):

This is the current period utility conditional on current asset i and next period asset h, for a given employment state j.

That is, all the possible utilities that arise from combinations of choices of assets for this period and the next one.

8

R j (i , h )= u [( r+ 1 )a i+ ws j − a h ] , i=1, . . . , n ; h=1, . . . , n





Define an operator T ([v1, v2]) that maps a pair of vectors [v1, v2] into a pair of vectors [tv1, tv2] (Eq. 4.3.1) :

LHS is a column vector n x 1

The term in the Max operator is n x n Matrix.

Intuition: each element brings the lifetime utility for a given pair of asset choices for current and next period. For each row (say current asset), the agent chooses the column (next period's asset) that allows max lifetime utility. Current consumption comes from the budget constraint in R

1.

Similarly, for unemployed state:

9

tv 1= Max {R 1+ β P 11 1 v 1 ' + β P 12 1 v 2 ' }

tv 2= M ax {R 2 + β P 21 1 v 1 ' + β P 22 1 v 2 ' }





Compact notation (Eq. 4.3.2):

Then the Bellman Equation can be viewed as:

And can be solved as:

The production dynamics

10

[tv 1

tv 2]=Max{[R1

R2]+β( P1)[tv1 'tv2 ' ]}

[v 1 v 2 ]=T ([v1 v2 ])

[v 1 v 2 ]m+ 1=T ([v1 v2]m)





Back in Chapter 3 we saw the Howard Improvement Algorithm.

The idea was to Pick a feasible policy, u = h0(x), and compute the value

associated with operating forever with that policy. Next generate a new policy u = h

j+1(x) that solves the two-period problem. Then iterate.

Now let there be a predetermined class M of (N × N) stochastic matrices P, which are the objects of choice.

I think that we can associate each stochastic matrix P to a pair (a,a'). That's why we have n x n matrices P. So choosing P is like choosing (a,a').

11





We can put the Bellman equation in the following form (Eq. 4.4.3):

For B = T – I , find Pn+1

such that (Eq. 4.4.4):

Intuition: Pick a feasible policy function and use it to iterate. I believe the advantage of the algorithm is that it is easier to compute (compare Eq 4.3.0 with 4.5.2):

12

( I − β P n) v P n=c P n

c Pn+ 1+ (β P n+ 1 − I ) v Pn= Bv Pn

v P n=( I −β P n)

−1 cP n

Bv = Max P ∈ M { cP+ β Pv } − v





Eq. 4.5.1

Eq. 4.5.2

Step 1. For an initial feasible policy function g j (k, j) for j = 1, form the r

h

matrices using an equation similar to (4.5.1), then use equation (4.5.2) to evaluate the vectors of values [vj

1, vj

2] implied by using that policy forever.

Step 2. Use [vj1, vj

2] as the terminal value vectors in equation (4.3.2), and

perform one step on the Bellman equation to find a new policy function gj+1(k, s) for j + 1 = 2. Use this policy function, update j , and repeat step 1.

Step 3. Iterate.13

Rh (a )=u [( r + 1)a i+ ws j − g (a , s h )] , h=1, . . . , n

[v1

v2]=[I −β(P 11 J 1 P12 J 1

P 21 J 2 P 22 J 2)]

−1

[r1

r 2]





Second approach for obtaining numerical approximations:

Judd (1998) describes a method for iterating on the Bellman equation using a polynomial to approximate the value function and a numerical optimizer to perform the optimization at each iteration.

We describe this method in the context of the Bellman equation for a particular problem.

Model of optimal unemployment insurance as in chapter 19.

14

MotivationThe model

Implications for saving and growthConclusions



Polynomial approximations

A planner wants to provide incentives to an unemployed worker to search for a new job while also partially insuring the worker against bad luck in the search process.

The planner seeks to deliver discounted expected utility V to an unemployed worker at minimum cost while providing proper incentives to search for work.

Hopenhayn and Nicolini show that the minimum cost C(V) satisfies the Bellman equation (Eq. 4.7.1).

15

MotivationThe model





C (V )= minv u {c+ β [1− p (a )]C (V u )}

The method uses a polynomial to approximate the i th iterate Ci(V) of C(V).

This polynomial is stored on the computer in terms of n + 1 coefficients.

Then at each iteration, the Bellman equation is to be solved at a small number m >= n + 1 values of V .

This procedure gives values of the i th iterate of the value function Ci(V) at

those particular V ’s.

Then we interpolate to fill in the continuous function Ci(V).

The minimum problem on the right side of equation (4.7.1) becomes a numerical minimizer (doable with simple algorithms in Matlab or Gauss).

16

MotivationThe model





Algorithm summary:

1. Choose upper and lower bounds for Vu [Vu, Vu] required to assure participation and a positive search effort, computed in chapter 19.

1. Choose a degree n for the approximator, a Chebyshev polynomial, and a number m >= n + 1 of nodes.

1. Generate the m zeros of the Chebyshev polynomial on the set [1,−1] , given by (4.7.6).

1. By a change of scale, transform the zi’s to corresponding points V

lu in

[Vu, Vu]. 17

MotivationThe model





5. Choose initial values of the n+1 coefficients. Use these coefficients to define the function Ci(Vu) for iteration number i = 0.

6. Compute the function

5. For each point Vlu , use a numerical minimization program to find C

i+1(V

lu ).

6. Compute new values of the coefficients by using “least squares”. Return to step 5 and iterate to convergence.

18

MotivationThe model





C̃i(V )≡c+ β [1 − p (a )]C i (Vu)

This chapter has described two of three standard methods for approximating solutions of dynamic programs numerically: discretizing the state space and using polynomials to approximate the value function.

The next chapter describes the third method: making the problem have a quadratic return function and linear transition law.

This will make solving a dynamic program easy by exploiting stochastic linear difference equations.

19



Practical Dynamic Programming in Ljungqvist – Sargent (2004) Presented by Edson Silveira Sobrinho

Documents

Transcript of Practical Dynamic Programming in Ljungqvist – Sargent (2004) Presented by Edson Silveira Sobrinho