Forecasting with Judgment - Stanford Universityfinmath.stanford.edu › seminars › documents ›...

Forecasting with JudgmentForecasting with Judgment

Simone ManganelliDG-Research

European Central Bank(Frankfurt am Main, Germany)

Disclaimer: The views expressed in this paperare our own and do not necessarily reflect the

views of the ECB or the Eurosystem

Classical Theory of ForecastingClassical Theory of Forecasting

• Forecasts serve to make decisions about the future.• Forecast errors impose costs on the decision-maker.• Agents want to minimize expected loss associated to

forecast errors.• Classical forecast is the minimizer of the sample

equivalent of the expected loss.• See, e.g., Haavelmo (1944), Granger and Newbold

(1986), Granger and Machina (2005).

Optimal Point ForecastOptimal Point Forecast

])[()(min 210 !!

!"# +TyEQ

! =

"=

T

t tT yT1

1#̂

0][2 1 =!! + "TyEFOC: ][ 1*

+= TyE!

Ttty 1}{ = Want to forecast yT+1

Simplest case:• i.i.d. normally distributed observations with variance 1• quadratic loss function

The ProblemThe Problem

T!̂

! =

" "#T

t tT yTQ1

21 ])[()(ˆmin $$$

is the minimiser of:

]})[(])[(])[({min 211

2121 !!!

!"""+" +=

"+ # T

T

t tT yEyTyE

)(!"T

#

)}(])[({min 21 !"!

!TTyE +#+

Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion

OutlineOutline

0][)( 10 =!"# + $$ TyEQ

!!!~

ˆ]~

[)~(ˆ

1

1 "="#$ % =

"T

T

t tT yyTQ

maker.decision theof guess subjective a is ~

where!

Start from a Subjective GuessStart from a Subjective Guess])[()(min 2

10 !!!

"# +TyEQ

The sample equivalent of the FOCis:

NOTE: This is a random variable, which may be differentfrom zero just because of statistical error.

Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion

• Choose a confidence level α

• ηα/2 is the critical value

Then Do Hypothesis TestingThen Do Hypothesis Testingmean true theis

~ :0 !H

)/1,0(~~

ˆ TNyT !"


If null is rejected:

→ First derivatives statistically ≠ 0

→ Can be confident to decrease the true objective function

→ Only up to the point where new H0 cannot be rejected*ˆT!

2/!)~(ˆ !TQ"

)~(ˆ !TQ" )ˆ(ˆ *

TTQ !"

Graphical IllustrationGraphical Illustration

Distribution of FOC under H0

0


11

ˆ y=!

First derivatives not statisticallydifferent from zero

2/1

ˆ!"# $

2/1

ˆ!"# +

!~

])[( 2!"yE2

1 )( !"y

100*(1-α)%confidence interval

)(2 1y!"

!


IntuitionIntuition

New estimator:

!!"

!!#

$

%<%+

<%

>%%

=

2/2/

2/

2/2/

*

~ˆ if ˆ

|~

ˆ| if ~

~ˆ if ˆ

&&

&

&&

'('

'((

'('

(

TT

T

TT

T

yy

y

yy


InterpretationInterpretation

• α is the probability of committing type Ierrors, i.e. of rejecting the null when it is true.

• Choose low α when confident in subjectiveguess or if the cost of type I errors is high.

• Classical paradigm sets α =1:– Always FOC equal to zero;– No room for subjective guess;– It commits type I errors with probability 1.


Problems with Pretest EstimatorsProblems with Pretest Estimators

!!~ˆ:

0=

TH

Test the following null hypothesis, for given confidence level α:

Then:

• If do not reject, keep the subjective guess

• If reject, take the maximum likelihood estimator

This is wrong!


11

ˆ y=!

First derivatives not statisticallydifferent from zero

2/1

ˆ!"# $

2/1

ˆ!"# +

!~

])[( 2!"yE2

1 )( !"y

(1-α)% confidenceinterval

)(2 1y!"

!


Introducing JudgmentRisk Analysis of New EstimatorRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion

Estimate the mean, given a single observation y1 (Magnus02)

)1,(...~ 0!NdiiytLet

!!"

!!#

$

%<%+

<%

>%%

=

2/12/1

2/1

2/12/1

*

1

~ if

|~

| if ~

~ if

&&

&

&&

'('

'((

'('

(

yy

y

yy

Compare the risk properties of two estimators:1) Subjective classical estimator (coincides with Magnus 02)

!"

!#$

>%

<%=

2/11

2/1

|~

| if

|~

| if ~

ˆ

&

&

'(

'(((

yy

yP

2) Pretest estimator


Risk Associated to Risk Associated to ff((yy))

0

1

2

3

4

0 0.5 1 1.5 2 2.5 3 3.5 4

OLS (_=1) Subjective Guess (_=0)

Pretest (_=0.10) Subjective Classical (_=0.10)

0~

,]))([( 200 =! ""

"yfE


Monte Carlo SimulationMonte Carlo Simulation

• Random draws from standard normal• Sample sizes = 5, 20, 60, 120, 240, 1000• Quadratic loss function• Two estimators: classical, new (α=0.10)• Evaluated expected loss with MC simulation• Repeat 5000 times and average


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

5 20 60 120 240 1000

Mean

0

0.05

0.1

0.5

1


ImplicationsImplications

• Good guess (gut feelings) are as important as goodeconometric models.

• Organization of forecasting process:– Subjective guess based on maximum likelihood estimates

can never be rejected by construction;– Clear separation b/w:

Who provides guess, based on judgment;Who tests the guess, based on econometric models.

• Shared responsibility for the quality of the forecasts:– High confidence in bad judgment results in bad forecast.


Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion

GeneralizationGeneralization),0()(ˆ 0

!"# NUT

d

T$$

]1,0[ ~

)1(ˆ)(*!"+# $%$%$$%

TT

0*

0 )(: null Under the !"! =T

H

))((ˆˆ))((ˆ))((ˆ *1*'* !"!"!" "" TTTTTTUUTz #$%$#&

'

kTT

TTkTT

kTT

z

Uz

z

,

*

*

]1,0[,

*

,

*

))((ˆ s.t.

))((ˆmaxargˆ ))0((ˆ if

0ˆ ))0((ˆ if

!

"!

!

#"$

"$"#$

"#$

=

=%>

=%&

'


Mean-Variance Asset AllocationMean-Variance Asset Allocation• Mean-variance optimizers tend to produce portfolio

allocations with little or negative investment value.

• “[They] overuse statistically estimated informationand magnify the impact of estimation errors. It is notsimply a matter of garbage in, garbage out, but,rather, a molehill of garbage in, a mountain ofgarbage out” (Michaud 1998)


Set upSet up

• N+1 assets

• yt,i return of asset i

• θi weight of asset i (they sum to 1)

!+

==

1

1 ,)(N

i itit yy "" portfolio return


OtpimizationOtpimization

})]([)]([{)]([

}]['{]['

)]([)]([];[

22 !!"!

!!"!

!"!!

ttt

tt

ttt

yEyEyE

yVyE

yVyEyU

##=

#=

#=

}])([)({)(];[ˆ 2

1

1

1

21

1

1 !!! =

"

=

"

=

" ""=T

t t

T

t t

T

t tt yTyTyTyU ##$##


Implementation DetailsImplementation Details• Monthly log returns from DJIA• From January 1987 to July 2005• λ = 4 (coefficient of risk aversion)• Rolling windows (M=60 and 120 as in

DeMiguel et al. 2007)• Benchmark portfolio: equally weighted• Report out of sample differences in average

utilities between optimal and benchmarkportfolios


)}(])()[()({)()(1

12 !!!"!! t

T

Mt tttt yyMTyyU # +=

$$$$=

! += ""

" ""#T

Mt tttMTUUMTZ

1

*

1

1 )]~())ˆ(([)( $%$

Out of Sample EvaluationOut of Sample Evaluation

Test for statistical significance using Giacomini and White (2006)


00.92**0.01000.02α=0.01

1.48*4.19***0.82*00.090.48*α=0.10

2.616.47*1.82-8.40***1.00-0.30α=1

N=30N=16N=4N=30N=16N=4

M=120M=60

ResultsResultsIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion

Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPExample 2: Forecasting US GDPRelationship with BayesianConclusion

Implementation IssuesImplementation IssuesDifficult to formulate guess on abstract model parameters.Assume the decision-maker can formulate a guess on the variable

to be forecast (U.S. GDP, in this example). This guess can bemapped into a guess for model parameters as follows:

11~)(ˆ s.t.

)(ˆmaxarg~

++ =

=

TT

T

yy

U

!

!!!

Subjective guess

Model’s forecast


Implementation DetailsImplementation Details

ti itityy !"" ++= # = $

4

10

The model, AR(4):

! =

" ""=T

t ttT yyTU1

21 )](ˆ[)(ˆ ##

The objective function (quadratic):

The data:

• Quarterly U.S. real GDP growth rates

• FRED data base, seasonally adjusted

• From Q1 1983 to Q3 2005


ResultsResults

3.77%5%3%3%3.19%

0.01-0.040.050.050.04

-0.12-0.03-0.18-0.18-0.16

0.340.300.370.370.36

0.290.420.210.210.23

1.992.731.541.541.65

%3~

1=+Ty

0!

)ˆ(* !"T

%5~

1=+Ty

T!̂ !

~

1!

2!

3!

4!

)(ˆ1 !+Ty

!~

)ˆ(* !"T


Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianRelationship with BayesianConclusion

Non sample information is summarised by:• priors, in Bayesian econometrics• subjective guess and confidence associated to it, in our case

Special cases:1. No uncertainty about optimal value of θ0:

– Bayesian: prior is degenerate distribution with total mass on θ0;– Classical: subjective guess=θ0, α=0 (i.e. never reject the null)

2. No information, besides the sample:– Bayesian: diffuse prior– Classical : α=1 (i.e. set FOC equal to zero)

In other intermediate cases, no clear mapping b/w the two.When non sample info if formulated:• via prior distributions, be Bayesian• via subjective guess and confidence, use subjective classical estimator

In general, the choice is dictated by the decision maker, throughthe format in which s/he provides the non sample info.

Relationship with Bayesian EconometricsRelationship with Bayesian Econometrics

ConclusionConclusion

• Ignoring non sample information and estimation errorare connected problems in the classical theory offorecasting.

• Forecasts should maximize the objective function in astochastic sense, not deterministic.

• Start from subjective guess and construct estimator(instead of the other way round)

• Test if FOC are statistically different from zero:– If not, subjective guess is retained as forecast– If yes, objective function is increased as long as FOC are ~= 0

Forecasting with Judgment - Stanford Universityfinmath.stanford.edu › seminars › documents ›...

Documents

Transcript of Forecasting with Judgment - Stanford Universityfinmath.stanford.edu › seminars › documents ›...