Eco No Metric Methodologies JES

8/6/2019 Eco No Metric Methodologies JES

1/22

THREE ECONOMETRIC METHODOLOGIES:A CRITICAL APPRAISALAdrian Pagan

University of RochesterAbstract. Three econometric methodologies, associated respectively with DavidHendry, Christopher Sims and Edward Learner have been advocated and prac-ticed by their adherents in recent years. A n umb er of go od pap ers have appearedabout each metho dology, but little has been w ritten in a comp arative vein. Thispaper is concerned with that task. It provides a statement of the main steps tobe followed in using each of the methodologies and com ments upo n the strengthsand weaknesses of each approac h. An attem pt is made to contrast and com parethe techniques used, the information provided, and the questions addressed byeach of the methodologies. It is hoped th at such a com parison will aid researchersin choosing the best way to examine their particular problem.Keywords. Econometric methodologies; Hendry; Sims; Learner; extremebounds analysis; vector autoregressions; dyna mic specification.

Methodological d eba te in economics is almost as long-standing as th e disciplineitself. Pro bab ly the first impo rtan t piece was written by Jo hn Stu art M ill (1967)and his conclusions seem as pertinent tod ay as when they w ere written in the 19thcentury. He observed that many practitioners of political economy actually heldfaulty conceptions of what their science covered and the methods used. At thesame time he emphasized that, in many instances, it was easier to practice ascience tha n to describe how one w as doing it. H e finally concluded th at a betterunderstanding of scope and method would facilitate the progi tss or economicsas a science, but that sound methodology was not a necessary cogdition for thepractice of sou nd m ethods. Get on with th e jo b seems the ap pro pria te message.It is interesting th at it was not until the 5th W orld Congress of the Econom etricSociety in 1985 tha t a session was devoted to methodological issues. There aregood reasons for this. Until the mid-1970s it would have been difficult to finda comprehensive statement of the principles guiding econometric research, andit is hard to escape the conclusion that econometricians had taken to Millsinjunction w ith a vengeance. Even the debat e between frequentists an d subjec-tivists that prevailed in statistics was mu ch m ore muted in econo metr ics. It is tru ethat there was a vigorous attempt to convert econometricians to a Bayesianapproach by Zellner (1971) and the Belgian connection at CORE (see Drt?ze an dRichard, 1983). But this attempt did not seem to have a great impact uponapplied research.All of this changed afte r 1975. Causes are always harde r to isolate th an effects,0950-0804/87/01 0003-22 %02.50/0JOURNAL OF ECONOMIC SURVEYS Vol.1. No.1 0 987 A. Pagan

3


2/22

4 P A G A Nbut it is difficult to escape the impression that the p roxima te cause was the predic-tive failure of large-scale models just when they were most needed. In retrospectit seems likely tha t the gunp ow der had been there for som e time, and th at theseevents just set it off. Most, for example, will know Ed Learners (1978, p. vi)account of his dissatisfaction with the gap between what he had been taught inbook s and the way practitioners acted, an d it seems likely that m any o thers hadfelt the same way about the type of econometrics then prevalent. But thesemisgivings were unlikely t o have an y impact until there w as evidence tha t the rewas something to complain about.Since 1975 we have seen a concerted a ttem pt by a num ber o f au tho rs to buildmethodologies for econometric analysis. Implicit in these actions has been thenotion tha t work along t he prescribed lines would better econometrics in a t leastthree ways. First, the m ethodology w ould (and should) provide a set of principlesto guide work in all its facets. Second, by codifying this body of knowledge itshould greatly facilitate the transmission of such knowledge. Finally, a style ofreporting should naturally arise from the methodology that is informative,succinct and readily understood.

In this paper we look at the current state of the debate over methodology.Three m ajor con tenders fo r the best methodology title may be distinguished. Iwill refer to these as the Hendry, Learner and Sims methodologies, afterthose individuals most closely idenrued with the ap proa ch. G enerally, each pro-cedure has its origins a good deal further back in time, a nd is the ou tco m e of aresearch programme that has had many contributors apart from the namedauthors above. But the references-Hendry an d Richard (1982), Learner (1978)and Sims (1980a)-are the most accessible an d succinct summ aries of th ematerial, a nd therefore it seems ap pro pria te to use the chosen appellations. In-evitably, there has been some convergence in the views, bu t it will be m ost usefulto present them in polar fashion, so as to isolate their distinct features.1. The Hendry methodologyPerhaps the closest of all the methods to the old style of investigation is theHen dry methodology. It owes a lot to Sarg ans seminal (1964) pape r, but it alsoreflects an oral tradition developed largely at the London School of Economicsover the past two decades. Essentially it comprises four steps.

(i) Formulate a general model that is consistent with what economic theorypostulates are the variables entering any equilibrium relationship and whichrestricts the dynamics of the process as little as possible.(ii) Re-parameterize the model to obtain exp lanatory variables tha t are nearorthogonal and which are interpretable in terms of the final equilibrium.(iii) Simplify the model t o the smallest version tha t is com patible with the d at a(congruent).(iv) Evaluate the resulting model by extensive analysis of residuals and predic-tive performance, aiming to find the weaknesses of the model designed in theprevious step.


3/22

THREE ECONOMETRIC METHODOLOGIES 5Steps (i) and (ii)Theory an d da ta continually interplay in this methodology. Unless there ar e goodreasons fo r believing otherwise, it is normally assumed th at theory suggests whichvariables should enter a relationship, and the data is left to determine whetherthis relationship is static or dynamic (in the sense that once disturbed fromequilibrium it takes time to re-establish it).

It may help to understand the various steps of Hendrys methodolog y if a par-ticular example is studied. Suppose tha t th e investigator is interested in the deter-minants of the velocity of circulation of money. Let mi be the log of the moneysupply, pr be the log of the price level and y r be the log of the real income.Theoretical reasoning suggests th at, for app rop riate ly defined mo ney,mr- pr- y f should be a function of the nominal interest rate ( I f ) along anysteady state growth path. With i f= log(Ir), we might therefore writem:- p : - y : = S i ; where the starred quantities indicate equilibrium values.Of course equilibrium quantities are not norm ally observed, leading to the needto relate these to actual values. For time series data it is natural to do this byallowing the relations between the variables m r ,pr, y r an d i f to be governed bya dynamic equation of the form

m i = 5 a jmr - j + b j p t - j + C j y r - j + 5; dji i - j . ( 1 )j = j = O j = O j = OTh e first step in Hendrys methodology sets p, q, r and s to be as large as prac-

ticable in view of the type of da ta (generally p = q = r = s = 5 for quarterly data),and to then estimate (1). This model, the general model, serves as a vehicleagainst which all other models are ultimately compared.Now (1) could be written in many different ways, all of which would yield thesame estimates of the unknown parameters, but each of which packages the in-formation differently and consequently may be easier to interpret and under-stand. Generally, Hendry prefers to re-write the dynamics in (1) as an errorcorrection mechanism (ECM). To illustrate this point, the simple relationXr= ~ x i - 1 box:+ b lX: -l , (2a)

where x, is the equilibrium value of x f , has the ECMA X r = ( a - ) ( X i - 1 - x;- 1 ) + boAX:+ ( a- 1 + bo + bl )x , -

= ( a - ) ( X i - I - x:- I )+ boAx:, (2b)since steady-state equilibrium in (2a) implies x = ax + box + b l x ora + bo + 61 = 1 . Although (2b) is no different t o (2a), He ndr y prefers it since Ax:and ( x i - - x:- I ) are closer to being orthogonal and he is able to interpret itselements as equilibrium (Ax:) an d disequilibrium ( x r - - x:- I ) responses.Moving away from this simple representation we can get some feeling for thetype of equation Hendry would replace (1 ) with by assuming that mr adjustswithin the period to p r , making the log of real money mi - pr the naturalanalogue of xr in 2(a). The equilibrium value is then x:= yr+ A i r , and by appeal


4/22


5/22


6/22

8 PAGANhe actually did the simplification. In Hendry (1986) for example, the transitionfrom a model with thirty-one parameters to one with only fourteen is explainedin the following way (p. 29):

These equations. . . were then transformed to a more interpretableparameterisation and redundant functions were deleted; the resultingparsimonious models were tested against the initial unrestricted forms bythe overall F-test.It is tru e that co nfidence in the simplified m odel is partly a function of t he valueof the F-test, but by its very natur e this evidence can only mean th at some of thedeleted variables dont matter. To see why, consider a general model with threeregressors XI, x2 and x3, all of which are orthogo nal. Suppose the F statistic forthe deletion of x3 is 5 and that for x2 is 0.5. Then the Fstatistic for the joint dele-tion of x2 and x3 is 2.75, and joint deletion is likely, even though it is dubious

i f x3 should be deleted at all. Thus an adequate documentation of the pathfollowed in any simplification process is desirable, rather than just accompanyingany simplification with a vague statement about it. More than that, I do believein the possibility o f situations in which sim plication may be do ne in a systematicway e.g. in choosing dynamics via CO M FA C (as in Hendry and Mizon, 1978 orMcAleer er a / . , 1985), polynomial orders within Almon procedures and varioustypes of demand and production restrictions that form a nested heirachy. As faras possible I am in favour of exploiting such well-developed strategies forsimplification. Research should also be encouraged with the aim of developingnew procedures or methods that require fewer assumptions.Knowledge of the path may be important for another reason. As discussedabove the critical value used in the decision rule is taken from the tables of th ex 2 or Fdi strib utio n. But under the conditions of th e story being used, this is onlytrue if the simplification path consists of a single step. When there has been morethan one step, the critical values cann ot normally be taken fro m a x 2 distribution,and it may be misleading if one proceeds as if it can. Some, for example Hill(1986), see this as a m ajor flaw in th e methodo logy, and o ther s feel that th e deci-sion rule needs to be modified quite substantially in the presence of such datamining. W hen the move fro m a general to a simplified model can be formulatedas a nested sequence, adjustments can be made to obtain the requisite criticalvalue (Mizon (1977) gives an account of this), but in the more common casewhere this is not possible theoretical analysis has made little progress. Never-theless, numerical methods of Type I error evaluation, such as the boots trap, d oenable the tracing of Type I error for any sequence of tests an d specified decisionrules. Veal1 (1986) provides an application of this idea.I am not certain th at it is worthwhile computing exact Type I errors. Ignoringthe sequence entirely produces a bias against the simplified model, but tha t doesnot seem such a bad thing. Moreover, the ultimate change in the criterion func-tion is independent of the path followed. It is frequently the change in thecriterion itself which is of interest, in that it displays the sensitivity of (say) thelog likelihood to variation in parameter value for the deleted variables as theserange from zero to the point estimates of the general model.


7/22

THREE ECONOMETRIC METHODOLOGIES 9Step (iv)Excellent accou nts are available of the necessity of this step-Hendry an dRichard (1982)-and the techniques fo r doing it-Engle (1984). Essentially, theseprocedures check if sample moments involving the product of specified variableswith functions of the d ata (typically residuals) a re zero. Very general treatm entsof diagnostic tests from this viewpoint have recently been given by Tauchen(1985) and Newey (1985).4 These procedures fulfil a number of roles within themethodology. They are important within a modelling cycle for the detection ofinadequate models, but they are also im por tan t in the reporting phase, where theyprovide evidence that the conventions underlying almost any m odelling exerciseare not violated by the chosen model. Routine examination of such items as theautocorrelation function and recursive estimation of parameters has proved to beindispensible t o both my own m odelling (Anstie et al., 1983; Pagan and Volker,1981) and to those of a large number of students studying applied econometricsat the Australian National University over the past decade (Harper, 1980; Kirby,1981 for example). More than anything else, it is step (iv) which differentiatesHendrys methodology from that which was standard practice in the 1960s.

2. The Learner methodologyProviding a succint description of Learners m ethodology is a good d eal m ore dif -ficult than doi ng so for the Hendry variant. Basically, the problem lies in a lackof applications of the ideas; consequently it is har d t o infer th e general principlesof the appro ach from any classic studies of how it is to work in practice. Despitethis qualification, I have reduced Learners methodology to four distinct steps.

(i) Formulate a general family of models.(ii) Decide what inferences are of interest, express these in terms ofparameters, and form tentative prior distributions that summarize the infor-mation not contained in the given data set.(iii) Consider the sensitivity of inferences to a particular choice of priordistributions, namely those that are diffuse for a specified sub-set of theparameters and arbitrary for the remainder. This is the extreme boundsanalysis (E BA) of Leamer (1983) and Leamer and Leonard (1983). Sometimesstep ( i i i ) terminates the process, but when it appears that inferences are sen-sitive to the prior specification this step is only a warm-up for the next one.(iv) Try to obtain a narrower range for the inferences. In some places thisseems to involve an explicit Bayesian approach, but in others it seems just toinvolve fixing a prior mean and interval for prior covariance matrices. If therestrictions in this latter step needed to get a narrow range ar e too implausible,one concludes that any inference based on this data is fragile.Collected a s in (i)-(iv), Learners m ethodology seems to be just an oth er sectin the Bayesian religion, and there is little point in my going over the debate instatistics concerning Bayesian procedures. Much of this is epistemological and Idoubt if it will ever be resolved. In practice, the limited appeal of Bayesian


8/22

10 P A G A Nmethods to econometricians seems to have been based on the difficulties comingfrom a need to form ulate high-dimensional priors in any realistic model, naggingdoubts about the need to have precise distributional forms to generate posteriordistributions, and the fact that many dubious auxiliary assumptions are fre-quently employed (for example, lack of serial correlation and heteroskedasticityin the errors). In theory, all of these doub ts could be laid to rest, but th e com put a-tional burden becomes increasingly heavy.Viewed as basically a n exercise in Bayesian econo me trics, I have therefore verylittle to say abou t Learners method. It is not t o my taste, but it may well be toothers. However, in attempting to sell his ideas, Learner has produced, par-ticularly in step (iii), an approach that can be interpreted in a classical ratherthan Bayesian way, and it is this which one tends to think of as the Leamermethodology. Th e reasons for such a belief lie in the advocacy of such ideas inLearners two most widely read articles, Leamer (1983) and Leamer and Leonard(1983), although it is clear from Learner (1985, 1986) that h e now sees the fo urthstep as the important part of his analysis. Nevertheless, applications tend to beof step (iii), and we will, therefore, analyse it before proceeding to (iv).Returning to steps (i) and (ii), it is apparent they do not differ greatly fromHendrys methodology (HM); the main distinction is that in HM the emphasisis o n building a model from which inferences will later be dra wn , whereas L eamerfocuses upon the desired inference from the beginning. Because of this concernabout a particular parameter (or, more correctly, a linear combination ofparameters), it is not clear if Leamer has a counte rpart t o the simplification stepin Hendrys methodology. In published applications he always retains thecomplete model fo r inferences, b ut he has suggested t o me th at s om e simplifica-tion may be practised as an aid to communication or in the interest of efficientprediction.Thus, cast in terms of (1 ) the essential distinction in these early steps betweenthe two methodologies is that Leamer would want a clear definition of what theissues in modelling money dem and ar e at the beginning. Suppose it w as the ques-tion of the impact of interest rate variations o n money dem an d, the question rais-ed by Cooley and LeRoy (1981) in one of the best known applications ofLearners ideas. Then either the size of individual djs or (1 - Ca j ) - C dj (thelong-run response) would be the items of interest, and the model would bere-parameterized to reflect these concerns. In Hendrys case it is rare to find aparticular set of coefficients being the centre of attention; it is variable inter-relationships as a whole that seem to dominate. As in McAleer et al. (1985),questions about the magnitude of the interest rate response in (1 ) are answeredafter the final model is chosen.Step (iii)To gain a better appreciation of what is involved in step (iii), particularly as acontrast to HM, it is necessary to expose the link between them. Accordingly,take the general model

Yr = XrP + Zry + er, (3 )


9/22

THREE ECONOMETRIC METHODOLOGIES 1 1where tr re a set of doubtful variables, and interest centres upon the pointestimate of the first coefficient in the vector , 01. In terms of th e variables in ( l ) ,x, would relate to the interest rate variables while zf would be the remainder. Instep (iii) Leamer examines the extreme values of the point estimates of 01 as allpossible linear combinations of zr are entered into regressions that always containxr (this being formally equivalent to diffuse priors upon 0 nd arbitrary priorson y). In McAleer er al. (1983, Appendix) it is shown that the ab solute differencebetween these bounds, scaled by the standard deviation of the OLS estimate of01 rom (3) , is given by:

(4)where 0 Q 4 Q 1 and xb is the chi-square statistic for testing if y is zero.Leamer refers to the left hand side of (4) as specification uncertainty. Let usfirst take the extreme case that 4 = 1 and ask what extra information is providedby an EBA that is not available to someone following HM. In the latter, if xbwas small, the recommended point estimate of 01 for someone following HMwould be that from the model deleting t t .From (4) an exactly equivalent state-ment would be that the specification uncertainty is very small, and the pointestimate of PI would not change very much as one moved from the general tothe simplified model. This is to be expected since, following Hausman (1978), alarge difference between 81 for the simplified and general models must meanevidence against any simplification. T hus th e two ap pro ach es provide a differentpackaging of the same info rm atio n, an d sh are exactly th e same set of difficulties.In particular, all the problems of nominating a critical value for & have theircounter-part in Learners methodology as providing critical values for specifi-cation uncertainty. As observed in McAleer er al. (1985), there has been no agree-ment on the latter question by users of the EBA method, with a range of defini-tions being proposed. Another interesting concomitant of (4) is that if, y # 0 in(3), x &-, as the sample size grows, a nd so, when 4 # 0, the range between thebounds tends to infinity. Thus Learners complaints about classical hypothesistesting apply also to his own methodology!Now, in HM it is an insignflcant x 6 that is important, but this need not benumerically small if the dimension of zc is large. Taking the previously citedexample from Hendry (1986), where seventeen parameters were set to zero,xz(17 ,0 .05 )= 27.59, allowing a potentially enormous gap between 6 1 , m i n andO l , m a x ; point estimates of the simplified model might the refo re de pa rt substa n-tially from those based upon other ways of combining the zr. If i f is poinrestimates ofP1 tha t are desired, it becomes very informative t o perform an EB A(i.e. to comp ute 4 ); knowledge of x b only sets an up per limit to t he specificationuncertainty, as it is the collinearity between regressors, reflected in 4, tha t deter-mines the exact value of the specification uncertainty. Whe never a largenumber of variables are deleted in a simplification exercise, the provision ofextreme bounds for any coefficients of interest seems desirable.Where the two methodologies really part compan y is over the interp reta tion ofa large x 6 . Followers of HM would argue that one should take point estimatesof 01 rom th e general model, concluding it would be a n error to ta ke them from

SD @i > - I 6 l . m a x - 1 , m i n I = 4xb


10/22

12 PAGANthe simplified model, as the data clearly indicate that the zr appear in therelationship. earner would presumably conclude that Because there are manymodels which could serve as a basis fo r a da ta analysis, there are many conflictinginferences which could be drawn fro m a given da ta set and therefore inferencesfrom these dat a are too fragile to be useful (Learner and Le ona rd, 1983, p. 306).I confess that I cannot be convinced that our response to a situation where thedata are clearly indicating that valid point estimates of 01 will not be found bydeleting zr from (1) should be to conclude that the dat a are not informative abo utPI!There is no denying that there would be comfort in narrow boun ds, as any co n-clusions that depend upon the precise value of 01 would then be unchanged byvariation in specifications. Some, for example Feldstein (1982), even see this asa desirable characteristic. But I think it hard to argue that the majority of m odel-ling exercises can be formulated in terms of interest in the value of a single co-emcient (or a linear combination of them). It is perhaps no accident that theexamples Learner provides in his articles do feature situations where singleparameter inference is par am ou nt, whereas Hendrys examples-money de m an d,consumption-are more concerned with the model as a whole. If the equati on (3)was being developed as part of a policy model, or even to provide predictions,knowledge of x$ is important, as a large value would presumably imply thatmodels which retained Zr would out-perform those that did not. Any modelshould be judged on all its dimensions and not just a few of them. One mightargue fo r an extension of Learners methodology t ha t chose 01 s representativeof many characteristics of a model. Since prediction errors can be estimated asthe coefficients of dumm y variables (Salkever, 1976) these might b e tak en as P I .Alternatively, why not look a t the extreme bou nds fo r the residual variance? Butthese must be the two estimates of u2 obtained by including and deleting all thezr in ( l ) , and so on e is essentially re -produ cing the x% statistics. Accordingly, onceattention shifts from a single parameter to overall model performance EBAbegins to look like a version of step (ii) of HM .

Step (iv)The fourth step constitutes the clearest expression of Bayesian philosophy inLearners work. Until this step it is not m andato ry t o fo rmu late a prior distribu-tion, but now at least the mean and variance of it must be provided (only twomoments ar e needed given the type of prio r assumed in his S EA RC H program).A proper Bayesian would then proceed to combine this prior knowledge with alikelihood, reporting the posterior distribution for the coefficient. If forced togive a point estimate of the coefficient, such an individual would probably givethe mode, median or mean of the posterior distribution. That would then be theend of the exercise, the data an d prior beliefs having been optimally comb ined t oprovide the best information possible about the parameter value. Consequently,when modelling money dema nd as in (l), a Bayesian w ould need t o form ulate a(p+ q + r + s)-dimensional prior distribution upo n the parameters of this model.


11/22

THREE ECONOMETRIC METHODOLOGIES 13A daunting task, although some progress has been made in automating theprocess of prior elicitation in recent years, and Learner (1986) is an excellentexample of how to do this in a context similar to that in (1).

What differentiates Learner from a standard Bayesian is his reluctance tofollow the above prescription rigidly. Rather, he prefers to study how the meanof the posterior distribution changes as the prior variances change. In Learner(1986) he stipulates a prior covariance matrix A , but them modifies i t to Vobeying the following constraint:

(1 - X)A < V < [ 1 / ( 1 - X ) ) A (O< X < 1).As X ranges from zero to unity the precision of the prior informationdiminishes and, for any given value of A , bounds for the posterior mean can becomputed corresponding to each side of the inequality. What is of primary in-terest t o Learner is how these bounds change in response t o variations in X, ratherthan just the values at X = 0. As he says in Learner (1985), what he is concernedwith is sensitivity analysis, and it is the question of sensitivity of inferences tovariation in assumptions which should pre-occupy the econometrician.If step (iv) is thought of as a tool to provide a Bayesian analyst with evidence

of how important prior assumptions are for conclusions based on the posterior,i t seems unexceptionable and useful. Is this also true for an investigator notoperating within the Bayesian paradigm? What is of concern to that individualis the shape of the likelihood. Step (iv) can provide some information on thisaspect. On the one hand, if the likelihood is completely flat the posterior andprior means would always coincide. On the other hand, if the likelihood wassharply defined around a particular point in the parameter space, changing Xwould cause the posterior mean to shift fro m the prior mean t o this point. Unf or-tunately, it is not entirely reliable as a guide to the characteristics of thelikelihood, as can be seen in the case of the linear model. With the prior meanset to BOLS and A proportional to (X'X)-',he posterior and prior meansalways coincide, so nothing is learnt about the likelihood as X is varied.From the above, the intention of step (iv) seems good, even if in execution itmay leave something to be desired. I think it certainly true that workers in theHM tradition do not pay enough attention to the shape of the likelihood (seeNote 7). The provision of second derivatives of the log likelihood (standarderrors) gives so m e feel for it, b ut they c an be very unreliable if problems are non-linear. Whether Learner's procedure is the best response is a moot point; at themoment it is one of the few methods we have of discovering information aboutcurva ture in the likelihood, an d its strategy to overcome the problems caused bya high dime nsional param eter space (index it by a single param eter A) may wellbe the best way to proceed. Certainly, we can use all the help we can get whenit comes to the analysis of data.My main reservation about step (iv), however, is that it does not do enoughsensitivity analysis, being restricted to the parameters of the prior distribution.As exemplified in his SEARCH progra m, there a re many conventions underlyingthe methodology (just as there were in H M ) , but those applying it have made


12/22

14 PAGANprecious little attempt to query the validity of such conventions fo r the data setbeing analysed. This is od d since, in principle, the re should be few dimcu lties inmimicking step (iv) of HM . Since most diagnostic tests can be formulated asmeasuring the sensitivity of the log likelihood t o the addition of variables design-ed to detect departures from th e conventions (Pa gan , 1984) they should be readilyadapted t o Learners framew ork. It seems imperative tha t this become part ofthe methodology. Leamer has indicated to me th at he does see the need toexamine the data for anom alies that suggest revision of th e model space or of theinitial prior distributions; the tools in this task ranging from unexpectedparameter estimates and peculiar residual patterns to (possibly) goodness-of-fitstatistics. But he emphasizes that adjustments must be made for any data-instigated revision of the model or prior. Because such adj ustm ent s are difficulthis first preference is for an initial selection of prior and model extensive enoughas to make any such revision unlikely. Nevertheless, when theory is rudimentaryand underdeveloped, comm itment to the original m odel and prior is likely to below, and the need for revision correspondingly high.3. Sims methodologyInterdependence of actions is one of the characteristics of economic studies.Hence, it might be argued that the evaluation of policy will normally need to bedone within a framework that allows for such interdependence. In fact, a gooddeal of analysis, and the econometrics supporting it, is done in a partial ratherth an general equilibrium way-see Feldstein (1982) fo r exam ple, where th e im-pact of taxes upon investment is assessed in a series of single equation studies.Traditionally, such questions were analysed with the aid of a system of stru ctu ralequations:

Byt - Cxf = er , ( 5 )where y, is a vector of endogenous variables, Xi a vector of predeterminedvariables, and el was the disturbance term. In (9 , ollowing the lead of theCowles Commission researchers, both B and Cwere take n t o be relatively sparse,so as to identify the separate relations, i.e. it was assumed that an investigatorcould decide which variables appeared in which equations.Both of the two previous methodologies would probably subscribe to thisframew ork, aiming to calibrate the non-zero elements in B and C (Leamer mightregard the exclusion restrictions as only approximately correct, but I know ofnowhere that he has explicitly stated his preferred procedure). By contrast, thethird methodology jettisons it. Sims (1980a) dissented vigorously from theCowles Commission tradition, resurrecting an old article by Liu (1960), which in-sisted tha t it was incredible to reg ard B and C as sparse. T he argument touchesa chord with anyone involved in the construction of computable generalequilibrium models. If decisions on consumption, labour supply, portfolioallocations, etc. are all determined by individuals maximizing lifetime utility sub-ject to a budget constraint, each relationship would be determined by the same


13/22

THREE ECONOMETRIC METHODOLOGIES 15set of variables. Consequently, theoretical considerations would predict nodifference in the menu of variables entering different equations, although thequa ntit ativ e importance of individual variables is most likely to vary with the typeof decision. Prescription of the zero elements in B a nd C herefore involvesexcluding variables with coefficients close to zero. In this respect, the action islittle different to what is done in any attempt to model reality by capturing thema jor influences at w ork. This was Fishers (1961) reply t o Liu, an d I find it aspertinent now as when it was written.Much more could be said about this issue of identifiability, but this is not theplace to do so. One cannot help wondering, however, if it is as serious as Simssuggests. There d o not seem many instances in applied work where identificationis the likely suspect when accounting for poor results. Despite the large amountof attention paid to it in early econometrics, it is hard to escape the impressionthat issues of specification and data quality are of far greater importance.Nevertheless, it would be silly to ignore these arg um ents if it was indeed possi-ble to do analysis without such a ssumptions. Sims claims th at it is. In th e CowlesCommission methodology, structure-free conclusions would have been derivedfrom the reduced form:

(6)but Sims chooses instead to work with a vector autoregressive representation(VAR) for the endogenous and exogenous variables. Defining z;= (y,!!;),where X r includes all members of xl that are not lagged values of variables, thishas the form:

~t = B - Cxr + B - let = nxr + v I ,

Zr = 2 Ajzr-j + e r .j = (7)

Although it is (7 ) that is estimated, two further manipulations are made for usein later stages of the methodology. First, (7) is inverted to give the innovations(o r moving average) form:m

Zr = C Ajer-jj = O

where & = cov(er) . Since Ao is a positive definite matrix there exists a non-singular lower triangular matrix P such that P A o P = I , allowing th e definitionq r = Per, where q1 has zero mean and covariance matrix I . (8) may then be re-written in terms of q r as:

m mz I = C A j P - P e r - j = C D , ~ i - j , (9)j = O j = O

where the q r are the orthogonalized innovations.methodology in four steps.Having dispatched the preliminaries it is possible to summarize Sims

(i ) Transform data to such a form that a VAR can be fitted to it.


14/22

16 PAGAN(ii) Choose as large a value of p and dim(zl) as is compatible with the size ofdata set available and then fit the resulting V A R .(iii) Try to simplify the V A R by reducing p or by imposing some arbitrarysmoothness restrictions upon the coefficients.(iv) Use the orthogonalized innovations representation to address the questionof interest.

Step ( i )This is an important step. The idea that zr can be expressed as a V A R has itsorigins in the theory of stationary processes, particularly in the Wold decom-position theorem. But that justification is not essential until the last step; untilthen the V A R might well have unstable roots. However, stable roots are indis-pensible t o step (iv), as the coefficients Aj only damp out for a stable V A R , i.e.Zr = azr-I + el ( z f a scalar) becomes

W

and a + O ( j + 00) only if I a I < 1. If step (iv) is to be regarded as an essentialpart of the methodology, the question of the appropriate transformation torender zr stationary must be faced at an early stage.In Sims (1980a) and D oan et al. (1984), as well as most applic ations, this seemsto be done by including time trends in each equation of the V A R . In th e latterarticle the attitude seems to be that most economic time series are best thoughtof as a stationary autoregression around a deterministic trend: after setting upthe prior that the series follow a rando m walk with drift (equation ( 3 ) , p. 7) theythen say:

While we recognise that a more accurate representation of generally heldprior beliefs would give less weight to systems with explosive roots. . . . ..It is not apparent to me that this is a generally held prior belief, particularlygiven the incidence of rand om walks with d rift in the investigation of N elson an d

Plosser (1982) into the behaviour of economic time series. If the series are of theran dom walk type, placing deterministic trends in to a regression does no t sufficeto induce stationarity, and an innovations form will not exist for the series inquestion. Of course, the sensible response to this objection would be to focusupon growth rates rather than levels for variables that are best regarded asA R I M A processes. I suspect that this makes som ewhat m ore sense in many con -texts anyway. In macroeconomic policy questions for example, interest typicallycentres upon rates of growth of output and prices rather than levels, and ittherefore seems appro priate to fo rmu late the V A R in this way. Consequently, t hedifficulties raised by the type of non-stationarity exhibited by many economictime series is not insurmountable, but it does suggest that much mo re care needsto be taken in identifying the form at of th e variables t o be modelled tha n has beencharacteristic of past studies employing Sims methodology.


15/22

THREE ECONOMETRIC M ETHODOLOGIES 17Step (ii)Both p and t he number of variables in zf need to be specified. T he first param eterwill need to be fairly large (the decomposition theorem sets it to infinity), andmost applications of Sims methodology have put p between four and ten. Doanet al. (1984, Footnote 3) indicate t hat , at least fo r prediction perform ance, con-clusions might be sensitive to the choice of lag length. Stronger evidence isavailable that the selection of variables to appear in zr is an im por tan t one-Simsconclusions about the role of money in Sims (1980a) were severely modified inSims (1980b) by expan ding t r o include an interest rate. Essentially step (ii) is theanalogue of step ( i) in the previous two m ethodologies, a nd th e need to begin witha model that is general enough h aun ts all the methodologies. Per hap s the dif-ficulties are greater in Sims case, as he wants t o model t he reduced for m rathertha n a single structural eq uat ion . To ad op t such a position it would be necessaryto respond to Sims contention that structural equations should also contain alarge number of variables, although what is really at issue is whether they arequantitatively more im portan t to the reduced form analysis.Step (iii)Step ( i i i ) is required precisely because of the fact that both p and dim(z,) needto be large, and so the number of unknown parameters, p x dim (zr), can easilybecome too large to be estimated from the available data. In his original articleSims chose p via a series of modified likelihood ratio tests in exactly the sam e wayas was don e in step (ii) of H endrys m ethodology. Because there ar e few degreesof freedom available in the most general model, this may not be a good way toselect p . Accordingly, in Doan et al. (1984) a different approach was promotedthat was Bayesian in spirit. In this variant the Aj were allowed t o vary over timeas

vec(Aj, ()= Tsvec(Aj , r - l )+ (1 - Ts)vec(Af) + v $ , ~ , (10)where the i indicates the ith equation and V j , r is a normally distributed randomvariable with covariance matrix V that is a function of T I . . ~ 7 . ixing the A:in (10) (at either unity, i f the coefficient cor resp ond s to t he first lag of t he depen-dent variable of the equation, or zero otherwise), there remain eight unknownparameters. (10) describes an evolving coefficient model. The likelihood for (9)and (10) was derived by Schweppe (1965) and can be written down with the aidof the Kalman filtering equ atio ns. Two of th e 7r parameters were then eliminatedby maximizing this likelihood conditional upon the fixed values of the others.One might well ask what the rationale for (10) is; Doan et al. claim (p. 6 ) :

What we do thus has antecedents in the literature on shrinkageestimation and its Bayesian interpretation (for example, the worksby . . .Shiller (1973). . . .I would dispute this connection. In the Bayesian formulation of shrinkageestimators, shrinkage occurs only in a finite sample, since the prior information


16/22

18 P A G A Nis dominated by the sample information as the sample size grows, i.e. changesin (say) the prior variance have a negligible effect upon the posterior distributionin large samples. This is not true for (10); changes in 7r always have an effect up onthe likelihood, since the variance of the innovations is always a function of the7r (see equa tion (10) of Doan et a / . ) .Reference t o Shillers work seems even m oremisleading. Shiller allows the coefficients to be random across the lag distribu-tions, not across time, i.e. he would have

and not

Thus, as (10) is a model f or coefficient evolution, an d not the im position of priorinformation, it is hard to see why this procedure is any less objectionable thanthat followed by the Cowles Commission; Malinvauds (1984) reaction t o th e ideais easy to sympathize with.Step (iv)As step (iv) has been the subject of a number of excellent critiques, particularlyCooley and LeRoy (1985), little will be said abo ut it. Th ere are two m ajo r objec-tions. First, the move from innovations to orthogonal innovations raises ques-tions. With the exception of the first variable in z I , the orthogonal innovationsare hard t o give any sensible meaning to; resort is frequently ma de to expressionssuch as that part of the innovation in money not correlated with the innovationsin other variables. In many ways the difficulty is akin to that in factor analysis;the mathematics is clear but the economics is not . U nfortunately, many users ofthe technique tend to blur the two concepts in discussion e.g. in Litterman andWeiss (1985) the orthogonalized soubriquet is dropped.A second query arises over the use of the orthogonalized innovations represen-tatio n. As Cooley and LeRoy (1985) point ou t, t o ascribe any m eaning t o impulseresponses for these innovations, it is necessary that the latter be treated asexogenous variables, and that requires the imposition of prior restrictions uponthe causal structure of the system in exactly the sam e fashion as was don e by theCowles Commission. The strong claims the methodology makes to being free ofprior information therefore seem to be largely illusory.As an aid to understanding the issues raised abo ve it may help to return to (1 )and the question of the response of m oney t o interest rate variations. Sims wouldfirst choose a lag length and a set of variables t o fo rm th e VAR . A minimal sub-set would be the variables m f ,p l , if and y f n ( l ) , but because one is attemptingto capture economy-wide interactions rather than just a money demand relation,extra variables that may need t o be included could be world activity, th e exchangera te, a nd fiscal policy variables. A lot of tho ught has to go into this choice. M ak-ing the set too small can seriously bias the answers, whereas making it too largerenders the method intractable unless other restrictions are imposed upon the


17/22

THREE ECONOMETRIC METHODOLOGIES 19VAR coefficients as in step (iii). Once the latter strategy is followed the clean-skin appeal of VARs begins to dissipate.Gra nte d th at steps (i)-(iii) have provided a satisfacto ry VAR representation form, s in (7) , it is then inverted to give the innovations representation (8) thatexpresses mr as a linear function of the innovations in the interest rate e; , randthe other variables in the VAR: e,, f , em, r ey, etc. The equa tion correspondingto mr in (8) would be of the form

m, i o , , n m em , r + I jo,mpep,r+ &,m;e i , t+ &,myey , r+ . . . . .and the response of mr to a unit innovation in the interest rate would be U O , ~ ; .This is to be contrasted w ith th e response of mr o a unit innovation in the interestra te provided by (1)--douo,;;-obtained by replacing if in (1) by if = CrO,;;e;,r+ . . . . . (the interest rate equation in (8)). Therefore different answers to thequestion of the response of mr to variations in if would be obtained frommethodologies concentrating upon (1) alone fro m those that in corpo rate systemresponses; in ( 1 ) the response is estimated by holding prices and incom e con stan t,whereas Sims seeks the effects on the quantity of money without such cet. parassumptions. T o some extent the methodologies a re not competitive, as they fre-quently seek to answer different questions.Sims aims to analyse a much broader set of issues tha n H endry or Leamer nor-mally do, but there are difficulties commensurate with this breadth. Making theset of variables to appear in the VAR large enough is one of these, and his fourthstep illustrates another. To speak of the response of mt to a unit innovation inthe interest rate it must be possible to carry out that experiment without disturb-ing current prices, incomes, etc. But that means the innovations e; , , must beuncorrelated with all the other innovations. When they are not, Sirns invokesartificial constructs, the orthogonal innovations, vi , v,, v,,,~, etc. These arelinear combinations of e ; , f ,ep, , , ey,fdesigned to be orthogonal to ?ne anothe r,and hence capable of being varied independen tly of each o the r. Ju st like principalcomponents, it is uncertain what meaning should be attached to these entities,leading to the controversy recounted above in discussion of step (iv).4. Summing upOu r review of th e methodologies now being com plete, it is time to sum up. Ignor-ing the criticisms of details that have been offered, how effective are themethodologies in meeting the three criteria o f goodness listed a t th e beginningof this essay, namely the provision of general research tools, the codification andtransmission of principles, and the reporting of results?None of the methodologies claims to be completely general. Sims explicitlydeals only with time series, while man y o f Hendrys concerns a re specific to suchseries as well. Learners techniques a re heavily based u po n the OLS estimator. Allhave the common deficiency of a failure to address explicitly the burgeoningfield of microeconom etrics. Whilst it is tru e th at the philosophies underlying H en -drys and Learners work transfers (see, for example, Cameron and Trivedi


18/22

20 P AG AN

(1985)), the actual techniques employed would need extensive modification, par-ticularly in light of the very large da ta sets tha t m ake traditional model selectionmethods inappropriate. Th ere is clearly a lo t to be done before an y of the threemethodologies provides a complete set of techniques for data analysis.Part of the objective of this paper has been to try to set out the general prin-ciples of each methodology, so as to assist in the communication and teachingroles. But this was don e at a high level of abs trac tion . When it comes to ap plica-tion m any questions arise which currently seem t o be resolved only by sitting atthe feet of the master. He ndr y, fo r example, is very vague abo ut how he managest o simplify his models, so little is learnt a bo ut ho w this is t o be don e by a readingof his articles. Learner recommends formulating multi-dimensional priors, butprovides little practical guidance o n how (say) the covariance matrices featuringin them are to be selected. Sims methodology seems clearest when it is appliedto the big issues of macroeconomics such as the neutrality of money, butaltogether vaguer when the question is of the m uch m ore prosaic kind such as theimpact of a quo ta upon import dem and. N o dou bt Sims would be able to handlesuch queries, but the personal ingenuity required seems a stumbling block t o thetransmission of knowledge.What a bou t reporting? Hendrys methodology seems to provide useful infor-mation in a concise form, although it is sometimes possible to be overwhelmedwith the detail on the statistics presented when judging the adequacy of a model.Perha ps this just reflects a lack of familiarity and a n early stage in learning abo utwhat a re the most useful tests. Learners extreme bo und s ar e easy t o understand;however, the extensions in which prior variances are restricted become muchharder t o interpret. T o my mind, it is Sims methodo logy which is the w orst whenit comes to the reporting role, w ith pages of g rap hs and impulse responses beingprovided. Whether this reflects a transition stage, or the problems mentionedpreviously abo ut ste p (iv), is still unclear, but a mo re consise method of rep ortingdoes seem to be needed.Granted that no methodology has managed to obtain a perfect score, whathave we learnt from all of this debate? First, a substantial clarification of the pro-cedures of model selection and auxiliary concepts such as exogeneity . Second,a pronounced recognition of the limits of modelling. Any reading of (say)Marschak (1953) makes it evident that the Cowles Commission researchers werenot deficient in this respect (doubters might no te the am ou nt of space M arschakdenotes to discussing the Lucas critique), but somehow it got lost in theeuphoria of th e 1960s. Th e much m ore critical attitude tow ards econometrics thatprevails today is generally a good thing, although there is a danger that theemergence of differing methodologies will be interpreted as a tacit admission ofa complete failure of econometrics, rather than as constructive attempts to im-prove it.What about the future? Constructing systematic theologies for econometricscan well stifle creativity, an d som e evidence of this has already become a ppa ren t.Few would deny tha t in the hands of the masters the methodologies perform im-pressively, but in th e han ds of the ir disciples it is all muc h less convincing. It will


19/22

THREE ECONOMETRIC METHODOLOGIES 21be important to rid econometrics of the black box mentality that always besetsi t . A po or m odelling strategy is unlikely to give useful results, but a good on e can-not rescue a project by rigidly following any m ethodology if it was badly conceiv-ed from the very beginning. What I see as needed is a greater integration of thedifferent methodologies. Although it is convenient to dr aw de m arc atio n lines bet-ween them in discussion, this should not blind a researcher to the fact that eachmethodology can provide insights that the others lack. Extreme bounds analysisis an imp ortant adjunct t o Hendrys methodology if large numbers of parametershave been om itted in any simplification. Exam ining the residuals fo r model defi-ciencies should be as automatic in Learners and Sims methodologies as it is inHendrys. Checking if the restrictions imposed by a model selected by Hendrysor Learners methodologies u pon th e VAR parameters ar e compa tible with thedat a should be part of an y analysis involving time series. O ur d ata are such th atwe cannot ignore the fact that the information therein may need to be extractedby a wide range of techniques borrowed from many different approaches.

NotesI Much of this paper was presented in the symposium on Econometric Methodology a tthe World Econometric Congress at Boston in August 1985. I am grateful to Ed Learnerfor his extensive comments upon the paper.

As is well known the importance of collinearity is a function of the parameterizationused. Thus the data may be very informative about certain parameters e.g. long-runresponses, but not others e.g. dynamics. It is not useful (or valid) to claim it is uninfor-mative about everything. Problems emerge if a decision rule is employed based on keeping Type I errorsconstant-see Berks on (1938). As the test statistic is the produ ct of th e sam ple size by theproportional change in variance, even very small changes in the latter become largechanges in the criterion when the sample size is large. Decision rules such as those inRissanen (1983) and Schwartz (1978) overcome this, but it might be better to modelformally the underlying conflict between Type I and Type I 1 error as in Quandt (1980).Both papers treat only the case where the observations making u p the sample momentsare i.i .d., but it is clear that the analysis extends t o th e case where the ortho gonality rela-tions follow a martingale difference process. This covers most cases of interest ineconometrics. Note, however, that Tauch ens results require tha t the maintained mod el beestimated by maximum likelihood.Breusch (1985) has an elegant proof of this.This division shows that interpretations which see the differences between themethodologies as due to different attitudes to collinearity are incorrect. Bounds can bewide even i f collinearity is weak (4 small). We have not dealt with the question of what inferences about P I might be then drawnfrom B l . Unfortunately, i t is not uncommon in econometrics to see sharp conclusionsdrawn ab out the value of 01on the basis of a test of a sharp hypothesis such as P I = 1or zero (H all, 1978; Barro, 1977). All that c a c be concluded, how ever, is that a range ofpossible values tor PI are compatible with 01, and this range is frequently found byexamining kSD(Pl), where k is some selected constant. Traditionally, k was set bystipulating the Type I error to be sustained, but Don Andrews (1986) has recently sug-gested a way of incorporating power requirements into the determination of k .


20/22


21/22

THREE ECONOMETRIC METHODOLOGIES 23Hendry, D. F. and Richard, J . F. (1982) On the formulation of empirical models inHill, B. (1986) Some subjective Bayesian considerations in the selection of models.Kirby, M. G . (1981) An investigation of the specification an d stability of the A ustralianLearner, E . E . (1978) Specifrcation Searches. New York: Wiley.- 1983) Lets take the con out of econometrics. American Economic Review 73,- 1985) Sensitivity analyses would help. American Economic Review 75, 308-13.

~ (1986) A Bayesian analysis of the determinants of inflation. In D. A . Belsley andLearner, E. E. and Leonard, H. (1983) Reporting the fragility of regression estimates.Litterman, R. and Weiss, L. (1985) Money, real interest rates and output: a re-Liu, T. C . (1960) Underidentification, structural estimation, and forecasting.McAleer, M., Pag an, A. R. and Volker, P. A . (1983) Straw-man econometrics. W orking- 1985) Wh at will take the con out of econometrics? American Economic Review 75,Malinvaud, E. (1984) Comment to forecasting a nd conditional projection using realisticMallows, C. L. (19?3) Some comments on C,. Technometrics 15 , 661-75.Marschak, J . (1953) Economic measurements for policy and prediction. In W . C . Ho o dand T. C. Koopmans (eds) Studies in Econometric M ethod (C ow les Comm issionResearch Monograph N o. 1 4 ). pp. 1-26. New Haven: Yale University Press.Mill, J . S. (1967) On the definition of political economy an d o n the method of investiga-tion proper to i t . Collected Works Vol. 4 . Toronto: Universi ty of Toronto Press.Mizon, G . E . (1977) Model selection procedures. In M. J . Art is and A . R. Nobay (eds)Studies in Modern Economic Analysis Oxford: Basil Blackwell.Nelson, C. R. an d Plosser, C. I. (1982) Tren ds and rand om walks in macroeconomic t imeseries: some evidence and implications. Journal of Monetary Economics 10, 139-62.Newey, W. (1985) Maximum likelihood specification testing and conditional momenttests. Econometrica 5 3 , 1047-70.Pagan, A. R. (1978) Detecting autocorrelation after Bayesian regression. CORE Discus-sion Paper No. 7825.- 1984) Mo del evaluation by variable addition. In D. F. Hendry and K . F. Wallis (eds)Econometrics and Quantitative Economics Oxford: Basil Blackwell.Pagan, A. R. and Volker , P. A. (1981) The sho rt-run deman d for transactions balancesin Australia. Economica 48, 381-95.Quandt , R. E. (1980) Classical and Bayesian hypothesis testing: a compromise.Metroeconomica XXXII , 173-80.Rissanen, J . (1983) A universal prior for integers and estimation by minim um descriptionlength. Annals of Statistics 1 1 , 416-31.Salkever, D. S. (1976) The use of dum my variables to com pu te predictions, prediction er-rors and confidence intervals. Journal of Econometrics 4, 393-7.Sargan , J. D. (1964) Wages and prices in the United Kingdom: a study in econometricmethodology. In P. E. Har t , G . Mills and J . K. Whitaker (eds) Econometric Analysis

for National Economic Planning London: Butterworth.Schwarz, G. (1978) Estimating the dimension of a model. Annals of Statistics 6, 461-4.Schweppe, F. C . (1965) Evaluation of likelihood functions for Gau ssian signals. I .E .E.E.

dynamic econometrics. Journal of Econometrics 20, 3-33.Econometric Reviews 4, 191-246.aggregate wage equation. Economic Record 57, 35-46.

31-44.

E. Kuh (eds) Model Reliability Cam bridg e, Mass. : M.I.T. Press.Review of Economics and Statistics 65, 306-17.interpretation of postwar U.S. data. Econometrica 53, 129-56.Econometrica 28, 855-65.Paper in Econometrics No. 097. Australian National University.293-307.prior distributions. Econometric Reviews 3 , 113-18.

Transactions on Information Theory 1 1 , 61-70.


22/22

24 PAGANShiller, R. J . (1973) A distributed lag estimator derived from smoothness priors.Sims, C. A. (1980a) Macroeconomics and reality. Econometrica 48 , 1-47.- 1980b) Comparison of interwar and postwar cycles: monetarism reconsidered.Tauchen, G. 1985) Diagnostic testing and evaluation of maximum likelihood models.Veall, M . R . (1986) Inference s on th e deterre nt effect of capital punishment: b ootstrappingZellner, A . (1971) An Introduction to Bayesian Inference in Econometrics New York:

Econometrica 41, 775-88.

American Economic Review 70( 1980), 250-7.Journal of Econometrics 30, 415-43.the process of model selection. Mimeo, University of Western Ontario.Wiley.

Eco No Metric Methodologies JES

Documents

Transcript of Eco No Metric Methodologies JES