1983-06-31 Problems in Measuring the Quality of Investment Information

8/7/2019 1983-06-31 Problems in Measuring the Quality of Investment Information

1/8

by T. Daniel Coggin and John E. Hunter

Pruhiemsi n MUeasuringt h e lualilym l InveulmentInlurmaliun:

T h e P e r i l s i it h e InhnrmalimnC o e f f H e le n !

Many investment firms and services use the "informationcoefficient"-the correla-tion between predictedand actualreturns-to assess the performanceof analystsorvaluationmodels. Whatmany may fail to realize is that any IC based on fewer thanseveralthousand stocks will suffer from significantsampling error.Forexample,an ICbasedon predictionsforallthe stocks on the New YorkStockExchangeis unlikely to be higher than 0.10. But an IC based on 30 of those stockswill varyrandomlyby plus or minus 0.35! Thus samplingerrorin the 30-stock IC islikely to createan illusion of massivevariationover time, or over analysts or over in-vestment models.Analysisof IC dataon the WellsFargoand Value Line valuationmodels revealsthat the largedifferencesin the models' ICs over time are due solely to samplinger-ror.Analysis of analysts'ICsreveals,similarly,thatthe differencesbetween analysts'ICs can be attributedentirelyto samplingerror.If there are differencesin analysts'performances,they must be measuredin some mannerother than comparingICs.

S EVERALrecentarticleshave recommend-ed that the quality of researchinformationin investmentanalysis be measuredby theinformationcoefficient(IC).' The IC is the (Pear-son) correlationbetween the returnon a commonstock predictedby a valuation model or analystand the actualreturn. Like any correlation,theICis subject to samplingerror-i.e., chance fluc-tuations in the value of the statistic due to ran-domness in the selection of the numbers used in

the analysis. The extent of randomerrordependson the number of stocks correlated.In the cur-rent applicationsof the IC, the number of stocksis usually small, hence the amount of samplingerroris large. Samplingerrorcanseriouslydistortthe conclusions drawn from ICs.This article outlines meta-analysis, a newmethodology for assessing the degree to whichthe difference in correlations cumulated acrossstudiesor time periods is due to samplingerror.2Meta-analysisis used to compare the ICs of theWells Fargo and Value Line valuation methodsand the ICs of professional investment analysts.In each case, all the observed variation in ICs isdue to sampling error.Sampling Errorand Meta-AnalysisTheideal evaluationof any forecastingprocedurewould be to test it against a large population of

1. Footnotesappear at end of article.T. Daniel Cogginis Managerof theInvestmentSystemsDepartmentat CenterreTrustCompanyof St. Louis.JohnHunteris ProfessorofIndustrialPsychologyat MichiganStateUniversity.TheauthorsthankStevenEinhorn,JamesFarrellandStephenLevkofffor theirhelpfulcomments.An earlierversionof thisarticlewaspresentedatthe31stAnnual Meeting of the Midwest FinanceAssociation,Chicago,1982.FINANCIAL ANALYSTS JOURNAL / MAY-JUNE 1983 O 25


2/8

stocks, such as allstocks tradedon the New YorkStock Exchange. This is not usually feasible,however, and mostempiricalresearchis confinedto a sample of stocks from a larger population.It has long been known that the correlationbe-tween forecast returns and actual returnsfor asample of stocks (denoted "r" in statistics)willdiffer from the correlationbetween forecast re-turns and actualreturnsfor the entire populationof stocks (denoted ",p" in statistics). The dif-ference is called "sampling error."The sampling erroris usually random in signand specificmagnitude. One can, however, com-pute its probabilityrangefrom known formulas.The results of many such computations indicatethat the size of the sampling errorwill dependlargely on the number of stocks in the sample:The larger the sample of stocks, the smaller thetypical sampling error.(Theformulasdevelopedin this article use the sample correlationcoeffi-cient ratherthan Fisher'sz transformation.Thereasons for this are given in the appendix.)Suppose we wish to use the ICto evaluatein-vestmentanalystsatAcme InvestmentSorcerers,Inc. An examination of Clara Clairvoyant'spredictionsfor every stock on the NYSEmighttell us that her IC is p =0.20. But an examina-tion of Clara'spredictionsforonly 30 stocks willgive us an observedICthatdepartsfrom the truevalue of p =0.20 by some unknown randomamount.The value of Clara's IC will have a normaldistributionwith a mean of p = 0.20 and a stan-dard deviation of:

- p2 1-0.20 0.18 (1)iN/-V i30-1where N is the number of stocks in the sample.This tells us that, 16 per cent of the time, Clara'sobserved IC will lie above 0.38 (=0.20 + 0.18),and 5 per cent of the time it will lie above 0.50(=0.20 + 1.64(0.18)).Another 16percent of thetime, her observed IC will lie below 0.02 (= 0.20- 0.18),and 5 percent of the timeit will liebelow-0.10 (=0.20 - 1.64(0.18)).Thus 90 per cent ofthe time, Clara's observed IC will fall in the in-terval -0.10 to +0.50, but there is a 10per centchance of even greatererror.Suppose we calculate Clara's IC for eachquarterof 1982,arrivingatvalues of 0.44, - 0.04,0.28and 0.12. At first glance, these values mightappearto be radicallydifferentfrom one another.We might ask why Claradid so well in the firstquarter,but so poorly in the second quarter.Yet

meta-analysiswould show that the varianceinthesevaluesis no largerthan would be expected,given the extent of the sampling error. In fact,the standarddeviationof these fourvalues is ex-actly the value of the samplingerror(0.18)com-puted from Equation(1).The same conclusion would apply if we weredealing with the performanceof four analysts,each of whom evaluated 30 stocks. Even if thepopulationICwere p = 0.20forallanalysts(i.e.,no actualdifferencein jobperformancebetweensorcerers),the observed ICs would varyat leastas faras from -0.10 to +0.50 for samples of 30stocks. There would appear to be huge dif-ferences between analysts, but this observedvariationwould actually be due solely to sam-pling error, hence would be of no real value incomparing the analysts' performances.Meta-analysissuggests thatthere is no reasonto believe there is any real variationin the ICsfor the different analysts. This result would beeven strongerif the meta-analysishad covereda largernumber of analysts. Forexample, if thevariationin the ICsfor 30 analystswere exactlyequal to the value predicted by samplingerror,then there is little likelihoodthatany other con-clusion would be possible.

Estimating Real VarianceThere is no way to estimate the samplingerrorin a sample IC fromthe informationin thatsam-ple. Thesamplingerrorin a sequenceof ICscan,however, be estimatedusing meta-analysis.Theprocedure is this:

(1) computethe meanandvarianceofobservedICs,(2) computethe variancein ICs that is attributableto samplingerror,(3) test the observed varianceagainst the chancevalue to see if there is any real variance inpopulation values,(4) subtract the sampling errorvariancefrom theobservedvarianceto produceanestimate of thevariancein population ICs, and(5) takethe squarerootof the correctedvariancetoestimate the standarddeviationof populationICs.

If the ICs in a sequence arebased on differingnumbers of stocks, they should be weighted ac-cordingly.Theweighted mean and the weightedvarianceof a sequence of observed ICsaregivenby:-C S(N1IC1) (2)

FINANCIAL ANALYSTS JOURNAL / MAY-JUNE 1983 O 27


3/8

and2 [NX(IC-IC)2] (3)

where Ni is the number stocks on which the ithIC is based. The variance due to sampling erroris approximately:

S2 =K(1 -1C2)2 (4)where K is the number of ICs and ENi is thetotal numberof stocks underlying all the coeffi-cients. The realvariance of population ICs isestimated by subtracting the sampling errorvariance from the observed variance:

S, = S2c - Se (5)Totestthehypothesis thatthereis no realvaria-tion in ICs, we can use the following statistic:XK K(sIC) (6)

Se

which has a chi-square distributionwith K-1degreesof freedomif the null hypothesisis true.This statisticis used as a formal test of no varia-tion, althoughit has so much power that it willrejectthe null hypothesis (of no variation)evenif onlytrivialamountsof variationexist.If the chi-squarevalue is not significant,there is trulynovariationacross studies; but if it is significant,then the variation may still be negligible inmagnitude.Considernow the IC values used in the aboveillustrations-O0.44,-0.04, 0.28 and0.12.Supposethat the number of stocksin each ICwas 32, 31,30and29. Then the weightedmean and varianceof the sequence would be:

IC 32(0.44)+31(- 0.04)+30(0.28)+29(0.12)32+31+30+2924.72= 0.2026122

2 32(0.44-0.2026)2+31(- 0.04- 0.2026)2+...Sic 32+31+30+294.0053- 1?223 = 0.0328

The sampling error variance would be:2 4(1-0.20262) - 00301122

Thus our best estimateof the true variancein ICswould be the difference:p= - S = 0. 0328-0.0301 = 0.0027

Forthese four values, the observed mean IC is0.20,andthe observedstandarddeviationis 0.18.Meta-analysissuggests thatthe true mean is 0.20and the true standard deviation 0.05 (i.e.,\10.0027).Applyingthe significancetest to see ifthe observedvariation is greaterthan samplingerroralone would suggest, we have:X= K(S2) - 4(0.0328) - 4.36S2 0.0301

which, for three degrees of freedom, is notstatisticallysignificantat the 5 percent level. Thuseven the correctedvarianceof 0.0027may be at-tributableto sampling error.Thechi-squaretestserves as an importantreminder that even thecumulated statistics suffer from samplingerror,the total sample size being only 122.3Variation Between Models:Wells FargoVersus Value LineTableI presents the results of a study by Am-bachtsheer comparing ICs for two valuationmodels over six independent six-monthperiodsfrom1973 to 1976.4As Ambachtsheernotes, theWellsFargo model is essentiallya dividend dis-count model that incorporates the SecurityMarket Line concept; the Value Line model isbased on historicalprice and earningsrelation-ships and incorporatesshort-termearnings sur-prisefactors.5Ambachtsheercontends thatthesedatademonstrate"considerableperiod-to-periodvariability in the ICs for both valuationmethodologies."TableIIpresents the results of applying Equa-tions (3), (4) and (5) to Ambachtsheer's data. Inboth cases, the observedvariance(SIC)is greaterthan the error variance (s2), which would sug-gest that some of the variabilityin ICs is real.Table I Wells Fargo and Value LineModel ICs

I~ICPeriod N WellsFargo ValueLine

911973-311974 200 0.12 0.173/1974-911974 200 0.16 0.049/1974-311975 200 0.01 -0.093/1975-9/1975 200 0.13 0.169/1975-3/1976 200 0.08 0.11311976-911976 200 0.31 0.01Weighted Average 0.135 0.067

FINANCIALANALYSTSJOURNAL/ MAY-JUNE1983 0 28


4/8

Table II SamplingErrorin Model ICsWells Fargo Value Line

S2C 0.0083 0.0083Se 0.0048 0.0049

SQ 0.0035 0.0034

However, the magnitudeof the real variance(SQ)may itself be due to chance.Applying Equation(6) to test the significanceof the real variances yields values of 10.37 forWells Fargoand 10.16 for Value Line (with fivedegrees of freedom). Neither is significantat the5 per cent level. Thus the chi-squaretest showsthat the observed variancesare not significantlygreater than the error variances. This analysisrefutes Ambachtsheer's claim of significantperiod-to-periodvariability.Our analysis of the data for each modelseparatelyshows that the variationover time isno greaterthan would be producedby samplingerror.If the variationover time is anillusion,thenthe true IC values for the two models are bestestimatedby the averagevalues. Thatis, ourbestestimates of the six successive values for WellsFargoare not the observed 0.12, 0.16, etc., butrather0.135, 0.135, etc.The Confidence Interval for the ICThe mean IC value for Wells Fargo (0.135)isjustovertwice as greatas thevaluefor ValueLine

(0.067).Is thereanypossibilitythisdifferencecanbe attributedsolelyto samplingerror?Eachmeanis based on a total sample size of only 1,200, sothere is room for sampling error.Sampling errorin a single value can only bedescribed as a probabilitydistribution. Thus,associatedwith a given IC,there is an errorbandor confidence interval that will contain thepopulation IC with a given probability.If theobservedvalue of the information coefficient isIC, then the width of the 95 per cent confidenceband is:w 1.96(1-IC2) (7)

where N is the number of stocks in the IC. Theendpoints of the confidence intervalare IC-wandIC+w, andthe populationIC will fallwithinthose endpoints 95 per cent of the time.The 95 per cent confidence intervals for WellsFargoand Value Line can be obtained by using

1,200 as the sample size. They are0.079to 0.191for WellsFargoand 0.011 to 0.123 for ValueLine.Because the confidence intervals do overlap, apossibility exists that the populationvalues arethe same for the two models. About four timesas much data would be needed to establishthedifferencebetween these models if the true dif-ference is no largerthan that observed.Theconfidence intervalis an unbiasedway ofdescribing sampling error. The alternativemethod of handling sampling error is thesignificancetest, which asks if the observed ICis significantlydifferentfrom 0.0. The appendixshows that the significancetest is contained inthe confidence interval and represents a biasedtreatment of sampling errorif used alone.

Combining PredictionsAmbachtsheer and Farrell have argued thatpredictive abilitycanbe improvedby combiningthe predictionsmadeby two investmentmodels.6Theirbasic argumentis as follows. Suppose twomodels areavailable,both workequallywell (i.e.,IC1 = IC2), and the predictions of the twomodels areuncorrelated.Accordingto multipleregressiontheory, the optimumpredictionwouldbe obtained by averaging the predictions madeby the two models. TheICfor the averagepredic-tion (equal-weighted) would be -/2(IC1or 2),meaningthatthe averaged predictionshould do41 per cent better than the predictions of eithermodel taken separately.This procedureworks if one assumes there isno correlationbetweenthe predictionsof the twomodels. Infact, however, most valuationmodelsrely on information derived from the samesources (e.g., the same industry reports andeconomicstatistics).One would thereforeexpecta substantialcorrelationbetween the predictedvalues of two investment models, hence onlymodest improvementfromaveragingthe predic-tions.Certainly,the recentemergenceof the divi-dend discountmodel as, perhaps, the dominantmodel of investment value does not weaken thisexpectation.7

Apparent Variation in ICs across AnalystsA number of firms use ICs to evaluate the per-formanceof theirinvestmentanalysts.At the endof each quarter,for example, they compute theIC for the predictions of each analyst and usethese coefficientsto ranktheiranalysts.Onecom-mercial service even calculates daily ICs foranalysts.FINANCIAL ANALYSTS JOURNAL / MAY-JUNE 1983 O 29


5/8

Table III Mean QuarterlyAnalysts' ICs----?--- MeanIC ?Analyst 1979 (N) 1980 (N) 1981 (N)

1 0.18 (72) -0.08 (76) 0.08 (101)2 0.12 (65) 0.02 (92) 0.11 (86)3 -0.06 (66) 0.23 (73) -0.04 (99)4 -0.03 (128) 0.16 (130) 0.17 (118)5 0.08 (32) 0.18 (44) 0.24 (37)6 0.12 (79) -0.06 (78) 0.27 (80)7* - 0.23 (88) 0.06 (108)8* - 0.17 (108) 0.16 (112)9* - -0.02 (54) -0.04 (60)WeightedAverage 0.06 (74) 0.10 (83) 0.11 (89)

*Analysts7, 8 and 9 were not employedfor the full year 1979.

Table IV Real Variancein Analysts' ICsReal Variance Chi-Square Degrees SignificantYear (s2) (X) of Freedom (cx=0.05)

1979 -0.0055 3.55 5 NO1980 0.0013 9.98 8 NO1981 -0.0019 7.44 8 NO

We analyzed the quarterlyICs of the invest-ment analystsat a large regionaltrust investmentdepartment over a three-year period, 1979 to1981. There were six analysts in 1979 and ninein 1980and 1981. Each analyst covered two tothree industries, following, on average,18 com-panies in 1979, 21 in 1980 and 22 in 1981.Eachanalystpredictedtotalreturnon the basisof a three-phase dividend discount model, pro-viding the model with near-term forecasts andlong-term growth rates for earnings anddividends. TableIIIpresents the mean quarterlyICfor each analystfor each year. The quarterlyICs wereaveragedovera yearto reducesamplingerror;thatis, the samplingerrorin the one-yearaverageof quarterlyICs based on 20 stocks is ap-proximatelythe same as an IC basedon 80 stocks.Averaging over a year is also consistent withmanagement's annual review of the analysts'performance.Theweighted averagequarterlyICfor all analystsfor each year and the 95 percentconfidence interval are as follows: 1979 = 0.06(-0.03, 0.15), 1980 = 0.10 (0.03, 0.17), 1981 =0.11 (0.04, 0.18).TableIVpresents the results of meta-analysisof the data in TableIII.Thereis no significantrealvariation in ICs across analysts for any of thethree years. Indeed, the chi-squaretest was un-necessary for 1979 and 1981, since the error

variance in these years was larger than theobserved variance.8To make distinctions be-tween these analystson the basis of these ICsisequivalentto throwing dice.A Replication of the Analysts'IC StudyAfter hearing of our results, an independentsource made availableto use two samples of itsIC data. Meta-analysis showed that, for bothsamples,the variancein ICs was due to samplingerror;in each case, there was no real differencein analysts' ICs. The first sample, covering 185analysts, had an observed variance(SIC) of0.00107, but with an error variance (s2) of0.00228. The second sample, covering 168analysts,had an observedvarianceof 0.00062andan errorvarianceof 0.00075.Both these samples had for the two time

periods a mean IC of 0.0, which differssharply(bymorethan sampling error)from ouranalysts'three-yearaverageof about 0.10. Otherresearchhas indicateda mean analysts'ICof about0.12.This suggests to us that mean analysts' ICs maybe period-specific,havingto dowithstock marketcycles.9Ourfindingsmake moot a worrisomequestionconcerningthe IC-the potentialnoncomparabili-ty of ICs based on different stock groups. Hadthere been significant variation across theanalysts' ICs for a particularperiod, we wouldhave had to address the question of the com-parability of ICs across industries or stockgroups-that is, whether differencesmight havebeen due to differences in stock groups, ratherthandifferencesin analysts' ability.Thefact thatno such differenceswere foundsuggest thatthereis simply no empiricalvalidityto the use of ICsfor the evaluation of analysts' relative per-formance.

ICs and Portfolio PerformanceThe theoretical relationship between themagnitudeof the IC and portfolio performancehas been derived in Treynor and Black andFerguson.10 Several empirical studies havediscussed the concept of a minimum IC levelnecessaryto indicatethatactiveportfoliomanage-ment produces a return greaterthan that of amarketindex." Unfortunately,these studies toohave failed to address properly the problem ofthe amount of sampling errorpresent in the sug-gested minimum IC level. Failureto estimatethedegree of sampling error can lead to highlyFINANCIAL ANALYSTS JOURNAL / MAY-JUNE 1983 0 30


6/8

suspect and possibly misleading IC guidelines forsuccessful active management. EAppendix

Distribution of the Sample CorrelationSome readersmaybe surprisedthatmeta-analysisdoesnot use the Fisherz transformationof the correlationcoefficient. Actually, early work by Schmidt andHunter did use the Fisherz transformation.12But laterwork discovered that the Fisherz transformationisbiased and that the analysisof the simple correlationcoefficient is both more understandableand morestatisticallyefficient.There is widespread misunderstanding about thepurpose of the Fisher transformation.Many believethat the transformation is intended to make thedistributionof the sample correlationmore normalinshape. Butexcept for extremecases, the distributionof r is already normal. Furthermore,the Fisher ztransformationwas derived for a very differentpur-pose. The purposeof the Fisherz transformationwasto develop a statistic whose standard error was in-dependent of the population value.The standard error of r is:

Ue= (1 - p2)/ N-i,which is known exactlyonly if the population correla-tion p is known. Thusif r2is used to estimatep2, thenthere is a small errorin the confidence interval cor-responding to the sampling errorin r2. On the otherhand, there is no errorin the standarderrorused toform the confidence interval for Fisher's z. Thetransformedvalue z has a known standarderrorof1/ N-3. The problemis that no one reallywants toknow the confidenceintervalfor the populationvalueof z. Therefore the confidence interval for z isbacktransformedto createa confidenceintervalfor p.Alas,thisconfidenceintervalforp has anupwardbias.For the purposeof creatingconfidenceintervalsforsingle correlations, there is only a decimal dust dif-ference between the use of r and the use of z in mostcircumstances(i.e., p < 0.80 or N > 20). The two willbe numericallyidenticalfor the tiny correlationsthatoccur in the present researchusing the informationcoefficient.Formeta-analysis,the situationis different.Averag-ing across studies eliminatesthe effectof samplinger-ror. The bias of Fisher's z then becomes a systematicoverestimate of the population correlation.Schmidt,Gast-RosenbergandHunterdiscoveredthis in a studyon computerprogrammers.13 The estimated popula-tion correlationusing the Fisherz transformationwasabout0.90, while the estimatedpopulationcorrelationusing samplecorrelationswas about0.80. Simulationandseries approximationsshowed that it was Fisher'sz that was wrong.It is ironicthat the standarderrorfor z of 1 I/ - 3

is actually only an approximation. The conditionsunder which the approximationis accurateareexact-ly the same conditions under which the sample cor-relation is normally distributed (such as p < 0.80 orN > 20). Thus for meta-analyticpurposes, it is neverappropriate to use the Fisher z transformation,although the error is not large unless the averagepopulation correlationis greaterthan 0.50.Significance Testsversus Confidence IntervalsMany people believe thatthe use of significancetestsguarantees an errorrateof 5 per cent or less. This isjust not true. Mathematicalstatisticianshave known(and have tried to teach practitioners)this for years,with the possibility of higher error rates shown indiscussions of the power of statisticaltests. The 5 percenterrorrateis a TypeI errorrate that is guaranteedonly if the null hypothesis (p = 0.0) is true. If the nullhypothesis is false, then the errorrate is a TypeIIer-ror rate that can go up to 95 per cent!Considerthe case of an "average" portfolio(i.e., IC= 0.15, N = 200).The one-tailedsignificancetest fora correlationcoefficientat the 5 percent level is N-1(r) > 1.64. In our case, this implies that the IC mustbe greaterthan 0.12 to be significantlydifferentfrom0.0. If the populationcorrelationis 0.15 and the sam-ple size is 200, then the mean of the sample correla-tions is 0.15, while the standard deviation is(1- p2)I /N-i = 0.07. Thus the probabilitythat theobservedcorrelationwillbe significantis theprobabilitythat the sample correlationwill be greater than 0.12when it has a mean of 0.15 and a standard deviationof 0.07:

P(IC?0.12) = p(IC-0.15 > 0.12-0.15)0.07 0.07= P(z> -0.43) = 0.67

Thatis, if all studies were done with a samplesize of200, then a population correlationof 0.15 would im-ply an errorrate of 33 per cent!What does this illustration tell us? Specifically, itshows thatif the populationICis 0.15 and the samplesize is 200, there will be positive informationin only67 per cent of the studies, even though the nullhypothesis(p = 0.0)is falsein allcases. Generally,andmoreimportantly,it shows thatthe t-statisticis not theappropriatemeasureof the amountof informationcon-tainedin an IC.Fortunately,therearetwo alternativesto the significancetest. At the level of review studies,there is meta-analysis,and at the level of the singlestudy, there is the confidence interval for the IC.If we consider our examplefromthe point of viewof the confidence interval for the population correla-tion froma singlestudy,we find0.01 ' p < 0.29, with95 per cent confidence. Confidence intervals aresuperior to the significance test for at least threereasons. First, the confidence interval is correctlycentered about the observed value rather than theFINANCIAL ANALYSTS JOURNAL t MAY-JUNE 1983 O 31


7/8

hypotheticalvalueof the null hypothesis. Second, theconfidence intervalis more appropriateforcomparinganalysts or models, since overlappingconfidencein-tervalsshow the extent of uncertaintysurroundingthetrue rankorderof the ICs. Thesignificancetest insteadcompareseach analyst separatelyto the hypotheticalvalue p = 0.0. Furthermore,the significancetest treatsall values above the minimum t-value the same (i.e.,"significant"),despite largepotentialdifferencesin themagnitude of the ICs.Consider, for example, a situation in which oneanalystis just barelyhigherthanthe t-testcutoffvalue,whereasa secondanalystis just barelybelow thecutoffvalue. The performanceof the analyst abovethe valueis judgedsignificantlybetterthanchance, whereas theperformanceof the analyst below the cutoff is notsignificantlybetterthanchance. Thus the firstanalystis judged better thanthe second analyst,even if theirobserved ICs are virtuallyidentical in value and farfrom significantlydifferentfromeach other. Thecon-fidence intervalsforthe two analysts,however,wouldalmostcompletelyoverlap,henceshow the trte uncer-tainty about the actualrankorder.Third, the confidenceintervalgives the researchera correctpictureof the degree of uncertaintyin smallsamples.Forexample, suppose that we wantthe con-fidence intervalfor the ICto definethe populationcor-relationto the firstdigit-i.e., to have a width of ? 0.05.Then it can be shown that the minimumsamplesizeis 1,538.Inorderfora samplesize of 1,000 to be suffi-cient, the populationcorrelationmust be > 0.44.Thusallsinglecorrelationstudieswithless than1,000stocksare dealing with a small sample size.As we have noted,confidenceintervalsgive thecor-rectpictureof the degreeof uncertaintyin correlationscomputedfrom smallsamples.Theonly way to reducethis uncertaintyis to construct large sample singlestudies or combine results across several small sam-ple studies. WithIC analysis, the most feasible solu-tion is to combineresults acrossstudies or test periodsand performmeta-analysis.Footnotes1. See, for example,KeithP. Ambachtsheer,"WhereAre All the Customers'Alphas?" TheJoumalofPortfolioManagement,Fall1977;AmbachtsheerandJames L. Farrell, Jr., "Can Active ManagementAdd Value?" Financial Analysts Journal,November/December 1979; and Ambachtsheer,"Evaluatingthe Quality of InvestmentInforma-tion," in Sumner N. Levine, ed., InvestmentManager'sHandbook(Homewood, IL: Dow Jones-Irwin, 1980).2. Foranintroductionto meta-analysis,see Gene V.Glass, BarryMcGawand MaryLee Smith,Meta-Analysis in Social Research(Beverly Hills: SagePublications, 1981). For a more advanced treat-ment,includinga discussionof samplingerrorand

errorof measurement,see JohnE. Hunter,FrankL. Schmidt and GreggB. Jackson,Meta-Analysis:CumulatingResearchFindingsAcrossStudies(Bever-ly Hills: Sage Publications, 1982).3. The formulasgiven here assume that the ICis thePearson correlationbetween predictedand actualvalues. However, some researchersconvertpre-dictedand actualvalues to ranksbeforecorrelating.This means their correlationis the Spearmanrankcorrelationcoefficient-Spearman's rho. (For adefinitionof Spearman'srho, see Sidney Siegel,NonparametricStatisticsfor the BehavioralScietnces(New York:McGraw-Hill,1956)orW.J. Conover,PracticalNonparametricStatistics,Second Edition(New York:JohnWiley, 1980).)Assumingbivariatenormality,Spearman'srho is less than Pearson'sr by an amountrangingfrom0.0, when both are0.0, up to about0.02, when the correlationsare inthe 0.70 to 0.80range.ThusSpearman'srhoslight-ly understatesthe strengthof therelationship.Thesamplingerroris alsoslightlydifferentforp * 0.0(see Maurice G. Kendall, "Rank and Product-MomentCorrelation,"Biometrika,Vol.36, 1949andM.G. Kendall, RankCorrelationMethods,FourthEdition(New York:HafnerPublishingCompany,1962)). Withinthe rangeof ICsin the investmentliterature,the sampling errorin Spearman's rhois the same as the sampling errorin Pearson'sr.Thereforethe formulasgiven in this articleworkequallywell forboth.If a studyis done on correla-tions greater than0.70, there will be a discrepan-cy between the meta-analysisformulasgiven hereand the correspondingformulas for Spearman'srho.4. Ambachtsheer,"Evaluatingthe Qualityof Invest-ment Information."5. The Wells Fargomodel is discussed in detail inWilliamF. Fouse, "Risk, Liquidityand CommonStockPrices,"FinancialAnalystsJournal,May/June1976and Fouse, "Riskand LiquidityRevisited,"FinancialAnalystsJoumal,January/February1977.TheValueLine modelis discussedin ArnoldBern-hard, Value LineMethodsof EvaluatingCommonStocks(New York:Arnold Bernhard&Company,1979).6. AmbachtsheerandFarrell,"CanActiveManage-ment Add Value?"7. Evidencefor our contention can be found in T.Daniel Coggin and JohnE. Hunter, "ForecastingAnnual Earnings Per Share: A Comparison ofThree Statistical Models to a Consensus ofAnalysts," The Journalof Business Forecasting,Winter 1982/83. This study found (using IBESanalysts'consensus earningsforecastdatafor1978and 1979) that the ideosyncraticcomponent inanalysts'forecasts is very small in proportiontothe systematic(consensus) errorcomponent.8. Our analysis of the reliabilityof the quarterlyICsfor the analysts(not presentedin this article)yields

FINANCIAL ANALYSTS JOURNAL / MAY-JUNE 1983 0 32


8/8

an average Spearman-Brown reliability coefficientof 0.0 for the three years, thus adding confirma-tion to the evidence that all the variability in theseICs is due to sampling error. For a discussion ofthe Spearman-Brown reliability coefficient, see J.P.Guilford and Benjamin Fruchter, FundamentalStatisticsin Psychologyand Education,Fifth Edition(New York: McGraw-Hill, 1973), Chapter 17.9. For a full description of the underlying model withan empirical test, see J.E. Hunter, T.D. Coggin andM.K. Conway, "Forecast Data and SystematicRisk" (Paper presented at the annual meeting ofthe Midwest Finance Association, St. Louis, 1983).10. See Jack L. Treynor and Fischer Black, "How toUse Security Analysis to Improve Portfolio Selec-

tion," The Journal of Business, January 1973 andRobert Ferguson, "Active Portfolio Management,"FinancialAnalystsJournal,May/June 1975.11. See Stewart D. Hodges and Richard A. Brealey,"Portfolio Selection in a Dynamic and UncertainWorld," Financial Analysts Journal, March/April1973 and Ambachtsheer, "Where Are All theCustomers' Alphas?"12. Frank L. Schmidt and John E. Hunter, "Develop-ment of a General Solution to the Problems ofValidity Generalization," Journal of AppliedPsychology, Vol. 62, 1977.13. F.L. Schmidt, I. Gast-Rosenberg and J.E. Hunter,"Validity Generalization for Computer Program-mers, " JournalofApplied Psychology, Vol. 65, 1980.

Booksconcluded from page 80bad. Existence of a bad system impliesthe existence of a superior one, and onewill prove as elusive as the other.Therefore, how-to books on investingare to be read with caution. And we arecautious.Mr. Bigus, an MIT-trained manage-ment engineer, faces the investingproblem squarely. He analyzes a longcatalogue of systems that investors usethat do not work. Then he presents hisown solution, low-risk investing-pureGraham and Dodd. His first require-ment is a very strong and conservativebalance sheet. Next he builds an invest-ment value for the stock based on anaverage price/earnings ratio and his-torical earnings. This investment valueis the sell price; the buy price is reachedby applying a severe haircut to the in-vestment value, based on company sizeand on the degree of past earnings fluc-tuation. That's it, one compact chapter.This reviewer automatically chal-lenged that chapter because of his feel-ing that reducing the downside riskmight reduce the upside potential com-mensurately; the method looked as ifthe investor would be out of the stockmarket much of the time. However, Mr.Bigus answered the important ques-tions in the appendix and elsewhere inthe text. He tested his resultsthroughout. It seems that he has re-duced fundamental risks through fun-damentals, ratherthan blind diversifica-tion. On the record, he seems to havetied systematic risk to individual stock

fundamentals rather than to overallmarket timing. Best of all, he has in-sulated his decision-making from a hostof investment methods that have notworked.Not that the book is perfect. Theorganization is somewhat confusing.

The tables are not always clearly lab-eled. The investment method, in ourview, is worth a try, but it is possiblethat its successful execution may proveto be too slow and too demanding formost would-be millionaires. -R.I. C.

Displayfinancialinformationatthe touch of a button.MULTIGRAPHApple? Il/lle VersionDevelopedbyRobertAbeyWhen you need to really understand data, look atit-with MULTIGRAPH for the Apple 11.Just selectthe type of graph you want-bar or line, pie chartor scatter diagram-input the values, and MULTI-GRAPH does the rest, displaying financial ornumerical data clearly, quickly, and conveniently.Change the values and the graphs change auto-matically. You can even change from one type ofgraph to another, and print at the touch of a button.One 5'/4" disk with documentation for the Apple IIor Apple lle with 48K memory. Available in May.$85.00Ask for Wiley Professional Software at your favoritecomputer store. Or use this coupon to order today-and examine any program free for 15 days.For faster service-CALLTOLL FREE:800-526-5368.In New Jersey, call collect: (201) 797-7809.OrderCode: 44-1022VISA, MasterCard, American Expressacceptedon phone orders.WILEYPROFESSIONALSOFTWAREa division of John Wiley & Sons, Inc.605 Third Avenue, New York, N.Y. 10158Apple?is a registeredtradernarkofAppleComputer,Inc.

[Please send me MULTIGRAPH(1-87460-j4) for a 15-DAY FREE EXAMINATION.| (Restrictedto continentalU.S. and Canada)O Payment enclosed plus sales tax.IWiley pays normal bookratepostage/han-I dling. We normally ship within 10 days. IfI shipment cannot be made within90 days, Ipaymentwillbe refunded.I Billme. O Billmy company. IPlease printNAMEI FIRM_I ADDRESSI CITYSTATE/ZIP__SIGN HERE ___ICREDITCARDORDERS:Tocharge your II order plus local sales tax and shipping/ lI handling,fillin the informationbelow. If Iyou are not satisfiedwiththe software, lreturnit within 15 daysfora full credittoI youraccount.E VISA O MASTERCARD lCARDNO- __ExpirationdateSignatureI Prices subjectto change without notice.Mailto: Wiley Professional SoftwareP.O. Box 092Somerset, N. J. 08873 i!4-1022

|Unlockingthe power of comrputinFINANCIAL ANALYSTS JOURNAL I MAY-JUNE 1983 C 33

1983-06-31 Problems in Measuring the Quality of Investment Information

Documents

Transcript of 1983-06-31 Problems in Measuring the Quality of Investment Information