Quality and Quantity, 14 (1980) 511-518 5 11 Elsevier Scientific Publishing Company, Amsterdam - Printed in The Netherlands

MODELLING AND EVALUATING THEORIES INVOLVING SEQUENCES: DESCRIPTION OF A FORMAL METHOD

STEPHEN TURNER

Department of Sociology, University of South Florida, St. Petersburg, FL, U.S.A.

Patrick Suppes' instructive foundational discussions of models in the empirical sciences (1960, 1962) argue for a highly general and formally simple concept o f the nature of fit between theory and data. He approaches the problem in a set-theoretic fashion by defining a "model of a theory" as a possible realization of the theory, a set-theoretic entity in which all o f the valid sentences of a theory are satisfied, and defines a "model of the data" as a possible realization of the data with a set-theoretic structure which corresponds to the set-theoretic struc- ture of the model of the theory to be tested. From a logical standpoint, the relation between a model of the data and the data itself is parallel to the relation between a model of the theory and the theory. A "the- o ry" typically contains elements, such as the physical theory concepts of force-fields and electron-spin, which are not included in models of the theory that are tested against models of data, for there is no data "measuring" force fields or electron-spin. Similarly, an experiment or survey contains much concrete experience, such as experience of exper- imental conditions, that does not end up on the researcher's computer cards and therefore is not and cannot be a part o f the test o f a theory. So the ultimate test of fit is between two highly "constructed" mathe- matical objects.

The mode of analysis to be presented here may be most readily understood in terms of Suppes' formulation. The method was designed in connection with a specific research problem, assessing relative ade- quacy of models of certain alternative theoretical formulations in the sociology o f science. The alternative formulations deal in one way or another with careers in science. None of the "theories" is highly struc- tured, indeed, they are not much more than collections of heuristic suggestions or "perspectives" on the phenomena. In this, and certain other crucial respects, these theories and the empirical problems on which they touch are characteristic o f sociological problems generally.

0033-5177/80/0000-0000/$02.25 �9 1980 Elsevier Scientific Publishing Company

512

The theories or "perspectives" do not cover precisely the same empiri- cal ground. Variables which figure prominently in one perspective may not be a concern of another perspective. And the hypothesized rela- tions between the variables within a given perspective or theory are often obscure. Often is it wrong or misleading to speak of the theories as hypothesizing "causal" relations between the variables, and rarely is it reasonable to make standard assumptions necessary for structural- equation models. Many times the variables are neither continuous nor scalable at the ratio level. Tabular analyses are often not easy to inter- pret when relatively large numbers of variables are concerned compari- sons between alternative perspectives are being made.

Sequence Models

Careers have a temporal dimension, so the events or properties which are acquired by a person in the course of a given career may be modelled as a semi-ordered set. For convenience, I will call these sets "sequences" in this discussion. These sets may be written as ordered tuples,

{a, b, c, d, e)

where a, b, c and so forth are events or properties acquired in the course of a career in the temporal order in which they were acquired or oc- curred. Ordered tuples are, as Suppes suggests, the basic form in which models of the data and models of a theory meet. This particular model of the data is a very simple one. A more complex model might involve such facts as the time between events. Data of any level of measure- ment may be expressed, i.e., modelled, in such tuples as, e.g., the data for the Cartesian coordinates used in ordinary correlational analysis may be expressed as ordered pairs.

A data set is a set of ordered tuples. In this application, each repre- sents an individual's career. The "theories" or perspectives which bear on the phenomena of scientific careers must be modelled in a way which allows their fit with data to be measured. Theories formulated in what philosophers of science call the "standard formalization" are statements about classes. Such theories would define certain ordered tuples as falling within or not falling within the class which the state- ments of the theory are about. One may formulate the perspectives or theories found in sociology in these terms only with some difficulty. Constructing a model of these theories as a suitable set-theoretic enti ty to be measured for fit with data modelled in "sequences" involves sim- ilar problems. The models must be stated with a precision which ordi-

513

narily cannot be warranted by the informal explanatory structure which is being modelled. So ordinarily the modeller must modify the original explanatory structure in the course of this reconstruction of the explanation in suitable set-theoretic terms. Part of the value of developing and introducing new techniques of modelling is that new techniques may prove to do less damage to the informal explanatory structure of the sociological theories that exist in a given research area than would be done if explanations are reformulated in terms of a small range of types of model. The greatest divergences between the struc- ture of models and the informal explanatory structure of sociological theories occur when sociological accounts with highly varied informal structures are restated exclusively as models of one particular type. For example, many sociological accounts can be stated as structural equa- tion models only by very substantial reconstructions. If the informal theories modelled using the techniques described here were restated as structural equation models, they would have had little resemblence to the informal theories, and therefore could not have been considered as a means of "testing" the informal explanations. In short, there is ordinarily a certain amount of non-rescmblence between model and informal explanations. But it is the modeller's responsibility to mini- mize it.

The mathematical form taken by a model of this sort is a partition. Each model partitions the possible sequences into "conforming" and "not conforming" semi-ordered sets. Each model, then, partitions both the set containing the actual, empirical, sequences and the set of possi- ble sequences. The "sequences" themselves are ordered in one of two ways. Since careers are temporally ordered, each tuple may be com- posed of temporally ordered events, or it may be ordered by having each place in the tuple represent one kind of fact about the career or it may be ordered in a combination of these ways. Consider the case where one is concerned with three facts about each scientific career, (a) prestige of Ph.D., (b) election to a scientific academy, (c) publica- tion of a heavily cited paper. A non-temporal expression of these facts about a career would require the designation of a place in the triple for each fact, if we order the facts as they are ordered in the sentence in which they are mentioned above. If "1" represents "yes" and "0" represents no, the triple {0, 1,0} would be read as "did not receive prestigious degree, did publish heavily cited paper, was not elected to scientific academy." To order these events temporally one would pro- vide a lower case letter for each event. If % " stands for "receives presti- gious degree," " b " stands for "publishes heavily cited paper," "c" stands for "elected to scientific academy," the triple {a, /), c} repre-

514

sents a career where the person received a prestigious Ph.D., then pub- lished a heavily cited paper, then was elected to a scientific academy. Information, in this case information about temporal order, is there- fore gained by stating the triple as a sequence. Some of the time the temporal order of an event may be theoretically irrelevant or the event is not plausibly treated as an event (e.g., the sex o f a scientist hardly is an "event"). Such facts may be given specific places in the triple, with the remaining places reserved for expressing in temporal order those events or facts whose temporal order is theoretically relevant. For rea- sons which will become clear below, it is also often necessary to stan- dardize sequences in a data set to length n. If one standardizes career sequences involving "a", "b ," and "c" as triples, one may fill in with a blank code (e.g., "z"). A scientist who published a heavily cited paper, but who did not receive a prestigious Ph.D. and was not elected to a scientific academy would be described by the triple {b, z, z}.

Hypothesis A is that during the generation in which a paradigm had been definitely established, which we will call the n generation, recogni- tion was dependent on prestigious connections, especially those that a scientist would expect to make as a student in a prestigious Ph.D. pro- gram. Hypothesis B holds that election to a scientific academy was invariably the consequence o f publication of a heavily cited paper for both generations. Hypothesis C holds that the path to scientific recog- nition was closed to members of the post paradigm establishment gen- eration who had not been trained in the prestigious Ph.D. programs in which the new "paradigm" in the field was fully established.

Prior to any consideration of actual data, it is necessary to construct a model for each of these hypotheses. The model for Hypothesis A would be a partition which included as conforming to the hypothesis all the quadruples of the set o f possible quadruples with these elements in the orderings which conform to the hypothesis. This partition would include all cases in the n generation who had been elected to scientific academies after publishing heavily cited papers, and, in the n genera- tion, all and only cases where persons had been elected to scientific academies who had received prestigious degrees. The informal explana- tion seems to allow for the possibility that graduates o f prestigious Ph.D. programs in the m generation and persons who published heavily cited papers in the n generation might not be recognized by election to a scientific academy, so these possibilities are not excluded by and there- fore "conform with" the model and are included in the partition. All other possible quadruples would be excluded from the partition. Par- titions for the other hypotheses may be constructed in a similar fash- ion. These constitute the "models of the theories."

515

These hypotheses could be evaluated by a data set containing qua- druples, such that each quadruple mixes temporally ordering and a place-ordering in this way: the first place in the quadruple represents membership in the post paradigm establishment scientific generation, and is symbolized m for membership, n for non-membership; the sec- ond place represents prestige of Ph.D., and is symbolized "a" for pres- tigious, "b" for non-prestigious; the third and fourth places represent temporal ordering of a set of elements including "g" for "elected to scientific academy" and "h" for "publication of first heavily cited arti- cle," with "o" for a blank code. The hypotheses that are consistent with each quadruple are listed on the right.

(n, a, g, o} ABC

{n, a, h, g} ABC

{n ,b ,g ,o} B

{n, b, o, o} ABC

{rn, a, g, h} ABC

{m, a, g, o} Be

{m, b, h, g} ABC

{m, b, o, o} ABC

{m, b, h, g} ABe

{m, b, o, o} ABC

Constructing models of the data and of the theories of this sort is only the first step in the evaluation of the explanations.

Evaluating Sequence Models

Evaluation of the "goodness of fit" between the model of the data and the model (not the theory of models)requires criteria of evalua- tion as well. The construction or adoption of such criteria involves problems which have classically formed the core of inductive logic. One might settle for a pure probability criterion for assessing fit for a given hypothesis, by defining fit as the dependent probability P(A IB) where B is inclusion in the particular modelling hypothesis and A is membership in the data set. The dependent probabilities for these hypotheses are,

for H A , P(A [BA) = 0.8

for H B , P(A IBB) = 1.0

for Hc, P(A IBc) = 0.9 .

516

Popper criticized Carnap [ 1 ] for treating these dependent probabilities as criteria for the relative acceptability of theories. The difficulty with this criterion, as Popper pointed out, is that it puts a premium on unin- formativeness - the theory that was the least uninformative, in the sense that it excluded the fewest possibilities, would be the theory that would produce the highest dependent probabilities:

Imagine in our list o f hypotheses on scientific careers of this sort that we had a hypothesis D which said that to be elected to a scientific acad- emy in the post-paradigm generation one must have either received a prestigious degree or have published a heavily cited article. This hypoth- esis excludes only one of the 20 possible sequences, so, on this ground alone it is an extremely uninformative hypothesis. Yet all the cases in the data set are consistent with it, so it rates a dependent probabili ty of 1.00. Intuitively, one must regard this hypothesis as virtually without value, in spite of its high "probabil i ty ." Such results moved Popper to reject such "probabilit ies" as a criterion of acceptability. He argues that "theoretical science . . . . aims at restricting the range of permitted events to a minimum; and, if this can be done at all, to such a degree that any further restriction would lead to an actual empirical falsifica- tion of the theory" (1968, p. 113). Hypotheses which have high a priori probabilities, such as D, do not serve this purpose, and are therefore less acceptable to us as scientific hypotheses, regardless of the high depen- dent probabilities that these hypotheses may attain.

In the case of hypotheses which take the set-theoretic form discussed here, we can construct a criterion of acceptability that deals with this concern. We can compute the a priori probability of each hypothesis by counting the number of sequences covered by the partition modelling each hypothesis and computing it as a proport ion of the number of total number of possible sequences. We can also compute the propor- tion of empirical cases falling in the partition for each hypothesis. If we conceive of the possible sequences as constituting a logical space, we can conceive of the empirical cases as distributed in this space. Each partition bounds a part of this space. The best non-"probabilist ic" hy- pothesis is the one which partitions all the actual cases in the smallest amount o f logical space. The best "probabil ist ic" hypothesis is the one which partitions the space such that the greatest density of cases is included within the partition [21. The computat ion formula for this criterion is:

P x . c~

f l (.D

where Px = total empirical cases in data set falling within Partition x,

517

n = number of cases in data set, co = total number of possible conse- quences [ 3 ], Px = number of possible sequences falling within Partition x.

This formula has a variety of useful properties. It specifies a level of complete valuelessness of a hypothesis. A hypothesis whose partition defines a space which is no more dense with empirical cases than is the space as a whole has a value of one. A hypothesis which excludes no possibilities (i.e., an analytic truth) has a value of one as well. Hypoth- eses with a value greater than one are preferable to these hypotheses. The "degree of preferability" may be said to correspond with one's intuitive sense of twice as good a hypothesis, in that a hypothesis with a score of 1.20 could intelligibly be claimed to be twice as much an improvement over no hypothesis as a hypothesis with a score of 1.10. This is not an useful number for signifying anything like a general "degree of confirmation," however, since as in a horse race, the degree of preferability depends on the hypotheses which it is preferred to for this reason: the extent to which a Partition excludes possibilities depends on the size of the space of possibilities it is partitioning. This space is constructed by considering the various hypotheses treated as relevant to the phenomena, and identifying all the possible sequences under all the hypotheses taken together. Thus the total size of the space depends on a decision about which hypotheses to compare and regard as relevant. So the score depends upon the "competi t ive" models. In this sense, the scores resemble odds in a horse race, which depend not just on the innate ability of a horse but on its competition.

If we compare the scores for the various hypotheses discussed here:

for H A , 1.333

fo rHB, 1.25

for H c , 1.058

with the dependent probabilities discussed above, it should be evident that these scores contrast sharply. HB, which produced a dependent probability of 1.0, ranks only second, while Hc proves to have been vir- tually without value, although it produced a 0.9 dependent probability. The best hypothesis, H A , had produced the lowest dependent probabil- ity [4].

Conclusion

The usefulness of this method, as with other formal methods, depends entirely on the relation between the informal structure of explanations of a particular social phenomenon and the model: if there is

518

a bad fit here, if the explanation must be forced to be plausibly restated in this set-theoretic formalism, it is of no value; if there is a good fit, the formalism can be of value. The formalism can be used to model dichotomous or categorical variables. Comparisons between alternative explanations of some complexity, rather than simple comparisons between variables, are possible. The stringent assumptions of, e.g., structural-equation models, are not required. Moreover, the objection that theories constructed through structural-equation models cannot be conjoined to produce more extensive theories (urged by Turner and Wilcox, 1974, p. 585), is not applicable to these models, which can be conjoined with other such models - t h e i r evaluation is dependent on competing hypotheses, but their meaning is not dependent on assump- tions such as the assumption of uncorrelated error terms which essen- tially contain reference to a limited universe of variables.

Notes

1 Unfairly, as it happens, for Carnap did not actually make this mistake, as Michalos points out in his useful and elaborate discussion ( 1971 ).

2 This notion of "best probabilistic hypothesis" is designed to accord with socio- logical practice in connection with hypothesis preferences, a practice which assumes large and undefinable sources of error.

3 The counting of possible sequences can be difficult with especially elaborate empirical problems, and there is no single formula for solving all counting prob- lems. However, "truth-tree" methods are especially useful in these counting problems, since it is necessary to count the number of possibilities under each partition as well as the total number of possibilities.

4 Since the ratio of cases covered on the partition to n is a proportion, it is easy to test the statistical significance of differences in proportions or test the significance of difference between 1.00 and the score, by using the usual computation formu- las for proportionality and distributions appropriate to them.

References

Michalos, Alex C. (1971). The Popper-Cqrnap Controversy. The Hague: Martinus Nijhoff.

Popper, Karl R. (1968). The Logic of Scientific Discovery. 2nd edition. New York: Harper.

Suppes, Patrick (1960). "A comparison of the meaning and uses of models in mathematics and the empirical sciences," Synthese 1 2 : 2 8 7 - 3 0 1 .

Suppes, Patrick (1962). "Models of data," in E.E. Nagel, P. Suppes and A. Tarski, (eds.) Logic, Methodology, and Philosophy of Science: Proceedings of the 1960 International Congress. Stanford, CA: Stanford University Press.

Turner, Stephen and Wilcoxl W.C. (1974). "Getting clear about the sign-rule," Sociological Quarterly i 5 : 571-588 .