A Reliability Study of the Mental Health Assessment Form for School-Age Children

Clarice J . Kestenbaum, M.D. and Hector R. Bird, M.D.

Abstract. This paper describes a reliability study of the Mental Health Assessment Form, a new instrument for the assessment of school-age children. An outline of the instrument and sample items are provided. Utilizing a half-hour videotaped, semistructured interview, three raters independently scored the Mental Health Assessment Form for 35 children from a nonpsychiatric population. The scoring was done in the absence of any other data. Inter- rater reliability was found to be high.

The clinical evaluation of children requires the gathering of infor- mation from multiple sources and the subsequent systematic orga- nization of that information. A thorough family history, develop- mental history, medical records, school reports, the clinical interview, and, where indicated, psychological tests, all enter into the composite picture of an assessment. Such an evaluation is, how- ever, impressionistic; the various sources provide information which is often conflicting and contradictory.

It is generally accepted that a thorough developmental history is probably the most important item in establishing a diagnosis; nev- ertheless its reliability is often questionable (Robbins, i 963). Mi- chael Rutter had emphasized how diagnostic reliability may be poor even in the face of extensive clinical material (Rutter and Graham, 1968; Graham and Rutter, 1968). T h e ideal situation is

Dr. Kestenbaum is Director and Lk. Bird is Assistant Director of the Division of Child and Adolescent Psychiatry, St. Luke’s Hospital Center, New York City.

Raters for the study included Drs. Hector R. Bird, Martha S. Troutman, and John Fogelman. Drs. Troutman and Fogelman are staff psychiatrish of t h Division of Child and Adolescent Psychiatly, St. Luke’s Hospital Center.

Consultants f o r the Study included Donald Rock, Ph.D., Associate Professor of Psychological Statis- tics, New School for Social Research, N.Y.C.; and of The Educational Division, Educational Testing Seruice, Princeton, NJ . , and Stephen Ellis, B.S., Senior Programmer, Sloan-Kettering Institute for Cancer Research, N.Y.C.

Reprints, as well as the complete Mental Health Assessment Form, may be requested from the authors at the Division of Child and Adolescent Psychiahy, St. Luke’s Hospital Center, 41 I West I 14 Street, New York, NY 10025.

002-7138/78/1702-338$00.70 @ 1978 American Academy of Child Psychiatry

338

Reliability of Mental Health Assessment Form 339

seldom realized and quite often a thorough, detailed evaluation is not possible. Particularly in clinic populations, many children are not seen after the initial contact. Thus there is need for an evalua- tion more inclusive and broader in scope than the usual mental status examination, yet which can be done in a single encounter.

The Mental Health Assessment Form (MHAF) for the school-age child was developed specifically as an instrument for clinical assess- ment of a child independent from all other data. A survey of the literature shows that available instruments (Arnold and Smeltzer, 1974; Bolland and Sandler, 1965; Burdock et al., 1960; Dibble and Cohen, 1974; Endicott and Spitzer, 1972; Fish and Shapiro, 1965; A. Freud, 1962; Graham and Rutter, 1968; Lyerly, 1973; McCon- ville, 1975; Rutter, 1967; Rutter and Graham, 1968; Spitzer and Endicott, 1968, 1972, 1973, 1974; Spitzer et al., 1973; Wing et al., 1967; Wittenborn, 1964) are designed to rate behavioral change on the basis of various treatment modalities. They consist either of questionnaires for parents or mental health workers, question- naires for children which cover a broad range of issues but are su- perficial in scope, or of “in-depth” psychological profiles which are too impressionistic and open-ended for scoring purposes. The MHAF, in which a wide variety of items can be scored in a system- atic fashion, has attempted to provide a compromise between these extremes.

The MHAF utilizes z semistructured interview. This confers the advantage of greater spontaneity and relatedness during the inter- viewer’s contact with the child. Information is obtained on a variety of topics without the use of cards or questionnaires. Some skill in child interviewing technique (Goodman and Sours, 1967; Werk- man, 1965; Yarrow, 1960) and a thorough familiarization with the items to be scored on the MHAF are assumed. The use of the form is intended for clinicians with a certain level of training and exper- tise in the basic concepts and principles of child development as well as clinical experience with children. The interview involves an inquiry into all areas of a child’s life and functioning, including specific questions about: (a) the child in his family (a description of home and family life, problems in family relationships); (b) the child in school (including relationship to teachers, peer rela- tionships, sports, hobbies, homework); (c) fantasy life (quality and content of fantasies and dreams as well as problem-solving ability); and (d) personality organization (self-concept, overall mood, per- ceptions, coping mechanisms).

The MHAF consists of 189 items divided into two major sections (table 1). Part I is scored according to observable data and is pri-

340 C. J . Kestenbaum and H . R. Bird

marily a mental status examination. It is subdivided into five major areas. Part I1 scores information derived from the content of the interview. Both historical and developmental data elicited from the child during the interview are included. Part I1 also deals with the child’s concepts and perceptions of himself in his world. It is di- vided into six major areas.

Most of the items in the MHAF are scored on a scale of 1 to 5, ranging from “no deviation” to “marked deviation” from the ex- pectable. The “expectable” represents generally accepted clinical

Table 1

Outline of the Mental Health Assessment Form

PART I-Mental Status I. Physical appearance

A. General attractiveness B. Physical characteristics C. Physical maturation D. Observable deviations in physical

E. Grooming and dress F. Gender differentiation

11. Motoric Behavior and Speech A. Motor activity B. Motor coordination C. Presence of unusual motoric pat-

characteristics

terns, habit patterns, and mannerisms

D. Speech

A. Quality of relatedness as judged

B. Quality of relatedness as judged

C. Social interaction

A. Inappropriate affect B. Constriction of affect C. Elated affect D. Depressive affect E. Labile affect F. Overanxious affect G. Angry affect H. Histrionic affect

V. Language and Thinking . A. Overall intelligence

B. Cognitive functions C. External reality testing D. Use of language E. Thought processes F. Attention span

111. Relatedness during Interview

by nonverbal behavior

by verbal behavior

IV. Affect

PART 11-Content of the Interview VI. Feeling States

A. Depression B. Elation C. Mood disturbance (other) D. Anger E. Anxiety

A. The child’s relationship to his

B. The child’s relationship with

VII. Interpersonal Relations

family

other adult authority figures

C. Relations with peers D. Relationship to pets E. Modes of interaction with others F. Aggressive behavior G. Sexual behavior

and Fantasies) A. Fantasy B. Dreams

A. Dissatisfaction with self B. Comparison of self to peers C. Comparison between self and

VIII. Symbolic Representation (Dreams

IX. Self-concept

ideal self X. Conscience-Moral Judgment

A. Deficit in development of con-

B. Antisocial behavior XI. General Level of Adaption

A. Personality characteristics B. Defense mechanisms C. Maladaptive solutions in dealing

science

with anxiety

Reliability of Mental Health Assessment Form 34 1

impressions about normality. The rater must use his clinical experi- ence, as well as his theoretical knowledge, to determine whether the particular item is to be scored within the expectable range, or, if deviant, the degree to which the child deviates from the “norm” for his age group. In most instances some degree of deviation from the expectable does not necessarily imply psychopathology. Certain items are bipolar and are scored in either direction away from the norm (or expectable) which lies at the midpoint. These may range from 1 to 7, from 1 to 5, or from 1 to 3 with the “expectable” being 5, 3, or 2. In many items, therefore, the deviation may point in the direction of superior functioning. Each item includes an “0” or “not scorable” category to be used when the information required cannot be elicited or if the item is not applicable. A number of items when scored at some point in the gradient may require elabo-

Table 2 Bipolar Item on Eye Contact

Expectable-There is frequent gre contact. The child’s body is turned toward the interviewer. Smiling is a#r@nate to the sitmtion. Mucks of expression are usedfiepently.

A. Quality of relatedness as judged by NONVERBAL behavior

* 153 a. Eye contact Score on a range from total avdance to staring into the interviewer’s eyes.

0. n.s. 1. marked avoidance 2. 3. 4. no deviation 5. 6. 7. staring

Table 3 Items on Depression

Score on a range from no deviation through pen‘& of feeling sad and tearjiul longer than what would be expectable for a pm‘od of mourning, through sepblame, or feelings unworthiness, to suictde wishes, feelings of hopelessness and despair, andlor suicidnl preoccu- pation.

A. Depression

0. 1. 2. 3. 4. - 5. n s . nodev. min. marked

239 a. Degree of depression related to a specific event (e.g., loss) 0 0 0 0 0 0

specific event 0 0 0 0 0 0

240 b. Degree of depression not related to a

342 C . J . Kestenbaum and H . R . Bird

ration or further specification. These items are so marked, and space is provided at the end of the scoring sheet for this purpose. An overall impression and a prognostic statement must also be given upon completion of the scoring. Examples of items in the MHAF are provided in tables 2, 3, 4, and 5.

Table 4 Item Measuring Anxiety during Dreams

Score on a range from no deviation, through varying degrees of anxiety, to nightmares from which the child is awakead.

326 c. Level of Anxiety during Predominant Dreams

0. ns . 1. no deviation 2. minimal 3. 4. 5. marked

Table 5 Items Providing Measures of Self-concept

Score on a range from no hiat ion, through a child who would pefw some change in himself without undue peoccupation or diminution of self-esteem (e.g., curly hair us. straight), through @eoccupataon with his “defect” so that self-esteem is unduly diminished, to p-e- occupation with “defect” so that it interferes with his functioning.

327-334 A. Dissatisfaction with self (even if there is a realistic handicap)

0. I . 2. 3. 4. n.s. nodev. min.

327 a. Physical appearance 0 0 0 0 0

328 b. Gender 0 0 0 0 0

329 c. Race/ethnic background 0 0 0 0 0 330 d. Physical handicap or

illness 0 0 0 0 0 331 e. Athletic ability 0 0 0 0 0

332 f. School performance 0 0 0 0 0

333 g. School acceptance- popularity 0 0 0 0 0

334 h. Other 0 0 0 0 0

If334 scored 2. through 5., specif31 on page included for Descriptive Information.

5. marked

0

0

0

0

0

0

0 0

MET1 10DOLOGY

The sample consisting of 35 children was drawn from a longi- tudinal study of children considered at high risk for schizophrenia (Erlenmeyer-Kimling, 1975).’ That study includes children from

‘“A Prospective Study of Children at Risk for Schizophrenia” supported by Grant MH 19560 of the National Institute of Mental Health and by the New York State Department of Mental Hygiene.

Reliability of Mental Health Assessment Form 343

four subgroups: the first two comprise children of either one or two schizophrenic parents; a control group of children whose parent(s) have been hospitalized for other psychiatric illness, and a population of “normals.” None of the children in the sample had been diagnosed as having significant psychopathology.

Each child receives a battery of psychological and physiological tests in a day-long laboratory visit (Rutschmann et a]., 1977). A half-hour clinical interview is included. The interview is videotaped for the purpose of scoring the MHAF. The interviewers were drawn from a pool of psychiatrists thoroughly familiar with the MHAF, and trained to elicit pertinent information. The inter- viewer has no previous knowledge of the child’s subgroup. Fur- thermore, any information which might indicate the subgroup is deleted from the videotape. Random videotapes are “edited” in a similar fashion.

The raters consisted of three child psychiatrists from different training backgrounds. The videotaped interviews were seen once by each rater, independent from the others. Like the interviewers, the raters were blind as to subgroup.

Upon completion of the scoring, raters were also asked to pro- vide a global assessment score. This score involves assigning a number from 0 to 100 on an anchored rating scale-the Children’s Global Assessment Scale-the latter representing an adaptation of the Global Assessment Scale (GAS) (Spitzer et al., 1973), itself a revi- sion of the Menninger Health Sickness Rating Scale of Luborsky.2 The GAS is divided into deciles, with a descriptive paragraph op- posite each 10-point block. The reliability and the validity of the GAS are described elsewhere (Endicott et al., 1976).

Interrater Reliability (table 6). Pearson correlation coefficients were computed between all pairs of raters. The stepped-up interrater reliabilities were then computed according to the formula k d l + (k - 1) f- where k is the number of raters and i- is the mean in- terrater correlation. Items having a stepped-up interrater reliability greater than 0.5 are considered to be reliable.

Since the population tested for the reliability study was essen- tially a healthy group, specific items describing a formal thought disorder or perceptual disturbance were characterized by small and sometimes zero variances. Hence, the reliability of most items geared toward a more pathological group could not be determined using classical methods because of lack of variance. Each of the 1 1 sections in the MHAF was given a total scale score when possible. Each item was scored individually as well.

2The scale for children was reworked by Doreen Rothman, Patricia Heldman, and James Sorrels in conformity with the emotional life of latency-age children.

344 C. J . Kestenbaum and H . R . Bird

Table 6 Interrater Correlates

I .

11.

111.

IV.

V.

VI.

VII.

Physical Appearance General attractiveness 0.5 Height 0.82

Motoric Behavior and Speech Total Scale Score 0.57

Motor activity 0.47 Tics 0.61 Disorder of articulation 0.76 Disorder of rhythm 0.87 Monotonous speech 0.54 Relatedness during Interview

Total Scale Score 0.62 Eye contact 0.76 Affect

Total Scale Score 0.53 Constriction of affect 0.55 Depressive affect 0.56 Overanxious affect 0.76 Language and Thinking Overall intelligence 0.57 Ability to abstract 0.6 Feeling States

Total Scale Score 0.85 Degree of depression related

to a specific event 0.86 Degree of depression not

related to a specific event 0.79 Anger 0.67 Anxiety 0.76 Interpersonal Relations Overall scale score 0.78 Child's perception of the degree of rejection, neglect or abuse from his family

Mother 0.86 Father 0.82 Sibling 0.8 1

Rater's impression of the degree of rejection, neglect, or abuse from members of the family

Mother 0.69 Father 0.75 Sibling 0.79

Weight 0.82

Disturbance in positive feelings to- ward family members

0.9 1 Child has a best friend 0.5 1 Several close friends 0.51 More of same sex than

opposite sex 0.65

Dominance-submission in relation- ship with

Mother 0.64 Sibling 0.64

Overly aggressive vs. overly inhibited behavior with

Mother 0.84 Sibling 0.56

Other manifestations of aggression (covert) 0.76

and Fantasies) Total Scale Score 0.84

Character of fantasy 0.43 Character of dreams 0.84 Level of anxiety during

predominant dreams 0.85

Mother

Feelings toward pets 0.9

VIII. Symbolic Representation (Dreams

IX. Self-concept Total Scale Score 0.78

Dissatisfaction with self: Physical appearance 0.94 Gender 0.89 Athletic ability 0.82

Comparison of self to

Degree of discrepancy between

School performance 0.80

peers 0.57

self and ideal self 0.73 X. Conscience-Moral Judgment

Deficit in development of conscience 0.57

Antisocial behavior 0.84 Child does these acts

alone 0.92

Special skills, talents 0.69 Special interests 0.77

XI. General Level of Adaptation

Reliability of Mental Health Assessment Form 345

RESVLTS

High reliability was found in the sections dealing with (a) Feeling States (such as the degree of depression, anxiety, and anger), (b) Zn- terpersonal Relationships (particularly those involving a child with the mother), (c) Self-concept (relating to the degree of discrepancy be- tween self and ideal self), and (d) Conscience-MoraE Judgment (par- ticularly in the item relating to antisocial behavior). Likewise, relia- bility in the 0.85 range was obtained in the section dealing with level of anxiety during dreams.

Section XI entitled General Level of Adaptation and Use of Defense Mechanisms was the only section where the results did not demon- strate meaningful reliability.

The interrater reliability using the Children’s Global Assessment Scale was found to be high: 0.80.

CONCLUSION

The MHAF permits the scoring of a number of important items germane to the assessment of a school-age child. Deviation in the pathological sense can be registered, along with “positive” devia- tions, such as special skills or talents, perseverance, superior in- telligence, frustration tolerance, creativity.

The MHAF appears to be a reliable instrument. Interrater relia- bility was significant not only in those items involving direct obser- vation, but also in those which are characteristically more inferen- tial, such as items about feeling states, quality of interpersonal relations, and self-concept. A second reliability study as well as a validity study using a more deviant general child psychiatric clinic population are currently under way.

The MHAF can be a valuable aid in the initial diagnostic ap- praisal of a child. It must be emphasized that it is not in itself a diagnostic tool, but is rather a screening instrument which can aid in singling out those children in need of additional work-up. Fur- thermore, the nature of the work-up is indicated by clusters of deviant scores in particular sections. The MHAF may serve as a research tool in the study of premorbid conditions. Items could be added which would focus, for example, on early affective disorder or gender identity. Lastly it has proved a most useful method of helping medical students and psychiatry residents refine their ob- servation of the patient, particularly regarding signs and symptoms of behavioral and developmental abnormality. For the inexperi-

346 C . J . Kestenbaum and H . R. Bird

enced interviewer the MHAF embodies a structured interview that can guide him toward the more orderly and complete accumula- tion of information about a patient.

REFERENCES

ARNOLD, L. E. & SMELTZER, D. J. (1974), Behavior checklist factor analysis for children and

BOLLAND, J. & SANDLER, J. (1965), The Hampstead Psychoanalytic Index. New York: Interna-

BURDOCK, E. I. , HAKEREM, G., HARDESTY, A. S., & ZUBIN, J. (1960), A ward behavior rating

DIBBLE, E. & COHEN, D. J. (1974) Companion instruments for measuring children’s compe-

ENDICOTT, J. & SPITZER, R. L. (1972), Current and past psychopathology scales (CAPPS).

-- FLEISS, J. L., & COHEN, J. (1976), The global assessment scale. Arch. Gen.

ERLENMEYER-KIMLING, L. (1975), A prospective study of children at risk for schizophrenia. In: Life H G t T Research in Psychopathology, ed. R. Wirt, G. Winokur, & M. Roff. Min- neapolis: University of Minnesota Press, pp. 22-46.

FISH, B. & SHAPIRO, T . (1965), A typology of children’s psychiatric disorders. This Journal,

FREUD, A. (1962), Assessment of childhood disturbances. The Psychoanalytic Study of the Child,

GOODMAN, J. & SOURS, J. (1967), The Child Mental Status Examination. New York: Basic Books. GRAHAM, P. J. & RUTTER, M. (1968). The reliability and validity of the psychiatric assessment

of the child: 11. Brit.J. Psychiat., 114:581-592. LYERLY, S . B. (1973), Handbook of Psychiatric Rating Scales. Bethesda: National Institute of

Mental Health, 2nd ed. MCCONVILLE, B. (1975), On attempting measurement in a child psychiatry in-patient unit. In:

Mental Health in Children, ed. D. V. Sankar. New York: P.J.D. Publications, 2:451-477. ROBBINS, L. C. (1963), The accuracy of parental recall of aspects of child development and

of child rearing practices.]. Abnurm. SOC. Psychol., 66:26 1-270. RUTSCHMANN, J., CORNBLATT, B., & ERLENMEYER-KIMLING, L. ( 1977). Sustained attention in

children at risk for schizophrenia. Arch. Gen. Psychiat., 34:57 1-575. RUTTER, M. (1967), Classification and categorization in child psychiatry. Int. J . Psychiat.,

- & GRAHAM, P. J. (1968), The reliability and validity of the psychiatric assessment of the child: I. Interview with the child. Brit. J . Psychiat., Il4:563-579.

SPITZER, R. L. & ENDKOTT, J. (19683, DIAGNO: A computer program for psychiatric diag- nosis using the differential diagnostic procedure. Arch. Gen. Psychiat., 18:746-756. -- (1972), The value of the interview for the evaluation of psychopathology. In:

Psychopathology, ed. M. Hammer, K. Salzinger, S. Sutton. New York: Wiley. -- (1973), Can the computer assist clinicians in diagnosis? Read at the Annual

Meeting of the American Psychiatric Association. -- (1974), Psychiatric rating scales in the evaluation of psychiatric treatment. In:

Cumprehensive Textbook of Psychiatly, ed. A. M. Freedman & H. I . Kaplan. Baltimore: Wil- liam & Wilkins. -- & GIBBON, M. (1973), Global Assessment Scale. New York: Biometrics Research,

New York State Department of Mental Hygiene. WERKMAN, S. L. (1965). The psychiatric diagnostic interview with children. A m r . J. Ortho-

WING, J. K., BIRLEY, J. L., COOPER, J. E., GRAHAM, P. J., & ISAACS, A. D. (1967), Reliability of

adolescents. Arch. Gen. Psychiat., 30:799-804.

tional Universities Press.

scale for use with mental hospital patients. J . Clin. Psychol., 16:246-247.

tence and parental style. Arch. Gen. Psychiat., 30:805-815.

Arch. Gen. Psychiat., 27:678-687.

Psych&, 33~766-77 I .

4~32-52.

17: 149- 158.

3:161-171.

psychiat., 35~754-771.

Reliability of Mental Health Assessment Form 347

a procedure for measuring and classifying ‘present psychiatric state.’ Brit. J. Psychiat.,

WITTENBORN, J . R. (1964), Manual: Wittenborn Psychiatric Rating Scales. New York: Psycho-

YARROW, L. J . (1960), Interviewing children. In: Handbook of Research Methods in Child Devel-

1131499-515.

logical Corporation.

opment, ed. P. H. Mussen. New York: Wiley.