McNamara Graesser Coh-Metrix

McNamara & Graesser 1

McNamara, D. S., & Graesser, A. C. (in press). In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing: Identification, investigation, and resolution. Hershey, PA: IGI Global.

Coh-Metrix: An Automated Tool for Theoretical and Applied Natural Language Processing

Danielle S. McNamara and Art Graesser Institute for Intelligent Systems

University of Memphis

Abstract

Coh-Metrix provides indices for the characteristics of texts on multiple levels of analysis, including word characteristics, sentence characteristics, and the discourse relationships between ideas in text. Coh-Metrix was developed to provide a wide range of indices within one tool. This chapter describes Coh-Metrix and studies that have been conducted validating the Coh-Metrix indices. Coh-Metrix can be used to better understand differences between texts and to explore the extent to which linguistic and discourse features successfully distinguish between text types. Coh-Metrix can also be used to develop and improve natural language processing approaches. We also describe the Coh-Metrix Text Easability Component Scores, which provide a picture of text ease (and hence potential challenges). The Text Easability components provided by Coh-Metrix go beyond traditional readability measures by providing metrics of text characteristics on multiple levels of language and discourse.


Coh-Metrix: An Automated Tool for Theoretical and Applied Natural Language Processing

Coh-Metrix is an automated tool that provides linguistic indices for text and discourse (Graesser,

McNamara, Louwerse, & Cai, 2004). Coh-Metrix was developed to meet three practical needs. First, at

the time that the Coh-Metrix research project began in 2002, there were no readily available tools that

provided an array of indices on words or texts. For example, if a researcher needed the word frequency

values for words or sentences in a text, one tool might be available (though challenging to find). But

another tool would have to be used for measures of word concreteness, familiarity, imagery, syntactic

complexity, and so on. In other words, there existed no linguistic workbench capable of providing a wide

array of measures on language and discourse. Second, traditional measures of text difficulty, referred to as

readability, were outdated given the maturation of our understanding of text and discourse (Clark, 1996;

Graesser, Gernsbacher, & Goldman, 2003; Kintsch, 1998). There was a growing recognition of a number

of factors contributing to text difficulty that are not considered within traditional measures of text

readability. Third, there existed no automated measures of text cohesion. Whereas recognition of the

importance of cohesion had flourished in the 80s and 90s (Gernsbacher, 1990; Goldman, Graesser, & Van

den Broek, 1999; Louwerse, 2001; McNamara & Kintsch, 1996; Sanders & Noordman, 2000), there were

no objective, implemented measures of cohesion available. Thus, with the overarching goal of providing

more informative measures of text complexity, particularly considering text cohesion, we embarked in

2002 on a mission to develop Coh-Metrix (initially funded by an Institute of Education Sciences). This

chapter describes some motivating factors that led to Coh-Metrix, an overview of the measures provided

by Coh-Metrix, some of the many NLP studies that have been completed over the last eight years, and the

ultimate outcome of our endeavors: Coh-Metrix Text Complexity Components.

Readability vs. Cohesion: Why Coh-Metrix was Developed

Readability measures are the most common approach to estimating the difficulty of a text and

hundreds have been developed over the past century. Readability formulas became popular in the 1950s


and by the 1980s over 200 readability algorithms had been developed, with over a 1000 supporting

studies (Chall & Dale, 1995; Dubay, 2004). The most well known readability measures include Flesch-

Kincaid Grade Level (Klare, 1974-5), Degrees of Reading Power (DRP; Koslin, Zeno, & Koslin, 1987),

and Lexile scores (Stenner, 2006). Measures of readability are highly correlated because they are based

on the same constructs: the difficulty of the individual words and the complexity of the separate sentences

in the text. However, the way in which these constructs are operationalized and the underlying statistical

assumptions vary somewhat across readability measures. The Flesch-Kincaid Grade Level metric is based

on the length of words (i.e., number of letters or syllables) and length of sentences (i.e., number of

words). DRP and Lexile scores relate these characteristics of the texts to readers’ performance on cloze

tasks. In a cloze task, the reader reads a text with some words left blank; the reader is asked to fill in the

words by generating them or by selecting a word from a set of options (usually the latter). Using this

methodology, the appropriateness of a text for a particular reader can be calculated based on the

characteristics of the texts and the reader’s performance on cloze tasks. A particular text would be

predicated to be at the reader’s level of proficiency if the reader can perform the cloze task at a threshold

of performance (75%) for texts with similar characteristics (i.e., with the same word and sentence level

difficulties). A text can be defined as too easy if performance is higher than 75% and too difficult to the

extent it is lower than 75%.

Readability measures based on word and sentence characteristics (i.e., usually length) have validity

as indices of text difficulty. When words contain more letters or syllables, they tend to be less frequently

used in a language. Readers need to have greater exposure to language and text in order to encounter less

frequent words and to know what they mean. Clearly, a requisite to comprehension is knowing the meaning

of the words in a text. In turn, to the extent that a sentence contains more words, there is a greater likelihood

that the sentence is more complex syntactically. Readers who have had less exposure to language and text

(i.e., younger and less skilled readers) face greater challenges in parsing syntactically complex sentences.


In sum, word and sentence factors go a long way in accounting for sources of text difficulty.

However, these factors alone explain only a part of text comprehension, and ignore many language and

discourse features that are theoretically influential at estimating comprehension difficulty. At the heart of

Coh-Metrix is the assumption that cohesion is one of the most important aspects of language that is ignored

by readability measures. Cohesion is the linguistic glue that holds together the events and concepts

conveyed within a text. Beyond the words and separate sentences in the text, are the relationships between

the sentences and larger units of text. Explicit cues in the text help the reader to process, understand, or infer

those relationships. A simple but powerful source of cohesion is referential and semantic overlap of adjacent

sentences, pairs of sentences in a paragraph, and adjacent paragraphs. Simply put, when words, concepts, or

ideas overlap between sentences, that content forms a link between the sentences. Another source of

cohesion is connecting words (or connectives) such as because and however. Connectives tell the reader that

there is a relationship between two ideas, and help the reader to understand the direction of the relationship.

Cohesive cues help the reader to understand connections among sentences and paragraphs. This, in

turn, facilitates understanding of the words and sentences, and enhances the reader’ global understanding of

the text. Many studies, across a variety of paradigms and dependent measures, have shown that cohesive

cues in text facilitate reading comprehension and help readers construct more coherent mental

representations of text content (Britton & Gulgoz, 1991; McNamara, 2001; Zwaan & Radvanksy, 1998). For

this reason, the primary bank of indices provided by Coh-Metrix assesses the cohesion of the text.

Coh-Metrix Measures of Language and Cohesion

The number and particular measures that are provided by Coh-Metrix depends on the version and

the type of tool. We have developed public versions of the tool that analyze individual texts and have

provided between 40 and 80 validated indices. We have also developed internal versions of Coh-Metrix

that analyze texts in batches and that include 600-1000 indices, many of which are redundant and many of


which have not been validated (and thus we don’t release them to the public). Although the specific Coh-

Metrix measures vary somewhat across versions and tools, the banks of measures are quite similar.

Descriptive. Coh-Metrix provides descriptive indices such as the number and length of words,

sentences, and paragraphs. These indices help the user to check the Coh-Metrix output (e.g., to make sure

that the numbers make sense) and also to interpret patterns of data.

Co-Referential Cohesion. Co-referential cohesion refers to overlap in content words between

local sentences. For example, this sentence overlaps with the previous sentence by means of the singular

form of the word sentence. The sentences below provide another example from Chapter 2 of the novel,

Jane Eyre:

"I resisted all the way: a new thing for me." Jane says this as Bessie is taking her to be locked in

the red-room after she had fought back when John Reed struck her. For the first time Jane is

asserting her rights, and this action leads to her eventually being sent to Lowood School.

In this excerpt, the explicit referential cohesion is provided only by the overlap in the subject, Jane. While

there is some semantic overlap, much of this must be inferred by the reader.

Coh-Metrix indices of co-referential cohesion vary along two dimensions. First, the indices vary

in terms of locality, from local to more global indices. Local indices include overlap between consecutive,

adjacent sentences. Indices become increasingly global to the extent that they consider overlap among 2,

3, 4, or all of the sentences in a paragraph or text. Second, the indices vary in terms of the explicitness of

the overlap. Noun overlap measures the proportion of sentences in a text for which there are overlapping

nouns. Argument overlap relaxes the noun constraint by also considering nouns that have the same stems

(e.g., baby, babies; mouse, mice) and also considers overlap between pronouns. It should be noted,

however, that Coh-Metrix does not attempt to determine the referents of pronouns (e.g., they refers to

Sally and John). Stem overlap includes overlap between nouns in one sentence with words that share a

common stem in another, including nouns as well as other types of content words (i.e., verbs, adjectives,


adverbs). Whereas the latter three indices are binary (i.e., there either is or is not any overlap between a

pair of sentences), content word overlap refers to the proportion of content words that are shared between

sentences.

Latent Semantic Analysis (LSA; Landauer, McNamara, Dennis, & Kintsch, 2007; also see

Kintsch & Kintsch, this volume) provides measures of semantic overlap between sentences or between

paragraphs. LSA considers meaning overlap between explicit words and also words that are implicitly

similar or related in meaning. For example, home in one sentence will have a relatively high degree of

semantic overlap with house and table in another sentence. LSA uses a statistical technique called

singular value decomposition to condense a large corpus of texts to 100-500 statistical dimensions. The

conceptual similarity between any two text excerpts (e.g., word, clause, sentence, text) is computed as the

geometric cosine between the values and weighted dimensions of the two text excerpts. The value of the

cosine typically varies from 0 to 1. Coh-Metrix measures LSA-based cohesion in several ways, such as

LSA similarity between adjacent sentences, LSA similarity between all possible pairs of sentences in a

paragraph, and LSA similarity between adjacent paragraphs.

Situation Model. Referential cohesion is one linguistic feature that can influence a reader’s

deeper understanding of text. However, there are deeper levels of meaning that go beyond the words. The

expression situation model has been used by researchers in discourse processing and cognitive science to

refer to the deeper meaning representations that involve much more than the explicit words (van Dijk &

Kintsch, 1983; Graesser & McNamara, in press; Graesser, Singer, & Trabasso, 1994; Kintsch, 1998;

Zwaan & Radvansky, 1998). For example, with episodes in narrative text, the situation model would

include the plot; and, in an informational text about the circulatory system, the situation model might

include the flow of the blood. The situation model comprises a mental representation of the deeper

meaning of the text (Kintsch, 1998). And, some researchers have described the situational model in terms

of the features that are present in the comprehender’s mental representation when a given context is

activated (e.g., Singer & Leon, 2007).


The content words and connective words systematically constrain and are aligned with aspects of

these inferred meaning representations, but the explicit words do not go the full distance in specifying the

deep meanings. Coh-Metrix provides indices for a number of measures that are potentially related the

reader’s situation model understanding. These include measures of causality, such as incidence scores for

causal verbs, intentional verbs, and causal particles (e.g., connectives like because, in order to), and ratios

of causal particles to causal verbs. There are measures of verb overlap. These indices are indicative of the

extent to which verbs (which have salient links to actions, events, and states) are repeated across the text.

There are measures of temporal cohesion and logical cohesion as well. These and other measures are

reflective of elements of a text that are more or less likely to support a reader’s construction of a coherent

situation model.

Lexical Diversity. Lexical diversity refers to the variety of words (types) that occur in a text in

relation to the number of words (tokens). When the number of word types is equal to the total number of

words (tokens), then all of the words are different. In that case, lexical diversity is at a maximum, and the

text is either very low in cohesion or very short. Lexical diversity is lower (and cohesion is higher) when

more words are used multiple times across the text. Coh-Metrix includes three indices of lexical diversity:

type-token ratio (TTR), vocd, and the Measure of Textual Lexical Diversity (MTLD). TTR is simply the

number of unique words divided by the number of tokens of the words. The index produced by vocd is

calculated through a computational procedure that fits TTR random samples with ideal TTR curves.

MTLD is calculated as the mean length of sequential word strings in a text that maintain a given TTR

value. TTR is correlated with text length because as the number of word tokens increases, there is a lower

likelihood of those words being unique. Measures such as vocd and MTLD overcome that confound by

using sampling methods (McCarthy & Jarvis, 2010).

Connectives. Coh-Metrix provides an incidence score (occurrence per 1000 words) for all

connectives as well as different types of connectives. Indices are provided on five general types of

connectives: causal (because, so), additive (and, moreover), temporal (first, until), logical (and, or), and


adversative/contrastive (although, whereas). In addition, there is a distinction between positive

connectives (also, moreover) and negative connectives (however, but).

Words and Sentences. Coh-Metrix also assesses text difficulty at the word and sentence level,

but in comparison to readability measures, does so in ways that are driven by theories of reading

comprehension. For example, in addition to word frequency, Coh-Metrix assesses the degree to which the

text contains abstract versus concrete words. When a text contains more abstract ideas, it is more difficult

to process. Coh-Metrix assesses words on abstractness, parts of speech, familiarity, multiple senses of a

word, age of acquisition, and many other features. At the sentence level, Coh-Metrix directly analyzes

syntactic complexity, such as the number of modifiers in noun-phrases and the number of words before

the main verb in a sentence.

Applied Natural Language Processing Studies using Coh-Metrix

Well over 70 studies have been conducted in our laboratory using Coh-Metrix, and the number

grows steadily each year. Indeed, there are nine studies reported in this volume that have made use of

Coh-Metrix tools (Adam, McCarthy, Boonthum, & McNamara, this volume; Baba & Nitta, this volume;

Bell, McCarthy, & McNamara, this volume; Crossley & McNamara, this volume; Jackson & McNamara,

this volume; McCarthy, Dufty, Hempelman, Cai, McNamara & Graesser, this volume; McCarthy &

McNamara, this volume; McNamara et al., this volume; Weston, Crossley, & McNamara, this volume).

In addition, a growing number of researchers are using Coh-Metrix to investigate differences among texts,

to develop stimuli, and to describe characteristics of texts. In sum, it is a very useful tool, particularly in

the context of ANLP.

In our laboratory, Coh-Metrix is often used to understand the features of texts that are used in the

context of experimental studies of text comprehension. For example, Ozuru, Dempsey, and McNamara

(2009) and Ozuru, Briner, Best, and McNamara (in press) used Coh-Metrix indices to guide cohesion

modifications to high and low cohesion texts and to validate their overall cohesiveness. Coh-Metrix has


also been used in the context of three main types of corpus studies. These include validation studies,

exploratory studies, and natural language studies. In this section of the chapter, we provide a few

examples of each type of study.

Validation Studies. One important part of the Coh-Metrix project is the validation of the indices.

It is necessary to verify that the indices measure what we think they are measuring and that they are

theoretically compatible with patterns of data corresponding to types of texts or human performance. We

have conducted many studies of this nature. One example of a validation study in this volume is

McCarthy, Dufty, Hempelman, Cai, McNamara, and Graesser who examined the validity of our LSA

Given/New measure.

Another example of a validation study is McNamara, Louwerse, McCarthy, and Graesser (2010).

This study investigated the validity of Coh-Metrix as a measure of cohesion differences in text. We

reviewed studies that had empirically investigated text cohesion. We collected 19 pairs of high and low

cohesion texts from 12 of these studies. The results of this study indicated that commonly used readability

indices (e.g., Flesch-Kincaid) inappropriately distinguished between low and high cohesion texts.

Specifically, according to readability estimates, the low cohesion texts should have been easier to read

and understand than the high cohesion texts, which is not the case. Using Coh-Metrix, we confirmed that

the co-reference indices provided by Coh-Metrix revealed significant differences between the high and

low cohesion texts, showing higher cohesion scores for the high-cohesion texts than the low-cohesion

texts. We also used discriminant analysis to examine which of the indices provided a greater distinction

between the high and low cohesion texts. These analyses indicated that a combination of two indices,

noun co-reference and causal cohesion, best discriminated between high and low cohesion texts. Notably,

the researchers who had modified the texts had done so with co-reference and causal cohesion in mind;

thus, it not surprising that these measures rose to the surface. Nonetheless, given the empirical results of

the studies indicating that these cohesion modifications do indeed affect comprehension, the results of this

corpus study provide additional evidence that referential and causal relationships play important roles in


the difficulty of texts. And, at a practical level, this study validated the ability of Coh-Metrix indices to

provide measures of text cohesion.

A third example of a validation study comes from Duran, McCarthy, Graesser, and McNamara

(2007). This study validated temporal cohesion measures using a corpus of high school texts from the

domains of science, history, and narrative. Assessing temporality in text is important because of its

ubiquitous presence in organizing language and discourse. Indeed, situation model theories of text

comprehension consider temporality to be one of the critical dimensions for building a coherent mental

representation of described events, particularly in narrative texts (Zwaan & Radvansky, 1998).

Temporality is partially represented through inflected tense morphemes (e.g., -ed, is, has) in every

sentence of the English language. The temporal dimension also depicts unique internal event timeframes,

such as an event that is complete or ongoing, by incorporating a diverse tense-aspect system (ter Meulen,

1995). The occurrence of events at a point in time can also be established by a large repertoire of

adverbial cues, such as before, after, then (Klein, 1994). These temporal features provide several different

indices of the temporal cohesion of a text.

To validate Coh-Metrix measures of temporality in Duran et al. (2007), experts in discourse

processing rated the 150 texts in terms of temporal coherence on three continuous scale measures

designed to capture unique representations of time. These evaluations established a gold standard of

temporality. A multiple regression analysis using Coh-Metrix temporal indices significantly predicted

human ratings. The predictors included in the model were a subset of five temporal cohesion features

generated by Coh-Metrix: incidence of temporal expression words, incidence of positive temporal

connectives, incidence of past tense, incidence of present tense, and temporal adverbial phrases.

Collectively, all but one of the predictors (i.e., the incidence of positive connectives) significantly

predicted the expert ratings. The indices accounted for 40 to 64 percent of the variance in the experts’

ratings (depending on the type of rating). The study thus demonstrated that the Coh-Metrix indices of

local, temporal cohesion significantly predicted human interpretations of temporal coherence. A


discriminant analysis further indicated that these indices were highly predictive of text genres (i.e.,

science, history, and narrative), and were able to classify texts as belonging to a particular genre with very

good reliability (i.e., recall and precision ranged from .47 to .92, with an average F-measure of .68). This

study validated the temporal indices both in terms of human ratings of temporal cohesion and their ability

to distinguish between text types.

Exploratory Studies. Another type of study can be characterized as exploratory. One such study

appears in this volume (Bell, McCarthy, & McNamara, this volume). In this type of study, we and others

have examined how far linguistic features can go towards characterizing and identifying different types of

texts. To some extent, these studies serve as validation studies because they provide additional evidence

that certain text features play important roles in characterizing text and discourse. Indeed, the distinction

between these two types of studies may be hazy. One difference is that exploratory studies are not

focusing on validating any particular measures or indices, but instead serve to validate the general utility

of Coh-Metrix. In sum, they explore the extent to which linguistic and discourse features alone

successfully characterize essential differences between texts and among corpora.

We have conducted many exploratory studies. One such study was conducted by Hall, McCarthy,

Lewis, Lee, and McNamara (2007). In this study, Coh-Metrix indices were used to examine variations in

American and British English, specifically in texts regarding the topic of Law. Their corpus included

400 American and English/Welsh legal cases. The results suggested that there were substantial

differences between the two, casting doubt on generalizations about British and American writing. In

addition, a discriminant function analysis including five indices of cohesion (referential, causal,

syntactic, semantic, and lexical diversity) was able to correctly classify an impressive 85% of the

texts in the test set. Thus, cohesion was found to be an important and highly significant predictor

of differences between American and British English.


Another example of an exploratory study using Coh-Metrix was conducted by McCarthy, Renner,

Duncan, Duran, Lightman, and McNamara (2008). In this study, a combination of 12 Coh-Metrix indices

was used to form a measure of topic sentencehood. Four experiments were conducted to assess two

models of topic sentencehood identification: the Derived Model and the Free Model. According to the

Derived Model, topic sentences are identified in the context of the paragraph and in terms of how well

each sentence in the paragraph captures the paragraph’s theme. In contrast, according to the Free Model,

topic sentences can be identified based on sentential features without reference to other sentences in the

paragraph (i.e., without context). The results of the experiments suggested that human raters were able to

identify topic sentences both with and without the context of the other sentences in the paragraph. The

authors also developed computational implementations for each of the theoretical models using Coh-

Metrix indices. When computational versions were assessed, the results for the Free Model were

promising; however, the Derived Model results were poor. These results collectively implied that humans'

identification of topic sentences in context may rely more heavily on sentential features than on the

relationships between sentences in a paragraph. The authors’ overall conclusion was that there are (at

least) two types of topic sentences. One type is ideal and can be represented by the computational

implementation of the Free Model (i.e., out of context). These types of topic sentences are often used as

examples in textbooks and they are computationally identifiable with a high degree of accuracy. The other

type of topic sentence is more naturalistic. Without the context of surrounding text, these sentences are

more difficult and perhaps impossible to recognize as topic sentences.

Natural Language Studies. By far, most of the studies we have conducted in the context of the

Coh-Metrix project focus on natural language. These include studies examining a wide range of topics

and tasks such as deception, paraphrasing, explaining, answering questions, and writing essays. Indeed,

there are seven chapters within this volume that describe such studies with Coh-Metrix (see in this

volume: Adam, McCarthy, Boonthum, & McNamara, this volume; Bell, McCarthy, & McNamara, this


volume; Crossley & McNamara; Jackson, Boonthum, & McNamara, this volume; McCarthy &

McNamara, this volume; McNamara et al., this volume; Weston, Crossley, & McNamara, this volume).

Studies of natural language using Coh-Metrix afford analyses of discourse under a wide variety of

circumstances, and thus have the potential to improve our understanding of multiple aspects of language

processing (e.g., Crossley, Salsbury, & McNamara, in press; Duran, Hall, McCarthy, & McNamara, 2010;

Hancock et al., 2010). Studies of natural language using Coh-Metrix are also motivated by our interests in

dialogue management in the context of intelligent tutoring systems. Coh-Metrix and Coh-Metrix tools

have played important roles in the context of AutoTutor (D’Mello & Graesser, this volume; Graesser et

al., this volume; Graesser, Person, Lu, Jeon, & McDaniel, 2005), iSTART (Jackson et al., this volume),

question asking and answering (Graesser, Rus, Cai, & Hu, this volume), and W-Pal (McNamara et al.,

this volume).

One example of such a study was conducted by Graesser, Jeon, Yang, and Cai (2007). In this

study, tutorial dialogues in the context of AutoTutor were analyzed using Coh-Metrix to investigate the

effect of college students’ background knowledge on various cohesion relations in the dialogue. Coh-

Metrix was used in this study to compare the tutorial dialogues of high-knowledge students who had

already taken the relevant topics in a college physics class as compared to novice students who had not

taken college physics. AutoTutor is an animated pedagogical agent that scaffolds students to learn about

conceptual physics and other topics by holding conversations in natural language (Graesser, this volume;

Graesser, Jackson, & McDaniel, 2007; Van Lehn, Graesser, et al., 2007). AutoTutor scaffolds students to

generate ideal answers to difficult questions requiring deep reasoning by using a variety of dialogue

moves (e.g., hints, prompts, assertions, corrections, answers to student questions). The findings of the

study conducted by Graesser, Jeon, et al. (2007) showed that the tutorial dialogues of high-knowledge

students shared substantially similar linguistic features with the dialogue of novice students in co-

reference, syntax, connectives, causal cohesion, logical operators, and other measures. However, there

were significant differences in the dialogue with high-knowledge students versus novice students in


semantic or conceptual overlap. This result supported the notion that background knowledge on subject

matter promotes deeper levels of comprehension (i.e., the mental models) of conceptual physics while

interacting with AutoTutor (see also, Graesser, Jeon, Cai, & McNamara, 2008; Jeon, 2008).

Another example of a study of natural language was conducted by McCarthy, Guess, and

McNamara (2009). This study examined the nature of paraphrases in the context of iSTART (see Jackson,

Boonthum, & McNamara, this volume) and developed a computational model to predict the quality of the

paraphrases. Two sentences can be considered to be paraphrases of one another if their meanings are

equivalent but their words and syntax are different. Understanding paraphrasing is important in a number

of contexts because paraphrasing can be used to aid comprehension, stimulate prior knowledge, and assist

in writing-skills development. However, computational studies of paraphrasing have been primarily

conducted using artificial, edited paraphrases and has most often examined only binary dimensions (i.e.,

is or is not a paraphrase). By contrast, McCarthy et al. examined a large database (N = 1,998) of natural

paraphrases generated by high school students that was collected in the context of the reading strategy

tutoring system, iSTART. The paraphrases were assessed by expert raters along 10 dimensions (see

McCarthy & McNamara, this volume). A model including LSA (semantics), minimal edit distances

(syntax), and the difference in length between the text and the paraphrase correlated .51 with the human

evaluations of the paraphrases. These results suggested that semantic completeness and syntactical

similarity were the primary components of paraphrase quality.

A third example of a natural language study is in the realm of writing. We are currently

developing a writing strategy tutoring system called the Writing-Pal that will provide high school students

with instruction and practice on writing prompt-based essays (see McNamara et al., this volume). One

part of the W-Pal project has involved collecting essays written by students so that we can better

understand the linguistic features of essays and how those features relate to their quality. One such study

was conducted by McNamara, Crossley, and McCarthy (2010). Coh-Metrix was used to examine the

degree to which high and low proficiency essays could be identified using linguistic indices of cohesion


(i.e., co-reference and connectives), syntactic complexity (e.g., number of words before the main verb,

sentence structure overlap), lexical diversity, and the characteristics of words (e.g., frequency,

concreteness, imagability). Contrary to expectations, the three most predictive indices of essay quality in

this study were syntactic complexity (as measured by number of words before the main verb), lexical

diversity (as measured by MTLD), and word frequency (as measured by CELEX, logarithm for all

words). Using 26 validated indices of referential and semantic cohesion from Coh-Metrix, none showed

differences between high and low proficiency essays and no indices of cohesion correlated with essay

ratings. Although lexical diversity might be associated with cohesion, the lexical diversity measure

indicated that the higher quality essays were higher in diversity and thus lower in cohesion. Otherwise,

cohesion as measured by Coh-Metrix played no role in predicting the quality of the essays. These results

indicate that the textual features that characterize good student writing are not aligned with those features

that facilitate reading comprehension such as text cohesion. Rather, higher quality essays were more

likely to contain linguistic features associated with text difficulty and sophisticated language.

Coh-Metrix Text Easability Components

One motivation for the development of Coh-Metrix was to provide better measures of text

difficulty (see e.g., Duran, Bellissens, Taylor, & McNamara, 2007), and particularly the specific sources

of potential challenges or scaffolds within texts. One limitation of traditional readability measures is that

they consider only the superficial features of text, which in turn tend to be predictive of readers’ surface

understanding: their understanding of the words and separate sentences. In turn, aassessments that are

used to validate or provide readability scores most often use a cloze task. Cloze tasks (i.e., filling in

missing words in text) assesses comprehension only within sentences based on word associations

(Shanahan, Kamil, & Tobin, 1982) and depends primarily on decoding rather than language

comprehension skills (Keenan, Betjemann, & Olson, 2008). However, many comprehension models

(Graesser & McNamara, in press; Kintsch, 1998; McNamara & Magliano, 2010; Van Dijk & Kintsch,

1983) propose that there are multidimensional levels of understanding that emerge during the


comprehension process, including (at least) surface, textbase, and situation model levels. Readability

formulas by contrast assume a uni-dimensional representation.

Uni-dimensional representations of comprehension ignore the importance of readers’ deeper

levels of understanding. Another limitation is that uni-dimensional metrics of text difficulty are not

particularly helpful to educators when specific guidance is needed for diagnosing what is wrong and

planning remediation for students. Readability formulas do not identify the particular characteristic of the

text that may be challenging or helpful to a student. One major advantage of Coh-Metrix is that it

provides metrics on multiple levels of language and discourse.

One reason that the difficulty or ease of a text should not be assessed at only one level of

language is because multiple levels often work to compensate for one another (McNamara, Louwerse, &

Graesser, in press). For example, in a recent analysis we found that narrative texts (stories, novels) tend to

have low co-referential cohesion and low verb cohesion (McNamara, Graesser, & Louwerse, in press). By

themselves, these indices may imply that narratives are difficult to read. However, narratives also tend to

be composed of more frequent words and they often have high causal cohesion and temporal cohesion,

allowing the reader to form a coherent mental model of the contents of the text. There are times when

language in narrative texts at the situation model levels compensates for more challenging sentences and

low overlap in words and concepts. By contrast, science texts are composed of rare words, making it

challenging for students to understand the concepts in the text. These challenges are sometimes offset by

reducing the syntactic complexity of the text and increasing the overlap in words and concepts (i.e., co-

referential cohesion).

Thus, Coh-Metrix, in contrast to traditional measures of text readability, has the potential to offer

a more complete picture of the potential challenges that may be faced by a reader as well as the potential

scaffolds that may be offered by the text. Coh-Metrix is motivated by theories of discourse and text

comprehension. Such theories describe comprehension at multiple levels, from shallow, text-based


comprehension to deeper levels of comprehension that integrates multiple ideas in the text and brings to

bear information that elaborates the ideas in the text using world and domain knowledge (Graesser &

McNamara, in press). Coh-Metrix assesses challenges that may occur at the word and sentence levels. In

addition, it is able to assess deeper levels of language in terms of cohesion and the situation model. By

doing so, it comes closer to having the capability to estimate how well a reader will comprehend a text at

deeper levels of cognition.

In a recent study (Graesser, McNamara, & Kulikowich, in prepartion), we have developed Coh-

Metrix Text Easability Components to describe the multiple levels of text ease and potential challenges

that arise from textual features. To do so, we examined 54 Coh-Metrix indices calculated on 37,651 texts

from the Touchstone Applied Science Associates (TASA) corpus. This corpus comprises excerpts

(M=287 words) from texts (without paragraph break markers) that a typical senior in high school might

have encountered throughout schooling in kindergarten through 12th grade. The text genres primarily

comprise language arts, science, and social studies/history texts, but also included text in the domains of

business, health, home economics, and industrial arts. The TASA corpus is the most comprehension

collection of K-12 texts currently available for research. A principal components analysis of the corpus

revealed that the following 8 dimensions (in order of robustness) accounted for 67% of the variability

among texts. We consider the first five of these components to be most highly associated with text ease

(and they were the most robust components, accounting for 54% of the variance).

1. Narrativity. Narrative text tells a story, with characters, events, places, and things that are

familiar to the reader. Narrative is closely affiliated with everyday oral conversation. This robust

component is highly affiliated with word familiarity, world knowledge, and oral language. Non-

narrative texts on less familiar topics would lie at the opposite end of the continuum.

2. Referential cohesion. This component includes Coh-Metrix indices that assess referential

cohesion. High cohesion text contains words and ideas that overlap across sentences and the


entire text, forming explicit threads that connect the text for the reader. Low cohesion text is

typically more difficult to process because there are fewer threads that tie the ideas together for

the reader.

3. Syntactic Simplicity. This component reflects the degree to which the sentences in the text

contain fewer words and use more simple, familiar syntactic structures, which are less

challenging to process. At the opposite end of the continuum are texts that contain sentences with

more words and use complex, unfamiliar syntactic structures.

4. Word Concreteness. Texts that contain content words that are concrete, meaningful, and evoke

mental images are easier to process and understand. Abstract words represent concepts that are

difficult to represent visually. Texts that contain more abstract words are more challenging to

understand.

5. Deep cohesion. This dimension reflects the degree to which the text contains causal, intentional,

and temporal connectives. These connectives help the reader to form a more coherent and deeper

understanding of the causal events, processes, and actions in the text.

6. Verb cohesion. This component reflects the degree to which there are overlapping verbs in the

text. When there are repeated verbs, the text likely includes a more coherent event structure that

will facilitate and enhance situation model understanding.

7. Logical cohesion. This component reflects the degree to which the text contains explicit

adversative, additive, and comparative connectives to express relations in the text. This

component reflects the number of logical relations in the text that are explicitly conveyed. This

component is related to the reader’s deeper understanding of the relations in the text.

8. Temporal cohesion. Texts that contain more cues about temporality and that have more

consistent temporality (i.e., tense, aspect) are easier to process and understand. In addition,


temporal cohesion contributes to the reader’s situation model level understanding of the events in

the text.

Using the results of the analysis, we can compute percentile scores for the eight components. A

percentile score varies from 0 to 100%, with higher scores meaning the text is likely to be more difficult

to read than other texts in the corpus. For example, a percentile score of 80% means that 80% of the texts

are easier and 20% are more difficult. Four such texts are presented below, including two narrative texts

and two science texts. We have only graphed the first five components because they account for the most

variance in the characteristics of the texts (and for ease of processing of the graphs).

As would be predicted, the novel and drama texts are substantially more narrative than the two

science texts. It is well documented that narrative is easier to read than informational texts (Bruner, 1986;

Haberlandt & Graesser, 1985; Graesser, Olde, & Klettke, 2002), a fact that should be incorporated in


assessments of text ease and complexity. However, it is possible for narratives to have informational

content that explains the setting or context, and it is possible for science texts to have story-like language

(e.g., the journey of a water molecule through the water system). In either case, the move toward

increased narrativity will generally make the text easier to understand.

The two example narrative texts (Jane Eyre and Glass Menagerie) are declared to be at a 9-10

grade band. However, they have very different profiles on the various dimensions. Jane Eyre is more

difficult when inspecting syntax (as well as verb cohesion, logical cohesion, and temporal cohesion,

which are not presented on the graph), but it is less difficult with respect to word abstractness and deep

cohesion; the two narratives have about the same referential cohesion demands. These are very different

profiles of text ease even though both texts were declared to be in the Grade 9-10 band.

The two science texts have very different grade bands declared. Elementary Particles is grade 6-8

whereas Gravity Reversed is grade 11-12. However, Gravity Reversed is not uniformly more difficult on

all Coh-Metrix dimensions. Gravity Reversed is more difficult with respect to referential cohesion,

syntax, and verb cohesions, but the opposite is the case with respect to narrativity and deep cohesion, (as

well as logical and temporal cohesion); it’s a draw with respect to word concreteness. Therefore, these

comparative profiles should raise questions about the grade levels of these texts.

Such a picture of texts is hoped to provide teachers with more information about text ease, and by

consequence, their ease of processing and at the lower end of the spectrum, their potential challenges.

This information can in turn inform their pedagogical practice. One goal is to bring to fruition the Coh-

Metrix project so that educators can more fully understand the complexity and nature of the texts that they

use in the classroom, particularly in concert with their pedagogical goals and the individual abilities of

their students.


Conclusion

Our goal in the Coh-Metrix project has been to provide indices for the characteristics of texts on

multiple levels of analysis, including word characteristics, sentence characteristics, and the discourse

relationships between ideas in text, within one tool. Our objective has been to go beyond traditional

measures of readability that focus on surface characteristics of texts, which in turn tend to principally

affect surface comprehension. Indeed, the validation of traditional readability algorithms using word and

sentence characteristics (e.g., number of letters or number of words) has been almost exclusively done

using assessments that primarily tap surface comprehension (e.g., cloze tests). Coh-Metrix can be used to

better understand differences between texts and to explore the extent to which linguistic and discourse

features successfully distinguish between text types. Coh-Metrix can also be used to develop and improve

natural language processing approaches.

In the course of using Coh-Metrix and validating its measures, we have come to better understand

differences between texts and which indices are most reliable in detecting differences between texts at

meaningful, consequential levels. This work has culminated most recently in the development of the Coh-

Metrix Easability Components. These components provide a more complete picture of text ease (and

hence potential challenges) that may emerge from the linguistic characteristics of texts. The Text

Easability components provided by Coh-Metrix go beyond traditional readability measures by providing

metrics of text characteristics on multiple levels of language and discourse. Moreover, they are well

aligned with theories of text and discourse comprehension (e.g., Graesser et al., 1994; Kintsch, 1998).

Coh-Metrix offers numerous opportunities to researchers, teachers, and students across a variety

of fields. With the unlimited wealth and diversity of texts that technology such as the internet brings,

computer assessment of text stands to increase dramatically in the next few years, and Coh-Metrix offers

what may be the best approach for such assessments to be conducted.


Acknowledgements

We are grateful to numerous people who have contributed to the Coh-Metrix project over the past decade

and who have provided feedback and help in writing this chapter. Without these collaborations, this

project would not have been possible. Many of the projects reported in this chapter were partially funded

by the Institute for Education Sciences (IES R305G020018-02). The development of the Coh-Metrix Text

Easability Components was funded by the Gates Foundation.


Suggested Readings

Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text

comprehension. Psychological Review, 101, 371–395.

Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, MA: Cambridge University

Press.


References

Baba, K., & Nitta, R. (in press). Dynamic effects of task type practice on the Japanese EFL university

student’s writing: Text analysis with Coh-Metrix. In P. M. McCarthy & C. Boonthum (Eds.), Applied

natural language processing: Identification, investigation, and resolution. Hershey, PA: IGI Global.

Bell, C., McCarthy, P.M., & McNamara, D.S. (in press). Gender and language. In P. M. McCarthy & C.

Boonthum (Eds.), Applied natural language processing: Identification, investigation, and resolution.

Hershey, PA: IGI Global.

Britton, B. K., & Gulgoz, S. (1991). Using Kintsch’s computational model to improve

instructional text: Effects of repairing inference calls on recall and cognitive structures.

Journal of Educational Psychology, 83, 329-345.

Bruner, J. (1986). Actual minds, possible worlds. New York: Plenum Press.

Chall, J. S. & Dale, E. (1995). Readability revisited, the new Dale-Chall readability formula. Cambridge,

MA: Brookline Books.

Clark, H.H. (1996). Using language. Cambridge: Cambridge University Press.

Crossley, S.A., Salsbury, T., & McNamara, D.S. (in press). The role of lexical cohesive devices in

triggering negotiations for meaning. Issues in Applied Linguistics.

Crossley, S., & McNamara, D. S. (in press). Interlanguage talk: What can breadth of knowledge features

tell us about input and output differences? In P. M. McCarthy & C. Boonthum (Eds.), Applied natural

language processing: Identification, investigation, and resolution. Hershey, PA: IGI Global.

D’Mello, S. K. & Graesser, A. C. (in press). Automated detection of cognitive-affective states from text

dialogues. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing:

Identification, investigation, and resolution. Hershey, PA: IGI Global.

DuBay, W. (2004). The Principles of Readability. Costa Mesa, CA: Impact Information.

Duran, N., Bellissens, C., Taylor, R., & McNamara, D. (2007). Qualifying text difficulty with automated

indices of cohesion and semantics. In D.S. McNamara and G. Trafton (Eds.), Proceedings of the 29th


Annual Meeting of the Cognitive Science Society (pp. 233-238). Austin, TX: Cognitive Science

Society.

Duran, N.D., Hall, C., McCarthy, P.M., & McNamara, D.S. (2010). The linguistic correlates of

conversational deception: Comparing natural language processing technologies. Applied

Psycholinguistics.

Duran, N.D., McCarthy, P.M., Graesser, A.C., & McNamara, D.S. (2007). Using temporal cohesion to

predict temporal coherence in narrative and expository texts. Behavior Research Methods, 39, 212-

223.

Gernsbacher, M.A. (1990). Language comprehension as structure building. Hillsdale, NJ: Erlbaum

Goldman, S., Graesser, A. C., & van den Broek, P. (1999). Narrative comprehension, causality, and

coherence. Mahwah, NJ: Erlbaum.

Graesser, A. C. (this volume). AutoTutor. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural


Graesser, A. C., D’Mello, S. K., Hu, X., Cai, Z., Olney, A., & Morgam, B. (in press). AutoTutor. In P. M.

McCarthy & C. Boonthum (Eds.), Applied natural language processing: Identification, investigation,

and resolution. Hershey, PA: IGI Global.

Graesser, A. C., Gernsbacher, M. A., & Goldman, S. (Eds.)(2003). Handbook of discourse processes.

Mahwah, NJ: Erlbaum.

Graesser, A. C., Jackson, G. T., & McDaniel, B. (2007). AutoTutor holds conversations with learners that

are responsive to their cognitive and emotional states. Educational Technology, 47, 19–22.

Graesser, A. C., Jeon, M., Cai, Z., & McNamara, D. S. (2008). Automatic analyses of language,

discourse, and situation models. In J. Auracher & W. van Peer (Eds.), New beginnings in literary

studies (pp. 72–88). Cambridge, U.K.: Cambridge Scholars Publishing.

Graesser, A. C., Jeon, M., Yang, Y., & Cai, Z. (2007). Discourse cohesion in text and tutorial dialogue.

Information Design Journal, 15, 199–213.


Graesser, A.C., & McNamara, D.S. (in press). Computational analyses of multilevel discourse

comprehension. Topics in Cognitive Science.

Graesser, A.C., McNamara, D.S., & Kulikowich, J. (in preparation). Theoretical and automated

dimensions of text difficulty: How easy are the texts we read?

Graesser, A.C., McNamara, D.S., Louwerse, M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on

cohesion and language. Behavior Research Methods, Instruments, & Computers, 36, 193-202.

Graesser, A. C., Olde, B., & Klettke, B. (2002). How does the mind construct and represent stories? In M.

C. Green, J. J. Strange, & T. C. Brock (Eds.), Narrative impact: Social and cognitive foundations (pp.

231–263). Mahwah NJ: Erlbaum.

Graesser, A.C., Person, N., Lu, Z., Jeon, M.G., & McDaniel, B. (2005). Learning while holding a

conversation with a computer. In L. PytlikZillig, M. Bodvarsson, & R. Bruning (Eds.), Technology-

based education: Bringing researchers and practitioners together (pp. 143-167). Greenwich, CT:

Information Age Publishing.

Graesser, A. C., Rus, V., Cai, Z., & Hu, X. (in press). Question answering and asking. In P. M. McCarthy

& C. Boonthum (Eds.), Applied natural language processing: Identification, investigation, and

resolution. Hershey, PA: IGI Global.

Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text

comprehension. Psychological Review, 101, 371–395.

Haberlandt, K. F., & Graesser, A. C. (1985). Component processes in text comprehension. Journal of

Experimental Psychology: General 114, 357-374.

Hancock, J.T., Beaver, D.I., Chung, C.K., Frazee, J., Pennebaker, J.W., Graesser, A., & Cai, Z. (2010).

Social language processing: A framework for analyzing the communication of terrorists and

authoritarian regimes. International Journal of Language, Culture, and Society.

Hall, C., McCarthy, P.M., Lewis, G.A., Lee, D.S., & McNamara, D.S. (2007). A Coh-Metrix assessment

of American and English/Welsh Legal English. Coyote Papers: Psycholinguistic and Computational

Perspectives. University of Arizona Working Papers in Linguistics, 15, 40-54.


Jackson, G. T., & McNamara, D. S. (in press). Applying NLP metrics to students’ self explanations. In P.

M. McCarthy & C. Boonthum (Eds.), Applied natural language processing: Identification,

investigation, and resolution. Hershey, PA: IGI Global.

Jeon, M. (2008). Automated analyses of cohesion and coherence in tutorial dialogue. Unpublished

doctoral dissertation. University of Memphis.

Keenan, J. M., Betjemann, R. B., & Olson, R. K. (2008). Reading comprehension tests vary in the skills

they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of

Reading, 12, 281– 300.

Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, MA: Cambridge University

Press.

Kintsch, W. &Kintsch, E. (this volume). In P. M. McCarthy & C. Boonthum (Eds.), Applied natural


Klare, G. R. (1974–1975). Assessing readability. Reading Research Quarterly, 10, 62-102.

Klein, W. (1994). Time in language. London: Routledge.

Koslin, B. L., Zeno, S., & Koslin, S. (1987). The DRP: An Effectiveness Measure in Reading. New York:

College Entrance Examination Board.

Landauer, T., McNamara, D. S., Dennis, S., & Kintsch, W. (Eds.). (2007). Handbook of Latent Semantic

Analysis. Mahwah, NJ: Erlbaum.

Louwerse, M.M. (2001). An analytic and cognitive parameterization of coherence relations. Cognitive

Linguistics, 12, 291-315.

McCarthy, P.M., Dufty, D., Hempelman, C., Cai., Z., Graesser, A.C., & McNamara, D.S. (in press).

Evaluating givenness/newness. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language

processing: Identification, investigation, and resolution. Hershey, PA: IGI Global.

McCarthy, P.M., Guess, R.H., & McNamara, D.S. (2009). The components of paraphrase evaluations.

Behavioral Research Methods, 41, 682-690.


McCarthy, P.M. & Jarvis, S., (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated

approaches to lexical diversity assessment. Behavior Research Methods, 381-392.

McCarthy, P. M., & McNamara, D. S. (in press). User language paraphrase corpus. In P. M. McCarthy &

C. Boonthum (Eds.), Applied natural language processing: Identification, investigation, and

resolution. Hershey, PA: IGI Global.

McCarthy, P.M., Renner, A.M., Duncan, M.G., Duran, N.D., Lightman, E.J., & McNamara, D.S. (2008).

Identifying topic sentencehood. Behavior Research and Methods, 40, 647-664.

McNamara, D.S. (in press). Computational methods to extract meaning from text and advance theories of

human cognition. Topics in Cognitive Science.

McNamara, D.S. (2001). Reading both high-coherence and low-coherence texts: Effects of text sequence

and prior knowledge. Canadian Journal of Experimental Psychology, 55, 51-62.

McNamara, D.S., Crossley, S.A., & McCarthy, P.M. (2010). Linguistic features of writing quality.

Written Communication, 27, 57-86.

McNamara, D.S., Graesser, A.C., & Louwerse, M.M. (in press). Sources of text difficulty: Across the

ages and genres. In J.P. Sabatini & E. Albro (Eds.), Assessing reading in the 21st century: Aligning

and applying advances in the reading and measurement sciences. Lanham, MD: R&L Education.

McNamara, D.S., & Kintsch, W. (1996). Learning from text: Effects of prior knowledge and text

coherence. Discourse Processes, 22, 247-287.

McNamara, D.S. & Magliano, J.P. (2009). Towards a comprehensive model of comprehension. In B.

Ross (Ed.), The psychology of learning and motivation. New York, NY: Elsevier Science.

McNamara, D.S., Louwerse, M.M., McCarthy, P.M., & Graesser, A.C. (2010). Coh-Metrix: Capturing

linguistic features of cohesion. Discourse Processes, 47, 292-330.

McNamara, D. S., Raine, R., Roscoe, R., Crossley, S., Jackson, G. T., Dai, J., Cai, Z., et al. (in press). The

Writing-Pal: Natural language algorithms to support intelligent tutoring on writing strategies. In P. M.

McCarthy & C. Boonthum (Eds.), Applied natural language processing: Identification, investigation,

and resolution. Hershey, PA: IGI Global.


Ozuru, Y., Briner, S., Best, R., & McNamara, D.S. (2010). Contributions of self-explanation to

comprehension of high and low cohesion texts. Discourse Processes, 47, 641-667.

Ozuru, Y., Dempsey, K., & McNamara, D.S. (2009). Prior knowledge, reading skill, and text cohesion in

the comprehension of science texts. Learning and Instruction, 19, 228-242.

Renner, A., McCarthy, P., Boonthum, C., & McNamara, D.S. (in press). Maximizing ANLP evaluation:

Harmonizing flawed input. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language

processing: Identification, investigation, and resolution. Hershey, PA: IGI Global.

Sanders, T. J. M., & Noordman, L. G. M. (2000). The role of coherence relations and their linguistic

markers in text processing. Discourse Processes, 29, 37–60.

Shanahan, T., Kamil, M. L., & Tobin, A. W. (1982). Cloze as a measure of intersentential comprehension.

Reading Research Quarterly, 17, 229–255.

Singer, M., & Leon, J. (2007). Psychological studies of higher language processes: Behavioral and

empirical approaches. In F. Schmalhofer & C. Perfetti (Eds.), Higher level language processes in the

brain: Inference and comprehension processes. Mahwah, NJ.

Stenner, A. J., Burdick, H., Sanford, E. E., & Burdick, D. S. (2006). How accurate are Lexile text

measures? Journal of Applied Measurement, 7(3), 307-22

Ter Meulen, A. G. B. (1995). Representing time in natural language: The dynamic interpretation of tense

and aspect. Cambridge, MA: MIT Press.

van Dijk, T.A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic.

VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are

tutorial dialogues more effective than reading? Cognitive Science, 31, 3–62.

Weston, J., Crossley, S. & McNamara, D. S. (in press). Computationally assessing human judgments of

freewriting quality. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing:

Identification, investigation, and resolution. Hershey, PA: IGI Global.

Zwaan, R.A. & Radvansky, G.A. (1998). Situation models in language comprehension and memory.

Psychological Bulletin, 123, 162-185.

McNamara Graesser Coh-Metrix

Documents

Transcript of McNamara Graesser Coh-Metrix