Computer-assisted essay assessment

Textbook paragraphs

………

………

………

………

………

Compute similarity betweentextbook paragraphs and

training essays

………

………

Essays of training m

aterial

………

Define the threshold values forgrade categories

Compute similarity betweentextbook paragraphs and

essay to be graded

Assign essay gradeComputed essay

grade

Essay to be graded

………

Similarity scores by Latent Semantic Analysis

Comparison material based on relevant passages from textbook

Defining threshold values for grade categories

Grading the essays

Results

Results

Training material

Wei

ghti

ngsc

hem

e

Nu

m. o

f gr

aded

essa

ys

Exa

ct

Exa

ct o

rad

jace

nt

Cor

rela

tion

5 sections and 70 essays LE 73 46.6 87.7 0.81

27 paragraphs and 70 essays LE 73 50.7 89.0 0.81

147 sentences and 70 essays LE 73 35.6 91.8 0.79

5 sections and 70 essays LE 73 43.8 87.7 0.80

27 paragraphs and 86 essays LE 57 43.9 84.2 0.80

5 sections and 70 essays N 73 28.8 58.9 0.56

Latent Semantic Analysis (LSA)aka Latent Semantic Indexing (LSI)

Several Applications Information Retrieval Information Filtering Essay Assessment

Documents are presented as a matrix in which each row stands for a unique word and each column stands for a text passage (word-by-document matrix)

Truncated singular value decomposition is used to model latent semantic structure

Resulting semantic space is used for retrieval Can retrieve documents that share no words

with query .

Latent Semantic Analysis (LSA)

Singular Value Decomposition Reduces the dimensionality of word-by-document

matrix Using a reduced dimension new relationships

between words and contexts are induced when reconstructing a close approximation to the original matrix

Reduces irrelevant data and “noise”

Latent Semantic Analysis (LSA)Document comparison

Semantic space is constructed from the training material

To grade an essay, a matrix for the essay document is built

Document vector of essay is compared to the semantic space

doc1 d o c 2 doc3 … d o c n

T1 w11 w12 w13 … w1n

T2 w21 w22 w23 … w2n

T3 w31 w32 w33 … w3n

… … … … …

Q u e ryv e c to r tm wm1 wm2 wm3 … wmn

t1 qw1Similarity scores

t2 qw2 doc1 doc2 doc3 … docn

t3 qw3 S1 S2 S3 … Sn

… …

tm qwm

Compute similarity between documentvectors and query vector

Word-by-document matrix

Latent Semantic Analysis (LSA)

Document comparison Euclidean distance Dot product Cosine measure

Cosine between document vectors

YX

YX

cos

Dot product of vector divided by their lengths

B

A

Latent Semantic Analysis (LSA)

Pros Doesn’t just match on terms, tries to match on

concepts

Cons Computationally expensive, its not cheap to

compute singular values Choice of dimensionality is somewhat arbitrary,

done by experimentation

Latent Semantic Analysis (LSA)

Word-by-document matrix

Latent Semantic Analysis (LSA)

Singular value decomposition

Latent Semantic Analysis (LSA)

Two dimensional reconstruction of word-by-document matrix

Latent Semantic Analysis (LSA)