8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
1/20
Clustering the Short Stories of
Edgar Allan Poe Using Galois Lattices
Joint Statistical Meetings, 2007
Roger Bilisoly, Ph.D.Department of Mathematical Sciences
Central Connecticut State UniversityNew Britain, Connecticut
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
2/20
Why Edgar Allan Poe?
Poe wrote in many styles Horror, Detective, Satire, Science Fiction
Poes styles are distinctive Hopefully easier for a computer to detect
Only ~70 short stories His novel, short stories and poems all fit in one
volume of about 1000 pages
Poe freely available on Web Lived 1809-49, so all works are out of copyright
Project Gutenberg has Poe (and many other authors)
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
3/20
Two Goals: 1) Cluster Poes stories2) Be able to explain clusters in terms of language
Text similarity is used in information retrieval (IR) to helpmatch queries and texts Can be used to measure distances between two texts
There are many ways to do this Vector Space Models: term-document matrix allows geometric
approach in high dimensional spaces (e.g., R20,000) Probabilistic Models: find maxi P(Texti | Query) Language Models: find max i P(Query | Language Modeli) And so forth. See Grossman and Frieder (2004)
These approaches are well tested Search engines are profitable and competitive Understanding these approaches require a good
knowledge of mathematics and probability
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
4/20
Vector Space Model uses
a term-document matrix
Each row has word counts for a particular word
Each column has word counts for a particular storyColumns can be viewed as vectors
Cos(Angle between two column vectors) is a text similarity measure
Just over20,000 rows
for Poes
short stories
Note allthe 0s
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
5/20
Zipfs Law: Word Freq c/Rank
(Applied to Poes Short Stories)
The Freq=24401
Of Freq=14622 And Freq=9559
8715 words where Freq=1Examples: acceleration,goats, loyal, theft,
This type ofword distribution
implies mostly 0sand low counts in a
term-documentmatrix
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
6/20
Clustering by Formal Concept Theory:Based on Incidence Matrices
Here rows represent authors (objects) & columns represent attributes
Formal concepts have the form {{objects}, {attributes}}
Idea: look for maximal submatrices that contain all 1s
0101001Whitman
1010100Eliot
0110100Dickens
1001100Stowe
0101011Poe
FemaleMaleUKUSANovelistShortStories
Poet
1 = author has attribute, 0 = author doesnt have attribute
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
7/20
Examples of Concepts {{Poe}, {Poet, Short Stories, USA, Male}} {{Poe, Whitman}, {Poet, USA, Male}} {{Stowe, Eliot}, {Novelist, Female}} {{Stowe, Dickens, Eliot}, {Novelist}} {{Poe, Dickens, Whitman}, {Male}}
0101001Whitman
1010100Eliot
0110100Dickens
1001100Stowe
0101011Poe
FemaleMaleUKUSANovelistShort
Stories
Poet
Example: Poe and Whitman share exactly Poets, USA & Male attributes
Moreover, Poets, USA & Male are all attributes of exactly Poe and Whitman.
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
8/20
Concepts in a Galois Lattice
Poe, Short Story Stowe Dickens Eliot
Whitman, Poet
American Male Female English
Novelist
We see that Dickens is a Male and English (go upwards)and All English are Novelists (go upwards again)
We see that Males are Whitman, and Dickens (go downwards),
as well as Poe (go downwards again)
These concepts are only true for the incidence
matrix we used. Adding further authors orattributes can change the formal concepts.
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
9/20
Goal: Find Galois lattice for Poe stories (as objects)
and word groups (as attributes)
Word groups are easy to understand
E.g., colors: black, dark, white, red, color, green, A thematic thesaurus lists related words, e.g., deathis
related to: dying, demise, killing, to kill, to decease, toexpire, the end,
These obtained from WordNet 2.1, an open source lexicaldatabase
Can produce a word group-document matrix Dimensionality much lower, so matrix is not sparse Word groups easier to link to literary ideas
For incidence matrix, let 1 = story in top 25%,0 otherwise
Rates of word groups are much higher than individualword rates, so sparseness not a problem
Other percentagestried, too
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
10/20
Word Groups Used in Analysis ofPoes Short Stories
7 word groups
Death: death, corpse, dead, murder, died,
Body: eyes, head, hand, body, feet, heart,
Sensation: see, seen, saw, heard, view, Religious: soul, god, spirit, heaven, moral,
Fear: horror, terror, fear, horrible, anxiety,
Family: family, wife, mother, daughter, Colors: black, dark, white, red, color, green, ...
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
11/20
Problem: Many word measures are
functions of sample size
We have seen that lexical measures such as mean
frequency and vocabulary size, as well as [modelparameter estimates] all depend on the sample size. p 24 of Word Frequency Distributionsby R. Harald Baayen
One solution used in corpus linguistics: take equal sized
samples from texts However, stories are whole entities
An approximate solution is to consider sets of storiesclose in size
Story sets used here have maximum difference in size < 33% Three groups: 2001 to 3000 words, 3001 to 4200 words and
4201 to 6000 words. (Includes 44 of 70 short stories)
Top 25% for each word group labeled 1, otherwise 0.
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
12/20
Galois Lattice for Poes Stories
of Length 2001-3000 Words
DeathMorella
Tell-Tale HeartMasque Red Death
BodyMorella
Tell-Tale HeartCask Amontillado
SensationTell-Tale Heart
Furniture4 Beasts in One
ReligiousMorella
EleonoraImp Perverse
FearMorella
Tell-Tale HeartImp Perverse
FamilyMorella
Eleonora3 Sundays
ColorsFurniture
Masque Red DeathEleonora
Death, ColorsMasque Red Death
Colors, SensationFurniture
Religious, FearMorella
Imp of the Perverse
Religious, FamilyMorella
Eleonora
Death, Body, FearMorella
Tell-Tale Heart
Death, Body,Fear, Sensation
Tell-Tale Heart
Religious, FamilyFear, Death, Body
Morella
Religious, Family, ColorsEleonora
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
13/20
Morella appears in 5 word groups;
Tell-Tale Heart appears in 4.
The Tell-Tale Heart is about a man who kills his older
roommate, hides the body under the floor, the police visit, andhe cracks and shows them the body. This story ranks 3rd (of 12) in Death (5.12 per K), 1st in Body (18.18 per K),
1st in Sensation (12.58 per K), 1st in Fear (9.32 per K) This story is, in fact, considered one of Poes iconic tales.
Morella is the narrators wife who is obsessed with mysticism,dies during childbirth, the daughter grows up to resembleMorella more and more, and upon baptism she cries I amhere, and dies.
Poe has several stories about wives who die (Berenice, Ligeia, The OvalPortrait, The Oblong Box and Eleonora who dies before she and narratorcan marry). So the plot is one Poe has explored several times.
Ranks 1st in Death (8.94 per K), 3rd in Body (13.18 per K), 1st in Religious (8.47per K), 2nd in Fear (5.65 per K), and 2nd in Family (5.18 per K)
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
14/20
Concepts for Stories with
3001-4200 words'Death''Metzengerstein''The Colloquy of Monas and Una''The Facts in the Case of M Valdemar''The Black Cat''A Tale of the Ragged Mountains'
'Body''The Devil in the Belfry''The Facts in the Case of M Valdemar''A Predicament'
'The Man That Was Used Up''The Angel of the Odd'
'Sensation''The Man of the Crowd''A Predicament''The Man That Was Used Up''A Tale of the Ragged Mountains''MS Found in a Bottle'
'Religious''The Colloquy of Monas and Una''Mesmeric Revelation''The Angel of the Odd''The Black Cat'
'Never Bet the Devil Your Head'
'Fear''Metzengerstein''Berenice'
'The Black Cat''A Tale of the Ragged Mountains''MS Found in a Bottle'
'Family''Metzengerstein''Berenice''The Black Cat'
'Never Bet the Devil Your Head''Diddling'
'Colors''Metzengerstein''The Devil in the Belfry''Berenice'
'The Black Cat''MS Found in a Bottle'
'Death' 'Body''The Facts in the Case of M Valdemar'
'Death' 'Sensation' 'Fear''A Tale of the Ragged Mountains'
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
15/20
'Sensation' 'Fear' 'Colors'
'MS Found in a Bottle'
'Death' 'Religious' 'Fear' 'Family' 'Colors''The Black Cat'
'Religious' 'Family''The Black Cat'
'Never Bet the Devil Your Head'
'Fear' 'Family' 'Colors''Metzengerstein''Berenice''The Black Cat'
'Fear' 'Colors''Metzengerstein''Berenice''The Black Cat''MS Found in a Bottle'
'Death' 'Religious''The Colloquy of Monas and Una''The Black Cat'
'Death' 'Fear''Metzengerstein''The Black Cat''A Tale of the Ragged Mountains'
'Death' 'Fear' 'Family' 'Colors''Metzengerstein''The Black Cat'
'Body' 'Sensation''A Predicament''The Man That Was Used Up'
'Body' 'Religious'
'The Angel of the Odd'
'Body' 'Colors''The Devil in the Belfry'
'Sensation' 'Fear''A Tale of the Ragged Mountains''MS Found in a Bottle'
Concepts for Stories with
3001-4200 words (cont.)
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
16/20
Galois Lattice for Poes Stories
of Length 4201-6000 Words
DeathLoss of Breath
Premature BurialThou Art the Man
BodyAssignation
Loss of BreathKing Pest
SensationAssignation
Landors CottageDomain Arnheim
ReligiousAssignation
Bon-BonDomain Arnheim
FearKing Pest
Premature Burial1002nd Tale
FamilyOblong Box1002nd Tale
Thou Art Man
ColorsLandors Cottage
1002nd TaleBon-Bon
Death, BodyLoss of Breath
Death, FearPremature Burial
Family, DeathThou Art the Man
Fear, BodyKing Pest
Religious,Sensation
AssignationDomain Arnheim
Colors, SensationLandors Cottage
Colors, ReligiousBon-Bon
Body, Religious,Sensation
The Assignation
Family, Fear, Colors1002nd Tale ofScheherezade
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
17/20
References
Word Frequency Distributions R. Harald Baayen
Corpus Linguistics: Investigating Language Structure and Use Biber, Conrad and Reppen
The Use of Color Words by Edgar Allan Poe PMLA, 45(2), Wilson Clough
Concept Data Analysis: Theory and Applications Claudio Corpineto and Giovanni Romano
Word Index to Poes Fiction Edited by Burton Pollin
The Last Haunting of Edgar Allan Poe: An Identification of Poe Preferences
contained in the The Beale Papers Robert Wald Text Mining: Predictive Methods for Analyzing Unstructured Information
Weiss, Indurkhya, Zhang and Damerau
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
18/20
Numbering for Poes Short Stories
1. The Unparalleled Adventures of One Hans Pfaall2. The Gold Bug3. Four Beasts in One
4. The Murders in the Rue Morgue
5. The Mystery of Marie Rogt6. The Balloon-Hoax7. MS. Found in a Bottle8. The Oval Portrait
9. The Purloined Letter10. The Thousand-and-Second Tale of Scheherezade
11. A Descent into the Maelstrm12. Von Kempelen and his Discovery13. Mesmeric Revelation
14. The Facts in the Case of M Valdemar
15. The Black Cat16. The Fall of the House of Usher17. Silence -- a Fable18. The Masque of the Red Death19. The Cask of Amontillado
20. The Imp of the Perverse21. The Island of the Fay22. The Assignation
23. The Pit and the Pendulum24. The Premature Burial
25. The Domain of Arnheim26. Landor's Cottage27. William Wilson
28. The Tell-Tale Heart29. Berenice30. Eleonora31. Ligeia32. Morella
33. A Tale of the Ragged Mountains34. The Spectacles
35. King Pest
36. Three Sundays in a Week37. The Devil in the Belfry
38. Lionizing39. X-ing a Paragrab
40. Metzengerstein41. The System of Doctor Tarr and Professor Fether42. How to Write a Blackwood article43. A Predicament
44. Mystification45. Diddling46. The Angel of the Odd47. Mellonta Tauta48. The Duc de L'Omlette
49. The Oblong Box
50. Loss of Breath51. The Man That Was Used Up
52. The Business Man53. The Landscape Garden
54. Maelzel's Chess-Player55. The Power of Words56. The Colloquy of Monas and Una
57. The Conversation of Eiros and Charmion58. Shadow -- A Parable
59. Philosophy of Furniture
60. A Tale of Jerusalem61. The Sphinx
62. Hop Frog63. The Man of the Crowd64. Never Bet the Devil Your Head65. Thou Art the Man
66. Why the Little Frenchman Wears his Hand in a Sling67. Bon-Bon
68. Some words with a Mummy69. Literary Life of Thingum Bob Esq.
70. Morning on the Wissahiccon
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
19/20
Word Variety Keeps Growing
Plot of
# Tokens/# Typesas function ofsize of sample
8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices
20/20
Inclusiveness of Concept Lattices
This is for a
Galois latticethat includes all70 short stories
Top Related