Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
-
Upload
lora-aroyo -
Category
Technology
-
view
413 -
download
0
Transcript of Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
![Page 1: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/1.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Truth is a Lie Rules & Semantics from Crowd Perspectives
![Page 2: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/2.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
“in embracing the diversity of human beings, we will find a surer way to true
happiness” Malcolm Gladwell, 2006
in his TED talk “Choice, happiness and spaghetti sauce”
![Page 3: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/3.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
“If we treat human brains as processors in a distributed system, each can perform a small part of a
massive computation” Luis von Ahn, 2006
“Games with a Purpose”, Computer Magazine
![Page 4: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/4.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
“The challenge for Watson was to answer questions posed in every nuance of natural language. The
first step in accomplishing this was to have a lot of questions and their correct answers. Any cognitive
system must take this first step.”
Chris Welty, 2014 “Cognitive Computing and the Future”, 1st Cognitive Computing Forum
![Page 5: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/5.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Take Home Message
True and False is not enough, There is diversity in human interpreta3on
CrowdTruth introduces a spa3al representa3on of
meaning that through crowdsourcing harnesses disagreement
Using CrowdTruth
untrained workers can be just as reliable as highly trained experts
![Page 6: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/6.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Is this song ….
Passionate Rousing
Confident Boisterous
Rowdy
Literate Poignant Wistful
Bittersweet Autumnal Brooding
Rollicking Cheerful
Fun Sweet
Amiable Good-natured
Humorous Silly
Campy Whimsical
Witty Wry
Aggressive Fiery Tense
Anxious Intense Volatile
![Page 7: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/7.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
?Passionate
Rousing Confident Boisterous
Rowdy
Literate Poignant Wistful
Bittersweet Autumnal Brooding
Rollicking Cheerful
Fun Sweet
Amiable Good-natured
Humorous Silly
Campy Whimsical
Witty Wry
Aggressive Fiery Tense
Anxious Intense Volatile
Is this song ….
![Page 8: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/8.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
?- category(“That’s not my name”, x) ?
Passionate Rousing
Confident Boisterous
Rowdy
Literate Poignant Wistful
Bittersweet Autumnal Brooding
Rollicking Cheerful
Fun Sweet
Amiable Good-natured
Humorous Silly
Campy Whimsical
Witty Wry
Aggressive Fiery Tense
Anxious Intense Volatile
(Lee and Hu 2012)
![Page 9: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/9.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects.
With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS.
ANTIBIOTICS are the first line treatment for indications of TYPHUS.
Does this sentence express TREATS(Antibiotics, Typhus)?
![Page 10: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/10.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
TREATS(Antibiotics, Typhus)
Is this True?
![Page 11: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/11.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
What is the treatment for typhus?
Antibiotic therapy is recommended for both endemic and epidemic typhus infections because early treatment with antibiotics (for example, az i thromyc in , doxycyc l ine, te t racyc l ine, or chloramphenicol) can cure most people infected with the bacteria. Consultation with an infectious-disease expert is advised especially if epidemic typhus or typhus in pregnant females is diagnosed. Delays in treatment may allow renal, lung, or nervous system problems to develop. Some patients, especially the elderly, may die.
TREATS(Antibiotics, Typhus)
Is this True?
![Page 12: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/12.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
TREATS(Corticosteroids, Lupus)
Is this True?
What is the treatment for lupus?
In more severe cases of Lupus, medications that modulate the immune system (mainly corticosteroids, immunosuppressants) are used to control the disease and prevent recurrence of symptoms (known as flares). Depending on the dosage, people who require steroids may develop Cushing's syndrome. This may subside if and when the large initial dosage is reduced, but long-term use of even low doses can cause elevated blood pressure and cataracts. Numerous new immunosuppressive drugs are being actively tested. Rather than suppressing the immune system nonspecifically, as corticosteroids do, they target the responses of individual [types of] immune cells.
![Page 13: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/13.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Human Interpretation
“I will never understand people” “They are the worst”
![Page 14: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/14.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
human subjectivity, ambiguity & uncertainty of expression are part of human semantics
![Page 15: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/15.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
current practices in computer science (AI)
are based on
fallacies
![Page 16: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/16.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
single truth
![Page 17: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/17.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
experts rule
![Page 18: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/18.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
disagreement is bad
![Page 19: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/19.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo simplification of context
this all results in
![Page 20: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/20.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
![Page 21: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/21.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
this means our systems understand only
simplified black & white
world
problem ..
![Page 22: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/22.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
reality is
complex diverse
but …
![Page 23: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/23.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
current gold standard practices
are not adequate any more
so...
![Page 24: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/24.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
there is a gap we need to bridge
and …
![Page 25: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/25.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
interestingly …
![Page 26: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/26.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
in humanities … diversity is well accepted
![Page 27: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/27.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
in sciences … diversity is taboo
![Page 28: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/28.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
despite numerous examples …
string theory debate
![Page 29: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/29.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
global warming debate and …
![Page 30: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/30.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
evolutionary bio. debate gradualism punctuated equilibrium
and …
![Page 31: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/31.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
What can we do?
so …
![Page 32: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/32.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
• collective decisions of large groups of people
• a group of error-prone decision-makers can be surprisingly good at picking the best choice
• when thumbs up or thumbs down - the chance of picking the right answer needs to be > 50%
• the odds that a most of them will pick the right answer is greater than any of them will pick it on their own
• performance gets better as size grows
1785 Marquis de Condorcet
“wisdom of crowds”
![Page 33: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/33.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
• asked 787 people to guess the weight of an ox
• none got the right answer
• their collective guess was almost perfect
1906 Sir Francis Galton
“wisdom of crowds”
![Page 34: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/34.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo WWII Math Rosies
1942: Ballistics calculations and flight trajectories
![Page 35: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/35.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo NASA’s Computer Room
transcribe raw flight data from celluloid film & oscillograph paper
![Page 36: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/36.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
“Who Wants to Be a Millionaire?”
• help from individual expert or audience poll
• majority of the audience right 91% of the time
• individuals right only 65% of the time
“wisdom of crowds”
2004 James Surowiecki
![Page 37: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/37.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Crowdsourcing Human Computing
![Page 38: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/38.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
the wise crowd
diversity of opinion independent perspectives
multitude of contexts gives the big picture
James Surowiecki
![Page 39: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/39.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
current practices in computer science (AI) are based on fallacies
![Page 40: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/40.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
today’s real-world complex problem solving needs understanding of
multiple perspectives
current practices in computer science (AI) are based on fallacies
![Page 41: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/41.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
single truth
![Page 42: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/42.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
multiple truths
single truth
![Page 43: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/43.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
experts rule
![Page 44: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/44.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
wisdom of crowds
experts rule
![Page 45: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/45.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
disagreement is bad
![Page 46: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/46.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
disagreement is bad
disagreement as signal
![Page 47: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/47.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
“best collective decisions are result of disagreement,
not consensus or compromise” James Surowiecki
because …
![Page 48: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/48.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
can we harness it?
![Page 49: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/49.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Is this song ….
?Passionate
Rousing Confident Boisterous
Rowdy
Literate Poignant Wistful
Bittersweet Autumnal Brooding
Rollicking Cheerful
Fun Sweet
Amiable Good-natured
Humorous Silly
Campy Whimsical
Witty Wry
Aggressive Fiery Tense
Anxious Intense Volatile
![Page 50: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/50.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
If “One Truth” & “No Disagreement” Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5
W1 1
W2 1
W3 1
W4 1
W5 1
W6 1
W7
W8
W9 1
W10 1
Totals 1 3 1 2 1
![Page 51: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/51.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W1 1 1 1
W2 1 1 1
W3 1 1 1
W4 1 1
W5 1 1
W6 1 1 1
W7 1 1 1
W8 1 1 1
W9 1 1
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
If “Many Truths” & “Disagreement”
![Page 52: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/52.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
can indicate alternative interpretations
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
Disagreement as Signal
can indicate ambiguity in the
categorisation
can indicate low quality workers
![Page 53: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/53.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Does this sentence express TREATS(Antibiotics, Typhus)?
Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects.
With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS.
ANTIBIOTICS are the first line treatment for indications of TYPHUS. 95%
75%
50%
Disagreement is signal here too
![Page 54: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/54.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Sentence Treat Prevent Cause Side Effect Diagnose
S1 2
S2 2
S3 2
S4 2
S5 1 1
S6 2
S7 2
S8 1 1
S9 2
S10 2
If “One Truth”, “No Disagreement” & “Experts”
![Page 55: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/55.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
• we rejected 3 fallacies: one truth, experts rule and disagreement is bad
• by encouraging disagreement • we exposed a richer set of possibilities • that help in identifying, processing &
understanding context
so …
![Page 56: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/56.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
• One truth: data collection efforts assume one correct interpretation for every example
• All examples are created equal: ground truth treats all examples the same – either match the correct result or not
• Detailed guidelines help: if examples cause disagreement - add instructions to limit interpretations
• Disagreement is bad: increase quality of annotation data by reducing disagreement among the annotators
• One is enough: most of the annotated examples are evaluated by one person
• Experts are better: annotators with domain knowledge provide better annotations
• Once done, forever valid: annotations are not updated; new data not aligned with old
7 Myths
myths directly influence the practice of collecting human annotated data; Need to be
revised with a new theory of truth (CrowdTruth)
Lora Aroyo, Chris Welty: Truth is a Lie: 7 Myths about Human Annotation, AI Magazine 2014.
![Page 57: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/57.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
![Page 58: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/58.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
![Page 59: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/59.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
a new approach to understanding semantics harnessing the power & diversity of the crowd
![Page 60: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/60.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
informs a vector space model of truth instead of a boolean, fuzzy or statistical model
![Page 61: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/61.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
observer referent
sign
has
triangle of reference
(Ogden & Richards, 1923)
![Page 62: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/62.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
crowd annotator annotation
example
annotation choices
Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014.
triangle of reference
![Page 63: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/63.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
crowd annotator annotation
example
annotation choices
passionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into rousing, cheerful, fun, poignant, wis9ul, campy, quirky, tense, anxious, any of the 5 confident, sweet, amiable, bi>ersweet, whimsical, wi>y, intense, vola?le, clusters boisterous, good-‐natured autumnal, wry visceral rowdy brooding
Category 1 Category 3 Category 5
Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014.
triangle of reference
![Page 64: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/64.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
crowd annotator annotation
example
annotation choices
treat prevent side effect diagnose cause other
Treat Cause
Prevent
Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014.
ANTIBIOTICS are the first line treatment for indications of TYPHUS.
triangle of reference
![Page 65: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/65.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
1 1 1
Worker Vector
![Page 66: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/66.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
1 1 1
1 1
1
1 1
1 1
1 1
1
1
1
0 1 1 0 0 4 3 0 0 5 1 0
Sentence Vector
OUR NEW THEORY OF
TRUTH
![Page 67: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/67.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Unclear relationship between the two arguments reflected in the disagreement
Sentence Clarity
Feeling the way the CHEST expands (PALPATION), can identify areas of the lung that are full of fluid.
?PALPATIONIs CHEST related to
diagnose location associated with
is_a otherpart_of
0 0 02 3 0 0 0 1 0 0 44 1
![Page 68: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/68.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Clearly expressed relation between the two arguments reflected in the agreement
Sentence Clarity
?CONJUNCTIVITISHYPERAEMIA related toIs0 0 0 1 0 0 0 013 0 0 0 0 0
symptomcause
Redness (HYPERAEMIA), irritation (chemosis) and watering (epiphora) of the eyes are symptoms common to all forms of CONJUNCTIVITIS.
![Page 69: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/69.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
1 13
4 3 1
7 2 1
6 2 1 1
1 1 1 1
3 3 1 1
8 3 1
5 7 1
1 2 1
Relation Similarity
![Page 70: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/70.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Measures how clearly a sentence expresses a relation
0 1 1 0 0 4 3 0 0 5 1 0
Unit vector for relation R6
Sentence Vector
Cosine = .55 confidence
Sentence-Relation Score
![Page 71: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/71.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Measured per worker averaged over all sentences (whether worker consistently disagrees with everyone else)
Worker-Sentence Disagreement
0 1 1 0 0 4 3 0 0 5 1 0
Worker’s sentence vector
Sentence Vector
AVG (Cosine)
Worker Quality
![Page 72: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/72.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Measured for pairs of workers / sentence averaged over all sentences
(identify communities of though)
Worker-Worker Disagreement
0 1 1 0 0 1 0 0 0 0 1 0
Worker 1 sentence vector
Worker 2Vector
AVG (Cosine)
Worker Similarity
![Page 73: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/73.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
![Page 74: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/74.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Example Results
![Page 75: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/75.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Example Results
![Page 76: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/76.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Example Results
![Page 77: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/77.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
crowdtruth.org
experimental results show that CrowdTruth is a better way of capturing context a more accurate way to predict & explain truth
![Page 78: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/78.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Training a Relation Extraction Classifier
F1 Cost per sentence
CrowdTruth 0.642 $0.66
Expert Annotator 0.638 $2.00
Single Annotator 0.492 $0.08
“wisdom of the crowd” provides training data that is at least as good
if not better than experts
only with proper analytic framework for harnessing disagreement from the crowd
![Page 79: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/79.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Learning Curves
(crowd with pos./neg. threshold at 0.5) above 400 sent.: crowd consistently over baseline & single above 600 sent.: crowd out-performs experts
![Page 80: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/80.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Learning Curves
![Page 81: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/81.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
Take Home Message
True and False is not enough, There is diversity in human interpreta3on
CrowdTruth introduces a spa3al representa3on of
meaning that through crowdsourcing harnesses disagreement
Using CrowdTruth
untrained workers can be just as reliable as highly trained experts
![Page 82: Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)](https://reader030.fdocuments.in/reader030/viewer/2022032513/55d183d6bb61ebe9738b45c7/html5/thumbnails/82.jpg)
Web & Media Group
http://lora-aroyo.org @laroyo
https://www.youtube.com/watch?v=CyAI_lVUdzM
To be AND not to be: quantum intelligence?
Lora Aroyo & Chris Welty