Identifying Implicit Relationships Within

Natural-Language Questions

Brandon MarloweID: 2693414

What is Watson?

Watson is a question answering computer system capable of answering questions posed in natural language, developed in IBM's DeepQA project by a research team led by principal investigator David Ferrucci. Watson was named after IBM's first CEO, industrialist Thomas J. Watson. The computer system was specifically developed to answer questions on the quiz show Jeopardy! and, in 2011, the Watson computer system competed on Jeopardy! against former winners Brad Rutter and Ken Jennings winning the first place prize of $1 million.

- Wikipedia

Watson - 2011

IBM Watson Hardware Specs (as of 2011)

● Cluster of 90 IBM Power 750 Servers

● Each server has a 3.5 GHz POWER7 Processor– 8 cores, 32 threads each (720 cores, 2880 threads total)

● 16 TB of RAM combined

● Can process 500 GB of data per second

Important Terms

What are Implicit Relationships Within Natural Language Questions?

● Implicit– “capable of being understood from something else though unexpressed” - Merriam

Webster Dictionary

● Related– “connected by reason of an established or discoverable relation” - Merriam Webster

Dictionary

● Language– “ the words, their pronunciation, and the methods of combining them used and

understood by a community” - Merriam Webster Dictionary

What is Machine Learning?

● Machine Learning: Machine learning is a field of computer science that gives computers the ability to learn without being explicitly programmed. - Wikipedia

● Features: In machine learning and pattern recognition, a feature is an individual measurable property or characteristic of a phenomenon being observed...Features are usually numeric, but structural features such as strings and graphs are used in syntactic pattern recognition. - Wikipedia

Types of Questions

● Common Bonds– Questions whose solution is a common element or

characteristic among all the entities within the question● These make up less than 0.2% of all Jeopardy! questions

● Missing Links– Questions whose solution relies on identifying a

missing entity explicitly or implicitly referred to within the question

Things with arches!

Common Bonds:

Missing Links:

On hearing of the discoveryof George Mallory’s body, this explorer toldreporters he still thinks he was first.

Answer: Sir Edmund Hillary

Missing Link: Mount Everest

How Does Watson Do It?

Watson’s Four Computational Steps

1) Question Analysis

2) Candidate Answer Generation

3) Candidate Answer Scoring

4) Merging and Ranking of Candidates

1) Question Analysis

● Analysis done using four components

– Spreading Activation Algorithm

– N-Gram Corpus

– PRISMATIC Knowledge Base

– Wikipedia link-crawling

Spreading Activation Algorithm

● A method for searching associative, neural, or semantic networks

● Begins at a set of source nodes with weights or “activation values”

● IBM developed a recursive S-A Algorithm

– Identifies related concepts

– Based on heterogeneous data resources

● Higher activation values between nodes = stronger relationship

Spreading Activation Algorithm Example Visualization

N-Gram Corpus

● Captures semantic relatedness between words using Normalized Google Distance (NGD)– NGD measures conceptual/semantic similarity between word pairs

– ie.) Football and Player

● Terms that frequently occur together in are more likely to appear in an N-Gram

PRISMATIC Knowledge Base

Determines conceptual and semantic relatedness based on syntactic arrangement

Uses VerbNet, FrameNet, and WordNet (minimally) as resources, each of which are manually built

WordNet contains synset information (definition, synonyms, antonyms, etc.)

FrameNet contains “frames” that describe the structure of selected words used in association

VerbNet maps verbs to their associated Levin-classes

PRISMATIC Knowledge Base

PRISMATIC Knowledge Base“In 1921, Einstein received the Nobel Prize for his original work on the photoelectric effect.”

SLOT

Parse Tree

Wikipedia Links

● Wikipedia metadata enables Watson to determine semantic relationships

<X> represents links where the anchor text and the target document are both “X.”

<X | Y> represent links where “X” is the anchor text and “Y” is the title of the target document.

2) Candidate Answer Generation

● Answer generation differs between Common Bond and Missing Link questions

– Common Bond● Identifies closely related concepts to entities in the question● Considers the union of all concepts as candidates● S-A Algorithm invoked on each question entity● Common bond solutions are directly related to entities in question → spreading activation depth = 1

– Missing Link● Candidate answers are generated, and used as “hypothesized” missing links● The missing links are then passed back into the algorithm along with the question● New candidate solutions are generated● Good missing links are:

– Highly related to concepts in the question – Must be ruled out as possible solutions

● Missing links are of the wrong answer type (ie. “Mount Everest” is not a person), but have high association with the question

3) Candidate Answer Scoring

● Answer ranking differs between Common Bond and Missing Link questions

– Common Bond● Scored on semantic relatedness to each entity in the question● Similarity score calculated using NGD and N-Gram Corpus● Candidates semantically close to all entities are ranked highly● Scores are used in final ranking step

– Missing Link● Watson performs worse when missing link is implicit● An additional answer scorer includes identified missing link to measure semantic relationship between all

entities– New answer scorer allows textual evidence scorers to operate more optimally

● Score is calculated by determining semantic relatedness between the missing link and candidate answers● Three instances of the scoring method are run in parallel

– One for each resource (N-Gram Corpus, PRISMATIC, and Wikipedia links)

Watson’s Confidence threshold

4) Merging and Ranking of Candidates

Experimental Evaluation

Experimental Evaluation● Separate experiments for Common Bonds and Missing Link

– Common Bonds Evaluation Setup● Tested end-to-end system performance● Trained Watson using a set of 14,770 questions (102 Common Bond)● Two versions of Watson: Enhanced (w/ N-Gram Corpus AKA Common Bond Answer Generator), baseline (w/o N-Gram Corpus)● 139 previously unseen common bond questions given to Watson● Two main benchmarks

– “Binary Recall” = percentage of questions for which the system chose the correct answer as a candidate answer– “Precision@70” = precision when answering the top 70% of questions it was most confident about

– Missing Link Evaluation Setup● Tested end-to-end system performance● Two version of Watson: Enhanced (w/ Missing Link Processing), baseline (w/o Missing Link Processing)● 1,112 previously unseen Missing Link questions given to Watson● Two main benchmarks

– “Binary Recall” = (same as Common Bonds)– “Question-Answering Accuracy” tests Watson’s ability to promote candidate answers produced by Missing-Link Answer Scorer to the top of candidate

list

Experimental Results

● Common Bond Evaluation Results– Enhanced System vs. Baseline System

● Common Bond Answer Generator produced at least one candidate answer for 113 of the 139 questions (81%)

● For 80 of those 113 (80%), the correct answer was one of the candidates● Binary Recall was improved for only 6 additional questions when combining all the systems● Fails to generate correct answer when the solution is an abstract concept

– ex.) Question: Modem, Quasar, Gestapo [Answer: Acronyms]

● Ultimately, the N-Gram Corpus was left out of the final system

Experimental Results

● Missing Link Evaluation Results– Of the 1,112 questions, presented to Watson

● Watson identified 259 as having a Missing Link● Just under 20% of them were not Missing Link questions

– ~60% of the Missing Links were explicit, ~40% were implicit

● Questions within the Missing Link subset are more difficult● Humans score ~48% in Missing Link Questions

Experimental Results

Correct type initially. Chooses answer lower in list, then identifies that as the missing link

Conclusion

● Three knowledge resources developed by IBM:– N-Gram Corpus

– PRISMATIC Knowledge Base

– Wikipedia Web Link Crawling

● Spreading Activation Algorithm:– Supported by all three knowledge resources

– Recursively traverses neural network to discover semantic relationships

● Massive implications in AI and widespread application– Health-care

– Law

– Advertising

Sources

https://www.merriam-webster.com/dictionary/implicit

https://www.merriam-webster.com/dictionary/language

https://www.merriam-webster.com/dictionary/related

https://en.wikipedia.org/wiki/Watson_(computer)

https://en.wikipedia.org/wiki/Machine_learning

https://en.wikipedia.org/wiki/Feature_%28machine_learning%29

Fan, James & Ferrucci, David & Gondek, David & Kalyanpur, Aditya. (2010). PRISMATIC: Inducing Knowledge From a Large Scale Lexicalized Relation Resource. 122-127.