Solving General Arithmetic Word Problems

Subhro Roy and Dan Roth

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 1 / 19

Part I

Problem and Main Idea

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 2 / 19

2: The Problem

Arithmetic Word Problem

Gwen was organizing her book case making sure each of the shelves hadexactly 9 books on it. She has 2 types of books - mystery books andpicture books. If she had 3 shelves of mystery books and 5 shelves ofpicture books, how many books did she have in total?

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 3 / 19

2: The Problem

Arithmetic Word Problem

Gwen was organizing her book case making sure each of the shelves hadexactly 9 books on it. She has 2 types of books - mystery books andpicture books. If she had 3 shelves of mystery books and 5 shelves ofpicture books, how many books did she have in total?

2 is irrelevant

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 3 / 19

2: The Problem

Arithmetic Word Problem

Gwen was organizing her book case making sure each of the shelves hadexactly 9 books on it. She has 2 types of books - mystery books andpicture books. If she had 3 shelves of mystery books and 5 shelves ofpicture books, how many books did she have in total?

3 and 5 need to be added and then multiplied by 9

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 3 / 19

2: The Problem

Arithmetic Word Problem

Gwen was organizing her book case making sure each of the shelves hadexactly 9 books on it. She has 2 types of books - mystery books andpicture books. If she had 3 shelves of mystery books and 5 shelves ofpicture books, how many books did she have in total?

Solution Expression

(3 + 5)× 9 = 72

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 3 / 19

3: Previous Work

Restrictions on the number and type of operations used

Hosseini 2014 : Only addition, subtractionRoy 2015 (My previous work) : Exactly one operation

Relying on templates seen in training data

Kushman 2014 : Template based approach

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 4 / 19

3: Previous Work

Restrictions on the number and type of operations used

Hosseini 2014 : Only addition, subtractionRoy 2015 (My previous work) : Exactly one operation

Relying on templates seen in training data

Kushman 2014 : Template based approach

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 4 / 19

4: Contribution

Contribution

An arithmetic word problem solver with

No restriction on the number and type of operations needed.

No assumption on expression template.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 5 / 19

4: Contribution

Contribution

An arithmetic word problem solver with

No restriction on the number and type of operations needed.

No assumption on expression template.

Before this, you could not solve ...Gwen was organizing her book case making sure each of the shelves had exactly 9 books on it.

She has 2 types of books - mystery books and picture books. If she had 3 shelves of mystery

books and 5 shelves of picture books, how many books did she have in total?

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 5 / 19

4: Contribution

Contribution

An arithmetic word problem solver with

No restriction on the number and type of operations needed.

No assumption on expression template.

Before this, you could not solve ...Gwen was organizing her book case making sure each of the shelves had exactly 9 books on it.

She has 2 types of books - mystery books and picture books. If she had 3 shelves of mystery

books and 5 shelves of picture books, how many books did she have in total?

Now you can !!

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 5 / 19

5: Main Idea

Main Idea

The prediction problem can be decomposed into simple decision problems.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 6 / 19

5: Main Idea

Main Idea

The prediction problem can be decomposed into simple decision problems.

Decomposition based on lowest common ancestor (LCA) of monotonicexpression trees.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 6 / 19

6: Expression Trees

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 7 / 19

6: Expression Trees

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

3

+

5

−

97 8×

+

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 7 / 19

6: Expression Trees

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

3

+

5

−

97 8×

+

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 7 / 19

6: Expression Trees

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

LCA

3

+

5

−

97 8×

+

LCA (3, 8, first tree) = +

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 7 / 19

6: Expression Trees

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

LCA

3

+

5

−

97 8×

+

LCA (3, 8, first tree) = +

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 7 / 19

6: Expression Trees

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

LCA

3

+

5

−

97 8×

+

LCA

LCA (3, 8, first tree) = + LCA (3, 8, second tree) = −

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 7 / 19

6: Expression Trees

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

LCA

3

+

5

−

97 8×

+

LCA

LCA (3, 8, first tree) = + LCA (3, 8, second tree) = −

LCAs dependent on the tree

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 7 / 19

7: Monotonic Expression Trees

Definition

An expression tree is called monotonic if

If + node is connected to − node, then − node is the parent.

If × node is connected to ÷ node, then ÷ node is the parent.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 8 / 19

7: Monotonic Expression Trees

Definition

An expression tree is called monotonic if

If + node is connected to − node, then − node is the parent.

If × node is connected to ÷ node, then ÷ node is the parent.

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

3

+

5

−

97 8×

+

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 8 / 19

7: Monotonic Expression Trees

Definition

An expression tree is called monotonic if

If + node is connected to − node, then − node is the parent.

If × node is connected to ÷ node, then ÷ node is the parent.

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

Non-monotonic

3

+

5

−

97 8×

+

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 8 / 19

7: Monotonic Expression Trees

Definition

An expression tree is called monotonic if

If + node is connected to − node, then − node is the parent.

If × node is connected to ÷ node, then ÷ node is the parent.

Expression Trees for (3× 5) + 7− 8− 9

3

+

5

−

9

7

8

×

+

Non-monotonic

3

+

5

−

97 8×

+

Monotonic

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 8 / 19

8: Theorem Time

Generality

Every expression can be expressed by a monotonic expression tree.

Uniqueness of monotonic tree LCA

For any expression E , LCA(qi , qj , T ) will be same for any monotonicexpression tree T for expression E .

⇒ Unique value of Monotonic tree LCA for any pair of quantities.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 9 / 19

8: Theorem Time

Generality

Every expression can be expressed by a monotonic expression tree.

Uniqueness of monotonic tree LCA

For any expression E , LCA(qi , qj , T ) will be same for any monotonicexpression tree T for expression E .

⇒ Unique value of Monotonic tree LCA for any pair of quantities.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 9 / 19

8: Theorem Time

Generality

Every expression can be expressed by a monotonic expression tree.

Uniqueness of monotonic tree LCA

For any expression E , LCA(qi , qj , T ) will be same for any monotonicexpression tree T for expression E .

⇒ Unique value of Monotonic tree LCA for any pair of quantities.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 9 / 19

Part II

System Description

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 10 / 19

9: System Pipeline

Problem Text

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 11 / 19

9: System Pipeline

Problem Text

LCA ClassifierLCA Classifier

For each pair of quantitiesqi , qj , predicts LCA ofmonotonic expression tree ofsolution.

Multiclass classifier.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 11 / 19

9: System Pipeline

Problem Text

LCA Classifier

IrrelevanceClassifier

Irrelevance Classifier

For each quantity q, predictswhether it is irrelevant for thesolution.

Binary classifier.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 11 / 19

9: System Pipeline

Problem Text

LCA Classifier

IrrelevanceClassifier

Features : Quantity Schema

For each quantity, we extract

Associated Verb

Subject of Associated Verb

Unit

Related Noun Phrases

Rate

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 11 / 19

9: System Pipeline

Problem Text

LCA Classifier

IrrelevanceClassifier

Features : Quantity Schema

For each quantity, we extract

Associated Verb

Subject of Associated Verb

Unit

Related Noun Phrases

Rate

All our features are from quantityschemas.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 11 / 19

9: System Pipeline

Problem Text

LCA Classifier

IrrelevanceClassifier

Constraints

Constraints

Question about physicalobjects⇒ Positive answer

Question asks “How many. . .”⇒ Integer answer

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 11 / 19

9: System Pipeline

Problem Text

LCA Classifier

IrrelevanceClassifier

Constraints

Inference

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 11 / 19

10: Inference

Score of Trees

Score(E) = Irrelevance score +LCA score

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 12 / 19

10: Inference

Score of Trees

Score(E) = Irrelevance score +LCA score

= λ∑

q not used in E

Irr(q) +∑

qi ,qj used in E

LCA(qi , qj )

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 12 / 19

10: Inference

Score of Trees

Score(E) = Irrelevance score +LCA score

= λ∑

q not used in E

Irr(q) +∑

qi ,qj used in E

LCA(qi , qj )

Inference Problem

arg maxall trees E respecting contraints

Score(E)

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 12 / 19

Part III

Evaluation

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 13 / 19

11: Datasets

AI2 Dataset : Addition, subtraction problems

IL Dataset : Exactly one operation problems

Commoncore Dataset : Multi-step problems New !!

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 14 / 19

11: Datasets

AI2 Dataset : Addition, subtraction problems

IL Dataset : Exactly one operation problems

Commoncore Dataset : Multi-step problems New !!

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 14 / 19

12: Commoncore Dataset

600 word problems equally divided in the following types :

Addition followed by Subtraction

Subtraction followed by Addition

Addition and Multiplication

Addition and Division

Subtraction and Multiplication

Subtraction and Division

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 15 / 19

12: Commoncore Dataset

600 word problems equally divided in the following types :

Addition followed by Subtraction

Subtraction followed by Addition

Addition and Multiplication

Addition and Division

Subtraction and Multiplication

Subtraction and Division

Challenging Evaluation Setting

Test on one type by training on all other types

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 15 / 19

13: Effect of Constraints

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 16 / 19

14: Comparison with Other Systems

AI2 IL CC

Hosseini 2014 77.7 - -

Roy 2015 - 52.7 -

Kushman 2014 64.0 73.7 2.3

Table: Accuracy in correctly solving arithmetic problems

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 17 / 19

14: Comparison with Other Systems

AI2 IL CC

Hosseini 2014 77.7 - -

Roy 2015 - 52.7 -

Kushman 2014 64.0 73.7 2.3

Our System 78.0 73.9 45.2

Table: Accuracy in correctly solving arithmetic problems

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 17 / 19

15: Conclusion

We developed a solver for a general class of arithmetic word problems.

Theory of Monotonic LCA can be applied to more general settings, inparticular, to algebra word problems.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 18 / 19

15: Conclusion

We developed a solver for a general class of arithmetic word problems.

Theory of Monotonic LCA can be applied to more general settings, inparticular, to algebra word problems.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 18 / 19

16:

Thank You

Datasets available athttp://cogcomp.cs.illinois.edu/page/resource view/98.

Subhro Roy and Dan Roth Solving General Arithmetic Word Problems 19 / 19