Spoken Language Support for Software Development

1

Spoken Language Support for Software

Development

Andrew BegelAdvisor: Susan L. GrahamComputer Science Division, EECSUniversity of California, Berkeley

2

Interactive Software Development

Human-ComputerInteraction

ProgrammingLanguages

Software Engineering

Human-CentricSoftware Tools

3

Motivation

• Programmers conventionally use keyboard– Long hours at keyboard leads to higher risk of RSI

• Can speech-based programming be an alternative?• Combines an unambiguous domain

(programming) with an inherently ambiguous input modality (speech)– Great for exploring ambiguity handling in a new

context

while (counter < limit) { }

4

Programming by Voice

• My Goal– Find out how developers use code verbally. Use this to

develop a naturally verbalizable input form.– Build development environment that supports verbal

authoring, navigation, modification.• Extend conventional compiler analyses to support ambiguities

generated by speech.– Learn how developers can use voice-based programming,

and iterate design.

while counter is lessthan limit do ...

5

Challenges

Programming languages were not designed to be spoken.

Speech is inherently ambiguous. Programming tools were not designed for ambiguity.

Speech tools are poorly suited for programming tasks.

Programmers are not used to verbal software development.

6

Talk Outline

• Introduction and Motivation Programming by Voice• Program Analyses for Ambiguous Inputs• Program Navigation and Editing• Conclusion

7

Programming by Voice

for (int i = 0; i < 10; i++ ) { ▌}

8

Current Tools are Awkward!

for (▌ ; ; ) { }

VoiceCode[Desilets 2004]

1234

for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace

9

for (int i = 0;▌ ; ) { }

1234

i


1234

i



10

for (int i = 0; i < 10;▌ ) { }


1234

i1234

i



11

for (int i = 0; i < 10; i++ ) { ▌}

1234

i


1234

i1234

i



12

Programming by Voice Related Work

Human-CentricComputer-Centric

MultipleTasks

AuthoringOnly

Arnold ‘00

Snell ‘00

Price ‘00 ‘02

Desilets ‘01 ‘04

Gray ‘03

Begel ‘05

13

How do Programmers Speak Code?

• 10 programmers read Java code out loud (Begel ‘05)

• Most programmers spoke the same way

14

for int i equals zero i less than ten i plus plus

for (int i = 0; i < 10; i++ ) { ▌}


15

Awkwardness by Design (Structural)

(int)foo

(3 + 5) * 7


16


Individual Inconsistency

System.out.println vs. System out println

bar sub i vs. bar of i vs. i from bar

17


Native English speakers vs. non-native speakers (Pronunciation)

tur vs. t u r

println

array[i++] vs. array[i]++

18

A Natural Way to Code

for int i equals zero i less than ten i plus plus

for (int i = 0; i < 10; i++ ) { ▌}

19

Too Many Ambiguities

for (int i = 0; i < 10; i++ ) { ▌}

4 int eye equals 0 aye less then ten i plus plus

KW or #?Spelling of ID?

KW or ID?for int i equals zero i less than ten i plus plus

20

Sometimes It’s Non-Obvious

for (times = 8; file(2, load); times == one) {▌

}

for times equals 8 file 2 load times equals one

fore *= 8; file.tooLode.times = won ▌

4; times = ate(file).to(load).equals(1) ▌

21

Design Tradeoffs

CommandLanguage

Easy to analyze,but prescriptive

NaturalLanguage

Flexible,but ambiguous

Programmingby Voice

22

Spoken Java• Semantically identical to Java• Syntactically easier to say than Java

– Methodology generalizable to any computer language

1. All punctuation has English equivalents• Open Brace, End For Loop

2. Most punctuation is optional3. Provide verbalization for all abbreviations4. Relaxed phrasing for better fit with English

• (int)foo “cast foo to integer”• foo = 6 “set foo to 6”• foo[i]++ “increment the ith element of array foo”

23

SPEED: Speech Editor• Build an editor that supports naturally verbalized

programs• SPEED: SPEech EDitor

• Based on IBM ViaVoice, Eclipse IDE, Harmonia

– Spoken Java Language for Composition– Spoken Command language for Navigation,

Editing, Template instantiation, Refactorings, Search

– Audible and visual feedback• Similar to JavaSpeak (Smith 2000)

24

Harmonia Analysis Framework

• Framework to support interactive editors– Language-based, programmer-oriented tools

• Incremental analyses– Lexing (Wagner ‘97), GLR Parsing (Wagner ‘97, Begel ‘04), Static

Semantics (Garrison ‘87, Begel, Jamison)

• C, Java, Titanium, Cool, Flex, Bison– Also, languages where indentation and CRs are

significant• Interactive Program Transformations (Java)• CodeLink (Toomim et. al. ‘04)

• Shorthand Editing

25

Talk Outline

• Introduction and Motivation• Programming by Voice Program Analyses for Ambiguous Inputs • Program Navigation and Editing• Conclusion

26

Traditional Compiler Analyses

LexicalAnalysis

FOR I

Parsing SemanticAnalysis

for (i = 0; i < 10; i++ ) { }

Programming languages are designed to be unambiguous

For Loop

FOR Assign Expr

I = 0

i LocalVar int

27

Ambiguity-Aware Analysesfor i equals zero ...

Handles input stream, syntactic and semantic ambiguities

LexicalAnalysis

FOR I

AmbiguousParsing

SemanticAmbiguityResolution

For Loop

FOR Assign Expr

I = 0

i LocalVar int

foureye

LocalVar ?4 EYE FOUREYE

Assign Expr

= 0

Ambig Stmt

28

Scan Input Stream

CommercialSpeech

Recognizer

HomophoneDictionary Lexical

Analysis

29

Homophones Cause Ambiguities

for i equals

for i =

4 i equals

foreeye ==

fore ayeequals foureyeequals

foriequals

Concatenated words cause them too

4

forefour

eyeaye

===

30


LexicalAnalysis

FOR I

XGLRAmbiguous

Parsing


For Loop

FOR Assign Expr

I = 0

i LocalVar int

foureye


Assign Expr

= 0

Ambig Stmt

31

FOOID

GLR Parsing [ Tomita 85 ]

ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

32

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

33

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

34

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

35

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

36

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

37

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

FOOID

IFKW

ExprIF

KW

Expr

38

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

FOOID

IFKW

ExprIF

KW

Expr

Stmt

39


ELSEKW

BARID

ELSEKW

FOOID

IFKW

ExprIF

KW

Expr

FOOID

IFKW

ExprIF

KW

Expr

Stmt

40


ELSEKW

BARID

ELSEKW

BARID

FOOID

IFKW

ExprIF

KW

Expr

FOOID

IFKW

ExprIF

KW

Expr

Stmt

41

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

Stmt

ELSEKW

BARID

FOOID

IFKW

ExprIF

KW

Expr

Stmt

42

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

Stmt

ELSEKW

BARID

FOOID

IFKW

ExprIF

KW

Expr

Stmt

Stmt

43

FOOID


ELSEKW

IFKW

ExprIF

KW

ExprBAR

ID

Stmt

ELSEKW

BARID

FOOID

IFKW

ExprIF

KW

Expr

Stmt

Stmt

Stmt

44

GLR Ambiguity Support

1. Fork when there is more than one possible interpretation (tree)

45

XGLR Ambiguity Support

1. Fork when there is more than one possible interpretation (tree)

2. Fork when the lexical input is ambiguous

46

XGLR Parsing [Begel 04]

< XIF FIFTY FIVE

47

XGLR Parsing [ Begel 04 ]

< XIF FIFTY FIVEKW

48


< XFIFTY FIVEIFKW

49


< XFIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

< X

< X

< X

< X

50


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

5

ID

ID

#

50#

55#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

< X

< X

< X

< X

< XOp

51


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

52


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

FIVE

5

FIVE

5#

ID

#

ID< X

< X

< X

< X

53


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

FIVE

5

FIVE

5#

ID

#

ID< X

< X

< X

< X

54


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

FIVE

5

FIVE

5#

ID

#

ID< X

< X

< X

< X

55


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

56


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID . FIVE

ID

FIFTYIFKW ID . FIVE

ID .

.(

(

< X

< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

57


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID . FIVE

ID

FIFTYIFKW ID . FIVE

ID .

.(

(

< X

< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

Op

Op

Op

Op

ID

58


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID . FIVE

ID

FIFTYIFKW ID . FIVE

ID .

.(

(

< X

< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

Op

Op

Op

Op

ID

59


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

60


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

ID

ID

ID

ID

ID

61


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

ID

ID

ID

ID

ID

62


55#

IFKW

FIFTYIFKW ID .

X<Op

FIVEID

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

ID

ID

ID

ID

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID< X

< X

Op

Op

ID

ID

63


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

ID

ID

ID

ID

ID

Expr

ExprExpr

ExprFuncCall

ExprFuncCall

64

XGLR Summary

• Generalization of traditional GLR algorithm– Forks on structural and lexical ambiguity– Preserves subtree sharing when parses have

different yields– Preserves efficiency when parses get out of sync

• Determine parse position w.r.t. ambiguous input

• Blender: Combined lexer and parser generator for XGLR

65

GLR Parsing Genealogy

Tomita1985

Farshi1991

Rekers1992

Wagner1997

Visser1997

van den Brand2002

Begel2004

Incremental

Scannerless

Johnstone et. al.2002

Input StreamAmbiguities

66


LexicalAnalysis

FOR I

XGLRAmbiguous

Parsing


For Loop

FOR Assign Expr

I = 0

i LocalVar int

foureye


Assign Expr

= 0

Ambig Stmt

67

Disambiguation Exampleclass Loader {

public void load() { String filetoload = null;InputStream stream =

getStream();... ▌

}}

file to load equals stream dot read string

filetoload = stream.readString();

68

Many Interpretations

file(2, load)

file(to, load) file(to.lode) file(to(lode)) file(toload)

(file, 2, load)

file.to.load file.to(load)

file.toload filetoload() filetoload

69

Incremental Semantics• What does this name mean? • What names are visible at this program

point?– Or, What can I say here?

• Visibility Graph [Garrison 1987]– Incrementally updated data structure for scopes, names

and bindings– Designed Visibility Graph algorithms for name

propagation and incremental update– Used for type checking, too

• Doesn’t <insert favorite IDE here> do this?

70

Program Context Can Helpclass Loader {

public void load() {String filetoload = null;InputStream stream =

getStream();... ▌

}}

class Loaderscope

[ load, Method, () void ]

method loadscope

[ filetoload, LocalVar, String ][ stream, LocalVar, InputStream ]

71

Program Context Can Helpclass Loader {

public void load() {String filetoload = null;InputStream stream =

getStream();... ▌

}}

class Loaderscope


method loadscope

[ filetoload, LocalVar, String ][ stream, LocalVar, InputStream ][ load, Method, () void ]

72

Semantic Disambiguation

file(2, load)


(file, 2, load)

file.to.load

file.to(load)


class Loaderscope


method loadscope


73


file(2, load)


(file, 2, load)

file.to.load

file.to(load)


class Loaderscope


method loadscope

Is “file” a visible variable name?


74


file(2, load)


filetoload() filetoload

class Loaderscope


method loadscope

Is “file” a visible method name?


75


filetoload() filetoload

class Loaderscope


method loadscope

Is “filetoload” a visible method name?


76

Manual Disambiguation• Some ambiguities cannot (and should not) be

automatically resolved:

print(“line”) vs. println()

if (pred1) then if (pred2) then foo() else bar()

• If ambiguities remain, ask the user how to resolve them. (e.g. [Mankoff 00])

if

foo()

if

bar()

if

if

foo()

bar()

77

Talk Outline

• Introduction and Motivation• Programming by Voice• Program Analyses for Ambiguous Inputs Program Navigation and Editing• Conclusion

78

Study - Navigation by Speech1. Eight navigation tasks with commercial VR tools2. Search through text (w/o Find dialog), e.g.

Find the sentence where Romeo cries out about his fate after killing

Tybalt.Metrics

1. Time to scroll to right page2. Number of commands3. Number of recognition

errors4. Number of system mistakes

Results1. Cognitive load is too high -

too many commands, misestimation errors

2. Voice recognition induces too much delay/errors for accurate control

79

SPEED Navigation

• Shorthand Editingclass Loader {

public void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”;

▌ }public void save() {

...}

}class MenuLoader extends Loader {

...}

80

SPEED Navigation

• Shorthand Editingclass Loader {

public void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”;


...}


...}

81

SPEED Navigation

• Shorthand Editingdown

class Loader {public void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”;


...}


...}

82

SPEED Navigation

• Shorthand Editingprotected

class Loader {protected void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”;


...}


...}

83

SPEED Navigation

• Shorthand Editingprivate

class Loader {private void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”;


...}


...}

84

SPEED Navigation

• Shorthand Editingdown

class Loader {private void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”;


...}


...}

85

SPEED Navigation

• Shorthand Editingpush down

class Loader {private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”;


...}


...}

86

SPEED Navigation

• Shorthand Editing • Context-Sensitive

Mouse Grid [Begel]– Related Work: Tree

Hierarchy Navigation [Smith 2004]



...}


...}

87

SPEED Navigation






...}


...}

1

2

88

SPEED Navigation




1



...}


...}

1

2

89

SPEED Navigation




1 1



...}


...}

1

32

4

90

SPEED Navigation




1 1 3 select



...}


...}

91

SPEED Editing




edit



...}


...}

filetoload equals quote file.txt quote

92

SPEED Editing




edit



...}


...}


93

SPEED Editing




put it back

class Loader {private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = stream.readString();


...}


...}


94

Study - SPEED Usability

Goal: Understand how SPEED can be used by expert programmers

1. Train expert Java programmers on SPEED2. Author new code

– Build a Linked List data structure with associated algorithms

3. Modify existing code– Change abstraction representation and update algorithms

95

Metrics• User Metrics

– Speed– Vocabulary mistakes, Forgotten vocabulary– Grammatical mistakes, Grammatical

substitutions• SPEED Metrics

– Word tokenization errors– Disambiguation errors– Irresolvable ambiguities

96

Talk Outline

• Introduction and Motivation• Programming by Voice• Program Analyses for Ambiguous Inputs• Program Navigation and Editing Conclusion

97

Contributions

1. A study of programmers to understand and design a naturally verbalizable input for programming

2. An interactive editor designed for spoken interaction

3. The use of syntax and semantics of programming for disambiguation– Enhanced lexical, syntactic, semantic analyses for

support of verbal ambiguities4. Evaluation of design and tools by studying

programmers using voice for software development

98

Future of Programming by Voice1. Improved automation of semantic disambiguation

– Use ideas from NLP, Machine Learning (team styles)

2. Early pruning of ambiguities using analysis feedback3. Higher-level linguistic programming tools

– Transformations, Paraphrasing– Phonetic search, Audible feedback

4. Support more software engineering tasks by voice– Debuggers, IDEs, Comments, Code reviews

5. Design spoken variants of other formal languages– General (C, C#) Scripting (PL, OS), Design (HCI),

Command (Robotics), Domain-specific languages (SQL)

99

Future Research Directions• Data mining the program interaction history

– Task inference, reminders, coding guides• Improving non-code programmer communication

– Design history, Code orientation, Software spelunking • Programming in the “Large”

– Multiple hi-res displays, whiteboards• Visualizing effects of compilation and optimization

– Scalable task-oriented program visualization• Scalable Debugging

– Concurrent apps, long-lived apps

100

Any Questions?

Andrew Begel: [email protected]

Spoken Language Support for Software Development

Documents

Transcript of Spoken Language Support for Software Development