Natural Human to Machine Communication

By Alon Eirew

Who Am I

• Software Architect at Intel Multimodal Language Understanding Group• Smart Home, Robotics, Smart Car, Mobile & Wearables

• Lead Developer at Intel New Devices Group• Personal Assistant for Mobile and Wearables

• Lead Developer Telmap (Navigation App)• Navigation Assistant for Mobile

https://il.linkedin.com/in/aloneirew

3

Agenda

What is Natural Communication?

Why is it interesting?

Overview the components and see some examples

4

Human to Human

Natural Communication Feature

Change the way we are communicating with our smart devices

8

Technologies that can’t support conventional input sources

The Era Of Conversational UI

How can I help you?

10

Building Blocks

Hardware

Recognition

Understanding

1

2

3

11

Hardware and Recognition Modules

Automatic Sound Recognition

Speech To Text

Computer VisionObject RecognitionPerson RecognitionGesture Recognition

Intent Extraction (NLU)

12

Intent extraction is the process of understanding commands, requests or promises conveyed in

human language.

Please turn off the lights in the living

room at 4PM

Location: living room

Time: 4PM

Object: lights

Action: turn off

Subject: (you)

World Knowledge

13

User Knowledge

14

15

Knowledge Use Cases

Semantic Fusion

Object [1] [N-Best]

Label Prob Location

Chair 0.7x1, y1, z1

Stool 0.4

Table 0.2

Label Prob Location

Table 0.8 x4, y4, z4

Stool 0.3 x4, y4, z4

Object 0.2 x4, y4, z4

Object [2] [N-Best]

Gesture List

Object Rec 1

Object Rec 2

End Point

Clean next to

this ???? ????

Word Prob

School Desk 0.7

Ship Deck 0.5

Sport Deko 0.2

ASR [N-Best]

Object [3] [N-Best]

Label Prob Location

Plant 0.7x7, y7, z7

Tree 0.4

Object 0.2

Dialog Features

17

Thank You!