The Connected World and Speech Technologies: It’s About “Effortless”
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
Jay Wilpon
Executive Director, Natural Language Processing and Multimodal Interactions Research
IEEE Fellow, AT&T Fellow
Jay Wilpon – 7/7/2014
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 2
“In the Beginning…” Genesis 1:1
Jay Wilpon – 7/7/2014
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
3 Front view of the spoken digit recognizer (J. Suzuki & K. Nakata, Radio Research Labs, Japan, 1961)
Jay Wilpon – 7/7/2014
“It would be too simple to say that work in speech recognition is carried out simply because one can get money for it. . . .The attraction is perhaps similar to the attraction of schemes for
turning water into gasoline, extracting gold from the sea, curing cancer, or going to the moon”
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 4
John R. Pierce, Executive Vice President at AT&T Bell Labs (1969)
Jay Wilpon – 7/7/2014
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 5
Good Morning America – 1987
Jay Wilpon – 7/7/2014
Virtual Assistants – Siri / Google Now
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 6
Start at 37 seconds
Jay Wilpon – 7/7/2014
AT&T Evolution of Speech Industry Evolution
1990 2000
Simple Commands
• Command & Control
(“Press or say”)
“Please say…”
“Collect call please”
“The New Frontier”
2010
Invention of Industrial Leading Technologies & Services “The New Frontier”
Beyond 1980
Pioneered speech service industry
Intelligent Multimodal Services
• Home automation
• Connected car
• Cloud API
• Big data analytics
• Emerging devices, wearables
• M2M
• Video and television
“You are running late for your meeting and you need gas”
“Thanks, can you find a good restaurant for Sunday brunch”
Seamless communications, information access, commerce –
Connected Life
Large-vocabulary
Speech Services
• Operator services
• Enterprise services
Natural Language
Services
• “How May I Help You”
Customer care
• Enterprise services
• Watson Speech mashup
Resulted in billions of dollars in savings
Paradigm shift in user experience and automation
“AT&T, how may I help you?”
“I have a problem with my bill”
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
Jay Wilpon – 7/7/2014
Devices
App Developers
Industries Connected Car
Home Automation
Customer Service
Healthcare
Finance
Consumer Applications
Machine Learning
Dialog Manager
Video Language Processing
Text to Speech
Speech Recognition
Voice Biometrics
3rd Party Developers
Solution Providers Systems
Integrators
Large Enterprises
Speech and multi-modal technologies universally accessible to all devices and industries through a wide ecosystem of partners
Connected World Ecosystem
Cloud Providers
Security
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
Jay Wilpon – 7/7/2014
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
Multimodal Virtual Assistants
• Cloud-based virtual assistant manifests itself appropriately based on the device and environment
- Personalized - Adaptive - Connected
• Natural and effective
- Spoken dialog - Language
understanding - Multimodal interaction
Web
Office
Social Media
Retail
Emerging Devices
Connected Car
Digital Home
Personal Data
Effortless access to information, services, transactions – anywhere, anytime
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
The Science Behind the Service
Speech and Language
Technology Suite
Natural Language
Processing
Speech Recognition
Dialog
Text to Speech
Language Generation
Biometrics
Speech Translation
Machine Learning
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
Speech Industry is Approaching a Cross Road
Market Trends Market Challenges
Hello
Hola
• Growth in connected devices (smartphones, tablets,
connected cars, wearables, home automation)
• Increased adoption of and access to cloud computing
• Desire to enhance and differentiate the consumer
experience (virtual assistants, multimodal solutions)
• Demand for business intelligence uses increasing
(speech, video, text)
Low customer adoption due to poor user experience
and performance
Low accuracy “in the real world”
Complexity due to variety of languages and devices
Competing / complementary input technologies
(touchscreen, keyboard, multi-touch and visual
gestures)
Slow development/adaption of applications
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
Its all about “Effortless”
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 13
• Scientists / Engineers
• Software vendors
• Platform vendors
• Application developers
• Customers
Jay Wilpon – 7/7/2014
Speech/Language Industry Ecosystem - Today
• Multiple code bases • Multiple version • Distribution • Integration • Testing
Expensive, Inefficient and Slow
Software Vendors
• Hardware management • Integration • Optimization • Testing • Version code • Prof. Services • Distribution • (Oh, did I add testing?)
Expensive and Slow
Platform Vendors
• Software maintenance • Platform dependence • Testing • Optimization • Version code • Prof. Services
Slow
App Developers Invention
• Ideas • Experiments • Trial • Ideas • Experiments • Trial • …
“Inspiration time”
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
Speech/Language Industry Ecosystem - Today
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 15
• Software maintenance • Platform dependence • Testing • Optimization • Version code • Prof. Services
Slow
App Developers Invention
• Ideas • Experiments • Trial • Ideas • Experiments • Trial • …
“Inspiration time”
Jay Wilpon – 7/7/2014
Its all about “Effortless” – Invert the Food Chain
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 16
• Customers – Intuitive, useful, secure, personalized, “anywhere, anytime”, device agnostic
• Application developers – technologies easy to integrate, scalable tools, simple interfaces, limited need for professional services
• Platform providers – technologies easy to integrate, interfaces simple and stable, tools
• Software Vendors – platform independence, scalable engines, tools, simple and stable interfaces
• Scientists / Engines – “it works”, fast to market, platform semi-independence, easy to update
Getting to ”Effortless” – Start with the Customer
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 17
• Software maintenance • Platform dependence • Testing • Optimization • Version code • Prof. Services
Fast
App Developers Invention
• Ideas • Experiments • Trial • Ideas • Experiments • Trial • …
“Inspiration time”
Innovative, Fast, Device Agnostic
Jay Wilpon – 7/7/2014
Page 18
Cloud Computing
Bandwidth
Network Connectivity
Jay Wilpon – 7/7/2014
Speech/Language Industry Ecosystem - Moving
19
Cloud Engines APIs and libraries
• “One” customer • “Single” code base • Low maintenance • Simple testing • Virtual ‘on-demand’ machines • Optimized • Personalization • Pricing << Software licensing
Fast
Innovative, Fast, Device Agnostic
Invention
• Ideas • Experiments • Trial • Ideas • Experiments • Trial • …
“Inspiration time”
• Software maintenance • Platform dependence • Testing • Optimization • Version code • Prof. Services
Fast
App Developers
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
Aggressive Market Segmentation
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 20
Google Apple Microsoft Nuance Amazon AT&T
Device/OS dependent
Speech APIs
Speech APIs
Device/OS independent
Google Now
Cortana Siri Kindle Nina mVA
Android Windows iOS Android “All” “All”
Jay Wilpon – 7/7/2014
Getting to ”Effortless” – Independence
21
Cloud Engines APIs and Libraries
Android Windows iOS Linux Other
Application Developers
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
Jay Wilpon – 7/7/2014
Getting to ”Effortless” – Scientists and Engineers
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 22
Are we in the … ?
“… Golden Age of Engineering for Speech Technologies”
“… Age of Invention for Language Technologies”
Data is driving innovation…
Jay Wilpon – 7/7/2014
Technical Challenges to Move Industry Forward
1990 2000
Simple Commands
• Command & Control
(“Press or say”)
“Please say…”
“Collect call please”
“The New Frontier”
2010
Invention of Industrial Leading Technologies & Services “The New Frontier”
Beyond 1980
Pioneered speech service industry
Intelligent Multimodal Services
• Home automation
• Connected car
• Cloud API
• Big data analytics
• Emerging devices, wearables
• M2M
• Video and television
“You are running late for your meeting and you need gas”
“Thanks, can you find a good restaurant for Sunday brunch”
Seamless communications, information access, commerce –
Connected Life
Large-vocabulary
Speech Services
• Operator services
• Enterprise services
Natural Language
Services
• “How May I Help You”
Customer care
• Enterprise services
• Speech mashup
Resulted in billions of dollars in savings
Paradigm shift in user experience and automation
“AT&T, how may I help you?”
“I have a problem with my bill”
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
Jay Wilpon – 7/7/2014
Seamless communications, information access and commerce – anywhere & anytime
• Phones/tablets/PCs: unified access to communications, information and transaction
• Enterprise customer care moving from IVR to mobile web
• E-commerce/retail sales
• Connected car & home
• Entertainment: easy access to content everywhere
• Wearable devices: new industry, hands free control
Market Opportunities
Multimodal Virtual Assistants
Customer Care
Entertainment
Enterprise Verticals
Wearables Home Automation
Connected Car
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
Seamless communications, information access and commerce – anywhere & anytime
• It has to work: Everywhere, be intuitive, be available whenever needed
• Always on: Handsfree, ROBUST, speech detection, sensors everywhere
• Multimodality: Connected devices need multimodal technologies
• Context-awareness: Pragmatics of the world around each user
• Conversational interaction: Effortless user experience require cooperative, personalized & multimodal human-machine dialog
• Secure and Personalized: Knows “its me” and learns about “me” – adaptive
• Proactive: Reasoning, M2M, not just Q/A
• Incrementality: Users expect fluid/engaging interaction
• Rapid application authoring: Enterprises demand easy ‘cold start’ for new domains and rapid support for ongoing improvements (e.g., tools); unsupervised (semi-supervised) training
• Internationalization: multilingual, translation
• Data, data, data: Vast amounts (content, personal, dynamic) – ingest, model, adapt, learn
Challenges Addressed by Scientists/Engineers
Multimodal Virtual Assistants – Getting to Effortless
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 26
AT&T Watson Multimodal Intelligent Virtual Assistant
Jay Wilpon – 7/7/2014
AT&T Watson Multimodal Intelligent Virtual Assistant Planning a “night out on the town” – events, movies and restaurants
Multimodal input and output – “effortless”
– Supports recognition, understanding, generation and
gesture
– Integration of ASR, NLP and Dialog maximizes performance
– Natural conversation supports contextually-aware NLU and
reference resolution
– Supports Targeted Clarification: error handling through
machine learning and natural language generation
Built for rapid authoring and adaptation
– Dynamic hierarchical language models (DHLM) enable ‘cold
start’ for new domains and rapid adaptation
– Bootstrap new domains with generic NLP pipeline
– Watson’s Semantic Abstraction Layer (SAL) facilitates easy
expansion to new domains and new content
Watson VA Features mVA Multimodal Concierge
19
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
AT&T Translator – TOUCH (Fox TV Show)
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 28 Jay Wilpon – 7/7/2014
Driving to Intelligence
Data
Natural Language
Processing
Speech Recognition
Dialog
Text to Speech
Language Generation
Biometrics
Speech Translation
Machine Learning
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Jay Wilpon – 7/7/2014
Natural Language Processing (NLP)
NLP
Analysis
Intent classification – semantics
Named entities
Part-of-speech
Syntactic parse
Coreference labels
Text normalization
Sentiments
Topic detection
Information extraction
Emotion detection
Anomaly detection
Summarization
Sentence planning
System intent
Semantic frames
NLP
Generation
Jay Wilpon – 7/7/2014
Intelligence – Information extraction
This authentic Venezuelan food is delicious homemade and great on price and service
Input sentence
Classify
Identify triggers
Identify related targets
Summarize
Sentiment/No Sentiment
This authentic Venezuelan food is delicious homemade and great on price and service
This authentic Venezuelan food is
delicious homemade and great on
price and service
Aspect/attribute Opinion Venezuelan food authentic
delicious homemade
Venezuelan food: Price great
Venezuelan food: Service
Sentiment analysis
My phone is not calling or receiving messages This girl told me Verizon is down
Problem/No Problem
My phone is not calling or receiving messages This girl told me Verizon is down
My phone is not calling or receiving
messages This girl told me Verizon is
down
Problem object Problem My Phone not calling or receiving
Verizon down
Problem Detection Basic Steps
Jay Wilpon – 7/7/2014
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 35 Jay Wilpon – 7/7/2014
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 36
“In the Beginning…” Genesis 1:1
Jay Wilpon – 7/7/2014
Rethink Possible
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
Jay Wilpon – 7/7/2014
Top Related