Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile...

www.AgeofMobility.com

http://www.bigbaldapps.com

@ActiveNick 2005-2014

Computer Speech is the Dream

Mobile Speech is a Responsibility

AccessibilityMedical

Conditions

Hands &

Eyes Free

Driving Apps

& Games

Learning &

Therapy

Travel Apps

Translation

Speech SDK

Features for DevsWindows Phone Windows Store

(iPhone/iPad)

Android

(Phone/Tablet)

Built-in Personal

Assistant with

Speech

Cortana* N/A Siri* Google Now

Launcher*

Personal Assistant

Extensions in third-

party apps using

Voice Commands

Yes N/A No

(first-party apps

Speech Synthesis

SDK for Devs

(Text-to-Speech)

Yes Yes Yes Yes

In-app Speech

Recognition SDK

for Devs

Yes Yes* No* Yes

<?xml version="1.0" encoding="utf-8"?><VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.1">

<CommandPrefix>MSDN</CommandPrefix><Example>How do I add Voice Commands to my application</Example>

<Command Name="FindText"><Example>Find Install Voice Command Sets</Example><ListenFor>Search</ListenFor><ListenFor>Search for {dictatedSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {dictatedSearchTerms}</ListenFor><Feedback>Search on MSDN</Feedback><Navigate Target="MainPage.xaml" />

</Command>

<Command Name="nlpCommand"><Example>How do I add Voice Commands to my application</Example><ListenFor>{dictatedVoiceCommandText}</ListenFor><Feedback>Starting MSDN...</Feedback><Navigate Target="MainPage.xaml" />

</Command>

</PhraseTopic>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

<Example>How do I add Voice Commands to my application</Example>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

</Command>

/Command>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

<Example>Find Install Voice Command Sets</Example>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

<ListenFor>Search</ListenFor><ListenFor>Search for {dictatedSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {dictatedSearchTerms}</ListenFor>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

<Feedback>Search on MSDN</Feedback>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

</PhraseTopic>

<CommandPrefix>MSDN</CommandPrefix><Example>Find Voice Commands</Example>

<Command Name="FindText"><Example>Find Windows Phone</Example><ListenFor>Search</ListenFor><ListenFor>Search {*}</ListenFor><ListenFor>Search for {listSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {*}</ListenFor><ListenFor>Find {listSearchTerms}</ListenFor><Feedback>Search on MSDN</Feedback><Navigate Target="MainPage.xaml" />

</Command>

<PhraseList Label="listSearchTerms" Disambiguate="false"><Item>Voice Commands</Item><Item>Windows Phone</Item>

</PhraseList>

</CommandSet>

</VoiceCommands>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

Windows Phone 8.0 Windows Phone 8.1

Simple Scenarios, Limited Accuracy More Powerful, Easier, More Accurate

<CommandPrefix>MSDN</CommandPrefix><Example>Find Voice Commands</Example>

<Command Name="FindText"><Example>Find Windows Phone</Example><ListenFor>Search</ListenFor><ListenFor>Search {*}</ListenFor><ListenFor>Search for {listSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {*}</ListenFor><ListenFor>Find {listSearchTerms}</ListenFor><Feedback>Search on MSDN</Feedback><Navigate Target="MainPage.xaml" />

</Command>

<PhraseList Label="listSearchTerms" Disambiguate="false"><Item>Voice Commands</Item><Item>Windows Phone</Item>

</PhraseList>

</CommandSet>

</VoiceCommands>

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

Windows Phone 8.0 Windows Phone 8.1

Simple Scenarios, Limited Accuracy … and enables Natural Language

</Command>

</PhraseTopic>

</CommandSet>

</VoiceCommands>

private async void

// SHOULD BE PERFORMED UNDER TRY/CATCH

Uri new ms-appx:///vcd.xml UriKind.Absolute

Windows Phone Silverlight Apps on Windows Phone 8.1

Windows Runtime Apps on Windows Phone 8.1

private async void

// SHOULD BE PERFORMED UNDER TRY/CATCH

Uri uriVoiceCommands = new Uri("ms-appx:///vcd.xml", UriKind.Absolute);

StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(uriVoiceCommands);

await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(file);

// Windows Phone Silverlight App, in MainPage.xaml.cs

protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)

base.OnNavigatedTo(e);

if (e.NavigationMode == System.Windows.Navigation.NavigationMode.New)

string recoText = null; // What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"

NavigationContext.QueryString.TryGetValue("reco", out recoText);

string voiceCommandName = null; // Which command was recognized in the VCD.XML file? e.g. "FindText"

NavigationContext.QueryString.TryGetValue("voiceCommandName", out voiceCommandName);

string searchTerms = null; // What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"

NavigationContext.QueryString.TryGetValue("dictatedSearchTerms", out searchTerms);

switch (voiceCommandName) // What command launched the app?

case "FindText":

HandleFindText(searchTerms);

break;

case "nlpCommand":

HandleNlpCommand(recoText);

break;

case "FindText":

break;

case "nlpCommand":

break;

case "FindText":

break;

case "nlpCommand":

break;

// What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"string recoText = null;NavigationContext.QueryString.TryGetValue("reco", out recoText);

// Which command was recognized in the VCD.XML file? e.g. "FindText"string voiceCommandName = null; NavigationContext.QueryString.TryGetValue("voiceCommandName", out voiceCommandName);

// What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"string searchTerms = null;NavigationContext.QueryString.TryGetValue("dictatedSearchTerms", out searchTerms);

case "FindText":

break;

case "nlpCommand":

break;

switch (voiceCommandName) // What command launched the app?{

case "FindText":HandleFindText(searchTerms);break;

case "nlpCommand":HandleNlpCommand(recoText);break;

// Windows Runtime App on Windows Phone 8.1, inside OnActivated override in App class

if (args.Kind == ActivationKind.VoiceCommand)

VoiceCommandActivatedEventArgs vcArgs = (VoiceCommandActivatedEventArgs)args;

string voiceCommandName = vcArgs.Result.RulePath.First(); // What command launched the app?

switch (voiceCommandName) // Navigate to right page for the voice command

case "FindText": // User said "find" or "search"

rootFrame.Navigate(typeof(MSDN.FindText), vcArgs.Result);

break;

case "nlpCommand": // User said something else

rootFrame.Navigate(typeof(MSDN.NlpCommand), vcArgs.Result);

break;

// Windows Runtime App on Windows Phone 8.1, inside OnActivated override// in App class

break;

// What command launched the app?string voiceCommandName = vcArgs.Result.RulePath.First();

// Navigate to right page for the voice command switch (voiceCommandName)

break;

// Navigate to right page for the voice command switch (voiceCommandName) {

case "FindText": // User said "find" or "search"rootFrame.Navigate(typeof(MSDN.FindText), vcArgs.Result);break;

case "nlpCommand": // User said something elserootFrame.Navigate(typeof(MSDN.NlpCommand), vcArgs.Result);break;

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo in FindText.xaml.cs

protected override void OnNavigatedTo(NavigationEventArgs e)

// Get recognition result from parameter passed in frame.Navigate call

SpeechRecognitionResult vcResult = e.Parameter as SpeechRecognitionResult;

if (vcResult!=null)

// What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"

string recoText = vcResult.Text;

// Store the semantics dictionary for later use

IReadOnlyDictionary<string,IReadOnlyList<string>> semantics = vcResult.SemanticInterpretation.Properties;

string voiceCommandName = vcResult.RulePath.First();

if (voiceCommandName == "FindText")

// What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"

if (semantics.ContainsKey("dictatedSearchTerms"))

HandleFindTextWithSearchTerms(semantics["dictatedSearchTerms"][0]);

HandleNoSearchTerms();

// Else handle other voice commands

} navigationHelper.OnNavigatedTo(e)

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In FindText.xaml.cs

if (vcResult!=null)

// Windows Runtime App on Windows Phone 8.1, inside // OnNavigatedTo in FindText.xaml.csprotected override void OnNavigatedTo(NavigationEventArgs e){

// Get recognition result from parameter passed in frame.Navigate callSpeechRecognitionResult vcResult = e.Parameter as SpeechRecognitionResult;

if (vcResult!=null){

// What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"string recoText = vcResult.Text;

// Store the semantics dictionary for later useIReadOnlyDictionary<string,IReadOnlyList<string>> semantics =

vcResult.SemanticInterpretation.Properties;

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In FindText.xaml.cs

if (vcResult!=null)

if (voiceCommandName == "FindText"){

// What did the user say, for named phrase topic or list "slots"?// e.g. "Windows Phone Voice Commands"if (semantics.ContainsKey("dictatedSearchTerms")){

HandleFindTextWithSearchTerms(semantics["dictatedSearchTerms"][0]);}else{

HandleNoSearchTerms();}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo in NlpCommand.xaml.cs

// Check for null!

string commandMode = vcResult.SemanticInterpretation.Properties["commandMode"][0];

if (commandMode == "voice") // Did the user speak or type the command?

SpeakText(audioPlayer, String.Format("MSDN app heard you say {0}", vcResult.Text));

HandleNlpCommand(vcResult);

else if(commandMode=="text")

messageTextBox.Text = string.Format("Working on your request \"{0}\"", vcResult.Text);

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In NlpCommand.xaml.cs

// Check for null!

// Windows Runtime App on Windows Phone 8.1, in OnNavigatedTo in NlpCommand.xaml.cs

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In NlpCommand.xaml.cs

// Check for null!

SpeakText(audioPlayer, String.Format("MSDN voice heard you say {0}", vcResult.Text));

msgTextBox.Text= string.Format("Working on your request \"{0}\"", vcResult.Text);

private void HandleNlpCommand(string recoText, bool actSilently)

string action = null;

string navigateTo = null;

string searchFor = null;

recoText = recoText.ToLower();

if (recoText.Contains("go to ") || recoText.Contains("goto ") ||

recoText.Contains("find ") || recoText.Contains("search ") ||

recoText.Contains("show me "))

action = "navigate";

if (recoText.Contains("windows phone dev center"))

navigateTo = "http://dev.windowsphone.com";

else if (recoText.Contains("learn how to "))

action = "find";

searchFor = recoText.Substring(recoText.IndexOf("learn how to ") + 13);

action = "find";

searchFor = recoText;

switch (action)

case "find":

// ...

action = "find";

switch (action)

case "find":

// ...

if (recoText.Contains("go to ") || recoText.Contains("goto ") ||recoText.Contains("find ") || recoText.Contains("search ") ||recoText.Contains("show me "))

{action = "navigate";if (recoText.Contains("windows phone dev center")){

navigateTo = "http://dev.windowsphone.com";}

action = "find";

switch (action)

case "find":

// ...

else if (recoText.Contains("learn how to ")){

action = "find";searchFor = recoText.Substring(recoText.IndexOf("learn how to ") + 13);

action = "find";

switch (action)

case "find":

// ...

action = "find";searchFor = recoText;

// Windows Phone Silverlight App

// Synthesis

private async void SpeakText(string textToSpeak)

SpeechSynthesizer synthesizer = new SpeechSynthesizer();

await synthesizer.SpeakTextAsync(textToSpeak);

// Recognition

private async Task<SpeechRecognitionUIResult> RecognizeSpeech()

SpeechRecognizerUI recognizer = new SpeechRecognizerUI();

// One of three Grammar types available

recognizer.Recognizer.Grammars.AddGrammarFromPredefinedType(

"key1", SpeechPredefinedGrammar.WebSearch);

await recognizer.Recognizer.PreloadGrammarsAsync(); // Optional but recommended

// Put up UI and recognize user's utterance

SpeechRecognitionUIResult result = await recognizer.RecognizeWithUIAsync();

return result;

// Calling code uses result.RecognitionResult.Text or result.RecognitionResult.Semantics

// Synthesis

// Recognition

return result;

// Synthesisprivate async void SpeakText(string textToSpeak){

SpeechSynthesizer synthesizer = new SpeechSynthesizer();await synthesizer.SpeakTextAsync(textToSpeak);

// Windows Phone Silverlight App// Synthesisprivate async void SpeakText(string textToSpeak){

SpeechSynthesizer synthesizer = new SpeechSynthesizer();await synthesizer.SpeakTextAsync(textToSpeak);

// Recognitionprivate async Task<SpeechRecognitionUIResult> RecognizeSpeech(){

// One of three Grammar types availablerecognizer.Recognizer.Grammars.AddGrammarFromPredefinedType(

// Put up UI and recognize user's utteranceSpeechRecognitionUIResult result = await recognizer.RecognizeWithUIAsync();

return result;}// Calling code uses result.RecognitionResult.Text or result.RecognitionResult.Semantics

// Recognitionprivate async Task<SpeechRecognitionUIResult> RecognizeSpeech(){

// One of three Grammar types availablerecognizer.Recognizer.Grammars.AddGrammarFromPredefinedType(

// Optional but recommendedawait recognizer.Recognizer.PreloadGrammarsAsync();

// Synthesis

// Recognition

return result;

// Put up UI and recognize user's utteranceSpeechRecognitionUIResult result

= await recognizer.RecognizeWithUIAsync();

return result;

// Windows Phone Store App

// Synthesis

// C# code behind

// Function to speak a text string

private async void SpeakText(MediaElement audioPlayer, string textToSpeak)

SpeechSynthesisStream ttsStream = await synthesizer.SynthesizeTextToStreamAsync(textToSpeak);

audioPlayer.SetSource(ttsStream, ""); // This starts the player because AutoPlay="True"

// Synthesis

// C# code behind

// Synthesis<MediaElement Name="audioPlayer" AutoPlay="True" .../>

// Synthesis

// C# code behind

// C# code behind// Function to speak a text stringprivate async void SpeakText(MediaElement audioPlayer, string textToSpeak){

SpeechSynthesisStream ttsStream= await synthesizer.SynthesizeTextToStreamAsync(textToSpeak);

// Recognition

private async Task<SpeechRecognitionResult> RecognizeSpeech()

SpeechRecognizer recognizer = new SpeechRecognizer();

// One of three Constraint types available

SpeechRecognitionTopicConstraint topicConstraint

= new SpeechRecognitionTopicConstraint(SpeechRecognitionScenario.WebSearch, "MSDN");

recognizer.Constraints.Add(topicConstraint);

await recognizer.CompileConstraintsAsync(); // Required

SpeechRecognitionResult result = await recognizer.RecognizeWithUIAsync();

return result;

// Calling code uses result.RecognitionResult.Text or// result.RecognitionResult.SemanticInterpretation

// Recognition

return result;

// One of three Constraint types availableSpeechRecognitionTopicConstraint topicConstraint

// Recognition

return result;

// Put up UI and recognize user's utteranceSpeechRecognitionResult result

= await recognizer.RecognizeWithUIAsync();

return result;}// Calling code uses result.RecognitionResult.Text or// result.RecognitionResult.SemanticInterpretation

http://aka.ms/CortanaDev

http://channel9.msdn.com/Events/Build/2014/2-530

http://www.bing.com/blogs/site_blogs/b/search/archive/2014/04/14/integrating-store-apps-with-cortana-in-windows-phone-8-1.aspx

http://www.bing.com/dev/en-us/speech

http://msdn.microsoft.com/en-us/magazine/jj721592.aspx

http://blogs.msdn.com/b/robch

http://www.windowsphone.com/en-us/store/app/msdn-voice-search/aed09797-b424-4030-a04d-5ec8ec63847f

http://code.msdn.microsoft.com/wpapps/MSDN-Voice-Search-for-95c16d92

https://voicecommands.codeplex.com

http://code.msdn.microsoft.com/wpapps/Voice-Commands-for-Windows-2f808ac6

http://code.msdn.microsoft.com/windowsapps/Speech-synthesis-sample-6e07b218

http://code.msdn.microsoft.com/windowsapps/Bing-Speech-Recognition-210c8978

http://code.msdn.microsoft.com/windowsapps/Speech-Recognition-19c452b4

https://developer.apple.com/technologies/ios/accessibility.html

https://developer.apple.com/library/ios/documentation/AVFoundation/Reference/AVSpeechSynthesizer_Ref/Reference/Reference.html

http://nshipster.com/avspeechsynthesizer

DEVFRIGHT Blog Post: http://www.devfright.com/quick-look-avspeechsynthesizer-class/

https://play.google.com/store/apps/details?id=com.google.android.launcher

http://developer.android.com/reference/android/speech/package-summary.html

http://developer.android.com/reference/android/speech/tts/package-summary.html

http://channel9.msdn.com/Events/Build/2014/

http://channel9.msdn.com/Events/Build/2014

Get Ready to Become a Windows Phone Developerdev.windowsphone.com

Learn More About Windows Phone Dev via Official Microsoft Videoshttp://aka.ms/wp81js

http://aka.ms/wp81devbegin

Check Out Additional Learning Resourceswww.pluralsight.com/training/Courses#windows-phone

www.developer.nokia.com

Download Additional Resources & Become an Expertphone.codeplex.com

http://bit.ly/nokiadevoffers

dev.windowsphone.com

dev.windows.com

blogs.windows.com/windows_phone/b/wpdev

blogs.windows.com/windows_phone/b/windowsphone

http://developer.nokia.com

http://bit.ly/godvlup

http://conversations.nokia.com

http://appstudio.windows.com

http://flip.it/95YFG

www.geekchamp.com

www.wpcentral.com

Thank You!Slides will be posted on my Slideshare account.

Slideshare: www.slideshare.net/ActiveNick

Blog: www.AgeofMobility.com

Twitter: @ActiveNick

Mobile Apps: www.bigbaldapps.com

LinkedIn: www.linkedin.com/in/activenick

Website: www.AgeofMobility.com

Email: nick.landry@microsoft.com

Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile...

Technology

Transcript of Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile...

Cortana Analytics Suite

Hey Cortana, set a reminder to …. Hey Cortana, send email to Jonas Brandell …. Hey Cortana, call Jonas Brandell Hey Cortana, text Jonas Brandell …

Accessibility in Windows 10download.microsoft.com/download/F/5/3/F53B8B72-4D01-4DE1...Interact with Cortana using speech Cortana will help you find things on your PC, manage your calendar,

fortwo - smart.com · systems for more safety and comfort, an unparalleled turning circle, ... Because whether it be Siri, Google now, Cortana or Samsung S-Voice, ...

Siri and Cortana have applied for your role: The rise of AI in Marketing

TODAY - HCCA Official Site...ey Siri, get directions to The Grill restaurant.” “Hey Cortana, what’s the balance in my checking account?” “Transfer $500 from savings to checking.”

Cortana / Speech Platform - sec.ch9.ms / Speech Platform 2. Cortana truly personal across all your devices get things done. ... Parse data and ensure the test result are at or above

TDC 2014 - Cortana

IMPROVEMENT OF VOICE TRIGGER RECOGNITION IN … · Hey Siri? Hey Cortana? Hello Jarvis? Open Sesame? Alexa? Sorry Master, I cannot recognize the voice trigger. Please, try again THANK

Microsoft Research Academic Services GTM Draft€¦ · Microsoft Cortana 2014 Amazon Echo 2014 Apple Siri 2011 Samsung S Voice 2014 Google Home 2016 Microsoft Invoke ... Microsoft

The spy at home without any problems; nevertheless all ...epub.oeaw.ac.at/0xc1aa5576_0x003ae397.pdf · Alexa, Google Assistant, Siri, Cortana or Bixby try to support their users by

The Dialog State Tracking Challenge Series: A Review · Apple’s Siri, Google Now, Nuance Dragon Go, Xbox and Cortana from Microsoft, and numerous start-ups. Figure 1 shows theprincipalcomponents

Cortana!Tutorial! - · PDF fileCortana’Community’Resources ... Cortana Tutorial

Jabra Talk 30 - dustinweb.azureedge.net · 6.3 Voice assistant To activate your smartphone’s voice assistant (e.g. Siri, Google Assistant, Cortana) press and hold the Answer/End

Accelerating Machine Learning and Deep Learning with Intel ... · Natural Language Processing –Siri*/Cortana*/Google Now*/Baidu* ... [Honglak Lee, NIPS 2010 Workshop Deep Learning

Personal digital assistants, rise up!fjeronimo.com/Anexos/54AB88E0-F620-446C-B660-F46CD1FA4E8E.p… · Microsoft’s Cortana, Apple’s Siri, Google Now and the battle over what —

Deep Learning for Speech Recognition in Cortana at AI NEXT Conference

Developing A Cognitive Assistant For Audit Plan ...accounting.rutgers.edu/docs/wcars/41wcars2/Li.pdfFrom left to right: Cortana, Siri, Google Now (Litchfield, 2015) 6/29 CognitiveAssistant

Cortana / Speech Platform - sec.ch9.mssec.ch9.ms/slides/winHEC/2_03_Cortana_Speech_Platform.pdf · Cortana / Speech Platform 2. Cortana truly personal across all your devices get

FoggyCache: Cross-Device Approximate Computation Reuse · For example, speech recognition based assistance is now common on smartphones (e.g., Siri, Cortana) and in homes (e.g., Alexa

Accelerating Machine Learning and Deep Learning with Intel ... · Natural Language Processing –Siri/Cortana/Google Now/Baidu ... [Honglak Lee, NIPS 2010 Workshop Deep Learning