Hey Cortana, set a reminder to …. Hey Cortana, send email to Jonas Brandell …. Hey Cortana, call...
-
Upload
elfreda-cole -
Category
Documents
-
view
241 -
download
4
Transcript of Hey Cortana, set a reminder to …. Hey Cortana, send email to Jonas Brandell …. Hey Cortana, call...
Adina Trufinescu - Principal Program ManagerKhuram Shahid - Senior Developer
Cortana Extensibility and Speech Platform In-Depth
3-716
Adina Trufinescu - Principal Program ManagerKhuram Shahid - Senior Developer
Cortana Extensibility and Speech Platform In-Depth
3-716
Hey Cortana, set a reminder to ….
Hey Cortana, send email to Jonas
Brandell ….
Hey Cortana, call Jonas Brandell
Hey Cortana, text Jonas Brandell …
Hey Cortana, what’s up with
Jonas Brandell on Facebook
Hey Cortana, create a station for Hozier on Pandora
Hey Cortana, send a message to Jonas Brandell on Viber
Hey Cortana, Tweet flying into San
Francisco tonight
• Differentiate your app with voice commands
• Delight your users with natural interactions
• Increase user engagement • Increase productivity• Increase discovery
Developer Opportunity
Voice Commands in the Foreground
Speech Recognition in your app
Voice Commands in the Background
Agenda
• Windows Universal Apps can register voice commands for users to say to Cortana
• Cortana launches the app to complete the task
• Available for Windows Universal Apps built with XAML and JavaScript
Voice Commands for Universal Windows AppsForegroundVoice Commands
Foreground Voice Commands Architecture
2
Windows SpeechPlatform
Call
Text
Remind
Voice Command
3 6
4
5
XApplication
Voice Command Definition
7
1
Microsoft SpeechRecognition Service
Adventure Works, show my trip to London
Foreground Voice Commands - How ToCreate the Voice Command Definition (VCD)
Register the VCD XML on App Startup
Handle Voice Command Activation
• Determine the task to enable and what should the user say
• Start a chat with {contact}• Send a message to {contact}
• Determine what Cortana should say as the app is launched
• Starting a chat with {contact}
• Provide an example of what the user can say for Cortana to show on “What can I do?”
• Start a chat with Jonas
Voice Command Intent
• Users need to say the app name as part of the voice command• Viber, start a chat with Jonas• Facebook, what’s up with Jonas• Adventure Works, show my trip to London
• By default uses the app’s display name
• Provide a alternate such as short form when the app name is long or hard to pronounce
Voice Command & App Name
• Explicitly specify the app name where feels most natural• Start a chat with Jonas on/with/using Viber• What’s up with Jonas on Facebook• Show my Adventure Works trip to London
• Increase the recognition accuracy by providing a phonetically-rich phrase• Don’t: Get Jonas on Viber• Do: Start a chat with Jonas on Viber
• Avoid using a Cortana reserved command, e.g. Call, Text, Start, Open, Find. If you have to, use the app name as a prefix• Viber, text Jonas I’m on my way
Voice Commands Guidance
Voice Command Registration
• Register the Voice Command Definition on app startup
• Use VoiceCommandDefinitionManager API in Windows.ApplicationModel.VoiceCommands namespace
• Update the VCD with application data
Handle the Voice Command Activation• Override App.OnActivated method
• Check for ActivationKind.VoiceCommand
• Use VoiceCommandActivatedEventArgs.Result to determine what the user said
• Navigate to page and continue the interaction
• Websites can be submitted to Windows Store
• Cortana can launch Web-Hosted Apps with Voice Commands
• Same Voice Command Definition schema and activation model as for JavaScript apps
Voice Commands for Web-Hosted Apps
Voice Commands for Web-Hosted Applications
Voice Activation for Web-Hosted Apps
2
Windows SpeechPlatform
Call
Text
Remind
Voice Command
3 6
4
5
1
7
XApplication
Package.appxmanifest
default.html
<meta name="msapplication-cortanavcd" content="https://contoso.com/vcd.xml"/>
<script src="voice.js"></script>
Vcd.xml
Microsoft SpeechRecognition Service
Contoso.com/default.html
if (typeof Windows != 'undefined') { var activation = Windows.ApplicationModel.Activation; Windows.UI.WebUI.WebUIApplication.addEventListener("activated", function (args) { if (args.kind === activation.ActivationKind.voiceCommand) { var speechRecognitionResult = args.result; var properties = speechRecognitionResult.SemanticInterpretation.Properties; //... } });
<Application Id="App" StartPage="https://flightarcade.azurewebsites.net">
<uap:VisualElements ... </uap:VisualElements>
<uap:ApplicationContentUriRules> <uap:Rule Type="include" WindowsRuntimeAccess="all“ Match="https://flightarcade.azurewebsites.net"/> </uap:ApplicationContentUriRules>
</Application>
Package.appxmanifest
Voice.js
Voice Commands for Web-Hosted AppsCreate the Voice Command Definition and host on web service
Register the VCD on the default HTML page
Handle Voice Command Activation in .js
• Check if the interaction was input through voice or text
• For voice, use Speech to continue the user interaction in the app
• Use Windows.Media.SpeechSynthesis APIs to talk to the user
• Use Windows.Media.SpeechRecognition for speech recognition
In-App Speech Recognition
• Use SpeechRecognizer.RecognizeWithUIAsync to present the user with System listening GUI that can be customized
• Use SpeechRecognizer.RecognizeAsync for on demand recognition and provide a listening UI experience
• System predefined grammars: Dictation, Web Search, Form Filling
• App grammars: a list of strings or SRGS file
On-Demand Speech Recognition
• SpeechRecognizer.ContinuousRecognitionSessionenables listening continuously
• Can be used with Dictation grammar for scenarios where users speak for long periods
• Can be used with app-provided grammars for always-listening in-app commands
Continuous Speech Recognition
• Universal App can register voice commands to run in an AppService
• AppService takes action on the voice command
• May use Cortana’s canvas and voice to present feedback to the user
• Available for Store Apps built with XAML and JavaScript
Voice Commands in the Background
Background Voice Activation
Background Voice Commands Architecture
2
Windows SpeechPlatform
36
4
5
Isolated Storage
Voice CommandApp Service
Render XAML
7
Web Service
Microsoft SpeechRecognition Service
Adventure Works, what trips do I have coming up? VCD
Your App
8
Go to Adventure Works
Here are your upcomingtrips
A
YosemiteAugust 15th
San FranciscoMay 29th
9
1
Background Voice Commands - How To
Create the Voice Command Definition (VCD)
Register the VCD XML on App Startup
Handle the Voice Command Activation in an App Service
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2"> <CommandSet xml:lang="en-us" Name=“en-us-CommandSet"> <AppName> Adventure Works </AppName> <Example> Show my trip to San Francisco </Example> <Command Name=“showTripToDestination"> <Example>Show my trip to San Francisco</Example> <ListenFor >show [my] trip to {destination}</ListenFor> <Feedback>Looking for trip to {destination}</Feedback> <VoiceCommandService Target=“AdventureWorksService"/> </Command>
Voice Command Definition
Register the Voice Command Definition on app startup
<Extensions>
<Extension
Category="windows.appService"
EntryPoint="AdventureWorks.AdventureWorksService">
<AppService Name=“AdventureWorksService"/>
</Extension>
</Extensions>
Create an AppService•Inherit from IBackgroundTask and implement Run()
• Register the AppService in the package.appxmanifest
public async void Run(IBackgroundTaskInstance taskInstance)
See Session 765 For Details on AppServices
using Windows.ApplicationModel.AppService;
using Windows.ApplicationModel.Background;
using Windows.ApplicationModel.VoiceCommands;• VoiceCommandDefinitionManager• VoiceCommandDefinition• VoiceCommandServiceConnection• VoiceCommand• VoiceCommandUserMessage• VoiceCommandContentTile• VoiceCommandResponse
WinRT Namespaces and APIs
VoiceCommandServiceConnection• Request actions from
Cortana:• RequestConfirmationAsync(…)• RequestDisambiguationAsync(…)• RequestAppLaunchAsync(…)
• Send updates to Cortana:• ReportProgressAsync(…)• ReportFailureAsync(…)• ReportSuccessAsync(…)
• Handle Events:• VoiceCommandCompleted
Cortana Canvas Interactions from Background• Present UI on
Cortana’s canvas
• Speak a message using Cortana’s voice
Which trip would you like to cancel?
May 14th 2015Las Vegas
May 28th 2015San Diego
Go to Adventure Works
August 24th 2015Phoenix
Content Area
App GUI Text
App Icon
Link to your app
App Screen
Text-To-Speech (TTS)
Content Tile
Which one do you wanna
cancel?
A
VoiceCommandResponse
Which trip would you like to cancel?
May 14th 2015Las Vegas
May 28th 2015San Diego
Go to Adventure Works
August 24th 2015Phoenix
Which one do you wanna
cancel?
A
Which trip would you like to cancel?
May 14th 2015Las Vegas
May 28th 2015San Diego
Go to Adventure Works
August 24th 2015Phoenix
Which one do you wanna
cancel?
A
VoiceCommandResponse
VoiceCommandResponse{ VoiceCommandUserMessage Message; VoiceCommandUserMessage RepeatMessage; List<VoiceCommandContentTile> VoiceCommandContentTiles; string AppLaunchArgument;}
VoiceCommandContentTile
VoiceCommandContentTile{ string Title; string TextLine1; string TextLine2; string TextLine3; Object AppContext; string AppLaunchArgument;
Windows.Storage.StorageFile Image; VoiceCommandContentTileType ContentTileType;}
Which one do you want to add to your trip?
Title NameText Line OneText Line TwoText Line Three
A
VoiceCommandContentTile – Text
Listening…
Listening…
Icon, Title
Which one do you want to add to your trip?
Listening…
Icon, Title, Line1, Line2, Line3
Title Name OneLong text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
Title Name One
Title Name Two
Title Name Three
Title Name FourTitle Name TwoLong text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
Title Name ThreeLong text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
Title Name Four
Go to Adventure Works
Which one do you want to add to your trip?
AA
VoiceCommandContentTile– Icons68x92xIcon, Title, Line1, Line2, Line3
Which one do you want to add to your trip?
Listening…
Title Name One
Long text two lines lorem ipsum consec tetuer amias
Title Name Two
Long text two lines lorem ipsum consec tetuer amias
Title Name Three
Long text two lines lorem ipsum consec tetuer amias
Title Name Four
Long text two lines lorem ipsum consec tetuer amias
240x140 Icon, Title, Line1, Line2, Line3
Which one do you want to add to your trip?
Listening…
Title Name OneOne line text
Title Name TwoOne line text
Long text two lines lorem ipsum consec tetuer amias odoran
68x68 Icon, Title, Line1, Line2
Which one do you want to add to your trip?
Listening…
Title Name One
Long text lorem ipsum
Title Name Two
Long text lorem ipsum
Title Name Three
Long text lorem ipsum
Title Name Four
Long text lorem ipsum
Title Name Five
Long text lorem ipsum
Go to Adventure Works
A A A
Which one do you want to cancel?
May 14th 2015
Vegas Tech Conference
August 18th 2015
Party in Phoenix
Listening…
Disambiguation
Vegas Tech Conference
A
Cancel this trip?
Listening…
May 14th 2015
Vegas Tech Conference
Yes No
Confirmation
Yes
A
Cancelling this trip…
Cancel
May 14th 2015
Vegas Tech Conference
Go to Adventure Works
Progress
A
Cancelled this trip.
May 14th 2015
Vegas Tech Conference
Go to Adventure Works
Completion
A
Which ‘Vegas’ trip do you wanna cancel?
Vegas Tech Conference or Party
in Vegas.
Do you wanna cancel ‘Vegas Tech
Conference’?Cancelling this trip… I’ve cancelled it.
[success-earcon]
Cancel my Adventure Works
trip
Which one do you want to cancel?
May 14th
Vegas Tech Conference
August 18th
Party in Phoenix
Listening…
A
Which trip do you wanna cancel?
Disambiguation
Vegas Tech Conference
The first one
The one on August 18th
• The user may select an option via speech or UI input.
• Cortana automatically listens for utterances that uniquely identify a content tile.
• RequestDisambiguationAsync() returns only after a selection has been made.
Do you wanna cancel ‘Vegas Tech Conference’?
Confirmation
Yes
No
• Always confirm before taking destructive or irreversible actionsCancel this trip?
Listening…
May 14th 2015
Vegas Tech Conference
Yes No
A
Progress
• Text To Speech is optional
• You may send multiple progress updates. Updates are coalesced where possible.
• Send progress during long-running operations. If there are no updates for 5 seconds, Cortana may cancel the flow.
Cancelling this trip…
Cancel
May 15th 2015
Vegas Tech Conference
Go to Adventure Works
Cancelling this trip…
A
Cortana Supplied Progress Screens
Listening…
Working on that…
Cancel
I’ll get Adventure Works on that.
Adventure WorksLooking for trip to San Francisco
Cancel
Search “Cancel my Adventure Works trip to Vegas”
Looking for your trip to San Francisco…
A
AMay 14th 2015
Vegas Tech Conference
Go to Adventure Works
• Shown if launching the App service, or the initial response takes > 500ms
• Feedback string from VCD, if any, is spoken out loud.
• Shown after RequestConfirmation or RequestDisambiguation if the App does not respond quickly (500ms)
• Content derived from the previous Confirmation/Disambiguation Response
• Ask the user to confirm before taking action on the voice command
• Ask the user to disambiguate if there are more than one action or target of the action
• Show progress as the voice command is handled
Interaction Summary:
Canvas Interactions from Background
Cortana Your AppServiceAdventure Works, cancel
my trip! I’ll get Adventure Works on that!
Inspect Voice Command
Las Vegas
yes
Invoke IBackgroundTask::Run()
RequestDisambiguationAsync
RequestConfirmationAsync
VoiceCommandConfirmationResult
VoiceCommandDisambiguationResult
Interaction Flow
Which one?Las VegasCancun//build/
check SelectedItem
Are you sure?
yes no
Process request
(Talk to web service,
save dataetc.)
Cortana Your AppAdventure Works, cancel
my trip! I’ll get Adventure Works on that!
Inspect Voice Command
Which one?Las VegasCancun//build/
Are you sure?
yes no
Las Vegas
yes
check SelectedItem
Invoke IBackgroundTask::Run()
RequestDisambiguationAsync
RequestConfirmationAsync
VoiceCommandConfirmationResult
VoiceCommandDisambiguationResult
Process request
(Talk to web service,
save dataetc.)
Canceling… ReportProgressAsync
foreground
Event: VoiceCommandCompletedTeardown
App Launch
Event: VoiceCommandCompleted
UI Interaction
• Be efficient. Less is more! • Be relevant. Provide information
that’s relevant to the given task, content, and context.
• Be clear. Avoid ambiguity. Use everyday language.
• Be trustworthy. Present accurate information. Be transparent. Respect privacy.
Conversational Writing Guidance
Canvas Interactions from Background
• For each supported language, localize the CommandSet in the Voice Command Definition
• When launched, inspect the value of Language property of the VoiceCommandServiceConnection
• Localize the dialog on the app provided screens to match Cortana’s language
Localization
Canvas Interactions from Background
Create a Voice Command Definition
Register the VCD when the app starts
Handle the Voice Activation in the app or AppService
Voice Commands for Universal Apps - Recap
• for tasks that require complex interactions
• for tasks that demand the user’s attention for a long time
Voice Command Handling Guidance
• for simple tasks that don’t require additional input beyond Confirmation and Disambiguation
• for tasks that complete within seconds
• to initiate longer tasks that execute remotely
• may use RequestAppLaunchAsync method to launch the app and complete remainder of the flow
Foreground Background
• Launch and deep link into your app from Cortana• Extend Cortana with your app’s capabilities through
voice commands in the background• Add speech recognition as an input modality to your
app
• References:• See AppServices Session 765• See Web-Hosted Apps Session 665• See Adventure Works Sample Code• See Design Guidelines for extending Cortana
Call to Action