“Avalon”: Using UI Automation to Improve Application Testability Robert Sinclair Group Manager...
-
Upload
antonia-adams -
Category
Documents
-
view
219 -
download
0
Transcript of “Avalon”: Using UI Automation to Improve Application Testability Robert Sinclair Group Manager...
“Avalon”: Using UI Automation to Improve Application Testability
Robert SinclairGroup ManagerMicrosoft [email protected]
Session Code: CLI307
AgendaAgenda
What is UI Automation? Demo: Windows UI Automation in actionA comparison of approachesOverview of Windows UI AutomationSample CodeSecurity ConsiderationsKey Take-Aways
What is UI Automation? Demo: Windows UI Automation in actionA comparison of approachesOverview of Windows UI AutomationSample CodeSecurity ConsiderationsKey Take-Aways
Tools
Client Application Model
Avalon Windows Forms
Web & Service Application Model
ASP.NET / Indigo Win FSCompact
FrameworkYukon Mobile PC Optimized
Data Systems Application Model
Presentation Data
Mobile PC & Devices Application Model
Communication
Command Line
NT Service
DataSetDataSet
MappingMapping
ObjectSpacesObjectSpaces
ObjectSpaceObjectSpace
QueryQuery
SchemaSchema
ItemItem
RelationshipRelationship
MediaMedia
AudioAudio
VideoVideo
ImagesImages
System.MessagingSystem.Messaging System.DiscoverySystem.Discovery
System.DirectoryServicesSystem.DirectoryServices
System.RemotingSystem.Remoting
System.Runtime.RemotingSystem.Runtime.Remoting
ActiveDirectoryActiveDirectory
UddiUddi
System.Web.ServicesSystem.Web.Services
Web.ServiceWeb.Service
DescriptionDescription
DiscoveryDiscovery
ProtocolsProtocols
System.MessageBusSystem.MessageBus
TransportTransport
PortPort
ChannelChannel
ServiceService
QueueQueue
PubSubPubSub
RouterRouter
Fundamentals
System.ComponentModelSystem.ComponentModel
System.CodeDomSystem.CodeDom
System.ReflectionSystem.Reflection
System.EnterpriseServicesSystem.EnterpriseServices
System.TransactionsSystem.Transactions
Security
System.Windows.TrustManagementSystem.Windows.TrustManagement
System.Web.SecuritySystem.Web.Security
System.MessageBus.SecuritySystem.MessageBus.Security
AuthorizationAuthorization
AccessControlAccessControl
CredentialsCredentials
CryptographyCryptography
System.Web.ConfigurationSystem.Web.Configuration
System.MessageBus.ConfigurationSystem.MessageBus.Configuration
System.ConfigurationSystem.Configuration
System.ResourcesSystem.ResourcesSystem.ManagementSystem.Management
System.DeploymentSystem.Deployment
System.DiagnosticsSystem.Diagnostics
Configuration Deployment/Management
System.WindowsSystem.WindowsSystem.WindowsSystem.WindowsSystem.Windows.Form
sSystem.Windows.Forms
System.ConsoleSystem.Console
System.ServiceProcessSystem.ServiceProcess
System.Windows.FormsSystem.Windows.Forms
System.WebSystem.Web System.StorageSystem.Storage System.Data.SqlServerSystem.Data.SqlServer
AnimationAnimation
ControlsControls
ControlControl
DesignDesign
PanelPanel
ControlsControls
DialogsDialogs
SideBarSideBar
NotificationNotification
System.WindowsSystem.Windows
DocumentsDocuments
Text ElementText Element
ShapesShapes
ShapeShape
InkInk
UI ElementUI Element ExplorerExplorer MediaMedia
System.Windows.FormsSystem.Windows.Forms
FormsForms
ControlControl
Print DialogPrint Dialog
DesignDesign
System.Web.UISystem.Web.UI
PagePage
ControlControl
HtmlControlsHtmlControls
MobileControlsMobileControls
WebControlsWebControls
AdaptorsAdaptors
DesignDesign
System.CollectionsSystem.Collections
GenericGeneric
System.SearchSystem.Search
AnnotationsAnnotations
MonitoringMonitoring
LoggingLogging
RelevanceRelevance
System.DataSystem.Data
SqlClientSqlClient
SqlTypesSqlTypes
SqlXMLSqlXML
OdbcClientOdbcClient
OleDbClientOleDbClient
OracleClientOracleClient
CoreCore
ContactContact
LocationLocation
MessageMessage
DocumentDocument
EventEvent
System.StorageSystem.Storage
System.WebSystem.Web
PersonalizationPersonalization
CachingCaching
SessionStateSessionState
System.XmlSystem.Xml
SchemaSchema
SerializationSerialization
XpathXpath
QueryQuery
PermissionsPermissions
PolicyPolicy
PrincipalPrincipal
TokenToken
System.SecuritySystem.Security
System.CollaborationSystem.Collaboration
RealTimeEndpointRealTimeEndpoint
TransientDataSessionTransientDataSession
SignalingSessionSignalingSession
MediaMedia
ActivitiesActivities
HttpWebRequestHttpWebRequest
FtpWebListenerFtpWebListener
SslClientStreamSslClientStream
WebClientWebClient
System.NetSystem.Net
NetworkInformationNetworkInformation
SocketsSockets
CacheCache
System.WebSystem.Web
AdministrationAdministration
ManagementManagement
NavigationNavigation
Peer GroupPeer Group
PolicyPolicy
RecognitionRecognition
System.SpeechSystem.Speech
SynthesisSynthesis
UI AutomationSystem.Windows.AutomationSystem.Windows.Automation
System.TimersSystem.Timers
System.GlobalizationSystem.Globalization
System.SerializationSystem.Serialization
System.ThreadingSystem.Threading
System.TextSystem.Text
System.DesignSystem.Design
PortsPorts
InteropServicesInteropServices
System.RuntimeSystem.Runtime
System.IOSystem.IO
SerializationSerialization
CompilerServicesCompilerServices
Base & Application Services
System.Windows.AutomationSystem.Windows.Automation
ProviderProvider
InteropProviderInteropProvider
“UI Automation” defined“UI Automation” defined
Gather information about the UIDynamically discover UI structureExtract property information Receive event notifications when UI changesQuery an element for its behavior
Interact with UI elementsClick a button, scroll a list, move a window, etc.Inject keystrokes and mouse input
Gather information about the UIDynamically discover UI structureExtract property information Receive event notifications when UI changesQuery an element for its behavior
Interact with UI elementsClick a button, scroll a list, move a window, etc.Inject keystrokes and mouse input
– code that programmatically drives another application’s UI to yield a desired
result
Windows UI Automation in Action
Goals:Traverse the UI treeQuery a control for its functionalityInteract with the control
Application #2Application #1
Today’s Approach to AutomationToday’s Approach to Automation
Custom 1Custom 1 WinForms SDM Win32Win32 Custom 2Custom 2Text PanelWin32Win32
Automation UtilityAutomation Utility
Win32Messages &
APIs
Win32Messages &
APIs
SDMObjectModel
SDMObjectModel
Custom 1APIs
Custom 1APIs
……
……Text Object
ModelText Object
Model
Custom 2APIs
Custom 2APIs
WinFormsMethods
WinFormsMethods
Interacting with a ControlComparison of ApproachesInteracting with a ControlComparison of Approaches
Today – Invest in each control libraryUse each library’s proprietary APIs to:
Traverse its treeCollect UI informationManipulate the controls
At runtime:Test the control’s classnameCall your library or control-specific routines
Longhorn – Use Windows UI Automation
Traverse UI structure using a single treeBase your app logic on control behaviorsObtain UI Automation security permissions
Today – Invest in each control libraryUse each library’s proprietary APIs to:
Traverse its treeCollect UI informationManipulate the controls
At runtime:Test the control’s classnameCall your library or control-specific routines
Longhorn – Use Windows UI Automation
Traverse UI structure using a single treeBase your app logic on control behaviorsObtain UI Automation security permissions
UI Automation ClientsUI Automation Clients
UI Automation Providers
Windows UI Automation Core
Longhorn Model: UI AutomationLonghorn Model: UI Automation
Assistive TechnologyProducts
ScriptingUtilities
TestingFramework
“Avalon”Implementation
Control Proxy CustomImplementation
…InteropProvider…Provider
MSAvalon.Windows.Automation
Leg
acy C
on
trol
ISV
Ob
ject
Mod
el
Atu
om
ate
d T
est
…InteropProvider
Windows UI AutomationWindows UI AutomationAutomation framework built into Longhorn
Platform-level support for automating all UI elements
Avalon, WinForms, Win32, Visual Basic, etc.
Exposes a consistent object model for all controls
3rd party controls easily integrate into model
Security Model – client must be trustedLocale, machine, and resolution independent
Creates new opportunities for innovation in:
Automated UI TestingAssistive Technology ProductsCommand-and-Control Utilities
Automation framework built into Longhorn
Platform-level support for automating all UI elements
Avalon, WinForms, Win32, Visual Basic, etc.
Exposes a consistent object model for all controls
3rd party controls easily integrate into model
Security Model – client must be trustedLocale, machine, and resolution independent
Creates new opportunities for innovation in:
Automated UI TestingAssistive Technology ProductsCommand-and-Control Utilities
UI Automation OverviewUI Automation OverviewLogical Tree – structure of the UI
Stitches all UI trees into one coherent structureEliminates unnecessary elementsResembles the structure perceived by an end user
Properties – important UI informationName, Bounding Rectangle, Persistent ID, etc.
Events – notification of UI changes Window creation, change in focus or selection, etc
Control Patterns – control behavior Scroll, Selection, Window, ExpandCollapse, etc.
Input – simple mouse and keyboard input
Logical Tree – structure of the UI Stitches all UI trees into one coherent structureEliminates unnecessary elementsResembles the structure perceived by an end user
Properties – important UI informationName, Bounding Rectangle, Persistent ID, etc.
Events – notification of UI changes Window creation, change in focus or selection, etc
Control Patterns – control behavior Scroll, Selection, Window, ExpandCollapse, etc.
Input – simple mouse and keyboard input
UI Automation Control PatternsUI Automation Control Patterns
Mutually exclusive classes of control behaviorControl developers (providers) expose these patterns for new or existing controlsAutomation developers (clients) use patterns to
Discover what functionality a control offersGather pattern-specific property informationReceive pattern-specific eventsProgrammatically manipulate the control
Examples: Button: InvokeListBox: Scroll, SelectionComboBox: Scroll, Selection, ExpandCollapse
Mutually exclusive classes of control behaviorControl developers (providers) expose these patterns for new or existing controlsAutomation developers (clients) use patterns to
Discover what functionality a control offersGather pattern-specific property informationReceive pattern-specific eventsProgrammatically manipulate the control
Examples: Button: InvokeListBox: Scroll, SelectionComboBox: Scroll, Selection, ExpandCollapse
Expand Collapse Pattern
Goals:Traverse Logical Element treeQuery for supported patternsFind an element that supports ExpandCollapseManipulate the control
Security ModelSecurity Model
No default automation permissions UI Automation functionality is protected according to the following permissions:
Read – Navigate tree, get properties, receive eventsWrite – Call control pattern methodsInput – Call methods in the Input class
Access to Rights Managed requires additional permissions
No default automation permissions UI Automation functionality is protected according to the following permissions:
Read – Navigate tree, get properties, receive eventsWrite – Call control pattern methodsInput – Call methods in the Input class
Access to Rights Managed requires additional permissions
3 Key Take-aways3 Key Take-aways
1. Platform-level support for automating UI
2. Control independence Simplifies your client logicMeans less code to write and maintain
3. New opportunities Integrate additional control librariesExpose custom object models, properties, events, etc.Extend the UI Automation framework
1. Platform-level support for automating UI
2. Control independence Simplifies your client logicMeans less code to write and maintain
3. New opportunities Integrate additional control librariesExpose custom object models, properties, events, etc.Extend the UI Automation framework
Call to ActionCall to Action
Next 30 daysStudy System.Windows.Automation.*Spend a day playing with UI AutomationSubmit feedback on client or provider side
Longhorn Beta 1Providers: Implement UI Automation for your custom controls Clients: Use UI Automation to develop your automation clients
Longhorn RTMShip your fully-enabled UI Automation providers and clients!!
Next 30 daysStudy System.Windows.Automation.*Spend a day playing with UI AutomationSubmit feedback on client or provider side
Longhorn Beta 1Providers: Implement UI Automation for your custom controls Clients: Use UI Automation to develop your automation clients
Longhorn RTMShip your fully-enabled UI Automation providers and clients!!
Question & AnswerQuestion & Answer
Join me for further discussion in the Client Track LoungePlease fill out an on-line session evaluation on COMMNET
Thank You for Coming!
Join me for further discussion in the Client Track LoungePlease fill out an on-line session evaluation on COMMNET
Thank You for Coming!
© 2003-2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.