Automation with Sikuli
-
Upload
aiacov -
Category
Technology
-
view
3.288 -
download
9
description
Transcript of Automation with Sikuli
![Page 1: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/1.jpg)
Automation with Sikuli
![Page 2: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/2.jpg)
AgendaWhat you’ll learn today
What is Sikuli?
How it works?
How to use it?
![Page 3: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/3.jpg)
What is Sikuli ?
Visual approach to search and automation of graphical user interfaces using screenshots.
![Page 4: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/4.jpg)
Motivation
Usually needed to automate GUIs: support from developers API access language/OS dependency position/naming dependencies
Just to see if it can be done
![Page 5: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/5.jpg)
Demo
![Page 6: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/6.jpg)
System design
![Page 7: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/7.jpg)
Finding GUI patterns on the screen
Invariance
Resized versions
Texturally similar, different color pallets
Template Matching for small patterns
![Page 8: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/8.jpg)
Invariant local features for big patterns
Learn Extract model from training pattern▪ Invariant to scale & rotation
Encode in the model the relative position of the center
Search Extract invariant features Infer possible model center position Cluster consistent features Validate supposition
![Page 9: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/9.jpg)
Find visual objects
find( )
![Page 10: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/10.jpg)
Finder
f = Finder(path-to-imagefile)
f.find(path-to-imagefile, [similarity])
// iterates through Match objectswhile(f.hasNext): print “found match: “ + f.next().getScore()
![Page 11: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/11.jpg)
Pattern
Abstraction for visual patterns Are used by finding operations Methods can be chained to refine the
pattern Can define click points
Pattern(string)
Pattern(string).similar(0.9).targetOffset(10,30)
![Page 12: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/12.jpg)
Region
Region(x, y, w, h)Region(region)Region(Rectangle)
Search in a given region Observe a region in background for changes Retrieve matches Optimize the search by chaining regions No need to be aware of the content
![Page 13: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/13.jpg)
Act on visual objects
click( )
![Page 14: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/14.jpg)
Actions
click(PSMLR), doubleClick(PSMLR)
dragDrop(PSMLR target, PSMLR destination)
![Page 15: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/15.jpg)
Actions
Uses the Tesseract OCR engine
type(PSMLR, text)
text()
![Page 16: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/16.jpg)
Actions
They can be used along with KeyModifiers
mouseDown(button), mouseUp(button)
keyDown(keys), keyUp(keys)
![Page 17: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/17.jpg)
Actions
wait(PS), waitVanish(PS)
![Page 18: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/18.jpg)
Actions
onAppear(PS, handler)
onVanish(PS, handler)
onChange([minChengedSize], handler)
![Page 19: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/19.jpg)
Performance
A typical call to find() for a 100x100 target on a 1600x1200 screen takes less than 200 msec
Improve your searches
How performant do you want it to be?
![Page 20: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/20.jpg)
Extensions
Record-playback Sikuli Guide …
Why not give it a try and make your own?
![Page 21: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/21.jpg)
Limitations
Ok, the moment of truth!
Screenshots – unstable interfaces Visibility constraints
A paradigm shift requires a thinking shift
![Page 22: Automation with Sikuli](https://reader033.fdocuments.in/reader033/viewer/2022051012/5468c137af7959a75e8b5577/html5/thumbnails/22.jpg)
DemoWhat we’ve made of it