Real-Time Communication Testing Evolution with WebRTC
-
Upload
alexandre-gouaillard -
Category
Engineering
-
view
124 -
download
0
Transcript of Real-Time Communication Testing Evolution with WebRTC
![Page 1: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/1.jpg)
Real-Time CommunicationTesting Evolution
with WebRTC
Dr Alex Gouaillard
CoSMo Software ConsultingIMTC, W3C, IETF
![Page 2: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/2.jpg)
RTC Interoperability: Web Apps
Chromecast
![Page 3: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/3.jpg)
• W3C Standardization process (features supported or not)
• Stand-alone tests
• Single browser (one at a time)
(JS API) Compliance and Usual Testing
Test the Web Forward
![Page 4: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/4.jpg)
What about WebRTC 1.0?
Why so low test coverage?
• Some parts of specifications were just written• Tests not needed early on in standardization process• Many tests need P2P communication / interop testing
Media devices APIs WebRTC 1.0 APIs
Test coverage as of November 2016 (W3C TPAC)
![Page 5: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/5.jpg)
Ex. of RTC feature not Testable before:Network and Protocol Testing: ICE
Client A Client B
STUN server
What is my public IP address?
Your public IP address is x.x.x.x:y
![Page 6: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/6.jpg)
Compliance and Usual Testing vs.Interoperability Testing
Compliance & Usual Testing
InteroperabilityTesting
• Interaction automation
• Stand-alone tests
• Single browser
• Interaction automation
• Two synchronized instances of the same test
• 2+ browsers run in parallel
New testing tools needed
![Page 7: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/7.jpg)
Browsers Interaction Automation:WebDriver protocol
ChromeDriver
geckodriver
Microsoft WebDriver
SafariDriver
![Page 8: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/8.jpg)
Browsers as a Service:hosted selenium nodes
![Page 9: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/9.jpg)
In-Progress: WebDriver needs updatefor WebRTC Security & others
• Permission prompt
– Geolocation
– Camera
– Microphone
– Bluetooth
– …
• Fake Media / Mock Capturer
• ICE Candidate Filtering
• Unsecure origin
![Page 10: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/10.jpg)
Wrap-Up: WebDriver for RTC Tests
• Interaction
• Multiple browsers in parallel
• Synchronization of instances of same test between browsers
![Page 11: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/11.jpg)
Lots of tools out thereaddressing a subset of the problem
WebRTC-test
Jitsi-Hammer
Kurento Testing Framework
Jattack
![Page 12: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/12.jpg)
Ok, existing tools are limited but …How bad can it be, really?
Rough Goal: Test all possible pairs of
[Browser] x [Browser Revision]x
[OS] x [OS Revision]
![Page 13: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/13.jpg)
Wh
at w
e W
AN
Tto
tes
t
496
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
![Page 14: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/14.jpg)
Wh
at w
e co
uld
test
to
day 115
23%
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
38177%
1 OS
2 Oses(Not tested)
![Page 15: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/15.jpg)
© Dr. Alex Gouaillard @ Citrix Systems, 2016. Document provided under CC BY-NC 4.0
65%
Wh
at w
e A
CTU
ALL
Yte
st t
od
ay
2118%
8877%
Tested
WIP
Not tested
![Page 16: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/16.jpg)
Ok, it’s REALLY bad …Get visibility: list and evaluate
![Page 17: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/17.jpg)
Evaluation of Interop Testing ToolsDimension 1: Tests Triggers and frequency
6 months
Weekly (2d)
Daily / Nightly (4h)
Commit-based and fast (20mn)
Experimental (manually triggered)
![Page 18: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/18.jpg)
Evaluation of Interop Testing ToolsDimension 2: Type of SE nodes I want to use
Self-Hosted, Pre-configured VM
Hosted, as-a-service (e.g. BrowserStack)
Self-Hosted, Ephemeral VM
Hosted, Ephemeral VM (e.g. Travis)
Local / physical machines
[Apple]
[Only stable browsers][Extensions and plug-ins][Modified browsers]
[no manual settings][Extensions and plug-ins][Apple]
[limited OS set]
![Page 19: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/19.jpg)
Evaluation of Interop Testing ToolsDimension 3: OS I want to support
![Page 20: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/20.jpg)
Evaluation of Interop Testing ToolsDimension 4: Which Browsers?
Easy, kind of tested already, great WebDriver support
No Apple VM on non-apple HardwareWebdriver is more recent and less mature than others
WebRTC? What’s your flavor~~ ?Permission prompt? Fake Media?
36+% of Enterprise market (Windows 7)which plugin?
![Page 21: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/21.jpg)
Evaluation of Interop Testing ToolsDimension 5: How early do I want to test?
=> How fast do you want your fix inEx: chrome 58 chrome 61
Stable
Windows Insider
Windows Insider
![Page 22: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/22.jpg)
Evaluation of Interop Testing Tools
Own cloud
On-premises
Dimension 6: Freedom, Convenience and Price
Hosted
![Page 23: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/23.jpg)
Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
testRTC
Hosted
![Page 24: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/24.jpg)
Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
Travis CI
Hosted
![Page 25: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/25.jpg)
Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
BrowserStack
Hosted
![Page 26: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/26.jpg)
Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
Sauce Labs
Hosted
![Page 27: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/27.jpg)
Ok, I see what’s missing for each.What’s the right design now?
![Page 28: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/28.jpg)
KITE: Karoshi InteroperabilityTesting Engine
Because for most of us, manually testing webrtcinterop felt like “death by overwork”
![Page 29: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/29.jpg)
KITE: Karoshi Interoperability Testing EngineAutomating RTC P2P tests
• Generic (any app, any back-end)
• Reusable (any process, trigger, …)
• Easy to maintain (no death )
Stable / Beta / Dev / Canary
+
Own cloudOn-premises Hosted
![Page 30: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/30.jpg)
KITE
Trigger
Type of SE node
# OSes
Browsers
Browser channel
Hosted/on-prem
KITE
Hosted
+
On-premises
![Page 31: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/31.jpg)
KITE Design
Easy maintenance by divide and conquer
• Selenium Grid
• Test engine
• Test
• Dashboard
![Page 32: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/32.jpg)
KITE Design
Conductor
Dispatcher
Test Runner Test RunnerTest Runner
Config => BrowserObjCompute list< N-Tuple< BrowserObj > >
• { Test, Callback }• list< N-Tuple< BO > >
handle concurrencyif resource: 1 N-tuple => 1 test runner =>1 thread
1 2 … N
• { Test, Callback }• 1 N-Tuple< BO >
Instruments N BORun TestCallback( results )
BrowserObj available• Cr/FF - Mac - Rem-BB• BrX - OSX - Rem-BaaS (SauceLabs)• BrY - OSY - Rem-VM (AWS)
ResultDashboard
• { N, Test, Callback }• list< configs >
![Page 33: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/33.jpg)
Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
![Page 34: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/34.jpg)
Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
Location of dashboard to send tests results
"callback": "http://localhost:8080/kiteweb/datacenter",
![Page 35: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/35.jpg)
Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
Registration of Selenium server
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
![Page 36: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/36.jpg)
Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
List of tests
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
![Page 37: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/37.jpg)
Configuration File{
"name": "local selenium example",
"callback": "http://localhost:8080/kiteweb/datacenter",
"remotes": [
{
"type": "local",
"remoteAddress": "http://localhost:4444/wd/hub"
}
],
"tests": [
{
"name": "IceConnectionTest",
"tupleSize": 2,
"testImpl": "org.webrtc.kite.IceConnectionTest"
}
],
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
}
List of browsers to run the tests
"browsers": [
{
"browserName": "firefox"
},
{
"browserName": "chrome"
}
]
![Page 38: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/38.jpg)
![Page 39: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/39.jpg)
![Page 40: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/40.jpg)
![Page 41: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/41.jpg)
KITE Dashboard
![Page 42: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/42.jpg)
KITE Dashboard
![Page 43: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/43.jpg)
That’s very nice but it feels like something is missing …
![Page 44: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/44.jpg)
Global RTC Interoperability
Chromecast
![Page 45: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/45.jpg)
Global RTC Interoperability: MOBILE (Browsers)?
Chromecast
![Page 46: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/46.jpg)
Global RTC Interoperability Beyond Desktop Browsers
Appium
• Mobile testing framework
• For native, hybrid and mobile web apps
![Page 47: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/47.jpg)
Global RTC Interoperability:NATIVE, IoT and embedded
ChromecastIoT
Electron
![Page 48: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/48.jpg)
Global RTC Interoperability Beyond Desktop Browsers
Electron
• Open source framework
• To build cross platform desktop applications (HTML, CSS, JS)
• Combines Chromium and node.js into a single runtime
![Page 49: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/49.jpg)
Global RTC Interoperability:Beyond 1-1
Jitsi
• Open source multiplatform
• Videoconferencing
• Instant messaging
![Page 50: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/50.jpg)
Global RTC Interoperability:Beyond 1-1
Jitsi
![Page 51: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/51.jpg)
Global RTC Interoperability Make it easy: Grid Manager
• Set up a grid through a web app
• Set up Selenium hubs and nodes
• Download and set up– Browsers– Browsers WebDrivers
• Backup everything for follow-up uses
• Tear-down and clean-up after tests
![Page 52: Real-Time Communication Testing Evolution with WebRTC](https://reader033.fdocuments.in/reader033/viewer/2022051123/5a647a637f8b9a7c568b46dd/html5/thumbnails/52.jpg)
Thank you!