RTP Payload format for Application and Desktop Sharing · Application Sharing Sharing an...
Transcript of RTP Payload format for Application and Desktop Sharing · Application Sharing Sharing an...
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 1
RTP Payload format for Application and Desktop Sharing
Omer Boyaci & Henning SchulzrinneNovember 18, 2008
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 2
Application Sharing
Sharing an application with multiple users
There is only one copy of the application
Participants do not need application itself
Briefly, participantsreceive screen updatessend keyboard and mouse events
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 3
Terminology
• Application Host (AH)• Participant• Remoting protocol• Human Interface Protocol (HIP)• Different modes of remote access
– Remote Desktop Connection– (Full or partial)Screen Sharing– Application Sharing
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 4
Screenshot
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 5
Screenshot (2)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 6
Architecture
Remoting protocol
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 7
Architecture
Human Interface Protocol (HIP)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 8
Architecture
Binary Floor Control Protocol (BFCP) manages the ownership of AH side human interface devices.
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 9
Multimedia Support (Movies)
• Composite image comparing JPEG and PNG: notice artifacts in JPEG versus solid PNG background.
draft-boyaci-avt-png-00
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 10
Comparison of Sharing Systems in terms of web page visiting performance
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 11
Multimedia Support (Movies)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 12
Comparison of Sharing Systems in terms of multimedia performance
Unlimited BandwidthBASS-TheoraBASS-JPEG
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 13
3Mb/s Bandwidth
Comparison of Sharing Systems in terms of multimedia performance
BASS-TheoraBASS-JPEG
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 14
Remoting Messages
• Application host (AH) to participants– WindowStateInfo– RegionUpdate– MoveRectangle– MousePointerInfo
• Participants to AH (NACK messages)– PLI (Picture Loss Indication)– NACK request
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 15
HIP Messages
• MousePressed• MouseReleased• MouseMoved• MouseWheelMoved• KeyPressed• KeyReleased• KeyTyped
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 16
VNC problems
• Client-pull based• No multicast support• Same encoding for all updates• CPU usage increases for each new user•
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 17
TeleTeachingTool (VNC based)
ProblemsModified VNC protocol and clients (TTT server is pushing updates)No compression (only hextile encoding) because packets can get lost
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 18
Open Issues
• Transport Protocol– RTP– MSRP– Custom Made
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 19
Open Issues
• How to do retransmission requests for UDP clients
• Current proposal– NACK-based solution– NACK suppression to prevent floods– RTP-library level retransmissions
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 20
draft-boyaci-avt-app-sharing-00draft-boyaci-avt-png-00
Current Internet Drafts
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 21
Backup Slides
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 22
Protocol Message Structures
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 23
Common remoting/hip Header
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 24
A Window Record
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 25
A Region Update Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 26
The MoveRectangle Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 27
The MousePressed Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 28
The KeyPressed Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 29
The KeyTyped Message
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 30
Comparison of Sharing Systems
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 31
Screenshot (Overlapped Windows)
1
23
4
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 32
Screenshot (Multicast App. Tool)
1
23
4
4
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 33
Screenshot (Ultra VNC)
1
2 3
4
4
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 34
Screenshot (BASS)
1
2
4
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 35
Comparison of Sharing Systems
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 36
VNC problems
• Client-pull based• No multicast support• Same encoding for all updates• CPU usage increases for each new user•
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 37
TeleTeachingTool (VNC based)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 38
Information about BASS
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 39
Supported Platforms/OS
Server Client*Windows + +*nix +Mac OS X +
-+-+
*Client is Java based.
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 40
Client (Viewer) Architecture
• Client can– Connect to server– Wait for incoming connections
• Client supports– TCP– UDP (+Multicast)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 41
BASS Windows Server Architecture
Kernel space
User space
Mirror Driver
0X 300 ,500Y 200 ,600
Server Main Thread
Window coordinates
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 42
BASS Windows Server Architecture
X 320 ,340Y 100 ,200Kernel space
User space X 320 ,340Y 100 ,200
X 320 ,340Y 100 ,200
X 320 ,340Y 100 ,200
Mirror Driver
2X 300 ,500Y 200 ,600
Server Main Thread
Window coordinates
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 43
BASS Windows Server Architecture
Kernel space
User space X 320 ,340Y 100 ,200
X 320 ,340Y 100 ,200
X 320 ,340Y 100 ,200
Mirror Driver
2X 300 ,500Y 200 ,600
Server Main Thread
Window coordinates
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 44
BASS Windows Server Architecture
K e r n e l s p a c e
U s e r s p a c e X 3 2 0, 3 4 0Y 4 5 0, 5 5 0
X 4 2 0, 4 4 0Y 2 0 0, 6 0 0
X 3 2 0, 3 4 0Y 4 0 0, 5 0 0 2
S e r v e r M a i n T h r e a d
W i n d o w c o o r d i n a t e s
C l i e n t T h r e a d- 1
C l i e n t T h r e a d- 2
C l i e n t T h r e a d- 3
X 3
20,3
40
Y 4
00,5
00
times
tamp
PN
G
X 32
0,340
Y 40
0,500
time
stam
pP
NG
X 32
0,34
0Y
400
,500
time
stamp
PN
G
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 45
Challenges
• Different client bandwidths/speeds• Late Joiner• The effects of packet loss• Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 46
Multimedia Support (Movies)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 47
Multimedia Support (Movies)
• Our system uses PNG to compress and transmit the region updates
• PNG is lossless and effective for computer generated images but ineffective for real world captures like pictures or movies
• JPG is more suitable for photographic images• However, JPG is lossy and not effective for
computer generated images (text, line, shapes,...)
• Our system should use both
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 48
Multimedia Support (Movies)
• Composite image comparing JPEG and PNG: notice artifacts in JPEG versus solid PNG background.
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 49
Multimedia Support (PNG vs JPG)
Size x
Size 4x
0
20
40
60
80
100
120
140
160
Image Size
PNGJPG
Size x360x150162K
Size 4x720x300648K
1:301:20
1:51:4
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 50
Multimedia Support (PNG vs JPG)
Ethernet (60Mb/s)Size x Size 4x
0
2.5
5
7.5
10
12.5
15
17.5
20
22.5
FPS
PNG
JPG
Size x Size 4x0
100
200
300
400
500
600
700
800
900
1000
1100
KB/s
PNG
JPG
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 51
Multimedia Support (PNG vs JPG)
Size x Size 4x0
2.5
5
7.5
10
12.5
15
17.5
20
22.5
FPS
PNG
JPG
Size x Size 4x0
100
200
300
400
500
600
700
800
900
1000
1100
KB/s
PNG
JPG
Wireless (4Mb/s)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 52
PNG/JPG Detection Algorithm
Region>40,000px
?
YES
New Region
?
NO
Use DetectedFormat
YES
-1,0,1
coordinates
PNG Size
Time Stamp
counter
Region recordCreate a record&
Start Checking
Detected?
Continue Checking
NO YES
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 53
Sharing a Movie (Media Player)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 54
Sharing a Movie File
• Capturing from the Frame Buffer is expensive.• Instead
– Transcode the movie to Theora beforehand
• Then stream the theora directly to participants– Java client supports theora playback
• Negligible CPU usage during playback on the server–
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 55
Sharing a Movie (Our Method)
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 56
Sharing a Movie File
Movie File (wmv,mpg,divx)Movie File (wmv,mpg,divx)
Theora Movie
FFMpeg2Theora
Java Streaming
Server
Java Client
Java Client
UDP/Multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 57
Challenges
• Different client bandwidths/speeds• Late Joiner• The effects of packet loss• Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 58
Different Client Bandwidths/Speeds
S e r v e r M a i n T h r e a d
C l i e n t T h r e a d- 1
C l i e n t T h r e a d- 2
C l i e n t T h r e a d- 3
X 3
20,3
40Y
400
,500
time
stamp
PN
G
X 3
20,3
40Y
400
,500
timesta
mp
PN
G
X 3
20,3
40
Y 4
00,5
00
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
times
tam
pP
NG
X 32
0,340
Y 40
0,500
time
stamp
PN
G
X 32
0,3
40Y
400
,500
time
stamp
PN
G
1 M b p s
2 M b p s
3 M b p s
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 59
Different Client Bandwidths/Speeds
• Possible Solutions– Slowest one
– Average speed
– Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 60
Different Client Bandwidths/Speeds
• Possible Solutions– Slowest one
• Problem: Penalize everybody except the slowest
– Average speed
– Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 61
Different Client Bandwidths/Speeds
• Possible Solutions– Slowest one
• Problem: Penalize everybody except the slowest
– Average speed• Possible solution
– Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 62
Different Client Bandwidths/Speeds
• Possible Solutions– Slowest one
• Problem: Penalize everybody except the slowest
– Average speed• Possible solution (Can we do better?)
– Fastest one
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 63
Different Client Bandwidths/Speeds
• Possible Solutions– Slowest one
• Problem: Penalize everybody except the slowest
– Average speed• Possible solution (Can we do better?)
– Fastest one• The best solution• Client bandwidths are fully utilized
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 64
S e r v e r M a i n T h r e a d
C l i e n t T h r e a d- 1
C l i e n t T h r e a d- 2
C l i e n t T h r e a d- 3
X 3
20,3
40Y
400
,500
time
stamp
PN
G
X 3
20,3
40Y
400
,500
timesta
mp
PN
G
X 3
20,3
40
Y 4
00,5
00
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
times
tam
pP
NG
X 32
0,340
Y 40
0,500
time
stamp
PN
G
X 32
0,3
40Y
400
,500
time
stamp
PN
G
1 M b p s
2 M b p s
3 M b p s
Different Client Bandwidths/Speeds
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 65
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 3
20,340
Y 4
00,500
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
timestam
pP
NG
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG
1Mbps
2Mbps
3Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 66
Challenges
• Different client bandwidths/speeds• Late Joiner• The effects of packet loss• Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 67
Late Joiner
• Force server to generate full screen update
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 68
Late Joiner
• Force server to generate full screen update– Problems
• Misbehaving clients can degrade performance• If Join/Leave rate is high, too much burden on server
– Solution• Generate full screen updates if really necessary• Otherwise start the new client from last full screen update
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 69
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG
1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 70
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG
1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
FIR
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 71
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
timestam
pP
NG
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG
1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 72
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
timestam
pP
NG
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG
1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 73
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
timestam
pP
NG
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG
1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 74
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
timestam
pP
NG
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG
1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 75
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
timestam
pP
NG
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
Client Thread -5
2Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 76
Different Client Bandwidths/Speeds
Server Main Thread
Client Thread -1
Client Thread -2
Client Thread -3
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
timesta
mp
PN
G
X 32
0,340
Y 40
0,500
timestam
pP
NG
X 32
0,34
0Y
400,5
00
time
stam
pP
NG
X 3
20,34
0Y
400
,500
timesta
mp
PN
G
X 3
20,340
Y 4
00,500
time
stamp
PN
G
X 32
0,34
0Y
400,5
00
timestam
pP
NG1Mbps
2Mbps
3Mbps
Client Thread -4
8Mbps
Client Thread -5
2Mbps
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 77
Challenges
• Different client bandwidths/speeds• Late Joiner• The effects of packet loss• Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 78
The effects of Packet Loss
• This problem applies to – Multicast– UDP
• The PNG images can be large– Regular desktop can be ~900KB– ~600 Ethernet packets– One packet loss wastes all PNG image
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 79
The effects of Packet Loss
• Solution– Small PNG images
• Around ~1500 bytes• Consist of a few scanlines
– Disadvantages• Increased CPU usage (client&server)• Lower compression ratio (%20 lower)
– Advantages• One packet loss = no update for a few scanlines
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 80
The effects of Packet Loss
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 81
Challenges
• Different client bandwidths/speeds• Late Joiner• The effects of packet loss• Reliable multicast
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 82
Reliable Multicast
• RTP Library stores last N rtp packets
• Clients send NACK for lost packets
• RTP Library resend the requested packets
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 83
The effects of Packet Loss
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 84
Overview
• Introduction• Demo• Architecture• Challenges• Features• Conclusion
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 85
Recording
• Clients can record the whole/part session• Anybody can play these files locally• These files can be streamed to receivers via
streaming server• Streaming server supports multiple receivers
– Also late joiners
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 86
Listening Client
• Client waits for incoming connections• It can display windows from multiple user• Can be used for RGB cable replacement
Boyaci & Schulzrinne, draft-boyaci-avt-app-sharing-00 87
Conclusion
• Application sharing allows users to share a single application with multiple participants.
• Participants don't need the application.
• It is not specific to a single application.
• Extra features like recording is added.