The Effect of Router Buffer Size on Subjective Gaming Quality Estimators based on Delay and Jitter

THE EFFECT OF ROUTER BUFFER SIZE ON SUBJECTIVE

GAMING QUALITY ESTIMATORS BASED ON DELAY AND JITTER

GTCTechnologies GroupCommunication

Jose Saldana Julián Fernández-Navajas

José Ruiz-Mas Eduardo Viruete Navarro

Luis Casadesus University of Zaragoza, Spain

Index - I. Introduction

- II. Related Works

- III. Tests and Results

- IV. Conclusions

CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012

Introduction - The Internet was not designed

for real-time services.

- First deployed ones: e-mail, file transfer


Introduction - But real-time services are being

widely used: VoIP, video conference, online gaming


- Problem: Using a best effort network for a real-time service.

- Users demand a quality similar to the one they are used to.

- Research: Find the relationship between network impairments and perceived quality.

- Delay, packet loss, bandwidth, jitter.

Introduction


Introduction - E-Model: ITU’s solution that

estimates perceived quality of voice, as a function of delay, packet loss, codec, etc.

- Battery of surveys in order to obtain a MOS (Mean Opinion Score) model


FPS online games - Very stringent Real-time

requirements:

- Interactivity (video)

- Players: Difficult to satisfy


FPS online games - Traffic characteristics

- UDP

- Small packets (100 bytes maximum)

- High frequency (25 to 85 pps)

- A universal MOS does not exist

- Some games are more sensitive to delay, or packet loss, or jitter, etc.


- II. Related Works


- IV. Conclusions

II. Related Works

- Buffer sizing problem

- Measuring subjective quality for online games


Buffer sizing problem - Players mainly use access networks

- We are considering low-end routers, i.e. the ones we can find in access networks

- Drop-tail FIFO tiny buffers, of some tens of kilobytes


Buffer sizing problem - Different proposals:

- Rule of the thumb: C x RTT

- Maximize buffer occupancy

- Too much delay

- Stanford model: C x RTT / sqrt(N)

- Tiny buffer: 20 to 50 packets


Buffer sizing trade-off

Bigger Buffer

Bigger delays

Bigger jitter

Smaller packet

loss

SmallerBuffer

Smallerdelays

Smallerjitter

Biggerpacket

loss

II. Related Works

- Buffer sizing problem

- Measuring subjective quality for online games


Subjective quality for FPS - MOS as a function of SRT (linear dependence)

C. Schaefer, T. Enderes, H. Ritter, M. Zitterbart. “Subjective quality assessment for multiplayer real-time games”. In Proc 1st workshop on Network and system support for games (NetGames '02). ACM, New York, NY, USA, 74-78. 2002.

- Influence of delay and packet loss (not developing a MOS)

S. Zander, G. Armitage, “Empirically Measuring the QoS Sensitivity of Interactive Online Game Players”. In Proc. Australian Telecommunications Networks & Applications Conference (ATNAC 2004), Sydney, Australia, Dec. 2004.

- Packet loss: different behavior:

- Halo: Does not work with 4% loss

- Quake III: Works with 35% loss


Subjective quality for FPS - Only delay and jitter, but separately

M. Dick, O. Wellnitz, L. Wolf. “Analysis of factors affecting players' performance and perception in multiplayer games”. In Proc. 4th ACM SIGCOMM workshop on Network and system support for games (NetGames '05). ACM, New York, NY, USA, 1-7, 2005.

- First developed MOS, adapted from E-model A. F. Wattimena, R. E. Kooij, J. M. van Vugt, O. K. Ahmed, “Predicting the perceived quality of a first person shooter: the Quake IV G-model”. In Proc. 5th SIGCOMM workshop Network and system support for games (NetGames '06), ACM, New York, NY, USA, 2006.

- Only delay and jitter were considered

- Game: Quake IV

- G-model: We will use it to study quality


- II. Related Works


- IV. Conclusions


Previous results with VoIP - E-Model uses delay and packet

loss.

- Jitter is not considered, as a de-jitter buffer is used


Previous results for VoIP - MOS presents a monotonically decreasing

behavior with background traffic

1

1.5

2

2.5

3

3.5

4

4.5

5

400 450 500 550 600 650 700 750 800 850 900 950 1000

MO

S

background traffic (kbps)

E-Model MOS

1 call

5 calls

10 calls

15 calls

20 calls


Scenarios of interest - A number of players connected

to the server sharing the same router

.

.

. Game Server

Users

buffer

Internet

Router

Game &

background

traffic


Test methodology - The same scenario as in VoIP

- FPS game with a MOS: Quake IV

- Traffic traces available from CAIA project

- Only client-to-server (the most restrictive one)


Test methodology - Inter-packet time and packet

size histograms - 40.7 kbps / user

40 50 60 70 80 90 100 110bytes

0 10 20 30 40 50 60 70ms

79.5 bytes avg

64 pps


Test methodology - 20 players sharing an Internet

connection

- Buffer: Drop-tail, fixed kB size

Bandwidth Buffer size 2 Mbps 10 kB

3 Mbps 20 kB

50 kB

100 kB


Test methodology - Background traffic sharing the

connection

- 50% packets 40 bytes

- 10% packets 576 bytes

- 40% packets 1,500 bytes

- Network RTT (avg 30 ms) added offline

.

.

. Game Server

Users

buffer

Internet

Router

Game &

background

traffic


Test results: delay

0

50

100

150

200

250

300

350

400

450

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms


RTT, 2 Mbps

10 kB

20 kB

50 kB

100 kB


Test results: delay

0

50

100

150

200

250

300

350

400

450

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms


RTT, 2 Mbps

10 kB

20 kB

50 kB

100 kB

Bandwidth limit 40.7 x 20 = 814 kbps


Test results: delay

0

50

100

150

200

250

300

350

400

450

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms


RTT, 2 Mbps

10 kB

20 kB

50 kB

100 kB Unacceptable delay


Test results: delay

0

50

100

150

200

250

300

350

400

450

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms


RTT, 3 Mbps

10 kB

20 kB

50 kB

100 kB


Test results: delay

0

50

100

150

200

250

300

350

400

450

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms


RTT, 3 Mbps

10 kB

20 kB

50 kB

100 kB Unacceptable delay


Test results: jitter

0

10

20

30

40

50

60

70

80

90

100

110

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms


Jitter (delay standard deviation), 2 Mbps

10kB

20 kB

50 kB

100kB



0

10

20

30

40

50

60

70

80

90

100

110

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms



10kB

20 kB

50 kB

100kB


Above the bandwidth limit, the jitter

decreases, as the buffer is always full



0

10

20

30

40

50

60

70

80

90

100

110

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms



10kB

20 kB

50 kB

100kB



0

10

20

30

40

50

60

70

80

90

100

110

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

ms



10kB

20 kB

50 kB

100kB

The peak is reduced when bandwidth grows


Test results: packet loss

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Packet Loss, buffer 10kB, 2 Mbps

game (79 bytes avg)

40 bytes

576 bytes

1500 bytes



0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000



game (79 bytes avg)

40 bytes

576 bytes

1500 bytes

Small packets have a clear advantage in a drop-tail

size limited buffer



0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000



game (79 bytes avg)

40 bytes

576 bytes

1500 bytes


Test results: MOS - G-Model MOS formula:

x = 0.104*ping_average + jitter_average

MOS = -0.00000587 x3 + 0.00139 x2- 0.114 x + 4.37

- Packet loss is not considered, unless it is above 35%


Test results: MOS

1

1.5

2

2.5

3

3.5

4

4.5

5

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

MO

S


G-Model MOS, 2 Mbps

10 kB

20 kB

50 kB

100 kB


Test results: MOS

1

1.5

2

2.5

3

3.5

4

4.5

5

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

MO

S


G-Model MOS, 2 Mbps

10 kB

20 kB

50 kB

100 kB

Good quality

Medium quality

Bad quality


Test results: MOS

1

1.5

2

2.5

3

3.5

4

4.5

5

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

MO

S


G-Model MOS, 2 Mbps

10 kB

20 kB

50 kB

100 kB

Delay and jitter grow

Delay grows, but jitter decreases

Not monotonically decreasing


Test results: MOS

1

1.5

2

2.5

3

3.5

4

4.5

5

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

MO

S


G-Model MOS, 2 Mbps

10 kB

20 kB

50 kB

100 kB

Small buffers present the best

behavior


Test results: MOS

1

1.5

2

2.5

3

3.5

4

4.5

5

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

MO

S


G-Model MOS, 3 Mbps

10 kB

20 kB

50 kB

100 kB


Test results: MOS

1

1.5

2

2.5

3

3.5

4

4.5

5

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

MO

S


G-Model MOS, 3 Mbps

10 kB

20 kB

50 kB

100 kB

Good quality

Medium quality

Bad quality


Test results: MOS

1

1.5

2

2.5

3

3.5

4

4.5

5

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

MO

S


G-Model MOS, 3 Mbps

10 kB

20 kB

50 kB

100 kB

Acceptable MOS values above

bandwidth limit


- II. Related Works


- IV. Conclusions


Conclusions - Access networks with low-end routers - Importance of the buffer size - Small buffers are better for real-time apps - Buffer implementation can penalyze big

packets - We cannot separately study each network

impairment - Need for subjective quality estimators to

calculate MOS - If delay and jitter are the considered

impairments, the jitter peak produces a MOS valley

THANK YOU

GTCTechnologies GroupCommunication [email protected]

Extra Slides


Online games: genres

Real-time strategy Sports

MMORPG FPS


FPS online games - Delay: Very important

The Effect of Router Buffer Size on Subjective Gaming Quality Estimators based on Delay and Jitter

Education

Transcript of The Effect of Router Buffer Size on Subjective Gaming Quality Estimators based on Delay and Jitter