Virtualization for the Win!Scaling Electronic Sports League’s

servers way up

Sreeram SammetaPaul Lindberg

Intel

Agenda

MMO hosting is well-understood, but hosting lots of LAN game servers can be hard.

What is virtualization?

Can we virtualize LAN game server code? Electronic Sports League (ESL) tests showed it can be done.

New hardware and software technologies let us virtualize more.

Do more with less!

Electronic Sports League*:Largest online gaming community in Europe

Electronic Sports League (ESL) has >1 million active members1

Mission-critical game servers

Sensitive to transaction latency

Often single-thread CPU intensive

1 Source: ESL web site August 19, 2009

Internet

Game Servers

…

Firewall

Industry knows how to host a “typical” MMO

Like any multi-tier IT shop

Dedicated network between tiers

SW/HW designed for throughput and availability,as needed

Database

Compute

Network

Internet

Game clients (1000s)

…

Hosting lots of LAN game servers can be harder

1 is easy How would

you host 100 LAN games?1000? 10,000?

Server

Internet

Game clients (16 typical)

Scaling up LAN game servers could be hard + expensive

LAN game server code sometimes not built to make hosting easy

Usually CPU-intensive Need lots of CPU “headroom”

Assumes it owns the machine? Network - single IP address? File system Impractical/unmanageable to run many server procs on

one machine

“Simple” way: A few game procs per server! …and lots of servers

But lots of servers $$, space, W

Is there a better way?

ESL had a challenge: Host lots of game servers without compromising

LAN game servers, like Counter-Strike* 1.6, are usually:

CPU Intensive: Single Thread vs. Multi-Thread Memory Intensive: Size/Throughput/Latency Network I/O Intensive: Throughput/Latency

How could ESL host lots of game servers (especially LAN games), to meet demand?

Must maintainQuality of Service (QoS)!

Virtualization can help!

Virtualization is a best practice used by many IT shops to consolidate servers for cost and efficiency

Can we really get enough performance in a virtual machine (VM) to satisfy gamers?

Perception: Can’t!

We say: Can! Here’s how…

Virtualization shares hardware

CS1 CS2 CS3

Windows* Server 2003

Physical hardware

CPU Memory

NetworkStorage

VMware* Hypervisor(OS + Virtual

Machine Manager)

Physical hardware

CPU Memory

NetworkStorage

Windows* Server 2003(1)

Windows* Server 2003(n)

Virtual HW Virtual HW

CS1

CS2

CS3

CS1

CS2

CS3

…

…

ESL Proof of Concept (PoC) showed it can be done

Hypothesis: Virtualization of gaming servers may be possible

Use the latest technologies

Intel® Xeon® 7400 processor based servers

Intel VMDq NICs VMware* ESX* 3.5U1 &

NetQueue* Test if virtualization adds

network latency, in the Intel lab

Private testing @ ESL lab Public testing on the

Internet with real ESL members

Success!

ESL Network

VM1 VMn

10 GbE VMDq

Firewall

10GbE Switch

InternetInternet

PoC Hardware makes it easier

Intel® Xeon® processor 7400 Series Performance boost from 6-core with 16 MB L3 cache Energy efficient boost from 45nm high-k process

technology Enhanced hardware assist features for virtualization

32 slots32 GB tested(256 GB max)

Configurable PCI Express

ESB2 I/O Bridge

4 sockets with 6-cores each

7400 Series

4x1066 MHz

FBDFBDFBDFBDFBD

FBDFBDFBDFBD

FBDFBDFBDFBDFBDFBDFBDFBDFBDFBD

Th

rou

gh

pu

t (G

b/s

)

0

2

4

6

10

8 >=60% of the NIC capacity unused

No VMDq

Network I/O can be slow if not tuned for virtualization

VMM overhead Switching load Interrupt bottleneck

Throughput measures receive side (Rx) I/O performance of 10GbE LAN.

Source: Intel.

VM1 VM2 VMnVirtual NIC Virtual NIC Virtual NIC

…

VirtualizationHypervisor

NIC

LAN

Queues for each VM give near-native throughput

VM1 VM2 VMn…

VMware* with

NetQueue*

NIC with VMDq

LAN

Virtual NIC Virtual NIC Virtual NIC

Th

rou

gh

pu

t (G

b/s

)

0

2

4

6

10

No VMDq

8

4.0

9.59.2

>2x throughput!Near native

10GbE

VMDq VMDq Jumbo Frames

Tests measure Wire Speed Receive (Rx) Side Performance With VMDq on Intel® 82598 10 Gigabit Ethernet Controller.Source: Intel.

VMDq & NetQueue* Optimize switching Load balance interrupts

PoC Software fits it together

VMware* ESX* 3.5 U1 Virtual Center* 2.5 NetQueue* enabled (16 queues) 1 virtual CPU per VM 2GB memory per VM

Windows* Server* 2003 32-bit Counter-Strike* 1.6

Intel® Xeon® Processor 7400 Series based Server

VMware® ESX 3.5 U1

Counter Strike 1.6

Windows Server 2003

32 bit

VM1

Counter Strike 1.6

Windows Server 2003

32 bit

VMn

It’s all about latency: Don’t make it worse

Player sends ~40-200 byte update to server

Server sends ~2000 bytes in return

In-game transaction latency = round-trip network latency + game server processing time

In-Game Latency (ms)

0

5

10

15

20

25

Best LANBest InternetTyp. Internet

Source: ESL observations.

VMDq keeps latency near native levels!

Virtualization with no VMDq increases latency VMDq latency is near-native Negligible effect on latency!

<< In-Game Transaction Latency (5 ms best case)

Source: Intel Lab. Performance measured using the netperf 2.4.4 (UDP latency test with 8 parallel streams) benchmark running on Intel® Xeon® processors 7300 (2.93 GHz).

64 256 10240

0.1

0.2

0.3

Native VMDq No VMDqPacket Size (bytes)

Avg.

late

ncy

(m

s)

Private ESL & Public Internet testing revealed no impact on In-Game Transaction Latency!

E SL v irtu a l g a m e s ervers o n X eo n 7 4 0 0

02 04 06 08 0

1 0 0

2 4 3 2 3 6 4 0

# o f V M s

CPU

(%)

6 0 0

6 5 0

7 0 0

7 5 0

Pow

er (W

atts)

Source: ESL Lab. Performance measured using esxtop & power meter with reference s/w stack running on Intel® Xeon® processors 7400 (2.67 GHz).

Live tests find ideal load

Replace 18 servers with 1!Before After

1P Core™2 Duo(2 cores)

4P Xeon 7400(24 cores)

6 game servers (3 per core)

108 game servers(3 per VM, 36 VMs)

72 gamers 1296 gamers

18x game servers per machine (18:1 consolidation ratio)

18x gamers per machineSame CPU headroom, same user

experience!Source: ESL Lab. Performance measured using esxtop & power meter

with reference s/w stack running on Intel® Xeon® processors 7400 (2.67 GHz). Power savings calculated based on ESL actual power rate

& Yahoo $/€ exchange rate as of 2008-08-12.

VMware* ESX 3.5 U1

Intel® Xeon® Processor 7400 series based server

Win 2003

CS (2)

CS (3)

CS (1)

Win 2003

CS (2)

CS (3)

CS (1)

18:1 consolidation yields big efficiency

VM (1) VM (36)

18 servers

into 1!

CS (1)

Windows Svr 2003Intel® Core™2 Duo

(1)

CS (6)

Intel® Core™2 Duo (18)

Windows Svr 2003

CS (1) CS (6)

18:1 consolidation gives big power savings

Before After

1P Core 2 Duo(2 cores)

4P Xeon 7400(24 cores)

350 W / machine(6300 W / 18 machines)

710 W / machine

Power $731K / 1000 machines

Power $83K / equivalent machines

Power savings: $648K per year for each 1000 machines converted!

Power and other factors give a good Total Cost of Ownership (TCO)

Source: ESL La. Performance measured using esxtop & power meter with reference s/w stack running on Intel® Xeon® processors 7400 (2.67 GHz). Power savings calculated based on ESL actual power rate & Yahoo $/€ exchange rate as

of 2008-08-12. Actual performance and savings may vary.

Source: ESL Lab. Performance measured using esxtop & power meter with reference s/w stack running on Intel® Xeon® processors 7400

(2.67 GHz). Power savings calculated based on 24x7x365 usage, ESL actual power rate & Yahoo $/€ exchange rate as of 2008-08-12.

ESL and gamers loved it!

"Playing on virtualized gameservers running on Intel® and VMware* technologies gives professional gamers no disadvantages compared with playing on a non virtualized server. Everything ran smoothly and I did not notice anything unusual. A perfect setup for professional gaming."

—Navid Javadi aka mousesports|Kapio

"The new Quad-core Intel® Xeon® processor 7400 series were completely overwhelming in all terms. The Intel Xeon MP processor … based servers with Intel VMDq technology enable us to efficiently run our servers with reduced costs and without any negative impacts."

—Bjoern Metzdorf

Director Information Technology

Electronic Sports League

New virtualization tech might help you, too! Do you have “non-virtualize-able”

apps? Really? Try them! Can have very low network latency Can consolidate many servers into 1

Consolidating servers can lead to big efficiency and power savings

Read more:http://software.intel.com/sites/billboard/

archive/performance-sensitive-application.php

24

Legal Disclaimer• INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH

INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.

• Intel may make changes to specifications and product descriptions at any time, without notice.

• All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.

• Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.

• Code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user

• Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance.

• Intel, Intel Inside, Xeon and the Intel logo are trademarks of Intel Corporation in the United States and other countries.

• *Other names and brands may be claimed as the property of others.• Copyright © 2009 Intel Corporation.