Evaluation of the 2-way Opteron 1U system Klaus Schossmaier CERN EP-AID Computing Seminar 3...

Evaluation of the 2-way Opteron 1U system

Klaus Schossmaier

CERN EP-AID

Computing Seminar

3 September 2003

Performance test of PCs based on AMD platforms

Computing Seminar, 3 Sep 2003

22-way Opteron 1U system

Welcome

The EP-AID group constitutes the core team of the Data Acquisition (DAQ) System for the LHC Experiment ALICE

Development of the DAQ software called DATE

Deployment of the DAQ system Computing fabric: ~500 nodes, GbE switches, ~400 optical links Permanent recording during Pb-Pb run: 1.25 GB/s during 1 month/year

Selecting products Evaluation of Linux platforms Intel: Pentium II, Pentium III, Xeon AMD: AthlonMP (32bit), Opteron (64 bit)



Overview

In March 2003 we received the Newisys 2100 evaluation system (“Beachhead”)

This 2-way 1U enterprise-class server is based on AMD Opteron processors (x86-64 architecture)

Our evaluation was done under NDA: Hardware, chassis, BIOS Linux operating systems Development tools Benchmarking Applications



Chassis & Periphery

Rack mountable Size: 42mm x 428mm x 725mm Robust construction

Easy to mount the rack module Easy to open the cover (no screws) Solid connectors

Not very hot or noisy! Good documentation

Labeling, e.g. MAC addresses Newisys “Getting Started Guide” Newisys support website



Hardware – Front View

2x hard disk bays:

- empty

- 36.7GB SCSI

hot swap

CD-ROM drive

Floppy drive

Operator panel

for the SP

4x 1GB DIMM

DDR266 SDRAM

4x 1GB DIMM

DDR266 SDRAM

2x Opteron 240, 1 MB L2 cache, 1388 MHz

2x Opteron 244, 1 MB L2 cache, 1793 MHz

Power supply

465W

(March – May)

(June – now)



Hardware – Rear View

PCI-X slot 64/133

full-length

PS/2 mouse

keyboard

SVGA video

Trident Blade 3D

8 MB

2x 10/100BaseT

for the SP

Service processor (SP)

2x 100/1000BaseT

BCM5703X chip

PCI-X slot 64/100

half-length

AMD-8131 PCI-X bridge AMD-8111 Southbridge



Booting & BIOS

Booting/Resetting/Shutdown is fast (~10 sec)

PhoenixBIOS 4.0 Release 6.0 Easy to enter the BIOS setup Well designed configuration menus Upgrade via the phlash16 flash-utility or the SP

System BIOS Revision 00.09.06 initial 00.09.08 bug fix 01.08.00 Opteron 1.8 GHz 01.09.00 Opteron “C” step



Linux Operating Systems

CERN Red Hat 7.3.2 Linux (2.4.18 kernel) Installation via boot floppy and NFS Everything works without any change in 32-bit mode

UnitedLinux 1.0 (2.4.19 kernel) Consortium of Conectiva, SCO, SuSE, Turbolinux Installation via CD-ROM (4 disks) using YaST Runs in 64-bit mode

Red Hat 8.0.95 = GinGin64 Installation via CD-ROM (6 disks) failed



Development Tools

GNU 2.96 C/C++ compiler default for Red Hat Linux installations only for 32-bit executables

GNU 3.2.1 C/C++ compiler default for UnitedLinux installations for 64-bit executables, 32-bit compatibility libraries

Portland Group Compiler Technology (PGI) Workstation 5.0-beta Fortan/C/C++ compiler commercial product for 64-bit executables

Other development tools (e.g. numerical libraries, debuggers, code migration, performance analyzers) exist, but not tested.



Benchmarking

CPU benchmarking Ubench 0.32, gcc 2.96, 32-bit mode Ubench 0.32, gcc 3.2.1, 64-bit mode bench/stress rootmarks, 64-bit mode

Memory bandwidth Stream 4.0, gcc 2.96, 32-bit mode Stream 4.0, gcc 3.2.1, 64-bit mode

Ethernet performance Iperf 1.2, gcc 2.96, 32-bit mode

Disk performance Bonnie, 64-bit mode



0

20000

40000

60000

80000

100000

120000

0 500 1000 1500 2000 2500 3000

MHz

ub

ench

/#C

PU

Pentium II Pentium III Opteron Athlon MP Xeon Itanium

Benchmark - CPU (Ubench)

64-bit

74955

97749

52122

66714

8806479798

1.0 GHz1.4 GHz

1.8 GHz

2.4 GHz



Benchmark - CPU (ROOT)

Done by Fons Rademakers (EP-SFT) Porting of ROOT and CINT was trivial

OO data analysis framework with ~900000 locs Porting to x86-64 under United Linux 1.0 in one afternoon Code was already 64-bit safe (no special x86-64 ifdef’s) No problem with gcc 3.2.1

Performance is very good

Opteron 1.4 GHz

1 MB cache, 8 GB RAM

Opteron 1.8 GHz


Itanium2 1.0 GHz


P4 Xeon 3.06 GHz

512 KB cache, 2 GB RAM

750 rootmarks

g++ 3.2.1 (-O2)

950 rootmarks

g++ 3.2.1 (-O2)

497 rootmarks

g++ 3.2.3 (-O2)

750 rootmarks

g++ 3.2.3 (-O2)

550 rootmarks

32-bit binary compiled on

P4 with g++ 3.2.2 (-O2)

1020 rootmarks

ecc 7.1 (-O)



Benchmark - Memory

1x Stream: 2x Stream: 4x Stream:

2x Opteron, 1.8 GHz, HyperTransport: 1006 – 1671 MB/s 975 – 1178 MB/s 924 – 1133 MB/s

2x Xeon, 2.4 GHz, 400 MHz FSB: 1202 – 1404 MB/s 561 – 785 MB/s 365 – 753 MB/s

4x DIMM

1GB DDR266

Avent Techn.

4x DIMM

1GB DDR266

Avent Techn.



Benchmark - Ethernet, Disk

Network benchmarking 2x Broadcom BCM5703X on motherboard 10/100/1000 Mbps Ethernet, auto-negotiation bcm5700 kernel module, 64-bit mode up to 105 MB/s, stable traffic during 4 days 20%+18% sys CPU as TCP/IP receiver 20%+2% sys CPU as TCP/IP sender

Disk benchmarking Seagate ST336753LC, 15kRPM LSI53C1030 dual channel Ultra320 SCSI controller 26.7 MB/s for writing a 6 GB file 36.2 MB/s for reading a 6 GB file



Service Processor (SP)

Dedicated SP on-board PowerPC based Own IP name/address Front panel Command line interface Web-server

Remote administration System status Boot/Reset/Shutdown Flash the BIOS



Applications - DAQ

Front-end computer / Event-Builder DATE v4.6 Red Hat 7.3.2 32-bit mode, binary compatible! Event-Building: 113 MB/s with 2 sources, 80% CPU Porting to 64-bit mode needs to be done

DAQ services NFS server Run control, message logger, database Online monitoring: performance, data quality



Applications – non DAQ

MySQL 4.0.13 for x86-64 64-bit mode, binary distribution UnitedLinux 1.0 ~3300 remote INSERT queries (80 chars) per sec

Web Servers HEP computing

Data analysis Simulation tools Visualizations

ALICE Performance

Data size

Hits ~1.4Gb

Digits ~1.1Gb

CPU time on 800MHz PIII

Hits ~24h

Digits ~15h

dNch/d ~ 8000

slice: 2o in

TPC

5000 PbPb events at 5.5 TeV/nucleon pair

were producedfor PPR

84’210 primary particles

ITS

PHOS

TRD

TOF

HMPID



Résumé

Merits Design: chassis, hardware, BIOS Execution: seamless 32-bit & 64-bit mode Benchmarking: CPU, memory, Ethernet, disk Applications: DAQ services, analysis, simulations

Improvements Linux operating systems: more in 64-bit mode Tool chain: more non-commercial tools

Questions Cost: processor and whole system Roadmap: AMD’s 64-bit processors

Thanks AMD: Pierre-Yves Ferrard, Francesco Torricelli, Bill Zipoy CERN ALICE: Pierre Vande Vyvre



References

Web pages www.amd.com/opteron www.x86-64.org www.hypertransport.org www.newisys.com

Technical Reports “Performance tests of PCs based on the AMD AthlonMP”,

ALICE-INT-2003-020 A report about the 2-way Opteron system is forthcoming

Further Talks 15 Sep 2003: “Linux platforms for the ALICE DAQ”

Access to our Opteron machine Please ask me for an account

Evaluation of the 2-way Opteron 1U system Klaus Schossmaier CERN EP-AID Computing Seminar 3...

Documents

Transcript of Evaluation of the 2-way Opteron 1U system Klaus Schossmaier CERN EP-AID Computing Seminar 3...