Use of storage inside SWITCH

SWITCH, 11.02.2014

Jens-Christian Fischer jens-christian.fischer@switch.ch

Thorsten Kleindienst thorsten.kleindienst@switch.ch

© 2013 SWITCH

• Zürich CLA, ETHZ • Zürich RZ, ETHZ, 600 m distance to CLA • Lausanne, Amphimax, 220 km distance to Zürich

Locations

2

© 2013 SWITCH

• 28 HP ProLiant • 2 SPARC Enterprise T5220 • 2 SPARC Enterprise M4000 • 3 SPARC T3,4 • 6 Sun Blade T6320 Server modules • 15 Apple Xserves

Real server

3

© 2013 SWITCH

• 151 Debian • 53 Solaris • 23 Ubuntu • 10 Scientific Linux • 5 Windows • 2 Suse

Virtual server

4

© 2013 SWITCH

• 2 NetApp FAS2040, 45 TB netto space • 2 NetApp FAS2240, 40 TB netto space • 2 NetApp FAS3020, 6 TB netto space

Distributed on disk types • 75 TB SATA • 16 TB SAS • 0.4 TB SSD, FlashPool Cache

Central storage

5

© 2013 SWITCH

• KVM images via NFS, live migration possible • Solaris zones via iSCSI, no live migration • Unix-Homes via NFS • Shared Storage for miscellaneous projects • Backup of databases, e.g. Domain Registry

Usage of storage

6

© 2013 SWITCH

• 5 TB for backup on disk • 3 TB for KVM images • 2 TB for Unix-Homes, Project-shares, ... • 0.5 TB for internal mail • 0.2 TB for solaris zones

Consumption of storage on NetApp Filer

7

© 2013 SWITCH

• 95% asynchronous by means of Snapmirror • 5% synchronous for mail by NetApp MetroCluster

Replication

8

© 2013 SWITCH

• “Pre-Production/ownCloud” – 6 Physical servers – 192 cores – Ceph cluster: 140 TB Raw

– Status: Up and running

• Production – 2 Datacenters (Zurich, Lausanne) – 32 OpenStack Compute Nodes – 512 cores – 32 Ceph Storage Nodes – 768 disks, around 1.5 PB Raw Storage

– Status: Hardware has been ordered, go live: June 2014

OpenStack / Ceph

9

© 2013 SWITCH

•  Integration into OpenStack – Glance – images/snapshots

•  1 TB – Cinder – volumes

•  2.5 TB

• RadosGW – Used for S3 compatible applications (SWITCHtube)

•  200GB

Ceph @ SWITCH

10

© 2013 SWITCH

• ownCloud based service for academia • Potentially 100’000 users • Go Live: 1.4.2014

• Storage: – 20 TB (for now) volume on Ceph – Exposed via NFS server

– 20 TB Backup volume

SWITCHdrive

11

© 2013 SWITCH 12 ceph

OpenStack Virtualisiert

rbd volume

web1 web2 web3

ha1

nfs1

Architektur Diagramm SWITCHbox v0.4jcf, 8.1.2014

ldap1

reg

Internet

ha - HA Proxy Load Balancerreg: Registration / LDAP account creation

web: Apache Web Servernfs: NFS Server

ldap: LDAP diectory Serverdb: Postgres DB Server

db1

AAI

SWITCHdrive Architecture

© 2013 SWITCH 13

SWITCHdrive – Preliminary findings

• Good support from ownCloud • Found (and experienced) a number of bugs – working with

support and development to fix them • Performance of PHP application not satisfactory (500 –

1000ms to upload a tiny file) – up to 1000 SELECT statements for one file upload

• Scalability concerns: – Database size (projected: 128 - 256 GB) – Storage (single mountpoint) – Application server performance for single clients (no parallelism due

to session locking)

© 2013 SWITCH

• Stock Ubuntu NFS Server on a VM • 2 vCPUs, 4 GB RAM • 20 TB Volume for data (backed by RBD) • 20 TB Volume for file system backups (again Ceph RBD)

• At one point we asked – is this server a bottleneck?

NFS Server Performance

14

© 2013 SWITCH

NFS Server Performance

15

© 2013 SWITCH 16

NFS Server Read performance

© 2013 SWITCH 17

NFS Server Random Write performance

© 2013 SWITCH 18

NFS Server Random Read performance

© 2013 SWITCH

Q & A

19

© 2013 SWITCH

• KVM-Images by Filer snapshots once per day • Filesystem Backup by Tivoli Storage Manager: Backup on

tape solution of ETHZ

Backup

20