Operation of the CERN Managed Storage environment; current status and future directions CHEP 2004 /...
-
Upload
alannah-ray -
Category
Documents
-
view
223 -
download
0
description
Transcript of Operation of the CERN Managed Storage environment; current status and future directions CHEP 2004 /...
Operation of the CERN Managed Storage
environment;current status and future
directions CHEP 2004 / Interlaken
Data Services team:Vladimír Bahyl, Hugo Caçote, Charles Curran, Jan van Eldik, David Hughes,
Gordon Lee, Tony Osborne, Tim Smith
2004/09/29 CERN Managed Storage: [email protected] 2 of 18
Managed Storage Dream
Free to open… Instant access Any time later… Unbounded recall Find exact same coins Goods integrity
0011010101001100111011111011
2004/09/29 CERN Managed Storage: [email protected] 3 of 18
Managed Storage Reality
Maintain + upgrade, innovate + technology refresh Ageing equipment, escalating requirements
Dynamic store / Active Data Management
0011010101001100111011111011
TapeStore
DiskCache
2004/09/29 CERN Managed Storage: [email protected] 4 of 18
CASTOR Service
CERN Managed Storage
Disk Cache
Tape Store
Stage Servers
CAST
OR S
erve
rs
CASTOR Grid Service
GRIDftp serversSRM Service
ReliabilityUniformityAutomation
New ServiceScalability
RedundancyScalability
Tape StoreTape StoreTape StoreTape StoreTape Store
Disk Cache
Stage ServersDisk Cache
Stage ServersDisk Cache
Stage ServersDisk Cache
Stage ServersDisk Cache
Stage ServersDisk Cache
Stage ServersDisk Cache
Stage Servers
Tape StoreTape Store
42 stager/disk caches370 disk servers6,700 spinning disks
70 tape servers35,000 tapes
HighlyDistributed System
2004/09/29 CERN Managed Storage: [email protected] 5 of 18
CASTOR Service Running experiments
CDR for NA48, COMPASS, Ntof Experiment peaks of 120MB/s
Combined average 10TB/day Sustained 10MB/s per dedicated 9940B drive
Record 1.5 PB in 2004 Pseudo-online analysis
Experiments in the analysis phase LEP and Fixed Target
LHC experiments in construction Data production / analysis (Tier0/1 operations) Test beam CDR
2004/09/29 CERN Managed Storage: [email protected] 6 of 18
Quattor-ising Motivation: Scale (See G.Cancio’s talk)
Uniformity; Manageability; Automation Configuration Description (into CDB)
HW and SW; nodes and services Reinstallation
Quiescing a server ≠ draining a client! Gigabit cards gymnastics; BIOS upgrades for PXE Eliminate peculiarities from CASTOR nodes Switches misconfigurations, firmware upgrades (ext2 -> ext3)
Manageable servers
2004/09/29 CERN Managed Storage: [email protected] 7 of 18
LEMON-ising Lemon agent everywhere
Linux box monitoring and alarms
Automatic HW static checks
Adding CASTOR server specific Service monitoring
HW Monitoring temperatures, voltages, fans etc
lm_sensors -> IPMI (see tape section) disk errors; SMART
smartmontools auto checks; predictive monitoring
tape drive errors; SMARTUniformly monitored servers
2004/09/29 CERN Managed Storage: [email protected] 8 of 18
Warranties
0
50
100
150
200
250
300
350
400
450
500
Jan-00 Jan-01 Jan-02 Jan-03 Jan-04 Jan-05 Jan-06 Jan-07 Jan-08 Jan-09
Num
ber o
f Dis
k Se
rver
s
Out of WarrantyX - 2.8 GHzX - 2.8 GHzELONEX - 2.4GHzELONEX - 2.4GHzELONEX - 2.0GHzELONEX - 1.1GHzJTT - 1.1GHzELONEX - 1GHzELONEX - 1GHzJTT - 1GHzELONEX - 900ELONEX - 900ELONEX - 900TECH - 800ELONEX - 700ELONEX - 650COGESTRA - 500ELONEX - 500ELONEX - 500COGESTRA - 450Out of Warranty
4th Generation
3rd Generation
2nd Generation
1st Generation
0th Generation
5th Generation
2004/09/29 CERN Managed Storage: [email protected] 9 of 18
Disk Replacement
10 months before case agreed: Head instabilities 4 weeks to execute
1224 disks exchanged (=18%); And the cages
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
Dec-03 Jan-04 Feb-04 Mar-04 Apr-04 May-04 Jun-04 Jul-04 Aug-04 Sep-04
% B
roke
n M
irror
s
1224 disks replaced
0th Generation
Unacceptablyhigh failurerate!
2004/09/29 CERN Managed Storage: [email protected] 10 of 18
Disk Storage Developments
Disk Configurations / File systems HW.Raid-1/ext3 -> HW.Raid-5+SW.Raid-0/XFS
IPMI: HW health monit. + remote access Remote reset + power-on/off (indep. of OS) Serial console redirection over LAN
LEAF: Hardware and State Management Next generations (see H.Meinhard’s talk)
360 TB SATA in a box 140 TB external SATA disk arrays
New CASTOR stager (JD.Durand’s talk)
2004/09/29 CERN Managed Storage: [email protected] 11 of 18
9940B228849840
8149
35908639
9940B50
984020
359014
LTO6
9940A4
Tape Service 70 tape servers (Linux)
(mostly) Single FibreChannel attached drives
2 symmetric robotic installations 5 x STK 9310 Silos in each
Drives Media
BulkphysicsFast
Access
Backup
2004/09/29 CERN Managed Storage: [email protected] 12 of 18
Chasing Instabilities Tape server temperatures?
2004/09/29 CERN Managed Storage: [email protected] 13 of 18
Media Migration Technology generations
Migrate data to avoid obsolescence and reliability issues in drives 1986 3480 / 3490 1995 Redwood 2001 9940
Financial Capacity gain in sub generations
2004/09/29 CERN Managed Storage: [email protected] 14 of 18
1% of A tapes unreadableon B drives – keep A drives
(drive head tolerances)
Media Migration
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
2001 2002 2003 2004
Num
ber o
f Tap
es
9940A60GB12MB/s
9940B200GB30MB/s
Replace A drivesby B drives
Capacity, Performance, Reliability
9 months;25% of Bresources
MigrateA to Bformat
2004/09/29 CERN Managed Storage: [email protected] 15 of 18
Tape Service Developments
Removing tails… Tracking of all tape errors (18 months)
Retiring of problematic media Proactive retiring of heavily used media (>5000
mounts) repack on new media
Checksums Populated writing to tape Verified loading back to disk
Drive testing Commodity LTO-2; High end IBM3592/STK-NG
New Technology; SL8500 library / Indigo
2004/09/29 CERN Managed Storage: [email protected] 16 of 18
CASTOR Central Servers Combined Oracle DB and Application
Daemons node Assorted helper applications distributed
(historically) across ageing nodes FrontEnd / BackEnd split
FE: Load balanced applications servers Eliminate interference with DB Load distribution, overload localisation
BE: (developing) clustered DB Reliability, security
2004/09/29 CERN Managed Storage: [email protected] 17 of 18
GRID Data Management GridFTP + SRM servers (Former)
Standalone / experiment dedicated Hard to intervene; not scalable
New load-balanced shared 6 node Service castorgrid.cern.ch DNS hacks for Globus reverse lookup issues SRM modifications to support operation
behind load balancer GridFTP standalone client
Retire ftp and bbftp access to CASTOR
2004/09/29 CERN Managed Storage: [email protected] 18 of 18
Conclusions
Stabilising HW and SWAutomationMonitoring and control
Reactive -> Proactive Data Management