UVic Advanced Networking Day 28 November 2005 University of Victoria Research Computing Facility...
-
date post
20-Dec-2015 -
Category
Documents
-
view
214 -
download
2
Transcript of UVic Advanced Networking Day 28 November 2005 University of Victoria Research Computing Facility...
UVic Advanced Networking Day 28 November 2005
University of Victoria Research Computing Facility
Colin Leavett-Brown
UVic Advanced Networking Day 28 November 2005
Introduction
• Established in 1999
• Supports National Laboratories on Vancouver Island– NRC Herzberg Institute for Astrophysics
– NRCan Pacific Forestry Centre
– NRCan Pacific Geoscience Centre
• Broad range of research– Earth Science, Physical Sciences, Engineering
– Social Science, Medical Sciences
• Users across Canada, US and Europe
• Centre of Grid Activity (Forestry and Grid Canada)
UVic Advanced Networking Day 28 November 2005
Minerva HPC (1999 CFI Award)
• 8 node 8 way IBM SP
• Upgraded to 8 node 16 way in 2000
• 167th in the TOP 500 (highest ranking of any C3 facility?)
• Funded in part with an IBM Shared University Research (SUR) Award for $830,000
• Primary users: cosmology, climate simulations, engineering, and geosciences
• Still operational and utilized today.
UVic Advanced Networking Day 28 November 2005
Storage and Cluster (2001 CFI Award)
• 4 year project; online in March 2003
• Currently 80 TB disk and 440 TB tape capacity
• Primary users: Astronomy, Forestry, Particle Physics– Grid enabled for Forestry Data Grid
• IBM Beta Test Site for LTO2 (Jan-Apr 2003)
• Upgraded to LTO3 (March 2005)
UVic Advanced Networking Day 28 November 2005
Mercury & Mars Xeon Clusters
• Operational in March 2003
• 84 Dual Xeon blades (2.4 & 2.8 GHz)
• Top500 in 2003
• Now have 154 Dual Xeon blades (including 3.2GHz)
• Primary users: particle physics, engineering, geoscience
• Back-filled with particle physics simulations – 1 TB of data shipped to Stanford (SLAC) each week
• Grid-enabled and connected to Grid Canada– Particle physics and other simulation applications
– Goal is to back-fill other sites for simulation production
UVic Advanced Networking Day 28 November 2005
TAPOR Project (2001 CFI Award)
• TAPor (Text-Analysis Portal) project will help Canadian researchers in online textual research
• Six leading humanities computing centres in Canada, including UVic, will contribute collections of rare tapes and documents. – ISE, Robert Graves Diary, Cowichan Dictionary
– http://web.uvic.ca/hrd/tapor
• Database servers are integrated into the Storage Facility
UVic Advanced Networking Day 28 November 2005
Kikou Vector Computational Facility
UVic Advanced Networking Day 28 November 2005
Kikou Vector Computational Facility
• $2.42 M CFI Award (2004) for a Vector Computational Facility
• Purchased NEC-SX6– $5.3M
– Commissioned March 2005
– 4 Nodes, 32 Processors, 11.67 GFlops, 32/64GB, 10TB.
– Primary use for Climate Modelling (A.Weaver)
UVic Advanced Networking Day 28 November 2005
Llaima Opteron Cluster
• 2004 CFI/BCKDF awards.
• 40 Node, Dual Opteron (2.0 Ghz), 4GB.
• 20/40 Commissioned September 2005.
• Will have direct SAN attachment.
• Primary use is for Astronomy (S. Ellison)
UVic Advanced Networking Day 28 November 2005
Networking• Campus Networks are primarily optical networks with Cisco 6500
series routers.
• Connections to the access layer are generally twisted pair copper.
• Both a general use and a dedicated Research Network are provided (GE), interconnected via access layer.
• Research network used for I/O, node interconnect, and direct connections BCNet/CA*Net4.
UVic Advanced Networking Day 28 November 2005
Networking• External connections: commodity
network (100E), ORAN/CA*Net4 (optical & GE) through BCNet.
• CA*Net4/ to other national networks– Abilene, NLR, ESNET (US), GEANT
(EUR), JANET (UK), JAXA (JPN), KREONET (KOR), AARNET (AUS)
• Lightpaths near future (Jan 06)– ~1GB going to ~10GB
– Optical Switching: BCNet, Member Sites, TXs.
– Applications: HEPNet/C$ Testbed, CCCma, Babar/SLAC, ATLAS/TRIUMF, WG2
UVic Advanced Networking Day 28 November 2005
Infrastructure Challenges
• March 2002, 43KW
• Research Load:– March 2000, Minerva, 6KW
– March 2003, Storage, 30KW
– March 2003, Mercury & Mars, 40KW
– April 2004, TAPoR, 5kw
– March 2005, Kikou, 45 KW
– September 2005, Llaima, 20KW
• November 2005, 150KW
UVic Advanced Networking Day 28 November 2005
Infrastructure Challenges
• Ran out of space, power, & A/C.
• $700K renovation.
• Increased room by 32m².• Added 44 Tons of A/C.
• Added 225KW UPS.
• Still growing, time for another renovation.
UVic Advanced Networking Day 28 November 2005
Support
• Managed by the University of Victoria Computer Centre– Under the Research and Administrative Systems Support Group
– 3 FTE dedicated to the Research Computing
– Augmented by approximately 9 other sysadmins, network and hardware technicians
– 24/7 almost.
• C3 TASP Support– Position was used to assist with parallel programming support very
successfully.
– Position is currently vacant.
UVic Advanced Networking Day 28 November 2005
User community
• Minerva (~150 user accounts)– NRC, Queens, UBC, SFU, NRCan, Alberta, Washington,
Dalhousie
• Storage Facility– HIA and PFC use facility to distribute images to their user
community – TAPOR will provide a database service for social science
• Cluster (~60 user accounts not including Grid Canada)– NRC (Ottawa), NRCan (Geoscience and Forestry)– Montreal, UBC, TRIUMF, Alberta, NRC– Germany, UK, US (SLAC)
UVic Advanced Networking Day 28 November 2005
Plans for 2006• Storage
– Currently at the beginning of 4th year
– 200 TB disk and 1000 TB Tape
– Higher bandwidth, remove single points of failure.
– Minimise ongoing maintenance costs.
• Mercury Cluster– 50 blades per year
• Llaima– Commission remaining 20 nodes.
– Upgrade to multi-core processors?
• Kikou– Another 8-way node?
UVic Advanced Networking Day 28 November 2005
Future plans
• Next CFI competition– Renewal of Minerva HPC
– Continued support of Storage
– Modest expansion of the Mercury Cluster
– Upgrade of TAPoR
– Part of WG2
• Venus and Neptune Projects– $50 Million CFI projects to instrument the floor of the Pacific
shelf
– Venus (inland strait) will take data in 2005
– Neptune begins data taking in 2006-2007
UVic Advanced Networking Day 28 November 2005
Summary
• UVic RCF supports a broad range of users – A large fraction of the users are not part of the C3 community
• Wide range of computational and storage requirements– HPC, Cluster, Storage, Database, Grid
• Very successful– Operational very quickly and efficiently
– 2 machines in the TOP500
– Large storage repository
– Willingness to experiment: Beta test site, 2 Grid modes