Post on 14-Jul-2015
Building High-Performance Inter-Cloud Infrastructure in Japan
Masaharu Munetomo
Professor & Vice Director,Information Initiative Center,Hokkaido University, Sapporo, JAPAN.
munetomo@iic.hokudai.ac.jp
HOKKA IDO UN IVERS ITY GU IDEBOOK 2014-2015
46Campus Maps
Campus Maps
1
2
3
4
5
6
7
8
9
10
11
P
P
P
P
P
To Hakodate Station
N
20
19
18
17
16
15
1413
12
11
10
9
8
7
8
6
5
4
3
2
1
... we are cosmopolitan, and accessible...
Picturesque Hakodate is home to Hokkaido University’s Faculty of Fisheries Science and is located on the south-west of the island.
With a population of approximately 280,000 people, the coastal city is at the base of Mount Hakodate, which boasts amazing natural beauty. The
view from the summit is renowned for having one of the most beautiful views in Japan, particulary at night. Since it opened in 1935, the Hakodate
Campus has had a proud history of being at the forefront of fisheries research and is one of the largest campuses of its kind in Japan.
Main Gate Main Building Annex BuildingMarine Bioresources Research Building Marine Frontier Research Building Marine Science Creative Research Building Lecture-room BuildingStudent Laboratories Controlled Environment RoomsTowing Tank RoomAuditorium
Hakodate Campus Map
Library Aquatic Biological Specimen House (Nakabe Hall)Fisheries Museum (main building) Fisheries Museum (annex) Gymnasium Student Center Student Activities Building Swimming Pool Athletic Field
12
13
14
15
16
17
18
19
20
Sapporo Campus
Hokkaido
Hakodate Campus
1
Masaharu Munetomo
• Professor & Vice director, Information Initiative Center, Hokkaido university, Sapporo, JAPAN.
• Chief examiner, Cloud computing research group of national supercomputing centers in Japan.
• Chief examiner, SIG Cloud, Academic eXchange for Information Environment and Strategy (AXIES) in Japan.
• Chief examiner, SIG Mathematical Problem-Solving, Information Processing Society of Japan (IPSJ)
• General advisor, Cloud Utilization Promotion Agency (CUPA) & Managed Service Providers associations in Japan (MSPJ)
• Founding member and of steering committee, Open Compute Project in Japan (OCPJ)
2
Information Initiative Center, Hokkaido University
• Founded in 1962 as a national supercomputing center.
• A member of High Performance Computing Infrastructure (HPCI) and Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructure (JHPCN) in Japan.
• University R&D center for Supercomputing, Cloud computing, Networking, IT systems for education
• Supercomputer (172TFlops) & Academic Cloud System (43TFlops)
@�(�*��[�( %!*��.9����
�7W1)�'*�%1�L�XN��"*��2G [@�(�*
��89=)+Y=)H4:<��"*���$*(�#�([,P=)
ZCQ=��"*���&(��#�(��B?������
;D�7S>3�RN EI�V��[57J������MO?A�/
-�Q0K�UF�@�(�*�6�T�����
���������� �
������
3
HPCI (High Performance Computing Infrastructure)
• Collaboration of national supercomputing centers in Japan.
• RIKEN AICS (K computer) & Supercomputing Centers (University, Research Institutes) connected via academic high-speed network (SINET4)
• Federations of users & systems management (GSI-SSH, Gfarm supported)
http://hpci-office.jp/4
Hokkaido University Academic Cloud System
• Largest Academic Cloud System in Japan started services from Nov. 2011: 43TFlops (5,000 cores), and more than 2,000 VMs can be deployed.
• Employing CloudStack to provide cloud management portal.
• High-performance cloud system: each physical node has 40-cores, 128GB memory. Network: 10GbE x 2, Shared Storage: 260TB (SAN) + 500TB (NAS) + 2PB (WebDAV, S3, Gfarm)
Hitach BladeSymphony BS2000 Xeon E7 8870 2.4GHz (10-core) x 4
128GB memory / 10GbE x 2
Hitachi NAS Storage AMS2300: 260TB AMS2500: 500TB 5
Use case: “Big Data” processing systems
• We provide “Big Data” service VM package consisting of Hadoop, Hive, Mahout, and R.
• Automated deployment of VM clusters, customizing scheduling policies in CloudStack to balance I/O overheads for cluster packages (Hadoop / MPI / Torque).
Storage #3�Virtual(Disk�
Storage #4�Virtual(Disk�
Storage #2�Virtual(Disk�
Zone!POD!
Shared Storage #1�Resource Pool #1�
HyperVisor #2�
HyperVisor #1�Virtual(Disk�VM(
Balancing!overheads!of!disk!I/O!with!round8robin!assignment!of!Virtual!disks.!�
Storage #1�
VM(
VM(
VM(
VM(
Virtual(Disk�Hadoop Cluster�Shared Storage #2�
Resource Pool #2�
HyperVisor #4�
HyperVisor #3�
Virtual(Disk�
VM(
Shared Storage #3�Resouce Pool #3�
HyperVisor #6�
HyperVisor #5�
Virtual((Disk�
VM(
Shared Storage #4�
Resouce Pool #4�
HyperVisor #8�
HyperVisor #7�
Virtual(Disk�
VM(
6
Use case: simulation environment to replace in-house computing servers or clusters
• Replacement of in-house clusters of laboratories employing L (10-core) or XL (40-core) project servers.
• Filling in the gap between PCs and super-computers.
7
Use case: development of in-silico screening system for drug design
• Center for Research and Education on Drug Discovery builds a Structure Based Drug Design (SBDD) system for in-silico screening with the academic cloud system
• A virtual private cloud system using XL servers (40-core): modeFRONTIER® and AutoDock are installed as docking applications.
AutoDock[1]
AutoDock[2]
AutoDock
AutoDock
AutoDock
AutoDockContinuous execution of analysis
servers
8
Use-case: Fishing ground prediction system
• Researchers in department of fishery build a fishing ground prediction system on Hokkaido university academic cloud system
• The system provides information on promising sea area for fishing boats to catch squids, employing satellite images and data assimilation results.
Portal System
Satellite image processing Data assimilation
Fishing ground prediction
INMARSAT
Satellite Earth station
Satellite Communications
Squid Fishing Boats Fishing ground prediction system portal
9
Use-case: Employing PaaS for scalable interactive evolutionary computation
• Building a scalable interactive evolutionary computation framework to evolve solutions according to the preferences of millions of users.
CloudStack
VM
Ubuntu
instance
VM
Ubuntu
Redis
VM
Ubuntu
Redis
VM
Ubuntu
Redis
Database
・・・
VM
Ubuntu
instance
VM
Ubuntu
instance
・・・
Applycation resource
iGA iGA iGA
Load Balancer
CloudFoundry
Sever
・・・
Interactive Evolutionary Computation using PaaS
Users select solutions according to their preferences
Present cadndates of solutions from the system
10
Japanese academic inter-cloud infrastructure
• Development of the inter-cloud system over Japanese universities to collaborate private clouds from Kitami (Northernmost) to Ryukyu (Southernmost) universities through Japanese academic high-speed network (SINET4).
Hokkaido University
Kitami Institute of Technology
University of Ryukyus (Okinawa)
National Institute of Informatics (NII)
@�(�*��[�( %!*��.9����
�7W1)�'*�%1�L�XN��"*��2G [@�(�*
��89=)+Y=)H4:<��"*���$*(�#�([,P=)
ZCQ=��"*���&(��#�(��B?������
;D�7S>3�RN EI�V��[57J������MO?A�/
-�Q0K�UF�@�(�*�6�T�����
���������� �
������
11
Related projects
• Remote collaborations of distributed cloud systems (JHPCN)
• Federations technologies development toward academic inter-cloud (Collaborative research project, National Institute of Informatics)
• Large-scale Distributed Design Exploration Framework (JHPCN)
• Development of distributed database infrastructure across Japan
• Inter-cloud resource optimization with multi-objective evolutionary algorithms
• Designing the next-generation Hokkaido university high-performance inter-cloud system
12
Remote collaborations of distributed cloud systems
• Prototyping an inter-cloud manager and authentication infrastructure for federation of academic cloud systems managed by different cloud middleware (CloudStack, OpenStack, etc.)
• Designing a VPC (Virtual Private Cloud) management framework in the distributed inter-cloud systems.
Cloud A IaaS �
Cloud B IaaS �
Cloud C IaaS �
User�
VPC 1�
Internet�
VM�VM�
VM�
VPC 2�
220km�
13
Large-scale Distributed Design Exploration Framework (LDDEF)• To establish a framework to support “parameter surveys” by
supercomputing simulations collaborating design engineers sharing information on promising solutions with distributed DBs
• “Multi-objective designexploration” explores Pareto-fronts stored indistributed DBs
• Optional info. Isstored in objectstorages forvisualization andanalysis Solutions DB
(distributed)
Automated replication for DR and
load balancing
Visualization
Simulation (Supercomputer)
Optimization & DB management (Cloud system)
Distributed Database
Product
14
Grid Unified Framework for Optimization (Grid-UFO) & MHGRID (Asim, Wahib, Munetomo, 2008-2010)
• A unified framework collaborating optimization algorithms libraries and simulation programs to evaluate “fitness” values registered by different developers in GRID computing distributed exec. environment.
GridUFO(Checks(compa3bility(of(sovler:obj(func(pair(
(Solvers(Database(
(Obj(Func(Database(
User(Develops(&(Registers(an((Op3miza3on(Problem(
User(Develops(&(Registers(a(Solver(
Solver(Developer(
User(Selects(a(Solver(&(an(Objec3ve(Func3on(
GridUFO(Deploys(the(Job(over(Grid(
Solver(Obj(Func(
MHAPI(
Ninf:IDL(
Distributed(Implementa3on(over(Grid(
MHML(
Obj(Func(Developer(
Ordinary(User(
Subm
its(Op3
miza3
on(Jo
b(
MHML(
15
LDDEF: System architecture overview
• Fully distributed and scalable architecture consisting of simulators in supercomputers, optimization engines, analyzers object storages and distributed database nodes in the inter-cloud environment.
DB�
Object)Storage(s)�
DB�
DB�
Simulator �
Optimizer �Simulator �
Optimizer �
<s,:f> �
<s,:?> �
<s,:f> �
<s,:?> �
replication �
<p>�<p>�
{:<s,:f>:} �
<s’> �
{:<s,:f>:} �
<s’> �
Analyzer:/ �Visualizer)
Controller:&)User:Interface �
Distributed:DBs �
{:<p>:}�
{:<s,:f>:} �(feedback) �
replication �
16
Cassandra distributed database nodes deployed across Japan
• We have built a testbed of Cassandra distributed database nodes across Japan from Kitami (Hokkaido) to Okinawa connected via SDN (Vyatta).
• We have tested performance with/without replications and availability and resiliency in cases of node and network faults.
0"
1000"
2000"
3000"
4000"
5000"
6000"
1" 11" 21" 31" 41" 51" 61" 71" 81"
0"
1000"
2000"
3000"
4000"
5000"
1" 11" 21" 31" 41" 51" 61" 71" 81"
Num
ber"of"requests�Num
ber"of"requests�
write8latency"(ms)�
read8latency"(ms)�
with"replicaCons� without"replicaCons�Hokkaido'University'Informa3on'Ini3a3ve'Center�
Kitami'Ins3tute''of'Technology�
University'of'the'Ryukyus'
70ms�
60ms�
10ms�
17
Cloud Resource Deployment Optimization (CReDO) in the Inter-Cloud Environment• Optimizing deployment of virtualized systems requested from
users according to their system specifications using multi-objective evolutionary algorithms such as NSGA-II/III.
• Semi-automated scheduling policy to “recommend” a variety of system deployment patterns at Pareto-front to users.
CReDO Solver / Optimizer
DB
Request with Spec. info
Response with Deploy. info
Public Cloud A Public Cloud B Private Cloud
System info., Accounting, etc
18
Multi-objective inter-cloud resource optimization using multi-objective evolutionary algorithms.
• We employ multi-objective evolutionary algorithms such as NSGA-II and NSGA-III to solve resource optimization problems in the inter-cloud environment.
• Solving multi-objective optimization considering cost, performance(response time), and greenness (CO2 emission) simultaneously.
19
Toward the next generation of Hokkaido university academic cloud as high-performance inter-cloud
• We are planning to develop a high-performance inter-cloud system as the next generation Hokkaido university academic cloud
• Inter-cloud (service layer): multi-cloud controller & broker with cloud exchange
• Inter-cloud (infrastructure layer): Inter-cloud connector with SDN controller
Private Cloud with Supercompter & BigData Storage�
Inter-Cloud Portal (multi-cloud controller)
�
Public Cloud A�
VPN (SDN)�Public Cloud B�
Inter-Cloud Connector�
Community Cloud C�Public/Comunity Clouds�
Cloud Exchange�
HPC�
20
High-performance inter-cloud design: an example
SW Super'computer40GbE'(x'142)
FC'or'IB
Tape'Archive
IaaS/'HaaS'
IaaS/'HaaS'
IaaS/'HaaS'
Cloud'Shared'Storage
App WebDAV'S3/Gfarm
HPC'Storage
Campus'LAN
FW'Router
SW
Management'servers'
Baremetal,'VMWare) Baremetal,'CloudStack'or'OpenStack) Baremetal)
100G'x'1 Tape'ArchiveIaaS/'
HaaS'IaaS/'HaaS'
Public'clouds'Community'clouds'
Remote'site'#1 Remote'site'#2
SW SW SW SW
IPS'
40GbE'(x'142)
SINET5'
Campus'DC
SDN' ontroller
22
Roadmap & Future direction
• 2016Q2: Upgrade network infrastructure (SINET5: 100Gbps)
• 2017Q2-Q3: Replacing inter-cloud infrastructure (including remote sites) & supercomputer at Hokkaido university
• Regional inter-cloud collaborations in Hokkaido
• National inter-cloud collaborations with other universities, NII and other research institutes to establish academic community cloud federations
• International inter-cloud collaborations (Asia-Pacific?)
• Investigations on future trends in inter-cloud applications such as IoT/IoE, extreme-scale parallel and distributed computing including big data processing and machine learning.
23
CloudWeek2015@Hokkaido University
• A collection of symposium, conference, and workshop related cloud computing technologies, sponsored by information initiative center, Hokkaido University.
• Sep.7th - 9th or 10th, 2015, at Hokkaido University, Sapporo, Japan.
• Academic Inter-Cloud Symposium 2015 for Universities, Research institutes
• Open Cloud Conference 2015 for Cloud service providers, vendors, etc.
• ITRC RICC (Regional Inter-Cloud Committee) Workshop
• Call for international speakers!
Joint usage/research base
Cloud Week@Hokkaido University (cloud symposium)
Promotion of open-type joint research
Introduction of Petabyte-class Data Science Unified Cloud Storage
Innovative high-performance computing infrastructure (HPCI)
Administration organization of the Information Initiative CenterMain activities of the Information Initiative Center
Cloud Week 2013@Hokkaido University (as of Jul. 1, 2014)
6142(2)
1141116153(2)
55
ProfessorSpecially appointed professorAssociate professorAssistant professor(Visiting researcher)Research support promotion workerClerical workerTechnical workerSpecific professionalPart-time workerResearch support promotion workerClerical assistantClerical support worker
Research divisions
Total
Clerical divisions (Information Environment Promotion Headquarters, Information Promotion Division)
Director of the Information Initiative Center
University of Tokyo Information TechnologyCenter
Kyoto UniversityAcademic Center for Computing and Media Studies
Tsukuba University Center for Computational Science
Tokyo Institute of Technology Global Scientific Information and Computing Center
Japan Agency for Marine-Earth Science and Technology Center for Earth Information Science and Technology
Kyushu University Research Institute for Information Technology
Osaka University Cybermedia Center
Nagoya University Information Technology Center
Hokkaido UniversityInformation InitiativeCenter
Tohoku UniversityCyber Science Center
RIKEN Advanced Institutefor Computational Science “K”
Institute of Statistical Mathematics Center for Engineering and Technical Support
「京」Faculty members
Organization chart Research divisions As a nationwide joint use facility, the Center aims to promote the advancement of education and research and the implementation and support of education taking advantage of information media, through R&D to facilitate informatization and the development and operation of an information infrastructure.
● Hokkaido University Information Initiative Center● Tohoku University Cyber Science Center● University of Tokyo Information Technology Center (core base)● Tokyo Institute of Technology Global Scientific Information and Computing Center● Nagoya University Information Technology Center● Kyoto University Academic Center for Computing and Media Studies● Osaka University Cybermedia Center● Kyushu University Research Institute for Information Technology
The Center is acknowledged as one of the joint use facilities among eight universities constituting the Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures (JHPCN), a network-type joint usage/re-search base established in accordance with the provisions of the Enforcement Regulations of the School Education Law.
■ Facilities constituting the network-type joint usage/ research base
In addition to open-type joint research conducted according to the purposes of JHPCN, the Center is conducting its own open-type joint research, in which the Center plays a central role in grand challenge-type research using information infrastructure and academic joint research projects to promote such grand challenges. In FY 2014, research projects in the following fields were sought from the public and 19 joint research projects are now being conducted in collaboration with teachers and researchers of Japanese and international research institutes.
Operation of HPCI was commenced in September 2012 by establishing an innovative computation base for joint use to meet diverse user needs, by connecting the next generation supercomputer “K” at RIKEN and supercomput-ers of universities and research institutes nationwide via a high speed network (SINET4). The Research Organization for Information Science & Technology (RIST) is in charge of the selection of projects, management of common receptions and other utilization promotion services. Our Center is participating as a system configuration organization and is providing computation resources. In FY 2014, six research projects using the Center were adopted.Purposes of HPCI● To develop and provide a system and mechanism so that a wide range of high-performance computing (HPC) users nationwide with different individual computation resource provision organizations can efficiently use HPC resources anywhere in the country● To accelerate extensive HPC utilization ranging from seeds to large-scale research, and to develop/expand the industrial use computational science community and return the results to society by enabling nationwide matching of needs and resources
The Center launched Petabyte-class Data Science Unified Cloud Storage service in April 2014. The system consists of a high-capacity storage (total physical capacity: approx. 2 PB) and a cloud system with 500 physical cores. As its purpose is to provide resources to HPCI (innovative high-performance computing infrastructure) and JHPCN (Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures), the service is provided mainly to those who were selected for these projects.
As the Hokkaido University Academic Cloud, which is one of Japan’s largest academic clouds, was established in November 2011, symposiums that contribute to exchange of opinions on the current status and future develop-ment of cloud research have been held yearly since FY 2012, by gathering cloud-related researchers from Japan and abroad. The FY 2013 symposium, which was held for three days and involved more than 300 participants contributed to the development of research technology by inviting leaders of various fields in cloud-related technology, holding lectures and exchanging detailed information.
■ Effective period of acknowledgment: April 1, 2010 – March 31, 2016
■ Purpose of the base The purpose of this network-type base is to contribute to further advancement and constant development of Japan’s academic and research bases through interdisciplinary joint usage/research concerning so-called grand challenges, which have been considered to be extremely difficult to solve or clarify, using super-large-sized computers, super-high-capacity storages/net-works and other information infrastructures. It covers information processing fields in general, including the global environment, energy, substances/materi-als, genome information, web data, academic information, time-series data from sensor networks, image data and program analysis.
■ Operation of the base JHPCN is operated by the Steering Committee and Joint Research Project Screening Committee established at the University of Tokyo’s Information Technology Center, which is its core base.
■ Promotion of open-type joint research For interdisciplinary research using large-scale information infrastructure, JHPCN is conducting 34 joint research projects in FY 2014 (of which 7 is related to our Center), by seeking research projects concerning application of ultra-large-scale numerical calculation and data processing systems and ultra-high-capacity network technology, as well as the field of ultra-large-scale information systems integrating these technologies from the public.
● Large-scale computer simulation
● Large-scale data science
● Digital contents
● Basic technology for solving large-scaleproblems
● Networks and cloud technology
● Educational information media
Board of delegates
Director Vice Director
Large-Scale Computation System Research Division
Information Network Research Division
Digital Contents Research Division
Research divisions
Media Education Research Division
Joint Usage/Research Committee
Special Committee of System Usage
Special Committee of System Technology
Special Committee of Libraries/Databases
Special Committee of Joint Research
Special Committee of System Usage
Special Committee of System Technology
Chief of the Information Promotion Section
Information Environment Promotion Headquarters
Head of the IT promotion group
System operation team
Information networkteam
Information security team
Software support team
Committee for On-Campus Joint Use of Educational Information Systems
Special Committee of System Usage
Special Committee of System Technology
Committee for On-Campus Joint Use of Information Networks Systems
In charge of joint usage/research
In charge of general affairs/planning
In charge of media education
In charge of business system operation
In charge of accounting
Assistant Section Chief
Large-scale Computation System Research Division
Information Network Research Division
Sophistication of large-scale scientific computing, R&D of application programs and research
on design technologies for large-scale information systems
Development of application software using large-scale/ul-tra-high-speed computer systemsResearch on the design and construction methods of large-scale computational system
●
●
●
●
●
●
●
●
Accumulation, processing, distribution, utilization and other handling of digital data contents
Research on information system infrastructure for accumu-lation, evaluation, processing and distribution of digital data contents centering around an academic cloudResearch on accumulation, evaluation, processing, distribution, utilization and other handling of digital data contents related to natural science, humanities, etc.
The Center has four research divisions as presented below, where cutting edge research on information technology, development of relevant technologies to support information infrastructure, education activities using information media and related research activities are conducted.
Research on advanced technologies for next-generation information networks,
research on information security and information distribution infrastructure
R&D of basic technologies for next-generation information network systems and basic research on ubiquitous computing and media applicationTheoretical research on the information distribution infrastructure in general, practical research on data analysis and research on the confidentiality and completeness of distribution information
Media Education Research Division
Digital Contents Research Division
Research on sophistication of education using ICT and information education and promotion
of informatized/open education in the global society
Research on sophistication of information education, infor-matization of education, open education, education/learn-ing using ICT and systems to support education/learningResearch on fostering communication abilities based on collaborative learning using ICT
24