Cray Cluster Solutions Cray Aries Interconnect · C O M P U T E | S T O R E | A N A LY Z E Cray...

C O M P U T E | S T O R E | A N A L Y Z E

Cray Cluster Solutions &

Cray Aries Interconnect

Mathias Pütz Pre-Sales Manager Cray Europe

Thanks for contributions from Thomas Alrutz & Christian Simmendinger

T-Systems – Solutions for Research

May 13, 2014 – HPCN Workshop / Göttingen

1


Cray Overview

Copyright 2014 Cray Inc. -

Confidential and

Proprietary

● Cray founded in 1972 ● ~1000 employees across 20 countries

● Headquarters in Seattle, WA

● Six Major Development Sites: ● Bristol, UK

● Chippewa Falls, WI

● Pleasanton, CA

● St. Paul, MN

● Seattle, WA

● Austin, TX

● UK based European Exascale Research Initiative

We build the world’s fastest solutions to help solve “Grand Challenges” in

science, engineering and analytics

5/13/2014

Manufacturing Earth

Sciences Energy Life Sciences Financial

Services

Government

and Defense

Higher

Education Cyber-

security


We Build Computational Tools That Help Change The World

Supercomputers

Flexible Clusters

Hybrid Architectures

Supercomputing Big Data

3

Compute

Integrated Storage

& Data Management

Tiered Storage Archive Store

Analyze Graph Analytics

Hadoop Solutions


4

Supercomputing Leadership

http://www.tennessee.edu/identity/graphics/utlogo-large.jpg


Why Cluster Supercomputers from Cray?

5

• End-to-End Cluster

Deployments

• Fast-to-Market Innovative

Technologies based on

Industry Standards

• Cluster Solution Expertise

• Experienced Building

Clusters Based on Workload

Requirements

• Air-Cooled or Direct

Liquid-Cooled Solutions

• Node and Network

Configuration Options

• Supported by Cray cluster

SW ecosystem

• Large Memory, Shared

Memory Configurations

• Problem/Solution

Customer Approach

• Fast Deployment Time

• Reliable and Scalable

Performance

• Outstanding Customer

Experience

Flexible Configurable Technologies to Solve Challenging Problems

Leadership Flexibility Results


Only from Cray

6

Unique offerings to Solve Challenging Problems

• Cray’s Scalable Cluster

Management Software

• Framework for Cray’s

Cluster HPC Software

Stack

• Deployed in some of the

largest production clusters

in the world

• Modular in design to

support 3rd party cluster

tools

Advanced Cluster Engine

• Cray’s Optimized C, C+ &

Fortran Compilers

• Cray’s Math & Scientific

Libraries

• LibSci

• LibSci-acc

Cray Compiler Environment


Cray Customers

7

http://www.google.co.kr/url?sa=i&rct=j&q=virginia+tech&source=images&cd=&cad=rja&docid=dINZgqFE5KNp5M&tbnid=f8IPalpKVCN_wM:&ved=0CAUQjRw&url=http://blogs.roanoke.com/theburgs/news/2012/03/30/virginia-tech-pleased-by-decision-overturning-schools-clery-act-fines/&ei=dm5HUbWxMKORiALhp4DwDg&bvm=bv.43828540,d.cGE&psig=AFQjCNEhkz5q92kRQEzbGz7oQY9g8yZ5Qw&ust=1363722224559393

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=fvbmqvToP_tSTM&tbnid=qSnW3lWaDk0lIM:&ved=0CAgQjRwwAA&url=http://en.wikipedia.org/wiki/File:University_of_Chicago_logo.svg&ei=u9LqUZ-OK6_YyQHc0oC4BQ&psig=AFQjCNEcd4wGPGQ5GxPjk1UOCV98O2qL5A&ust=1374430267784851


“Catalyst”, Lawrence Livermore National Laboratory

Based on the Cray CS300-AC™, the Catalyst system is a

collaborative system between Lawrence Livermore,

Intel and Cray Inc. The cluster will be managed through

LLNL’s HPC Innovation Center and used to explore non-

volatile memory in HPC simulation and big data analytics

Research Science

Allows for the exploration of new approaches to

big data analytics and HPC simulations requiring

superb memory and I/O capabilities for :

• Bioinformatics

• Business analytics

• Machine learning

• Natural language processing

• Simulations

For more information: http://hpcinnovationcenter.llnl.gov/

https://asc.llnl.gov/

System Overview

• 150TF of performance with a total system cores of 7,776

• Intel Xeon ES-2695v2 processors (12 cores, 2.4Ghz)

• 42TB of dynamics random access memory (DRAM)

• 282TB of non-volatile memory (NVRAM)

• Over 600GB/s of I/O bandwidth, over 60 million IOPS

• 12 Lustre and 2 NFS router nodes (128GB DRAM)

• VRAM: Intel SSD 910 Series (800GB)

• Fabric: Intel True Scale Fabric (dual-rail QDR-80)

• Storage software: Intel Lustre, Livermore DIMMAP (Data

Immersion MMAP), and SCR (Scalable Checkpoint Restart)

• Delivered to LLNL in October 2013

Catalyst

8

http://hpcinnovationcenter.llnl.gov/

http://hpcinnovationcenter.llnl.gov/

https://asc.llnl.gov/

C O M P U T E | S T O R E | A N A L Y Z E9

Cray CS300 Series

Cluster Supercomputers

Air-Cooled Liquid-Cooled

http://inside.us.cray.com/depts/ccs/Documents/CCS BU MARKETING/Images - WIP/Stock and Created Images/Created Graphics/Datasheet Templates/cs300-AC datasheet template.jpg

http://inside.us.cray.com/depts/ccs/Documents/CCS BU MARKETING/Images - WIP/Stock and Created Images/Created Graphics/Datasheet Templates/CS300 LC datasheet template.jpg


Cray CS300-AC™ Cluster Supercomputer Overview

Flexible and Scalable • Up to 80 nodes/rack configured with 2 CPUs/node

• Interconnect: 3D Torus/Fat Tree, single/dual rail,

QDR/FDR/EDR, IB/GigE, STL-1 and STL-2

• Supports a wide range of parallel Storage File

Systems

Manageable • Integrated with Cray’s HPC Cluster Software Stack

• Includes Cray’s Advanced Cluster Engine™

Management software

• Includes network, server, cluster and storage

management

Reliable • Node and System level redundancy

• Hardened software stack tested by CCS

Development team

End-To-End Supercomputer Solutions • From design, integration, delivery to on-site

installation and customized services

10

Performance and Flexibility

Air-Cooled Solution

http://inside.us.cray.com/depts/ccs/Documents/CCS BU MARKETING/Images - WIP/Stock and Created Images/Created Graphics/Datasheet Templates/cs300-AC datasheet template.jpg


2x Motherboards

Compute Blades

Service Blades

Hybrid Blades

Chassis

Cray CS300™ Cluster Supercomputer

Optimal Building Blocks for Cluster Supercomputer

11


Compute Platform

• Design optimized for Compute platform

• Support for 2x Intel Ivy Bridge EP CPUs

• Support for 8x DDR3 1866MHz memory modules

• Access to up to 32 PCIe Gen3 lanes

— 1x PCIe Gen3 x16 expansion slots

— 2 x PCIe Gen3 x8 expansion slots

• Option for on-board PCIe Gen3 Mellanox Connext-X3

• Future generation supporting Haswell/Broadwell in plan

Hybrid Platform

• Design optimized for Hybrid platform

• Support for 2x Intel Haswell (Broadwell) EP CPUs

• Support for 16x DDR4 2133MHz memory modules

• Access to up to 48 PCIe Gen3 lanes

— 2x PCIe Gen3 x16 expansion slots

— 1 x PCIe Gen3 x8 expansion slots

• Option for on-board PCIe Gen3 Mellanox Connect-X3

• Future generation supporting Haswell/Broadwell in plan

CS300 Cluster Supercomputer Roadmap Romley System Boards


● 0.5RU effective density Compute Blades ● Supports Intel Xeon E5 Ivy Bridge

CPUs

● Supports up to 128GB DDR3 1866 MHz system memory

● Integrated PCIe Gen3 QDR/FDR IB (optional)

● Support for one internal 2.5” HDD

● 1RU effective density Hybrid

Blades ● Host + Expansion Blade

● Supports 2x nVIDIA Keplers

● Supports 2x Intel KNCs

Cray CS300-AC™ Cluster Supercomputer Configurable Compute Nodes

13


● SR5000 Chassis supports either: ● 10x Compute Blades

● 5x Hybrid Blades

● Compute Blades are 0.5RU 2P x86 servers

● Hybrid Blades are 1RU 2P x86 combined with either 2x nVIDIA Keplers or 2x Intel KNCs

● Three hot-swappable, redundant Cooling Fan Units

● Up to four hot-swappable, redundant 1630W PS

● Can support one or two redundant iSCB chassis managers

Cray CS300-AC™ Cluster Supercomputer Reliability & Serviceability – SR5000 Series

14


Cray proprietary design ● Common interface to all blade servers

with iSCB: Chassis Management Appliance

ARM-based appliance running embedded Linux

● Node health monitoring

● Supports concurrent console sessions to each node (terminal concentrator)

● Reliable power control

● Active dynamic fan control

● Power monitoring

● Dedicated powerman/conman interface

● GPU/MIC power control

Can easily upgrade features

Cray CS300™ Cluster Supercomputer Reliable Management Designed for HPC


2RU effective density

● Service Node w/ expanded IO + 3x 3.5” HDDs ● Up to three 3.5” SATA/SAS HDDs

● Up to four PCIe expansion slots

● Ideal as a GW/Login Node

● Service Node w/ expanded IO + 6x 2.5” HDDs ● Up to six 2.5” SATA/SAS HDDs

● One DVD ROM Drive

● Up to four PCIe expansion slots

● Ideal as a Management/Boot Node

Cray CS300-AC™ Cluster Supercomputer Highly Configurable Service & Hybrid Nodes

16

Used for ACE Management Nodes, Login Nodes,

Data Transfer Nodes, and Ancillary Nodes


● 2RU effective density ● Hybrid Node w/ 4x Accelerators + 2x 2.5” HDDs

● Two (2) IVY BRIDGE Processors (to 115W)

● Washington pass motherboard (Intel PCSD Mfg) ● Will support Taylor Pass for Haswell (Broadwell)

● Four nVIDIA Keplers or Intel KNCs

Cray CS300-AC™ Cluster Supercomputer Highly Configurable Service & Hybrid Nodes

17


● SR8000 Chassis family supports either: ● 4x double-wide Service Blades

● Compute Blades are 0.5RU 2P x86 servers ● Hybrid Blades are 1RU 2P x86 combined with

either 2x nVIDIA Keplers or 2x Intel KNCs

● Double-wide Service Blades are 2RU 2P x86 servers

● Double-wide Hybrid Blades are 2RU 2P x86 servers with either 4x nVIDIA Keplers or 4x Intel KNCs

● Six hot-swappable, redundant Cooling Fan Units

● Up to six hot-swappable, redundant 1630W PS


Cray CS300-AC™ Cluster Supercomputer Reliability & Serviceability – SR8000 Series

18


Cray CS300-LC™ Cluster Supercomputer Overview

Energy-Efficient Architecture • Liquid-Cooled architecture

• Uses warm water instead of chilled water

• Directly cools the processor and memory

• Uses low-pressure isolated secondary loop

• Leak detection and prevention features

integrated with remote monitoring and

reporting

Flexible and Scalable • Up to 60 nodes/rack configured with 2

CPUs/node

• Interconnect: 3D Torus/Fat Tree, single/dual

rail, QDR/FDR, IB/GigE

• Supports a wide range of parallel Storage File

Systems

End-To-End Supercomputer Solutions • From design, integration, delivery to on-site

installation and customized services

Less Energy – Lower TCO – Faster ROI

Liquid-Cooled Solution

http://inside.us.cray.com/depts/ccs/Documents/CCS BU MARKETING/Images - WIP/Stock and Created Images/Created Graphics/Datasheet Templates/CS300 LC datasheet template.jpg


Liquid-to-liquid “Heat Exchangers”

exchange heat between datacenter

liquid loop and server liquid loops.

Datacenter and server liquids are

kept separate and are never mixed

Warm water from datacenter

dry cooler or cooling tower

enters RackCDU and the

heated water returns

Tubes move cooling liquid to and

from RackCDU to blade servers

Pump/cold plate units atop CPUs

(or GPUs) circulate liquid through

server and RackCDU, collecting

heat and returning to RackCDU for

exchange with datacenter liquid

Cray CS300-LC™ Cluster Supercomputer How It Works


Cray CS300-LC™ Cluster Supercomputer Pump Details


● Two expansion blade models – one

supporting 2x Intel MICs, the other

NVIDIA Keplers

● Remove 80-90% of heat generated

● Two compute blade models – one

based on CPU cooling only, the

other CPU+memory cooling

● Remove 65-75% of heat generated

22

Cray CS300-LC™ Cluster Supercomputer Configurable Compute Nodes


Cray CS300-LC™ Cluster Supercomputer Highly Configurable RAS Chassis

● SR6000 Chassis supports either: ● 10x Liquid-cooled Compute Blades

● 5x Liquid-cooled Hybrid Blades

● Compute Blades are 0.6RU 2P x86 servers

● Hybrid Blades are 1,2RU 2P x86 combined with either 2x nVIDIA Keplers or 2x Intel KNCs

● Three hot-swappable, redundant Cooling Fan Units

● Up to four hot-swappable, redundant 1630W PS


● Enhanced monitoring – leak detection, water pressure, pump failure detection


InfiniBand Network

Compute Nodes

Management Group

Storage Network

Redundant

Management

Nodes

Core Operational

NW ( 10 GbE )

Operational

GigE Network

Login Nodes

External Network

( 10 GbE )

Core Management

Network (1GbE)

1 GbE (Operational)

10 GbE (Operational)

High Speed Interconnect

1 GbE (Management)

NFS Servers

Sub-Management Nodes

(Active/Active pairs)

Operational

GigE Network

Operational

GigE Network

Operational

GigE Network

CN CN

CN CN

CN

CN CN

CN CN

CN

CN CN

CN CN

CN

CN CN

CN CN CN

Global Storage

LSN LSN

MN MN

SMN NFS

SMN SMN SMN SMN SMN SMN SMN SMN

24

Cray CS300™ Cluster Supercomputer System Architecture


Cray HPC Cluster Software


Intel® Cluster Studio PGI (PGI CDK) GNU

MVAPICH2 OpenMPI Intel® MPI-

(Cluster Studio) Application Libraries

Development Tools

HPCC IOR netperf Performance Monitoring Perfctr PAPI/IPM

Cray® Compiler Environment (CCE)

Cray® LibSci

HPC Programming Tools

Essential Software and Management Tools Needed to Build a

Powerful, Flexible, and Highly Available Supercomputer.

Linux (Red Hat, CentOS, SUSE) Operating System Operating Systems

Operating Systems

Cray ACE (iSCB and OpenIPMI)

Cray ACE

Cluster Monitoring

Remote Power Mgmt

Remote Console Mgmt Cray ACE

Grid Engine Integrated

Resource Management / Job Scheduling

Altair PBS Pro

NFS File System Lustre Local FS

(ext3, ext4, XFS)

Cray® Advanced Cluster Engine (ACE) management software Provisioning

PanFS

SLURM Integrated

IBM Platform LSF

Torque/Maui

Middleware Applications and Management

MOAB

Cray HPC Cluster Software Stack Flexible SW Architecture for our flexible HW


Programming Environment from Cray

27

The Cray Programming Environment Suite for Clusters is a fully integrated programming environment with compilers, tools, and libraries designed to maximize programmer productivity, application scalability, and performance. It consists of ● The Cray Compiling Environment

● Compilers for Fortran, C and C++ supporting OpenMP 3.1 and OpenACC 2.0

● Fortran 2003 & 2008 compliance ● Significant performance gains over competitive compilers in particular for

well vectorizing applications ● Highest performance without sacrificing accuracy

● The Cray Scientific Libraries

● Autotuned BLAS library, LAPACK, ScaLAPACK ● Highly optimized non-square DGEMM/SGEMM/ZGEMM routines

● Iterative Refinement Toolkit (IRT) ● Cray LibSci_ACC providing accelerated BLAS and LAPACK routines for

GPUs


• Diskless/Stateless Operation

– Simplifies System Administration

– Improves Performance

– Multi-Level Cached Root File System

– Does not require a Light Weight Kernel

– Local Storage also Supported

– State maintained in two HA management servers

• “Instant” Provisioning

– Multiple Logical Clusters

– Multiple OS Configurations

– Provisioning Time = Reboot Time = Less than 10

minutes

• Configuration Management

– Multiple Revisions with Roll-Back Capability

– Rolling Upgrades on Individual servers between jobs

28

Cray HPC Cluster Software Stack ACE Features


Management

Compute

Servers

Storage

Networks

Clusters

ACE-Managed Objects

Root

File

System

Image

Hosts

prod-0001 prod-0002 prod-0003 prod-0004

...

Root

File

System

Image

Hosts test-0001 test-0002

test-0003 test-0004

...

iSCB

29

Cray HPC Cluster Software Stack ACE Managed Objects


Cray Aries Interconnect

Scale out to 1000s of nodes


0%

5%

10%

15%

20%

25%

30%

35%

40%

32 64 128 256 512 1024 2048

IB Cost % (Fat Tree)

IB Cost %

Node Count

Inflection Points

Infiniband scale-out cost Infiniband cost as a % of system cost

Compute Nodes

Service Nodes

InfiniBand

Ethernet

Software

Infrastructure

Services

32 Node

Compute Nodes

Service Nodes

InfiniBand

Ethernet

Software

Infrastructure

Services

1024 Node

Introduce

islands to keep

cost scaling flat


Aries connectivity to Compute Nodes

32


Aries Rank-1 network – adaptive routing

o Chassis with 16 compute blades

o Inter-Aries communication over

backplane

o Per-Packet adaptive Routing

o Proactive vs. Reactive

Copyright 2014 Cray Inc. -

Confidential and

Proprietary


16 Aries connected

by backplane

“Green Network”

Cray XC30 Rank-2 Copper Network

4 nodes

connect to a

single Aries

6 backplanes

connected with

copper cables in a 2-

cabinet group:

“Black Network”

Active optical

cables interconnect

groups

“Blue Network”

2 Cabinet

Group

768 Sockets


Configurable Global Bandwidth

● An all-to-all pattern is wired between the groups using optical cables

● Installed in integrated overhead trays

● Site-specific lengths are ordered to minimize excess

● The global bandwidth can be tuned by varying the number of optical cables in the group-to-group connections


● Sandia miniApp, models performance of CTH

● Running on 2256 node CSCS system (¼ global bandwidth)

● Four concurrent 512 node parallel jobs, different layouts

● Runtime in seconds for 100 cycles

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 2 4 4 4 4 4 4 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

3 1 3 2 1 3 2 3 3 1 1 4 4 1 3 3 4 2 1 2 3 2 2 2 3 2 3 1 4 2 4 3 4 1 3 1 4 2 1 1 4 1 3 2 3 3 2 3 4 4 3 2 1 3 3 3 1 1 2 4

1 4 3 3 3 3 4 2 3 3 4 4 3 4 4 2 3 2 2 3 2 3 1 1 1 1 1 2 2 2 2 2 1 4 1 4 2 4 3 1 2 2 3 2 4 2 3 4 4 3 3 1 4 2 2 3 3 3 1 2 4 3 2 1

3 1 2 1 3 2 3 4 4 4 2 1 3 4 1 1 1 4 1 2 1 4 2 1 3 2 4 1 2 1 2 4 4 3 2 3 1 2 1 1 2 1 1 1 1 4 1 4 3 1 3 2 1 3 2 2 4 1 4 1 4 1 3 4

3 3 4 2 4 4 4 3 3 4 3 4 4 4 3 1 1 2 3 1 4 2 1 2 2 1 2 4 2 1 4 4 2 1 2 1 4 1 4 3 4 1 2 4 2 2 4 4 4 1 4 2 3 1 4 2 3 1 2 2

4 3 2 4 3 4 3 1 2 3 2 4 1 2 4 2 2 3 3 2 4 1 1 3 4 4 2 1 1 2 3 1 2 1 4 4 2 1 2 3 4 3 1 2 1 4 2 2 4 4 1 2 4 4 4 2 1 3 3 3 1 4 3 3

1 1 2 4 1 3 4 4 4 2 2 1 1 1 1 4 4 4 1 4 2 2 3 3 1 1 4 4 2 2 1 1 4 1 1 1 2 2 2 3 1 2 4 1 1 1 3 4 2 2 4 1 3 4 3 4 4 4 4 1 1 4 4 3

3 3 4 2 1 4 2 3 3 1 3 3 4 3 1 1 3 4 1 3 2 2 4 3 2 3 4 1 2 3 2 1 2 1 4 4 3 2 1 3 2 1 4 2 3 4 3 4 2 4 4 4 1 3 2 2 3 2 3 4

4 3 1 2 1 1 4 2 4 3 3 4 3 4 3 3 2 3 4 1 2 4 2 4 4 3 1 2 4 1 1 3 4 3 3 4 4 2 4 4 4 1 4 2 4 4 2 4 4 1 2 3 1 3 2 1 1 3 1 2 3 4 3

1 2 3 2 1 4 3 1 2 2 4 4 4 3 3 3 3 2 2 4 4 2 4 4 1 2 3 2 1 2 4 1 2 1 1 3 2 4 2 2 4 3 1 2 4 3 3 2 4 4 1 1 2 2 1 1 4 4 2 3 2 4 3 1

2 2 2 2 2 3 2 4 3 2 3 4 4 4 2 2 1 1 2 3 2 3 1 1 4 2 1 2 2 1 2 1 2 1 4 3 2 3 3 2 2 1 3 4 1 2 3 3 4 2 2 4 4 3 4 2 3 1 3 1

1 1 1 1 1 1 3 1 3 4 4 4 2 1 1 4 2 1 1 2 4 3 4 1 3 4 2 2 4 1 1 2 2 1 4 4 3 3 2 2 3 4 3 3 1 1 4 2 1 2 2 2 2 3 1 3 1 4 2 1 3 1 2 3

3 3 4 2 4 3 2 1 2 4 3 2 1 3 4 3 4 3 1 3 4 3 1 4 4 2 2 2 1 2 3 1 2 1 3 4 1 4 2 1 2 1 4 2 2 4 2 3 4 2 2 2 4 3 3 1 1 3 1 1 1 3 3 4

4 2 2 1 1 3 3 4 1 2 2 2 2 3 2 2 1 2 1 2 1 2 4 1 3 1 3 3 3 2 3 4 1 2 2 1 1 3 1 4 4 3 1 3 3 1 3 4 3 3 1 4 4 1 3 2 4 1 4 3

3 1 3 1 4 1 1 1 3 2 3 2 3 4 1 4 4 2 2 1 1 2 4 3 4 3 1 3 2 4 4 2 2 4 1 3 2 3 4 4 1 1 1 3 2 1 2 1 1 2 2 1 4 1 1 4 1 2 1 4 2 1 3 3

3 1 4 2 3 1 4 3 2 3 4 1 4 2 4 3 3 4 4 4 4 2 4 2 3 4 3 4 4 4 1 2 2 1 2 1 3 2 3 1 3 2 4 1 2 4 3 1 1 4 3 1 1 3 1 1 3 1 1 1 3 1 3 4

4 3 3 2 2 1 3 3 3 3 4 2 4 2 3 3 3 4 4 2 3 2 4 2 4 2 1 1 1 4 3 4 4 1 1 3 3 4 2 4 2 2 4 2 2 4 1 2 2 2 2 1 1 2 1 4 2 3 3 1

4 2 2 3 2 4 1 2 4 2 2 4 4 2 1 4 3 2 3 2 4 3 2 3 4 3 4 3 1 4 4 1 1 1 4 4 2 4 1 1 3 4 1 4 2 3 3 3 1 4 2 1 4 2 3 1 4 2 2 4 2 4 3 1

3 3 1 4 1 4 1 1 4 4 1 3 3 3 1 1 2 2 2 1 3 4 2 1 1 1 1 3 4 4 1 1 4 3 2 1 2 1 4 2 1 4 1 2 3 4 4 4 3 1 4 2 3 1 3 4 2 2 2 2 3 3 2 1

2 1 1 2 1 2 3 4 4 2 4 4 2 3 3 2 3 3 2 3 2 4 1 2 4 4 1 4 4 2 1 1 2 1 3 4 3 4 1 3 2 4 4 1 1 3 3 1 4 2 1 4 3 2 4 2 3 3 2 2

2 1 2 4 4 4 1 1 3 3 3 3 3 1 4 2 3 1 3 2 4 1 2 2 3 3 1 2 3 3 2 2 1 3 2 4 4 4 1 1 1 3 4 3 3 2 4 2 4 1 4 1 4 4 3 4 4 2 4 1 3 4 3 4

1 1 1 4 1 3 4 3 1 3 1 1 1 1 4 2 1 3 1 2 4 4 2 3 1 4 4 2 1 3 2 2 3 3 2 3 4 4 3 3 4 4 1 3 3 3 3 1 1 2 3 1 3 3 1 2 2 4 2 3 3 2 4 3

2 1 2 2 1 3 4 2 1 1 1 3 3 1 3 3 4 2 2 2 2 3 2 1 4 4 1 2 2 3 3 4 2 2 4 1 4 4 1 3 4 3 1 3 1 2 3 1 2 1 4 4 2 2 3 1 2 2 2 4

2 1 4 3 4 2 4 1 4 3 1 2 1 2 4 4 3 2 3 2 4 1 4 1 4 2 2 1 4 2 4 1 2 4 4 4 3 3 4 4 3 1 3 4 1 1 2 2 3 1 3 3 1 3 4 3 3 2 1 2 2 1 4 2

3 4 1 2 3 3 1 2 3 1 1 4 3 1 4 1 2 3 3 3 1 2 1 4 2 2 2 3 1 3 1 1 1 2 1 1 4 2 2 1 4 2 1 3 4 2 2 1 4 1 3 2 1 3 2 2 4 3 4 4 4 4 4 2

4 4 2 1 3 2 1 4 4 2 1 4 2 3 3 2 1 2 3 1 4 2 3 2 2 3 2 2 3 3 4 3 4 4 2 4 3 1 2 3 2 4 4 4 3 1 4 4 1 1 3 3 2 2 1 4 4 4 2 4

3 1 2 3 4 1 4 3 4 3 2 1 1 1 2 2 2 4 4 3 3 4 3 1 3 4 4 4 1 1 3 3 3 1 1 1 1 3 3 4 3 2 4 4 2 3 2 2 4 4 3 2 4 2 4 2 4 4 3 2 3 2 1 2

3 1 2 3 3 2 1 2 3 4 1 2 4 4 2 1 4 4 3 3 4 4 4 3 4 1 3 2 4 3 3 1 3 3 1 3 2 2 1 2 4 1 1 4 1 2 1 2 2 4 1 4 2 4 4 2 3 4 3 4 1 4 3 1

4 3 1 3 1 4 2 3 2 3 3 2 3 3 3 2 2 1 4 2 4 3 4 2 1 4 4 3 1 2 4 2 1 2 2 1 2 2 2 1 2 4 3 1 3 2 4 4 1 3 1 1 4 2 1 4 3 3 1 1

2 3 2 2 2 4 1 2 3 2 4 1 3 4 2 3 1 4 4 4 1 2 3 1 3 4 1 3 4 3 3 4 2 1 1 3 3 4 2 2 1 3 4 3 1 4 4 4 1 3 3 3 1 1 4 2 4 1 1 4 1 4 2 2

2 3 1 1 1 4 4 1 3 1 2 1 4 4 4 2 2 4 1 4 2 1 4 4 1 1 2 1 4 3 4 1 2 1 2 3 3 1 4 3 3 4 2 1 1 4 3 4 2 4 2 4 2 2 4 3 4 1 2 1 3 4 4 2

2 4 4 1 2 2 1 4 3 1 3 3 3 3 4 1 3 2 2 2 4 3 3 3 2 3 2 1 1 3 4 1 1 2 3 4 4 1 4 3 4 2 3 2 3 2 4 3 1 2 1 2 4 1 2 3 3 1 3 4

3 3 4 2 4 4 3 3 3 3 4 2 1 1 1 2 3 4 2 4 2 4 3 1 3 4 1 1 3 2 3 3 4 1 1 1 3 3 4 2 2 3 2 3 4 1 1 4 1 3 3 2 4 4 4 2 3 3 3 3 1 2 4 1

4 4 3 2 2 2 3 2 2 2 1 4 3 2 4 3 4 4 2 3 4 3 4 2 3 3 1 3 2 1 1 2 3 2 1 4 4 1 2 3 2 4 2 3 3 4 4 4 1 2 4 4 2 3 4 4 4 3 2 3 2 3 1 3

2 4 4 1 3 4 2 3 2 4 4 1 4 3 2 2 2 3 1 1 2 2 2 4 4 4 2 4 1 4 2 3 4 1 3 3 1 4 3 4 2 3 4 2 2 4 2 1 1 4 3 2 4 3 3 3 2 2 3 2

3 1 2 2 1 3 3 4 3 3 2 2 3 1 2 2 4 2 4 2 2 4 1 3 3 2 3 4 2 1 2 1 3 1 3 1 1 3 3 2 4 3 1 4 4 1 1 1 4 4 3 1 1 3 3 2 1 2 1 2 2 3 1 2

1 1 3 2 2 3 3 3 1 2 1 2 2 1 2 3 1 3 3 3 1 2 3 3 1 1 3 3 3 1 3 1 3 3 3 1 3 3 1 3 3 1 3 1 1 1 3 3 1 1 3 1 1 3 1 1 1 1 1 1 1 1 1 1

Blocks of 512 nodes Blocks of 64 nodes Average of 32 nodes Random nodes

69.0 68.8 68.9 68.9 69.4 69.4 69.4 69.5 69.7 69.7 69.4 69.7 70.9 71.0 70.6 70.5

Application Explorations for Future Interconnects, R.F. Barrett, C.T. Vaughan, S.D. Hammond, and

D.Roweth, Workshop on Large-Scale Parallel Processing, at the International Parallel and Distributed

Processing Symposium: IPDPS 2013, Boston, MA, 2013.

Performance independent of placement

http://hpc.pnl.gov/conf/LSPP/



http://ipdps.org/

http://ipdps.org/

http://ipdps.org/


Real world-performance: TAU code single-grid

37

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

240 480 960 1920 3840 7680 15360

FDR efficiency (CASE-2)

Aries efficiency

F6-DPWII benchmark single-grid parallel scaling efficiency

normalized on 240 core performance

CASE-2: Intel XEON E5-2695v2 (12-core, 2.4 GHzwith turbo), Mellanox Connect-X3 FDR

XC30:Intel XEON E5-2697v2 (12-core, 2.5 GHz w/o turbo, Aries interconnect


Real world-performance: TAU code multi-grid

38

F6-DPWII benchmark 4w multi-grid parallel scaling efficiency

normalized on 240 core performance

CASE-2: Intel XEON E5-2695v2 (12-core, 2.4 GHz with turbo), Mellanox Connect-X3 FDR

XC30:Intel XEON E5-2697v2 (12-core, 2.5 GHz w/o turbo, Aries interconnect

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

240 480 960 1920 3840 7680 15360

FDR efficiency (CASE-2)

Aries efficiency


STAR-CCM+ Scaling: Cray XC30 compared to FDR InfiniBand Cluster

39

cores Cray XC30 FDR IB Cluster IB/XC30 32 34.08 33.34 0.98

64 16.53 17.11 1.04

128 7.75 8.99 1.19

256 3.98 4.85 1.22

512 2.11 2.72 1.29

1024 1.11 1.53 1.38

2048 0.59 0.96 1.62

3072 0.49 NA*

Automotive External Aerodynamics: 100M cells

STAR-CCM+ version: 7.04.011

Average Time per Iteration [s]

Intel “Sandy Bridge” processors on both systems, 2.6 GHz

XC30 results obtained in cluster ompatibility mode

* Only 2048 cores available in the CCS cluster


LS-DYNA Scaling: Cray XC30 compared to InfiniBand Cluster

40

cores Cray XC30 FDR IB Cluster IB/XC30 32 41,474

64 20,652 22,499 1.09

128 11,278 12,427 1.10

256 6,849 7,781 1.14

432 4,781 5,729 1.20

512 4,054

Automotive “crash”: 2.4M elements LS-DYNA, mpp971

Elapsed time (sec)

Intel “Sandy Bridge” processors on both systems, 2.6 GHz

Native LS-Dyna binary used on XC30

* Only 464 cores available in the CCS cluster


Legal Disclaimer

Information in this document is provided in connection with Cray Inc. products. No license, express or implied, to any intellectual property rights is granted by this document.

Cray Inc. may make changes to specifications and product descriptions at any time, without notice.

All products, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.

Cray hardware and software products may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Cray uses codenames internally to identify products that are in development and not yet publically announced for release. Customers and other third parties are not authorized by Cray Inc. to use codenames in advertising, promotion or marketing and any use of Cray Inc. internal codenames is at the sole risk of the user.

Performance tests and ratings are measured using specific systems and/or components and reflect the approximate performance of Cray Inc. products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance.

The following are trademarks of Cray Inc. and are registered in the United States and other countries: CRAY and design, SONEXION, URIKA, and YARCDATA. The following are trademarks of Cray Inc.: ACE, APPRENTICE2, CHAPEL, CLUSTER CONNECT, CRAYPAT, CRAYPORT, ECOPHLEX, LIBSCI, NODEKARE, THREADSTORM. The following system family marks, and associated model number marks, are trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT, and XT. The registered trademark LINUX is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other trademarks used in this document are the property of their respective owners.

Copyright 2014 Cray Inc.

41


Special thanks to

Thomas Alrutz & Christian Simmendinger

T-Systems – Solution for Research

For providing TAU benchmark results on CASE-2 system, native XC30 binaries and input data sets


Thank you !

Questions ?

[email protected] [email protected]

mailto:[email protected]

mailto:[email protected]

Cray Cluster Solutions Cray Aries Interconnect · C O M P U T E | S T O R E | A N A LY Z E Cray...

Documents

Transcript of Cray Cluster Solutions Cray Aries Interconnect · C O M P U T E | S T O R E | A N A LY Z E Cray...