Red Hat Technologies for HPC on Wall St Overvie 5-Red Hat 4.9... · 2013-04-11 · Red Hat...
Transcript of Red Hat Technologies for HPC on Wall St Overvie 5-Red Hat 4.9... · 2013-04-11 · Red Hat...
Red HatTechnologies
for HPC on Wall St Overview
Jan Mark [email protected] Office of CTORed Hat Inc.
William [email protected] Office of CTORed Hat Inc.
2 HPC on Wall St 4/2013
Agenda
● Red Hat Technology Overview
● Linux at the core of Enterprise IT● Enterprise Virtualization ● Emerging Technologies
● HPC & Low Latency Messaging
3 HPC on Wall St 4/2013
Open Source Communities as the Foundation
4 HPC on Wall St 4/2013
COMPLETE RED HAT PRODUCT PORTFOLIOIT
A
DM
IN S
DE
VE
LO
PE
RS
PaaS
Open Hybrid Cloud
Management
App Platforms / Messaging / Cache / Grid / SOA
PhysicalServers
Network & Storage Infrastructure
VirtualServers
CloudServers
Scale-Out, High-Performance Storage Software
RED HATENTERPRISE VIRTUALIZATION
RED HAT NETWORKSATELLITE
JBoss Operations Network
SY
SA
DM
INS
5 HPC on Wall St 4/2013
6 HPC on Wall St 4/2013
Red Hat Enterprise Linux Roadmap*
CY2010 CY2011 CY2012 CY2013 CY2014
RHEL 6
RHEL 5
RHEL 4
RHEL 3
Extended Life PhaseProduction 3Production 2Production 1
.4
.10
RHEL 7
.11
*All dates are approximate and subject to change
7 HPC on Wall St 4/2013
Red Hat Enterprise Linux 7:Performance Management
● Simplified instrumentation and tuning capabilities
● Ability to dynamically tune processor workloads through TUNA
● Faster tuning profile setup with predefined templates that can be customized
● Support for kernel parameter tuning
● Superior out-of-the-box performance leveraging advances in hardware
8 HPC on Wall St 4/2013
● Pool of resources (compute, storage, network)
● Nodes can run native or virtualized
● (Re)deployment/growth as needed and required by business
9 HPC on Wall St 4/2013
Red Hat Enterprise Linux 7:Manageability
● Easily configure, monitor, and manage high impact aspects of Red Hat Enterprise Linux through tools and interfaces
● Manage and automate RHEL instances through scripting and APIs
● Increase efficiency via a standards-based common management interface that abstracts complexity of underlying systems
● Support for Red Hat, third-party and custom management frameworks
10 HPC on Wall St 4/2013
● Scale out and scale up
● Consistent platform with RHEL
● Virtualized mission critical workloads with RHEV
11 HPC on Wall St 4/2013
● Load balancing
● Seamless workload migration
12 HPC on Wall St 4/2013
New Storage Infrastructure
RHEV 4.0 – Target H2 - 2013
- Remove need for SPM- Mixed storage types in same pool – iSCSI, FC, NFS
- Based on RHEL7 HypervisorRHEL 7.0
- Cluster-wide Service Level / QOS ManagementSLA Manager
- Third party plugin framework for RHEV-HRHEV-H Plugins
- Network Management ServiceQuantum
- Multilayer Virtual Switch Open vSwitch
RED HAT ENTERPRISE VIRTUALIZATION
RHEV 3.2 – Target H1-2013
- Based on RHEL 6.4 HypervisorRHEL 6.4
- SLA / QoS for CPU, memory and networkHost Level SLA
- Extension framework for RHEV pluginsUI Plugins
- Offload basic storage operations to array – clone, delete, etcArray Offload
13 HPC on Wall St 4/2013
Red Hat OpenStack Contribution Highlights
● nova-core, glance-core, quantum-core
● Stable tree maintenance
● Release and vulnerability management teams
● Packaging for Fedora / EPEL
● Upstream trunk chasing (SmokeStack)
● LDAP integration for Keystone
● Puppet modules for Fedora
● openstack-common
14 HPC on Wall St 4/2013
Red Hat will include the following in it's Red Hat OpenStack distribution
● All core OpenStack Foslom packages, including Quantum and Cinders
● Support for OpenvSwitch via userspace tools in Red Hat OpenStack + kernel support in RHEL 6.4
● Puppet modules for installing all services for OpenStack● A multi-node installer for small deployments● Reference architectures for large scale deployments● Bug-fixes and features selectively back-ported from Grizzly
Red Hat's OpenStack Offering
15 HPC on Wall St 4/2013
Red Hat OpenStack Preview
● Preview Program
● openstack.redhat.com
16 HPC on Wall St 4/2013
● Gluster as platform for big data
● Integration with Hadoop
● Build on demand big data storage clusters
17 HPC on Wall St 4/2013
● Take advantage of resource wherever they are
● Cater for security, SLA and location requirements
● Create seamless pool of resources
18 HPC on Wall St 4/2013
● Drives the need for management tooling
● Consistent tools independent of deployment and location
19 HPC on Wall St 4/2013
● Cloudforms/ManageIQ provides a common and open framework
● Execute administrative tasks across clouds
● Define “once” , deploy “multiple”
● Describe security, configuration and dependencies once
20 HPC on Wall St 4/2013
Red Hat Open Hybrid Cloud
21 HPC on Wall St 4/2013
Advanced Messaging
22 HPC on Wall St 4/2013
APIs and Protocols
Messaging API Messaging API
Application Application
Protocol
23 HPC on Wall St 4/2013
APIs and Protocols
REST Web Server/CGI
Application Application
HTTP
24 HPC on Wall St 4/2013
APIs and Protocols
JMS JMS
Application Application
<proprietary>
25 HPC on Wall St 4/2013
Sender
Receivermytopic
Topic - mytopic
Receivermytopic
Sender
Receiverworkq1
Queue - workq1
Receiverworkq1
msg1,msg2,msg3,msg4
msg1,msg2,msg3,msg4
msg1,msg2,msg3,msg4
msg1,msg2,msg3,msg4
msg1,msg3
msg2,msg4
26 HPC on Wall St 4/2013
Broker
Publisher
PublisherW/ Reply Q
Subscriber
Subscriber
Subscriber
Traditional MOM
Message Queues
PersistentStore
27 HPC on Wall St 4/2013
APIs and Protocols
JMS JMS
Application Application
AMQP
28 HPC on Wall St 4/2013
APIs and Protocols
JMS Qpid .NET API
Application Application
AMQP
29 HPC on Wall St 4/2013
Router or Broker
ClientPublisher
ClientSubscriber
Message To Address//hostname/topic
Message To Address//hostname/topic
ClientPublisher
ClientSubscriber
Message To Address//hostname/topic
AMQP 1.0 Supports peer-to-peer addressing
30 HPC on Wall St 4/2013
ClientPublisher
Message To Address//hostname/topic
ClientSubscriber
AMQP Network of Smart Routers
31 HPC on Wall St 4/2013
AMQP Routing
AMQP Network Architecture
DNS / Geo Load Balancing
Server Load Balancer / VIP
Proxy/Gateway Proxy/Gateway
Broker Broker
ManagementMonitoring
ProvisioningMulti-tennancy
Clients / Services
Back Office
32 HPC on Wall St 4/2013
Proton: An AMQP Protocol Engine
AMQP
Application
TransportProton
Application
Transport
Driver
API
Designed to promote adoption of AMQP
Integrates below the API
Not specific to any execution/threading model
Not specific to any transport API (sockets, etc.)
Implementations: C, Java, JavaScript (planned)
33 HPC on Wall St 4/2013
AMQP and Vlans across the Clouds
Qpid Dispatch Router
Red HatLab
Machine
Mylaptop LOGICAL
Public Cloud
34 HPC on Wall St 4/2013
Message To Address//hostname/topic
ClientPublisher
NYC
ClientSubscriber
London
AMQP Network of Smart Routers in a Cloud
ClientSubscriberHong Kong
Private or Public Cloud
35 HPC on Wall St 4/2013
MOMBroker
Back Office Apps
JEE Container
JEEApps
JEE Container
JEEAppsWS
Qpid/Proton Lib
ClientPublisher
ClientSubscriber
Intranet/Internet
The AMQP Network
36 HPC on Wall St 4/2013
Challenges For AMQP
Finalize addressing – for truly interoperable and portable end points and intermediaries
Some standardized configuration for broker artifacts. ● AMQP 0-10 -> AMQP 1.0+ based assets
Shifting the focus from MOM based
38 HPC on Wall St 4/2013
Red Hat Enterprise Linux 7: Networking
Team Device● Mechanism for bonding multiple network devices into a
simple logical interface at the data link (L2)● Alternative to the existing Linux Bonding driver
Ethernet Data Center Bridging (DCB) support● Priority-based Flow Control (PFC) [802.Qbb], Enhanced
Transmission Selection (ETS) [802.1Qaz], and Congestion Notifications (QCN) [802.1Qau]
Network Manager expanded to support more network configurations
Additional support for Virtual Ethernet Port Addressing (VEPA), Network Namespaces (netns), IEEE 1588 Precision Time Protocol (PTP), and Stream Control Transmission Protocol (SCTP)
39 HPC on Wall St 4/2013
Red Hat Enterprise Linux 7: Linux Containers and RHEL guest Easily partitions a system into multiple secure compartments
● Simple, low overhead method of deploying multi-tenant applications
Simplified installation process for RHEL as a guest on supported hypervisors
Ability to handle 3D graphics and audio for virtual desktop applications
IP Virtual Server (IPVS)● Support for Linux Containers (LXC) allowing process
virtualization with network namespace transport-layer load balancing
40 HPC on Wall St 4/2013
Red Hat Enterprise Linux 7: Virtualization and cloud NUMA capabilities in KVM for better virtualization performance
Strengthening security isolation and fine-grained capabilities in the KVM stack
Smaller virtualization overhead such that SSD-based I/O device and SATA drives overhead is less than 10%
41 HPC on Wall St 4/2013
Red Hat Technology Differentiation
OpenStack is an exciting and fast moving project with a very vibrant community. We expect it will be for IaaS what RHEL is for Linux
Red Hat is a founding member of the OpenStack foundation
Committed significant financial and development resources
Numerous Red Hat engineers part of key OpenStack projects
Red Hat is now the #2 contributor to OpenStack
Working with partners and customers to make OpenStack a stable, robust and scalable enterprise solution
Plans in place to offer a Red Hat based OpenStack distribution
42 HPC on Wall St 4/2013
Why is Red Hat involved with OpenStack● OpenStack is a growing, exciting and vibrant open-source project. Red Hat is a
leader in open source, cloud and virtualization technologies. OpenStack is a natural fit and strategic to our efforts in advancing open source cloud technologies
Red Hat's achievements to date● Established ourselves as leaders of OpenStack community
● Red Hat engineers began quietly contributing to OpenStack mid-2011
● Red Hat has grown into a significant OpenStack engineering team● Red Hat was the #2 overall code contributor to the Folsom release
and the #3 contributor to Essex● Red Hat engineers are highly influential in the project's technical
meritocracy● Established a long-term governance structure for OpenStack
● Red Hat was a key driver in establishing the OpenStack Foundation to “protect, promote and empower” the project
● Red Hat is a Platinum member of the OpenStack Foundation
Red Hat's OpenStack History
43 HPC on Wall St 4/2013
Red Hat's achievements to date (continued)● Released preview versions of Red Hat’s commercial OpenStack distribution
– Essex and Folsom versions available, both based RHEL 6.3– Freely available for download from Red Hat’s Customer Portal– Preview is a non-production, unsupported version aimed at developers and
integrators for testing OpenStack on RHEL● Introduced an invitation-only high-touch, supported preview of Red Hat
OpenStack● Created a Lighthouse Program to enable paid engagement supported by Red
Hat’s OpenStack engineering team
What is Red Hat’s current focus● Creating a commercial, fully-supported OpenStack product
– Paid subscription– Based on OpenStack Folsom with RHEL 6.4– Available early 2013
Red Hat's OpenStack History
44 HPC on Wall St 4/2013
Developers with the most patches:
termie 238 (6.8%)Gabriel Hurley 207 (5.9%)Brian Waldon 195 (5.6%)Johannes Erdfelt 146 (4.2%)Vishvananda Ishaya 116 (3.3%)Dolph Mathews 98 (2.8%)Dan Prince 84 (2.4%)Ziad Sawalha 80 (2.3%)Jason Kölker 77 (2.2%)Mark McLoughlin 73 (2.1%)Jake Dahn 73 (2.1%)Rick Harris 71 (2.0%)Alex Meade 70 (2.0%)Trey Morris 62 (1.8%)Joe Heck 58 (1.7%)Chris Behrens 52 (1.5%)Russell Bryant 50 (1.4%)Eoghan Glynn 50 (1.4%)Joe Gordon 47 (1.4%)Jesse Andrews 46 (1.3%)
Covers 54.380925% of changesets
Top patch contributors by employer:
Rackspace 1921 (55.2%)Nebula 348 (10.0%)Red Hat 275 (7.9%)HP 101 (2.9%)Canonical 92 (2.6%)Citrix 83 (2.4%)Nicira 78 (2.2%)Cloudscaling 47 (1.4%)Delta Electronics 44 (1.3%)eNovance 41 (1.2%)SINA 38 (1.1%)Cisco Systems 25 (0.7%)Nimbis Services 22 (0.6%)[email protected] 20 (0.6%)FathomDB 20 (0.6%)Everbread 19 (0.5%)Midokura 19 (0.5%)Wikimedia Foundation 18 (0.5%)[email protected] 14 (0.4%)[email protected] 14 (0.4%)
Covers 93.047975% of changesets
45 HPC on Wall St 4/2013
Red Hat's product offering beyond what's available upstream
● RHOS will be to OpenStack what RHEL is to Linux● In many ways, the RHOS and RHEL value propositions are identical● Products. Solutions. Support. Training● Stable, Secure, Scalable, Certified● Product evolution will follow a similar path to RHEL● Customers will influence that evolution as they do for RHEL● Customers benefit from our insight into upstream roadmap● Customers have the ability to leverage Red Hat's upstream
leadership● “Upstream first” development mentality● Killer combination of our Linux, KVM and OpenStack leadership
Red Hat OpenStack and the Community Version
46 HPC on Wall St 4/2013
Red Hat leadership in the upstream community● Two members on the Technical Committee● Project Technical Lead for Oslo● Members of Nova, Quantum, Glance, Keystone, Oslo, Heat and
Ceilometer “core” teams● Leading the stable tree maintenance effort● Members vulnerability management and release teams● Upstream continuous integration via Smokestack● Overall, Red Hat Engineers were #3 contributor to Essex and #2 to
Folsom
Red Hat's Upstream Leadership Commitment
47 HPC on Wall St 4/2013
Examples of Red Hat's upstream engineering work● Packaging for Fedora and RHEL● LDAP integration● PKI Tokens based authentication● Configuration rationalization● Generalization of messaging infrastructure to support Apache Qpid● Improved KVM performance● Improved file injection security via libguestfs integration● Groundwork for supporting rolling upgrades● Improved Quantum/Nova integration● Enterprise networking support in Quantum● Development of the Heat orchestration service
Red Hat's Upstream Areas of Focus
48 HPC on Wall St 4/2013
Focus areas for Grizzly and beyond● Rolling upgrades● Provisioning, management and monitoring tools● Orchestration (ala CloudFormations)● Monitoring and auto-scaling (ala CloudWatch)● Metering and billing● Improved PostgreSQL support● Paying down technical debt (e.g. Oslo)● GlusterFS integration● Moving from Nova networking to Quantum● Quantum support for complex enterprise environments● Cells work for scaling● Better integration of guest image creation tools like Oz● Minimal image based compute node deployments (ala ovirt-node)● Ongoing OpenShift, CloudForms and RHEV integration improvements
Red Hat's Future Focus Areas