Sysplex Networking Technology Overview - SHARE · Sysplex Networking Technology Overview Thursday,...
Transcript of Sysplex Networking Technology Overview - SHARE · Sysplex Networking Technology Overview Thursday,...
© Copyright International Business Machines Corporation 2010. All rights reserved.
Summer 2010 Technical Conference
Enterprise Networking SolutionsGus Kassimis - [email protected]
Sam Reynolds - [email protected]
Sysplex Networking Technology Overview
Thursday, August 5, 2010, 11:00 AM - 12:00 PM
APP
APP
Sysplex
Distributor
WLM
Sysplex
Distributor
Hot Standby
VIPA1
Hidden
VIPA1
Hidden
VIPA1
z/OS Sysplex
Pagent
Inbound data path
Outbound data path
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
• Sysplex Overview
• Communication Server enablement for key Sysplex value points
• Network access - SNA and TCP/IP
• The Virtual IP Address concept
• Sysplex-internal or external IP load balancing decision point
• SNA Availability and load balancing
• Subplexing - Isolating network resources
Agenda
APP
APP
Sysplex
Distributor
WLM
Sysplex
Distributor
Hot Standby
VIPA1
Hidden
VIPA1
Hidden
VIPA1
z/OS Sysplex
Pagent
Inbound data path
Outbound data path
112
2
3
4
56
7
8
9
10
11
112
2
3
4
56
7
8
9
10
11
SwitchSwitch
PPRC
Application A
CEC-1
Application A
CEC-2OS and middle- ware infra- structure supporting data sharing
OS and middle- ware infra- structure supporting data sharing
112
2
3
4
56
7
8
9
10
11
112
2
3
4
56
7
8
9
10
11
SwitchSwitch
PPRC
Application A
CEC-1
Application A
CEC-2OS and middle- ware infra- structure supporting data sharing
OS and middle- ware infra- structure supporting data sharing
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
I want to use Application A and its data, but I do not care where it is in this Sysplex!
The promises of the Parallel Sysplex cluster environment are:
•Application location independence•Ability to shift application workload between LPARs
•Application single system image from the network
•Application capacity on-demand•Component failure does not lead to application failure
Gaining the benefits, depend on:• Carefully designed redundancy
of all key hardware and software components in symmetric configurations
• Supporting functions in z/OS and middleware
• Cooperation by applications• Operations procedures
Application services to be always available - both during planned and unplanned outages
Unleashing the benefits of the Parallel Sysplex cluster
3
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
NN
CouplingFacilityEscon
NN
CS
CSCSCS
Network
Sysplex Enables Single System Image
Transparent location of applicationsMultiple images of same application appear as single application to end user
• Balance Workload within Sysplex• Minimize Application failure impact• Freedom to move application workload to
other images
Single SystemImage
4
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Connectivity Services• Discovery of new sysplex
members• Dynamic connectivity via XCF
linksDirectory Services
• Dynamic registration for applications
• Transparent location of resources
System Administration Services
• System Cloning• Application Cloning
Sysplex Enables Horizontal Growth
CouplingFacility
Network
CS CS
CS
Move Application
CS
Add New Image
CS
5
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Reduced Definitions via System SymbolicsExploit Cloning Support
• System Cloning • System Symbolics in VTAMLST members
•Allows VTAMLST definitions to be shared among Sysplex members
• System Symbolics in TCP Config files •Allows TCP Config Files to be shared among Sysplex members
• VTAM Application Cloning• Dynamic Definition of VTAM Applications
•System Symbolics and wildcards in APPL names (e.g. APPL.&sysclone, APPL*)
• Reduction in VTAM resources used for APPL definitions •Network Address allocated at OPEN ACB and released at CLOSE ACB
•APPL definition created at OPEN ACB and deleted at CLOSE ACB
• Allows for easy APPL relocation • Cloning for TN3270 Server
• Support includes TN3270 clients represented by VTAM APPL definitions
•Reduction in VTAM resources used to represent TN3270 clients
•Simplifies TN3270 Server relocation
Network
NN
CouplingFacilityEscon
NN
CS
CSCSCS
Single SystemImage
6
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Automatic Recovery
Exploit zSeries Automatic Restart Manager (ARM)• Registered applications automatically restarted on failure
• ARM policy provides an ordered list for recovery • VTAM registers with ARM for restart• TCP/IP stack registers with ARM for in-place restart
• ARM facility is open interface which can be exploited by any application • Exploited by CICS, IMS, DB2
ARM
XCF Address Space
Applicationsdiscontinue
on SYSB
Appl1Appl2Appl3
GRP01
GRP02
Appl4Appl5
Restart on SYSA
Restart on SYSC
SYSA
SYSB
GRP01
Appl1Appl2Appl3
GRP02
Appl4Appl5
SYSC
CF
ARMPolicy
Couple Data Set
7
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Network access to the down-stream network from a Sysplex
(IP Communications)
Layer-3Layer-3
CEC-1 CEC-2
Network Services LPAR
Network Services LPAR
Application LPAR
Application LPAR
Application LPAR
Application LPAR
Application LPAR
Application LPAR
Switch-1 Switch-2
Layer-2Layer-2
VLAN1 VLAN2 VLAN3 VLAN4
OSA
QDIO
OSA
QDIO
OSA
QDIO
OSA
QDIO
Layer-3Layer-3
CEC-1 CEC-2
Network Services LPAR
Network Services LPAR
Application LPAR
Application LPAR
Application LPAR
Application LPAR
Application LPAR
Application LPAR
Switch-1 Switch-2
Layer-2Layer-2
VLAN1 VLAN2 VLAN3 VLAN4
OSA
QDIO
OSA
QDIO
OSA
QDIO
OSA
QDIO
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Downstream network connectivity to the Sysplex: OSA-Express with QDIO
No single point-of-failure!
• Redundancy at all levels
• Application• LPARs• zSeries HW• CF• OSA-E• Switch• Routers• Network
• Automated fail-over technologies at all levels
9
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Network access to the down-stream network from a Sysplex
(SNA Communications)
RTP RTPANR
ANR ANR
ANRx
Application
RTP TCP
ANR IP
DLC
API Sockets
RTP RTPANR
ANR ANR
ANRx
Application
RTP TCP
ANR IP
DLC
API Sockets
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
High Performance Routing (HPR)
• High Performance Routing (HPR) is a high-availability extension to the original SNA architecture
• HPR preserves sessions across intermediate node/link failure•RTP reroutes sessions in event of a planned or unplanned node or link failure
•Sessions rerouted to new route determined by Class of Service
•Sessions may be maintained across link "hit" without any switch occurring
•Discarded data retransmitted using HPR selective retransmission
11
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Enterprise Extender (HPR Over UDP)
• Preserves SNA application/device investment
• No changes required to SNA application
• Enables single WAN protocol• Eliminates native SNA in the WAN• End-to-end SNA over IP transport
•Includes preservation of SNA prioritization
• Improves datacenter connectivity and access
• Exploits OSA Express and HiperSockets
• Simplifies APPN network design• Significantly reduces network flows in
WAN as compared to base APPN
• Can replace SNI with IP technologies• Uses Extended Border Node (EBN)
connectivity
IP Backbone
HPR
TN3270,or Webclient
SNAClients
Cisco SNASw, Communications Server
for Windows, or other
System z Servers
SNA Network
IBM
IBM
IBM
TCP sessions/routes
SNA routes for SNA sessions
IBM
EE routes for SNA sessions 12
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
z/OS CS TN3270E Server
TN3270 server
TCP/IP address space
VTAM address space
SNA appli- cation
z/OS LPAR
•TN3270 is a standard protocol for transmitting 3270 data streams over an IP network
•The TN server has an LU-LU session with the SNA application for each TN3270 client and transforms the datastream back and forth between native SNA and TCP/IP.
•The TN3270 server can take advantage of many TCP/IP high-availability functions:
•VIPA Takeover•Sysplex Distributor
•To improve the ability of a set of TN3270 servers in a sysplex to present a single system image to the network, z/OS V1R10 added a TN3270 LU Name Management capability to coordinate the assignment of LU names from an LU name pool shared between multiple TN3270 servers.
SNA TCP/IP
TN3270
13
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Virtual IP Addressing
TN3270e Server
VIPA#1
CICS Appl-A
VIPA#2
FTP Services
VIPA#3 DB2 subsystem
VIPA#4
OSA OSAOSA
CICS Appl-B
VIPA#5
Web Services
VIPA#6
IP#10 IP#11 IP#12
Connect to VIPA#1
Connect to CICS-Appl-A.xyz.com
My virtual z/OS IP host
Resolve CICS-Appl-A.xyz.com
Use IP address VIPA#2
Name server
TN3270e Server
VIPA#1
CICS Appl-A
VIPA#2
FTP Services
VIPA#3 DB2 subsystem
VIPA#4
OSA OSAOSA
CICS Appl-B
VIPA#5
Web Services
VIPA#6
IP#10 IP#11 IP#12
Connect to VIPA#1
Connect to CICS-Appl-A.xyz.com
My virtual z/OS IP host
Resolve CICS-Appl-A.xyz.com
Use IP address VIPA#2
Name server
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
The network view of a Parallel Sysplex - a single large server with many network interfaces and many services
The objective is to make the Sysplex look like one large server that has a
number of physical network interfaces
for performance and availability - and that provides a number of highly available
and scalable services.
• Single-system image (SSI)
• Scalable• Highly available• Secure
15
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Why do I need virtual IP addresses (VIPA)?
What does the virtual IP addressing (VIPA) technology promise?
Interface resilience: •Communication with a server host is unaffected by server physical network interface failures. As long as just a single physical network interface is available and operational on a server host, communication with applications on the server host will persist.
Application access independent of network topology:• Separates network topology from server application topology - a VIPA address can be used to identify a server application instead of a physical network interface.
• Allows network administrators to renumber physical network topology • no impact to end-user accessing server applications by IP address• no changes needed in DNS or hosts file configuration• no impact to firewall filtering rules
Single system image: •Allows the Sysplex to be perceived as a single large server node, where VIPA addresses identify applications independently of which images in the Sysplex the server applications execute on.
•Applications retain their identity when moved between images in a Sysplex.•Multiple instances of a server application can be accessed as one server.
TN3270e Server
VIPA#1
CICS Appl-A
VIPA#2
FTP Services
VIPA#3 DB2 subsystem
VIPA#4
OSA OSAOSA
CICS Appl-B
VIPA#5
Web Services
VIPA#6
IP#10 IP#11 IP#12
Connect to VIPA#1
Connect to CICS-Appl-A.xyz.com
My virtual z/OS IP host
Resolve CICS-Appl-A.xyz.com
Use IP address VIPA#2
Name server
16
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Dynamic VIPA Movement - Stack Managed DVIPAs
ESCON
Network
COUPLINGFACILITY
VIPA 192.168.253.1
VIPA 192.168.253.2
VIPA 192.168.253.3
VIPA 192.168.253.4 VIPA
192.168.253.5
VIPA 192.168.253.6
192.168.253.4Cached IP address
1 2Dynamic VIPA Support
• VIPAs can survive any outage by moving to another stack in Sysplex via VIPA Takeover
• VIPAs exchanged by TCP/IP stacks in sysplex via XCF messaging
• Another appl instance can pick up workload or Appl can be restarted on takeover stack
• Connections broken but Reset sent to client upon takeover
• Significantly reduces down time
Dynamic VIPA Takeback • VIPA moves back to recovered primary owner
• New Connections Handled By Primary Owner again• Connections Established To Backup are allowed to
continue• Data forwarded from primary owner to backup
• Allows Movement Of Application Server Without Impacting Existing Workload
Useful for planned outages as well• Operator commands allow you to move
Dynamic VIPAs non-disruptively
17
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Are dynamic routing protocols required on z/OS in order to use VIPA?
Appl-1
Appl-1
VIPA2
VIPA2
VIPA1
VIPA1
OSA OSA OSA
R R R
Connect to VIPA-1 (Appl-1)
Connection resilience to network and interface
outages
A
Move appl-1 and its associated dynamic VIPA address
BMove a dynamic VIPA address between z/OS images
C
DLoad-balance
outbound IP traffic (multipath)
z/OS-A
z/OS-B z/OS-C
Base IP recovery as well as VIPA address movement were designed and implemented with the use of dynamic routing functions in mind!
The recommended dynamic routing protocol in the Sysplex is OSPF.
Dynamic routing is not an absolute requirement, but it is highly recommended when using VIPA addresses (it makes life a whole lot easier)!
Always remember that a z/OS Sysplex is not a host, it is an IP network in itself and as any IP network it needs the capability to react to topology changes in its own network and in the adjacent networks.
18
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
• Single-instance applications are applications that only run in one instance in the Sysplex. Either because the application needs exclusive access to certain resources, or because there is no need to start it in more than one instance.
• Availability from an IP perspective then becomes an issue of being able to restart the application on the same LPAR or on another LPAR with as little impact to end-users as possible.
•Speed of movement - ARM or automated operations procedures•Retain identity from a network perspective (its IP address) - Application Instance DVIPAs
DNS
cicsappl1.mycom.com:10.1.1.1
Either1 Resolve cicsappl1.mycom.com 2 connect to returned address
or3 Connect to cached (or hardcoded!) address
cicsappl1
cicsappl1
Resolve cicsappl1.mycom.com
Use 10.1.1.1
Connect to 10.1.1.1
10.1.1.1 10.1.1.1 Application-specific dynamic VIPA addresses come in very handy for this purpose.
Restart application
Basic principles for recovery of single-instance IP application in a Sysplex
19
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Single system image (SSI) from an IP perspective in the Sysplex
z/OS z/OS z/OS
Sysplex
Connect to DRVIPA1
Connect to ? from
SRCVIPA1
Connect to ? from
SRCVIPA3
Connect to ? from
SRCVIPA2z/OS z/OS z/OS
Sysplex
Connect to DRVIPA1
Connect to ? from DRVIPA2
• We have single system image capability for inbound connections where a single distributed VIPA address can represent all images in the Sysplex - and remote users do not need to select a specific image when connecting to their server application.
• But if we establish outbound connections from the images in the Sysplex, each image has its own source VIPA address - so there is no single system image from an outbound connection perspective - which has implications in firewall filter setup, etc.
•TCP/IP provides several facilities that allow you to specify the IP address to be used on outbound connections from the sysplex
• SOURCEVIPA, TCPSTACKSOURCEVIPA, and SRCIP statement in TCP/IP profile
• Can be specified on a system basis or an application basis (even if the application can execute on any system in the sysplex)
Inbound SSI
Outbound SSI
20
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
How to balance your IP workloads so that you can maximize your availability and productivity?
A1 A1 A1
Sysplex
z/OSz/OS
TCP/IP VTAM
LB
TCP/IP VTAM
A1 A1 A1
Sysplex
z/OSz/OS
TCP/IP VTAM
LB
TCP/IP VTAM
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Connection load balancing technologies:
Between z/OS images:• Internal: Sysplex Distributor, Generic
Resources• External: Cisco CSM, CSS, F5 Big IP, etc.
Inside single z/OS TCP/IP stack:• Port sharing
Application Characteristics:•Multiple instances of the server are able to provide the exact same services to clients (will typically require data sharing)
•No state preserved at server between two connections (application protocol has to include support for such behavior or store state data in shared storage)
Benefits of Intelligent Load Balancing:•Performance - improving response time•Availability - If one instance goes down, connections with it break, but new connections can be established with remaining instance(s)
•Scalability - more server instances can be added on demand (horizontal growth)
Examples:•Web server•TN3270 server•CICS applications•FTP server •DB2•MQ•WAS•LDAP•RYO...
Workload balancing: a question of both performance, availability, and scalability
22
23
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Sysplex Distributor: z/OS-integrated intra-Sysplex workload balancing• Independent of network attachment
technology. Will work with both direct (including OSA Express) and channel-attached router network connections.
• All z/OS images communicate via XCF. Each TCP/IP stack has full knowledge of IP addresses and server availability in all stacks.
• A network-connected stack owns a given VIPA address and acts as the distributor of new connection requests to that VIPA address.
Distribution of new connection requests is based on real-time information• State of target application, system and TCP/IP stack• WLM recommendations
• LPAR CPU capacity or WLM Server specific recommendations (are the target server applications meeting their WLM policy goals?)• Additional workload distribution methods
• Round robin, Weighted Active, Hot/Standby • Application Server and TCP/IP health
• Are the target applications accepting new connections? Do they have network connectivity back to the clients?• Network Quality of Service (QoS) metrics (with z/OS QoS policy agent)
APP
APPSysplexDistributor
WLM
SysplexDistributor
Hot Standby
VIPA1
HiddenVIPA1
HiddenVIPA1
z/OS SysplexPagent
Inbound data path
Outbound data path
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Server instance
Server instance
Server instance
z/OS LB agent
Work requests
Work requests
Private protocol control flows
SASP control flows
Load Balancer
z/OS workload balancing• Support for clustered z/OS servers in a z/OS Sysplex
• Not focused on HTTP(S) only, will support all IP-based application workloads into a z/OS Sysplex
• Based on Sysplex-wide WLM policy
• Scope is a z/OS Sysplex
z/OS Load Balancing Advisor (LBA) for outboard load balancers
z/OS LB agent
z/OS LB agent
z/OS Sysplex
z/OS LB advisor
The SASP (Server/Application State Protocol) control flows will provide relative weights per server instance (based on WLM weight, server availability, and server processing health taking such metrics as dropped connections, size of backlog queue, etc. into consideration)
24
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
SNA Sysplex Capabilities for Maximizing Availability
CICS
ESCON
Network
VTAM1 VTAM2 VTAM3 VTAM4
VTAMA VTAMB
NNBNNA
EN1 EN2 EN3 EN4
CICS2 CICS3
COUPLING
FACILITY
CICS5
CICSCICS1 - EN1
CICS2 - EN2
CICS3 - EN3
CICS4 - EN4
CICS5 - NNB
"Directory"
DB2DB2a -EN1
DB2b -EN2
DB2c - EN3
DB2d - EN4
IMSIMS1 - EN1
IMS2 - EN2
IMS3 - EN3
IMS4 - EN4
CICS1 CICS4
IMS1 IMS2 IMS3 IMS4 DB2a DB2b DB2c DB2d
LU62A
LU62A-IMS3
IMS
LU3270A
"Affinities"
CICS
ESCON
Network
VTAM1 VTAM2 VTAM3 VTAM4
VTAMA VTAMB
NNBNNA
EN1 EN2 EN3 EN4
CICS2 CICS3
COUPLING
FACILITY
CICS5
CICSCICS1 - EN1
CICS2 - EN2
CICS3 - EN3
CICS4 - EN4
CICS5 - NNB
"Directory"
DB2DB2a -EN1
DB2b -EN2
DB2c - EN3
DB2d - EN4
IMSIMS1 - EN1
IMS2 - EN2
IMS3 - EN3
IMS4 - EN4
CICS1 CICS4
IMS1 IMS2 IMS3 IMS4 DB2a DB2b DB2c DB2d
LU62A
LU62A-IMS3
IMS
LU3270A
"Affinities"
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
SNA Generic Resources• Multi- system application seen as single
application to end user• Balances sessions within the parallel sysplex• New logons not affected by application outage
• Dynamic registration performed by application on activation
• Application is de-registered when no longer available
• Exploited by CICS, IMS, DB2, APPC/MVS, and TSO/VTAM
• Available to any SNA application
MNPS Support for SNA Sessions • Support for both RAPI and APPC sessions
• Eliminates or reduces outage (VTAM, z/OS, or hardware) impact
• Persistence support also available for planned application workload takeover
• Requires HPR within a parallel sysplex
• Exploited by CICS, APPC/MVS, IMS, and the IBM Session Manager (ISM)
Generic Resources and Multi-Node Persistent Sessions
26
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
The usual answer: It depends!• The most important aspect to understand is whether session affinities exist for the
application workloads involved• 3270 workloads typically mean no session affinity exists upon application outage
•Generic Resources allows the end user to logon to another application instance immediately after the original application outage
• Recovery is faster than MNPS and requires far less overhead during normal operations• LU 6.2 workload typically means that a session affinity does exist upon application outage
•Generic Resources is not allowed to choose another application instance upon a subsequent logon from the same end user after the original application outage
• Original application must be recovered
• Another important aspect is to determine the importance of availability vs overall system performance
• MNPS provides superior availability but does impact performance during normal operation•Storage impact, CF access, etc. •The steady-state CPU utilization associated with an application may increase by up to 50% when MNPS-enabled due to the overhead of maintaining the state in the coupling facility
• Generic Resources does not impact performance of the data path •Resolution only done during session establishment
Recommendation:• Implement Generic Resources for IMS, CICS, and DB2 workloads
• Consider implementing MNPS for select applications only if LU 6.2 workload is critical enough to justify the extra CPU cycles
MNPS or Generic Resources
27
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
How to mesh connect your sysplex images so that you can promote data sharing and grow your business without impact to your customers?
CouplingFacility
Network
CS CS
CS
Move Application
CS
Add New Image
CS
CouplingFacility
Network
CS CS
CS
Move Application
CS
Add New Image
CS
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
XCF Dynamics Enables Horizontal GrowthXCF uses Coupling Facility Links for Data Transport
• Eliminates requirement for ESCON definitions for SNA and IP
• Provides dynamic discovery and connectivity of other nodes in Sysplex• No coordinated definition required to
add new images• Provides notification of new
sysplex members as well as members who have failed• Dynamically updates existing
members of sysplex
29
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Is XCF signaling always used for the DYNAMICXCF IP network?
zSeries CEC-1
CEC-2
LPAR-1 LPAR-2
LPAR-3
TCP/IP Stack-1
TCP/IP Stack-2
TCP/IP Stack-3
TCP/IP Stack-4
IUTSAMEH HiperSockets
XCF Signaling
CS z/OSTCP/IP
CS z/OSTCP/IP
CS z/OSTCP/IP
CS z/OSTCP/IP
From an IP topology perspective, DYNAMICXCF automatically establishes fully meshed IP connectivity to all other z/OS TCP/IP stacks in the Sysplex that also have DYNAMICXCF specified.
• One end-point specification in each stack for fully meshed connectivity to all other stacks in the Sysplex:•IPConfig DynamicXCF 192.168.5.1 255.255.255.0 1
• Automatic connectivity to new stacks as they start up in the Sysplex• Only one dynamic XCF network supported per Sysplex
Under-the-covers DYNAMICXCF will choose one of three transport technologies depending on availability and location of partner z/OS TCP/IP stack:
• Inside same LPAR: IUTSAMEH (memory-link inside a z/OS system)• Inside same zSeries CEC: HiperSockets (if enabled for that purpose via the
IQDCHPID VTAM start option)• Outside zSeries CEC: XCF signaling 30
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
SNA Intra-Sysplex Connectivity: MPC+, XCF, or EE using QDIO?
• The usual answer: It depends!
•All things being equal, MPC+ throughput should exceed XCF throughput, but using multiple XCF links can significantly increase throughput.
•XCF links can bypass the coupling facility, increasing throughput if the CF is being used for other functions (GR, MNPS)
•XCF will use more VTAM CPU cycles due to the API to the XCF facility.
• Many customers want to define both MPC+ and XCF links, but want to prefer MPC+, with XCF available for backup.
•This can be accomplished by adding COSTBYTE=1 to the XCF TGP (in IBMTGPS) which is automatically associated with XCF TGs (assuming IBMTGPS has been activated). This makes the XCF link have a higher weight (and therefore be less desirable) than the MPC+ link for the IBM-supplied APPN Classes of Service.
• EE using QDIO is a valid option for intra-sysplex connectivity but consider:•Cross CEC traffic must go out onto the data center LAN and via 2 OSA Express cards•Cross CEC traffic may not realize a significant performance advantage over well tuned XCF or MPC+ •For SNA workloads within the same CEC, EE over HiperSockets (or shared OSA) will provide superior performance unless CPU availability is limited
31
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Sysplex Network Partitioning
A1 A2
Intranet IP
A1 A2
Intranet IP
A1 A2
DMZ IP
A1 A2
DMZ IP
Intranet DMZ
A1 A2
Intranet IP
A1 A2
Intranet IP
A1 A2
DMZ IP
A1 A2
DMZ IP
Intranet DMZ
Dedicated LPARs with single TCP/IP stack
Multi-purpose LPARs with dual TCP/IP stacks
A1 A2
Intranet IP
A1 A2
Intranet IP
A1 A2
DMZ IP
A1 A2
DMZ IP
Intranet DMZ
A1 A2
Intranet IP
A1 A2
Intranet IP
A1 A2
DMZ IP
A1 A2
DMZ IP
Intranet DMZ
Dedicated LPARs with single TCP/IP stack
Multi-purpose LPARs with dual TCP/IP stacks
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
• How to control level of automatic connectivity •XCF signaling (group name) - both IP and SNA•IUTSAMEH (same host IP links inside an LPAR)•HiperSockets (as enabled via IQDCHPID in VTAM)
• How to control level of IP and SNA resource awareness
•Dynamic IP address discovery across the Sysplex•VTAM generic resource and MNPS resource scope spans the full Sysplex
• How to control scope of IP workload balancing using Sysplex Distributor
•SD requires Dynamic XCF to be enabled, and Dynamic XCF will establish automatic IP connectivity to all stacks in the Sysplex that also have Dynamic XCF enabled
z/OS Sysplex connectivity to multiple security areas has been an issue every since CS began using Sysplex capabilities
To support environments such as these, installations typically end up implementing complex resource controls and disabling
many of the dynamic networking functions that are provided by TCP/IP and VTAM.
33
VTAM VTAM VTAM VTAM VTAM
IP-1 IP-1 IP-1 IP-1 IP-1
IP-2 IP-2
IP-3 IP-3
DMZ SNA Subplex
DMZ IP Subplex
Intranet SNA Subplex
Intranet Primary IP Subplex
Research IP Subplex
Development IP Subplex
LPAR1 LPAR2 LPAR3 LPAR4 LPAR5
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Enable use of networking Sysplex functions in a Sysplex that is connected to multiple security areas
• One SNA subplex per LPAR
• A TCP subplex cannot span multiple SNA subplexes
• Different IP stacks in an LPAR may belong to different IP subplexes
• Standard RACF controls for stack access and application access to z/OS resources need to be in place.
• Networking subplex scope:•VTAM Generic Resources (GR) and Multi-Node Persistent Session (MNPS) resources
•Automatic connectivity - IP connectivity and VTAM connectivity over XCF (including dynamic IUTSAMEH and dynamic HiperSockets based on Dynamic XCF for IP)
•IP stack IP address (including dynamic VIPA) awareness and visibility•Dynamic VIPA movement candidates•Sysplex Distributor target candidates 34
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
• Sysplex Overview
• Communication Server enablement for key Sysplex value points
• Network access - SNA and TCP/IP
• The Virtual IP Address concept
• Sysplex-internal or external IP load balancing decision point
• SNA Availability and load balancing
• Subplexing - Isolating network resources
Agenda
112
2
3
4
56
7
8
9
10
11
112
2
3
4
56
7
8
9
10
11
SwitchSwitch
PPRC
Application A
CEC-1
Application A
CEC-2OS and middle- ware infra- structure supporting data sharing
OS and middle- ware infra- structure supporting data sharing
VTAM VTAM VTAM VTAM VTAM
IP-1 IP-1 IP-1 IP-1 IP-1
IP-2 IP-2
IP-3 IP-3
DMZ SNA Subplex
DMZ IP Subplex
Intranet SNA Subplex
Intranet Primary IP Subplex
Research IP Subplex
Development IP Subplex
LPAR1 LPAR2 LPAR3 LPAR4 LPAR5
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
For More Information....
URL Content
http://www.ibm.com/systems/z/ IBM System z
http://www.ibm.com/systems/z/hardware/networking/index.html IBM System z Networking
http://www.ibm.com/software/network/commserver/zos/
http://www.ibm.com/software/network/commserver/z_lin/
http://www.ibm.com/software/network/ccl/
http://www.ibm.com/software/network/commserver/library
http://www.redbooks.ibm.com
http://www.ibm.com/software/network/commserver/support
http://www.ibm.com/support/techdocs/
http://www.rfc-editor.org/rfcsearch.html
IBM z/OS Communications Server
IBM Communications Server for Linux on zSeries
IBM Communication Controller for Linux on System z
IBM Communications Server Library - white papers, product documentation, etc.
IBM Redbooks
IBM Communications Server Technical Support
Technical Support Documentation (techdocs, flashes, presentations, white papers, etc.)
Request For Comments (RFCs)
IBM Education Assistanthttp://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp
http://www.twitter.com/IBM_Commserver IBM Communications Server Twitter Feed
IBM Communications Server Facebook Fan Pagehttp://www.facebook.com/IBMCommserver
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
QDIO QDIO10.1.1.1MAC: M1
10.1.1.2MAC: M2
PortA PortB
Router-1
10.1.1.5
IP address Mac address10.1.1.1 M1
10.1.1.2 M2
Router's initial ARP Cache
Example: OSA PortA fails or is shut down
1 The z/OS TCP/IP stack moves address 10.1.1.1 to the other QDIO adapter (PortB), which is on the same network (same network prefix) as PortA was.
2 The z/OS TCP/IP stack issues a gratuitous ARP for IP address 10.1.1.1 with the MAC address of PortB (M2) over the PortB adapter
3 Downstream TCP/IP nodes on the same subnet will update their ARP caches to point to M2 for IP address 10.1.1.1 and will thereafter send inbound packets for both 10.1.1.1 and 10.1.1.2 to MAC address M2
IP address Mac address
10.1.1.1 M2
10.1.1.2 M2
Router's ARP Cache after movement of 10.1.1.1 to PortB
z/OS TCP/IP Stack
IP Layer-2 based network interface recovery functions
Requirement for this feature to function properly:
• At least two adapters attached to the same network (broadcast media).
• Adapters must use either LCS or QDIO
• The two adapters should be two physical adapters for real availability benefits
Inbound to 10.1.1.1
Inbound to 10.1.1.2
Friendly advice: Make sure you are current on OSA-Express micro code upgrades!
10.x.y.0/24
38
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Load-balancing outbound IP packets over multiple first-hop routers (MULTIPATH)
Destination Via
10.1.1.0/24 Direct delivery
Default 10.1.1.5 / PortA
Default 10.1.1.5 / PortB
Default 10.1.1.6 / Port A
Default 10.1.1.6 / Port B
z/OS-1's IP Routing Table (extract)
IPCONFIG MultiPath [PerConnection or PerPacket]
Static route definitions on z/OS:•If an adapter fails in such a way that z/OS TCP/IP gets informed, it will skip over the corresponding entries from the routing table•If one of the first-hop routers loses its connection to the backbone network or if it "dies" - z/OS TCP/IP doesn't know anything about it since it doesn't participate in dynamic routing updates - and it will continue to attempt to use the corresponding routing table entries - connections will time out, UDP packets will be lost, etc.
•If the two routers deploy VRRP or HSRP between them on the interfaces towards the z/OS systems, then the fact that one of them turns into a black hole can be hidden for z/OS - z/OS continues to send packets to both first-hop addresses, they are just both serviced by the one surviving router
Dynamic routing updates:•z/OS TCP/IP will know both if the adapter itself fails or if the first-hop router fails - and dynamically update the routing table entries and recover from the router outage..
PortA PortB
10.1.1.1 10.1.1.2
10.1.3.1
PortC PortD
10.1.1.3 10.1.1.4
10.1.3.2
VIPA2: 10.1.2.2
z/OS-1 z/OS-2
Router-1 Router-2
10.1.1.5 10.1.1.6
VIPA1: 10.1.2.1
QDIO QDIO QDIO QDIO
Be careful if using Multipathing without
dynamic routing!
1 2 3 4
10.x.y.0/24
z/OS V1R5 raised the number of dynamic multipath routes from 4 to 16. HSRP/VRRP
39
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
QDIO QDIOPortA PortB
10.1.1.1 10.1.1.2
10.1.3.1
QDIO QDIOPortC PortD
10.1.1.3 10.1.1.4
10.1.3.2
VIPA2: 10.1.2.1
z/OS-1 z/OS-2
Router-1 Router-2
10.1.1.5 10.1.1.6
Some QDIO basics with respect to VIPA addresses
• All HOME IP addresses will be registered in the OATs dynamically by the TCP/IP stacks and the OAT content will be changed as the HOME lists change due to movement of IP addresses.
• When an IP address is registered, the adapter will do a gratuitous ARP if the address belongs to the same network as to which the adapter is attached (in this example the 10.1.1.0/24 subnet) or if the address is a VIPA address (independent of which subnet the VIPA address is defined on).
• Gratuitous ARPs are done for two purposes:• to enable downstream routers to update their ARP cache if an adapter malfunctions and the TCP/IP stack decides to move an
address to another adapter (example: if PortA fails, then 10.1.1.1 will be moved to PortB and PortB will grat ARP 10.1.1.1) - Note that downstream routers normally will ignore gratuitous ARPs for IP addresses that do not belong to the subnet on that physical network (in this example the 10.1.1.0/24 subnet)
• to check for duplicate IP addresses on the subnet - will continue for up to 5 seconds, but the adapter will accept incoming packets for the new address immediately
VIPA1: 10.1.1.10
Grat ARP with:• 10.1.1.4• 10.1.2.1 (will be ignored by router)
Grat ARP with:
• 10.1.1.1• 10.1.1.10
10.x.y.0/24
40
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
What is DNS/WLM?•Domain Name Service which interfaces with MVS Work Load Manager
•Targeted for long duration connections•DNS resolution for every connection
•More availability than DNS round-robin methods•Provided caching not done at clients or other DNS nodes
•Work load distribution on user defined goals •Clustered host names, server names or Weighted IP Addresses
Benefits of DNS/WLM•Distributes connections based on current load and capacity
•Distributes load across adapters on a host•Dynamically avoids crashed hosts and servers
• Client can reconnect to same Server instance if required•Dynamically avoids crashed TCP/IP stacks
•When using sysplex name•Highly scalable
•New servers added without DNS administration•Inexpensive to deploy
•Uses existing technology
z/OS DNS with MVS Workload Manager
Client 1
mvsplex1TN3270.mvsplex1
Client 2
Client 3
mvsa.mvsplex1LAN
mvsplex1
TCP/IPWLMTN3270myserve
TCP/IPWLMDNSTN3270myserve
TCP/IPWLMmyserve
mvsa mvscmvsb
** z/OS V1R10 was the last release in which DNS/WLM (BIND 4.9.3) was supported **
Not Strategic
41
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Server instance
Server instance
Server instance
z/OS LB agent
Private protocol control flows
Replacing the dynamic DNS registration part of the DNS/WLM component with a Dynamic DNS (DDNS-based) solution
z/OS LB agent
z/OS LB agent
z/OS LB advisor
z/OS SASP DDNS
SASP control flows
z/OS BIND 9 DNS Server DDNS
update flows
Central configuration file with information to identify which servers, server groups, host groups and individual hosts (TCP/IP stacks) to register dynamically.
• No requirements to have applications register themselves
DNS can be on z/OS or any other platform that supports a BIND 8 or later name server
• DDNS registration component will use existing z/OS load balancing advisor infrastructure and appear to the load balancing advisor as an external load balancer
•Potentially possible to extend the dynamic registration capabilities to any SASP-server based implementation, such as a global e-WLM manager.
•Registration/de-registration triggered by the same events that trigger when a server instance is available/not available from an external load balancer perspective.
•LBA controls to quiesce and resume server instances also apply to SASP-DDNS.•Sysplex-wide scope.
• Central Sysplex-wide definitions of which servers, server groups, and stacks to register under which names and in which name servers (DNS domains).
•Registration/de-registration driven by start/stop of the actual resources as reported by the LBA infrastructure.
• The z/OS load balancing advisor may serve both the SASP DDNS registration component and external load balancers at the same time
42
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
Selected name server contents:tn3270.mvsplex.mycorp.com 10.1.10.3 sysa.tn3270.mvsplex.mycorp.com 10.1.10.3 10.1.10.11 sysb.tn3270.mvsplex.mycorp.com 10.1.10.11ftp.mvsplex.mycorp.com 10.1.10.3 sysa.ftp.mvsplex.mycorp.com 10.1.10.3
10.1.10.11 sysb.ftp.mvsplex.mycorp.com 10.1.10.11
SYSA SYSB
TCP/IP S1 TCP/IP S2
Agent Advisor
Name Server
Agent TN3270TN3270 ADNRFTPD FTPD
10.1.10.22 10.1.10.1
10.1.10.55..53
10.1.10.11VIPAsVIPAs
10.1.10.3
z/OS Sysplex
Example of Dynamic Application Domain Name Registration
43
IBM Software Group | Enterprise Networking Solutions
© 2010 IBM Corporation and SHARE
DNS/WLM - going away or not going away or what?• DNS/WLM implemented two distinct functions:
•Dynamic name registration of servers, server groups, and TCP/IP stacks•Workload balancing based on name resolution requests and interaction with WLM
• WLM-based TCP/IP workload balancing into a z/OS Sysplex is today better handled by more modern technologies, such as Sysplex Distributor or external load balancers using the z/OS load balancing advisor technology:
•Less overhead - balancing at connection set up time and not at name resolution time•Not sensitive to DNS caching •Better load balancing decisions - the new technologies have more metrics available than DNS/WLM had
• However, the dynamic name registration capabilities of DNS/WLM are still very useful from an availability perspective and are not replaced by any of the currently available alternative load balancing technologies:
•Dynamic registration of individual application instances when they start up•Dynamic registration of groups of application instances when they start up•Dynamic registration of TCP/IP stacks when they start up
• General dynamic registration in modern DNS servers (BIND 8 or later) is supported by a set of DNS protocols that are known as Dynamic DNS (DDNS)
•CS z/OS V1R8 implemented a new infrastructure that supports DDNS registration of the same type of entries that were supported by DNS/WLM
•DDNS is a standard protocol•Any DDNS capable name server can be the target of the DDNS registrations
44