Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

IMS in a Sysplex Environment: Challenges and Solutions

Nick Griffin

World Wide IMS Software Group Sales Advisor

IBM Silicon Valley Lab

Agenda

� Product Highlights

� IMS Sysplex Review

� IMS Sysplex Challenges and Solutions– Shared Queues

– Data Sharing

– RM Resources

� Q & A

IMS Sysplex Manager Highlights

– Real-time management of the IMS Sysplex Environment• Single system image thru local and aggregate view of data

• Simplified User Interface (TSO/ISPF)

• Structured displays of IMS resources and CF structures

• Global Type-1 command, OM Type-2 and IMS SPOC

• Basic z/OS performance information and SVC dump capture

• Statistics for CSL (OM, RM and SCI), IRLM, VSO, and CQS

• Dashboard with key system indicators and threshold monitoring

• Intercept System exceptions and generate Console alerts

• Produce real-time IRLM Long Lock Report

• Browse, delete and recover messages on Shared Queues

• Delete RM resource structure entries

• Assign affinity for transactions in Shared Queues environment

• Protect against buffer overflow in Shared Queues environment

– Support IMS DB/TM, DBCTL, and DCCTL for IMS v11 and later

MVS1MVS4MVS3MVS2

CQS1 CQS2

SCI1 RM1 OM1 SCI2 RM2 OM2

IMS Sysplex Manager Sample Configuration

IMSSM DC

IMS1 IMS2

IMS Connect IMS Connect

IMSSM UIS

FDR1 FDR2

IMS Sysplex Review

Source: IMS Redbook (SG24-6908-00)

IMS Sysplex Review - Components

V13V13

Source: IMS RedbookSource: IMS Redbook (SG24-6908-00)

IRLM Lock

IMSplex Configuration

IMS Sysplex Challenges and Solutions

� Shared Queues– Transaction affinity

– CF structures and log streams

– Queue monitoring and management

– Full Queue Protection

� Shared Databases– Long lock detection

– Long lock detection

– Investigate lock holders and waiters

– IRLM statistics

– VSO stats

� Shared Resources– RM structure content, status, ownership, and removal

� Shared Queues

–Transaction affinity– CF structures and log streams

– IRLM statistics

– VSO stats

Transaction affinity – The Challenges

�Force local affinity for efficiency

�Assign affinity to specific IMS due to resource availability

�Reduce false-scheduling overhead

�Reduce contention

Sysplex Manager Tool - Transaction Affinity Highlights

�Finer control of transaction scheduling

�Non-invasive to existing definition and operation– No omission of transaction definitions in sysgen– No stopping of transactions– No re-classing of dependent regions – No operational impact for loss of a system

201410

�User defined affinity to route transaction messages– Any IMS in the shared queues group– Any subset of IMS systems– Equal or weighted distribution

Affinity Routing Statistics – System View

201411

Affinity Routing Statistics – Update System

201412

Affinity Routing Statistics – Destination View

201413

Affinity Routing Statistics – Update Destination

201414

Affinity Routing Statistics – Add Destination

201415

–CF structures and log streams– Queue monitoring and management

201416

– IRLM statistics

– VSO stats

IMS Sysplex Review – IMS CF structures and log streams

Log streams

(CQS, OM,

201417

(CQS, OM, RS)

IMS CF Structures and log streams – The Challenges

�Growing use of Coupling Facility Structures – Data Sharing, Shared Message Queues, Resource Manager, CQS log streams

– No single source for list of in use structures and details

�Check log stream attributes

201418

�Monitor offload activities

SM Solution - IMS CF Structures and log streams

�Real-time display of structure list

�Statistics, Connections, Coupling Facility information

�Structure rebuild and checkpoint stats

201419

�Log streams attributes and offload monitoring

IMS Coupling Facility Structures

201420

IMS Coupling Facility Structures

201421

Shared VSO structures

201422

Shared VSO structures

201423

CQS Structures

201424

CQS Structures

201425

CQS Structures

201426

CQS and System Logger

201427

SM Solution - CQS Log Stream info

201428

SM Dashboard

201429

–Queue monitoring and management– Full Queue Protection

201430

– IRLM statistics

– VSO stats

IMS Shared Queues

201431 Source: IMS Redbook (SG24-6908-00)

Shared Queue Management – The Challenges

�Visualize queue content

�Browse, Delete, and Recover messages

�View CQS processing counters

201432

SM Solution - Shared Queue Management

�Filter based on Destination name or ID

�High level queue depth information, Drill down to levels of detail

�Browse, Delete and Recover messages from Cold queue

201433

�CQS processing counters

Destination Queue Depth

201434

201435

201436

201437

201438

–Full Queue Protection

201439

– IRLM statistics

– VSO stats

Full Queue Protection Challenges

�Local Queue– Local buffer overflow can abend or stall IMS control region

�Shared queue structures– Full structures will stall IMS

201440

SM Solution - Full Queue Protection

�Local Queue– Real-time local queue buffer counts and stats

– Reject new messages by employing the Queue Space Notification Exit

(DFSQSSP0)

– Automatic actions for warning, critical or Severe thresholds

– Identify programs using large number of buffers

201441

�Shared queue structures– Alert structure utilization exceeding predefined thresholds

– Unable to reject new messages (a possible future enhancement)

� Shared Databases

–Long lock detection

201442

–Long lock detection– Investigate lock holders and waiters

– IRLM statistics

– VSO stats

Data Sharing Components

201443

Diagnosing Long Locks - The Challenges

� DB Lockouts by applications holding IRLM locks for an inordinate amount of

– Could go unrecognized until it becomes critical

– Lack of supported tools to assist in recognition and identification of problem

– Manual intervention required to resolve

201444

SM Solution - Data Sharing Long Locks

�Automatic real-time recognition when IRLM detects

� Information consolidated, analyzed for top blocker, and presented

� Information recorded in exceptions file and sent to z/OS console

�Messages can be sent to z/OS console using user exit so that

automated operations can resolve

201445

automated operations can resolve

�Problem quickly resolved with automation

Data Sharing Long Lock Exceptions

201446

Data Sharing Long Lock Exceptions

201447

201448

–Investigate lock holders and waiters– IRLM statistics

– VSO stats

Real-time IRLM / PI Locking Display – The Challenges

� Identify resource contention – Which IMS, dependent region, application is holding resources

– Untangle queue of holder/waiters

– Determine resources with contention

201449

SM Solution - Real-time IRLM / PI Locking Display

�Display of resources with contention– Select by dependent region and drill down to resource

– Select by resource and drill down to dependent region

201450

Real-time IRLM Locking Display

201451

Real-time IRLM Locking Display

201452

Dependent Region Display

201453

201454

–IRLM statistics– VSO stats

IRLM Statistics – The challenges

�Managing the well being of IRLM(s) – Deadlocks, false contentions, storage utilization?

– Multiple IRLMs to check

201455

SM Solution - Aggregated IRLM Statistics

Information gathered from IRLMs across Sysplex– Aggregated into single system image

– Drill down for information from individual IRLMs

201456

Aggregated IRLM Statistics

201457

201458

– IRLM statistics

–VSO stats

VSO cache hit stats

201459

VSO cache hit stats

201460

201461

– IRLM statistics

– VSO stats

� Shared Resources

–RM structure entries’ status, ownership, and removal

IMSplex Resource Management

201462 Source: IMS Redbook (SG24-6908-00)

IMSplex Resource Management

201463

IMS Resource Structure Content

201464

Sysplex Serialized Program Management – The Challenges

� The IMS who scheduled the serial program abnormally terminated and not

coming back for a while, the scheduled serial program is locked and no

other IMS in the Sysplex can use it.

201465

SM Solution - Sysplex Serialized Program Management

�Delete the locked RM entry to allow the serial program to be

scheduled by other IMS via ISPF

201466

201467

201468

201469

201470

201471

Summary

Challenges Sysplex Manager

Transaction Affinity management Seamless, persistent, dynamic update of

transaction affinity

CF Structures monitoring All IMS CF info in one place including VSO

Current utilization and checkpoint stats

Log streams offload monitoring Alert of too many offloads

Shared Queues Management Queue depths for destinations and message details

201472

Long Lock Detection Real time WTO alerts

Lock Contention report Untangle Holder/Waiter queue for locks

Dependent Region activities

IRLM Statistics report Deadlock and false contention stats

VSO stats Buffer hit stats for shared and non-shared VSO

RM structure management Check ownership and removal of troubled entries

201473

Other Product Use Scenarios

201474

Scenarios

� IMS– Scenario 1 – Taking Inventory and capture diagnostic data– Scenario 2 – Managing IMS System Parameters– Scenario 3 – Verifying IMS Resource Definitions– Scenario 4 – Issuing IMS Commands– Scenario 5 – Maintaining Command Audit Trail– Scenario 6 – Managing Dependent Regions– Scenario 7 – Viewing IMS CF Structures

� IMSplex

201475

� IMSplex– Scenario 9 – Viewing Aggregate SCI, RM, OM Statistics

� Shared Queue and CQS– Scenario 10 - View local buffer utilization– Scenario 11 – Managing Destination Queue Depth– Scenario 12 – Viewing CQS Structures

� Dashboard– Scenario 13 – Dashboard and Thresholds

Scenario 1 – Taking Inventory and Capture Diagnostics

�Many address spaces – IMS Control Region, IMS DLI/SAS, IMS DBRC, IRLM, CQS, RM, OM, SCI, etc..

– How do you identify related IMS components across the Sysplex?– What is the status of these components?– What version of IMS components are involved?– How much resource are they using from z/OS perspective?– How do you collect diagnostic data to debug sysplex problem?

201476

� IMS Sysplex Manger structured TSO/ISPF interface – Guided display of IMS components– Provides component id, task or job name, version, status and basic z/OS

information such as CPU time and EXCP counts– Drill-down to detailed component information– Easily capture console dumps for IMS components across the plex– Check DBRC RECON datasets placement and VSAM stats

Component List

201477

Component List (cont)

201478

Capture Console Dumps

201479

Capture IMS CF structures Dump

201480

z/OS perspective for IMS address spaces

201481

DBRC RECON datasets and VSAM stats

� Show RECON datasets placement (COPY1, COPY2, SPARE) for all

IMS systems

� Built-in LISTCAT for RECONs when drill down

201482

DBRC RECON VSAM stats

201483

Scenario 2 – Managing IMS System Parameters

�Many system run-time parameters – Sources: DFSPBxxx, overrides via Control Region PARM=

– Which ones are being used?

– Are the parameters the same across the Sysplex?

�System parameter display

201484

�System parameter display – Real-time scrollable display of “resolved” values

– Parameter values across all IMS systems for easy comparison

– New – System Parameter Tutor for instant description

IMS System Parameters

201485

IMS System Parameters

201486

IMS System Parameters – Showing Unequal Parms

201487

IMS System Parameters – Tutor

201488

Scenario 3 – Verifying IMS Resource Definitions

� Resource definitions – Transactions, Programs, Data Bases, Nodes, LTERMS, etc. – Are the definitions the same across the Sysplex?– What is the resource status across the Sysplex?– How do you alter the status across the Sysplex?

� Resource definition display – Query resource by status or attribute– Real-time scrollable display of resources

201489

– Real-time scrollable display of resources– Resource attributes and status across all IMS systems for easy comparison– Integrated IMS Type 1 / Type 2 command interface to alter resource status– Ability to drill-down to related resource (from transaction to PSB, from PSB to

databases… )

IMS Resource Definitions – Filter Specification

201490

IMS Resource Definitions – Drill-down to PSB

201491

IMS Resource Definitions – Drill-down to Database

201492

IMS Resource Definitions – Databases for a PSB

201493

IMS Resource Definitions – Manage Transactions

201494

IMS Resource Definitions – Manage Transactions

201495

Scenario 4 – Issuing IMS Commands

� Issue Type 1 command to all IMS systems – Display same resource type across Sysplex

– Alter resource status across Sysplex

�Command issued to each IMS system– Output recorded to command file and logged history file

201496

– Output recorded to command file and logged history file

– ISPF Browse(view) used to display result

– Scrollable, Primary and line commands (find, exclude, etc.)

– Retrievable list of previously entered commands

IMS Commands

201497

IMS Commands – Type-1

201498

IMS Commands – Type-2

201499

Scenario 5 – Maintaining command Audit Trail

�Keeping track of operations of your IMS systems– Need for audit trail? – Easy way of viewing IMS commands & system messages?– Automatic archive of audit log?

�Sysplex Manager with its history datasets– Log most type-1 IMS commands, command responses and MTO messages

from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI

2014100

from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI programs using CMD or ICMD interface)

– Log OM processed command and output– Searchable, filterable online viewing of log data using ISPF dialog– Automatic archive for full history dataset– Archive Directory to assist locating archived data– Issue type-1 and type-2 IMS commands while viewing log data (similar to z/OS

SDSF)– Check SM History datasets status

Command Audit Trail

2014101

Command Audit Trail

2014102

SM History database status and usage

� Up to 8 VSAM Linear datasets make up SM history database

� Used in a circular fashion similar to IMS OLDS

� Automatic archive process to save full datasets to user-defined output location

� Directory to help users keep track of archive output

� Real-time information on status of these history datasets

2014103

View and retrieve archive datasets

� Locate which archive dataset has the info you need

� Browse the archive dataset instantly

2014104

Scenario 6 – Managing Dependent Regions

�Transaction workload back-logged– What dependent regions are available?

– What transaction classes do they handle?

– Are they occupied with work?

– What are they doing?

2014105

�Scrollable list of all dependent regions across Sysplex– Enhanced IMS /DISPLAY ACTIVE REG

– Displays overall DL/I DB/TM call counts

– Sort the lock held column to see which reg holds the most locks

– Resequenced by primary or secondary classes

– Drill down to lower level of detail

2014106

2014107

2014108

2014109

2014110

2014111

Scenarios

� IMS– Scenario 1 – Taking Inventory and capture diagnostic data

– Scenario 2 – Managing IMS System Parameters

– Scenario 3 – Verifying IMS Resource Definitions

– Scenario 4– Issuing IMS Commands

– Scenario 5 – Maintaining Command Audit Trail

– Scenario 6 – Managing Dependent Regions

– Scenario 7 – Viewing IMS CF Structures

� IMSplex

2014112

� Shared Queue and CQS– Scenario 9 - View local buffer utilization

– Scenario 10 – Managing Destination Queue Depth

– Scenario 11 – Viewing CQS Structures

Scenario 8 – Aggregated SCI, RM, OM Statistics

�Managing the well being of Common Service Layer (CSL) address

spaces– Lack of tools to obtain CSL statistical information

– Multiple instances to check

� Information gathered from CSL address spaces across Sysplex

2014113

� Information gathered from CSL address spaces across Sysplex– Aggregated into single system image

– Drill down for information from individual address space

Aggregated RM Statistics

2014114

2014115

2014116

Scenarios

� IMSplex

2014117

Scenario 9 - SQ local buffer utilization

2014118

Scenarios

� IMSplex

2014119

Scenario 12 – Dashboard Sysplex at a Glance

�Managing the IMS Sysplex through key system indicators – Determine overall Sysplex health at a Glance

– Alert when indicators violate predefined thresholds

– Set multiple monitoring thresholds depending on workloads, time of day…

�Critical data from key areas summarized on one or more screens

2014120

�Critical data from key areas summarized on one or more screens– User customized content, positioning, threshold value

– Threshold exceptions are logged for future review

– One or more per user, default supplied

– Automatic screen refresh capability for operational ease

– Drill down for additional detail

Configure Your Dashboard

� Create and customize a new dashboard

2014121

When to monitor

� Define monitoring periods

2014122

Low and High Thresholds

� Define threshold values

2014123

Threshold Processing Interval

� Set dashboard data collection interval

2014124

Dashboard

2014125

Dashboard – Drill Down

2014126

References

�For publications and demo, visit IMS Sysplex Manager for z/OS at– http://www-306.ibm.com/software/data/db2imstools/imstools-library.html

2014127

Thank You!

Your Feedback is Important to Us

2014128

Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

Technology

Transcript of Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

IMS Parallel Sysplex Best Practices - IMS UG May 2013 Oslo

Sysplex in a Nutshell

Parallel Sysplex Operational Scenarios - IBM … Defining SDSF as a new ARM element ... 11.5.3 XDC command ... MACHINES CORPORATION . IBM z/OS Parallel Sysplex Operational Scenarios

IMS Parallel Sysplex Best Practices

Sysplex Networking Technology Overview - SHARE · Sysplex Networking Technology Overview Thursday, August 5, ... PPRC Application A CEC-1 ... capability to coordinate the assignment

Boston, Massachusetts Parallel Sysplex InfiniBand Aetna…€¦ · © 2010 IBM Corporation Boston, Massachusetts Parallel Sysplex InfiniBand & Aetna’s Implementation Experience

zPDT 2016 Sysplex Extensions - IBM · PDF file6.1 CICSplex overview ... zPDT® ® ® Frank Kyne Bill Ogden. zPDT 2016 Sysplex Extensions. zPDT 2016 Sysplex Extensions

IBM z/OS V2R2: Sysplex · 2015-12-16 · x IBM z/OS V2R2: Sysplex Mark Brooks (z/OS Sysplex Design and Development, Poughkeepsie Center) for offering valuable advice and guidance

Parallel Sysplex Overview: Introducing Data Sharing and ...largescalecomputing.wikispaces.com/file/view/IBM+sysplex+overview.… · Introducing Data Sharing and ParallelisminaSysplex

parallel sysplex resiliiency - Confex...A Sysplex Failure Management (SFM) policy that implements best practices is a critical component of a resilient sysplex A good SFM policy enables

s8326 Sysplex and network topology - the Conference Exchange · review various Sysplex access network topology examples and discuss aspects of each. In particular the session will

Presented by Jim McNeill Vanguard Professional Services · CICSPlex DB2 eServer IBM IBM z IBM z Systems IBM z13 IMS MQSeries MVS NetView OS/390 Parallel Sysplex RACF RMF S/390 System

2Q2008 System z High Availability – Parallel Sysplex TGVL: System z Foundation 1 System z High Availability – Value of Parallel Sysplex IBM System z z10.

2011 Sysplex Failure Management October - SHARE Performance is in Internal Throughput Rate ... •SFM - Sysplex Failure Management Terminology •Each system has an FDI - …

Parallel Sysplex Clustering Comparison - IBM

Sysplex and Network Technologies and Considerations ... · also include best practices recommendations for intra-sysplex connectivity and external connectivity into the sysplex environment,

Sect-10826 Leveraging DataPower XI50z and Sysplex...SECURE your SOA, Web 2.0, B2B, and Cloud environments GOVERN your evolving IT architecture 6 IBM Confidential IMS Integration (1)

Parallel Sysplex Resiliency August 2011 · 2011-08-12 · Parallel Sysplex Resiliency Mark Brooks – Parallel Sysplex Development ... user will experience will vary depending upon

Parallel Sysplex Overview: Introducing Data Sharing and

Exploiting Parallel Sysplex: A Real Customer Perspective · Adabas data sharing guideline ... vi Exploiting Parallel Sysplex: A Real Customer ... 4 Exploiting Parallel Sysplex: A