Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

IMS in a Sysplex Environment: Challenges and Solutions

© 2014 IBM Corporation

Nick Griffin

World Wide IMS Software Group Sales Advisor

IBM Silicon Valley Lab

Agenda

� Product Highlights

� IMS Sysplex Review

� IMS Sysplex Challenges and Solutions– Shared Queues

– Data Sharing

– RM Resources


20141

� Q & A

IMS Sysplex Manager Highlights

– Real-time management of the IMS Sysplex Environment• Single system image thru local and aggregate view of data

• Simplified User Interface (TSO/ISPF)

• Structured displays of IMS resources and CF structures

• Global Type-1 command, OM Type-2 and IMS SPOC

• Basic z/OS performance information and SVC dump capture

• Statistics for CSL (OM, RM and SCI), IRLM, VSO, and CQS

• Dashboard with key system indicators and threshold monitoring

• Intercept System exceptions and generate Console alerts


20142

• Intercept System exceptions and generate Console alerts

• Produce real-time IRLM Long Lock Report

• Browse, delete and recover messages on Shared Queues

• Delete RM resource structure entries

• Assign affinity for transactions in Shared Queues environment

• Protect against buffer overflow in Shared Queues environment

– Support IMS DB/TM, DBCTL, and DCCTL for IMS v11 and later

MVS1MVS4MVS3MVS2

CQS1 CQS2

SCI1 RM1 OM1 SCI2 RM2 OM2

IMS Sysplex Manager Sample Configuration

IMSSM DC

IMSSM DC

IMS1 IMS2

IMS Connect IMS Connect

FDR1


20143

IMSSM UIS

IMSSM UIS

FDR1 FDR2

IMS Sysplex Review


20144

Source: IMS Redbook (SG24-6908-00)

IMS Sysplex Review - Components

V13V13


20145

Source: IMS RedbookSource: IMS Redbook (SG24-6908-00)

IRLM

IRLM Lock

V13

IMSplex Configuration


20146


IMS Sysplex Challenges and Solutions

� Shared Queues– Transaction affinity

– CF structures and log streams

– Queue monitoring and management

– Full Queue Protection

� Shared Databases– Long lock detection


20147

– Long lock detection

– Investigate lock holders and waiters

– IRLM statistics

– VSO stats

� Shared Resources– RM structure content, status, ownership, and removal


� Shared Queues

–Transaction affinity– CF structures and log streams





20148



– IRLM statistics

– VSO stats


Transaction affinity – The Challenges

�Force local affinity for efficiency

�Assign affinity to specific IMS due to resource availability

�Reduce false-scheduling overhead

�Reduce contention


20149

�Reduce contention

Sysplex Manager Tool - Transaction Affinity Highlights

�Finer control of transaction scheduling

�Non-invasive to existing definition and operation– No omission of transaction definitions in sysgen– No stopping of transactions– No re-classing of dependent regions – No operational impact for loss of a system


201410

�User defined affinity to route transaction messages– Any IMS in the shared queues group– Any subset of IMS systems– Equal or weighted distribution

Affinity Routing Statistics – System View


201411

Affinity Routing Statistics – Update System


201412

Affinity Routing Statistics – Destination View


201413

Affinity Routing Statistics – Update Destination


201414

Affinity Routing Statistics – Add Destination


201415



–CF structures and log streams– Queue monitoring and management




201416



– IRLM statistics

– VSO stats


IMS Sysplex Review – IMS CF structures and log streams

Log streams

(CQS, OM,


201417


(CQS, OM, RS)

IMS CF Structures and log streams – The Challenges

�Growing use of Coupling Facility Structures – Data Sharing, Shared Message Queues, Resource Manager, CQS log streams

– No single source for list of in use structures and details

�Check log stream attributes


201418

�Monitor offload activities

SM Solution - IMS CF Structures and log streams

�Real-time display of structure list

�Statistics, Connections, Coupling Facility information

�Structure rebuild and checkpoint stats


201419

�Log streams attributes and offload monitoring

IMS Coupling Facility Structures


201420

IMS Coupling Facility Structures


201421

Shared VSO structures


201422

Shared VSO structures


201423

CQS Structures


201424

CQS Structures


201425

CQS Structures


201426

CQS and System Logger


201427


SM Solution - CQS Log Stream info


201428

SM Dashboard


201429




–Queue monitoring and management– Full Queue Protection



201430



– IRLM statistics

– VSO stats


IMS Shared Queues


201431 Source: IMS Redbook (SG24-6908-00)

Shared Queue Management – The Challenges

�Visualize queue content

�Browse, Delete, and Recover messages

�View CQS processing counters


201432

SM Solution - Shared Queue Management

�Filter based on Destination name or ID

�High level queue depth information, Drill down to levels of detail

�Browse, Delete and Recover messages from Cold queue


201433

�CQS processing counters

Destination Queue Depth


201434



201435



201436



201437



201438





–Full Queue Protection



201439



– IRLM statistics

– VSO stats


Full Queue Protection Challenges

�Local Queue– Local buffer overflow can abend or stall IMS control region

�Shared queue structures– Full structures will stall IMS


201440

SM Solution - Full Queue Protection

�Local Queue– Real-time local queue buffer counts and stats

– Reject new messages by employing the Queue Space Notification Exit

(DFSQSSP0)

– Automatic actions for warning, critical or Severe thresholds

– Identify programs using large number of buffers


201441

�Shared queue structures– Alert structure utilization exceeding predefined thresholds

– Unable to reject new messages (a possible future enhancement)






� Shared Databases

–Long lock detection


201442

–Long lock detection– Investigate lock holders and waiters

– IRLM statistics

– VSO stats


Data Sharing Components


201443


Diagnosing Long Locks - The Challenges

� DB Lockouts by applications holding IRLM locks for an inordinate amount of

time

– Could go unrecognized until it becomes critical

– Lack of supported tools to assist in recognition and identification of problem

– Manual intervention required to resolve


201444

SM Solution - Data Sharing Long Locks

�Automatic real-time recognition when IRLM detects

� Information consolidated, analyzed for top blocker, and presented

� Information recorded in exceptions file and sent to z/OS console

�Messages can be sent to z/OS console using user exit so that

automated operations can resolve


201445

automated operations can resolve

�Problem quickly resolved with automation

Data Sharing Long Lock Exceptions


201446

Data Sharing Long Lock Exceptions


201447








201448


–Investigate lock holders and waiters– IRLM statistics

– VSO stats


Real-time IRLM / PI Locking Display – The Challenges

� Identify resource contention – Which IMS, dependent region, application is holding resources

– Untangle queue of holder/waiters

– Determine resources with contention


201449

SM Solution - Real-time IRLM / PI Locking Display

�Display of resources with contention– Select by dependent region and drill down to resource

– Select by resource and drill down to dependent region


201450

Real-time IRLM Locking Display


201451

Real-time IRLM Locking Display


201452

Dependent Region Display


201453








201454



–IRLM statistics– VSO stats


IRLM Statistics – The challenges

�Managing the well being of IRLM(s) – Deadlocks, false contentions, storage utilization?

– Multiple IRLMs to check


201455

SM Solution - Aggregated IRLM Statistics

Information gathered from IRLMs across Sysplex– Aggregated into single system image

– Drill down for information from individual IRLMs


201456

Aggregated IRLM Statistics


201457








201458



– IRLM statistics

–VSO stats


VSO cache hit stats


201459

VSO cache hit stats


201460








201461



– IRLM statistics

– VSO stats

� Shared Resources

–RM structure entries’ status, ownership, and removal

IMSplex Resource Management


201462 Source: IMS Redbook (SG24-6908-00)

IMSplex Resource Management


201463


IMS Resource Structure Content


201464

Sysplex Serialized Program Management – The Challenges

� The IMS who scheduled the serial program abnormally terminated and not

coming back for a while, the scheduled serial program is locked and no

other IMS in the Sysplex can use it.


201465

SM Solution - Sysplex Serialized Program Management

�Delete the locked RM entry to allow the serial program to be

scheduled by other IMS via ISPF


201466



201467



201468



201469



201470



201471

Summary

Challenges Sysplex Manager

Transaction Affinity management Seamless, persistent, dynamic update of

transaction affinity

CF Structures monitoring All IMS CF info in one place including VSO

Current utilization and checkpoint stats

Log streams offload monitoring Alert of too many offloads

Shared Queues Management Queue depths for destinations and message details


201472

Long Lock Detection Real time WTO alerts

Lock Contention report Untangle Holder/Waiter queue for locks

Dependent Region activities

IRLM Statistics report Deadlock and false contention stats

VSO stats Buffer hit stats for shared and non-shared VSO

RM structure management Check ownership and removal of troubled entries

Q & A


201473

Other Product Use Scenarios


201474

Scenarios

� IMS– Scenario 1 – Taking Inventory and capture diagnostic data– Scenario 2 – Managing IMS System Parameters– Scenario 3 – Verifying IMS Resource Definitions– Scenario 4 – Issuing IMS Commands– Scenario 5 – Maintaining Command Audit Trail– Scenario 6 – Managing Dependent Regions– Scenario 7 – Viewing IMS CF Structures

� IMSplex


201475

� IMSplex– Scenario 9 – Viewing Aggregate SCI, RM, OM Statistics

� Shared Queue and CQS– Scenario 10 - View local buffer utilization– Scenario 11 – Managing Destination Queue Depth– Scenario 12 – Viewing CQS Structures

� Dashboard– Scenario 13 – Dashboard and Thresholds

Scenario 1 – Taking Inventory and Capture Diagnostics

�Many address spaces – IMS Control Region, IMS DLI/SAS, IMS DBRC, IRLM, CQS, RM, OM, SCI, etc..

– How do you identify related IMS components across the Sysplex?– What is the status of these components?– What version of IMS components are involved?– How much resource are they using from z/OS perspective?– How do you collect diagnostic data to debug sysplex problem?


201476

� IMS Sysplex Manger structured TSO/ISPF interface – Guided display of IMS components– Provides component id, task or job name, version, status and basic z/OS

information such as CPU time and EXCP counts– Drill-down to detailed component information– Easily capture console dumps for IMS components across the plex– Check DBRC RECON datasets placement and VSAM stats

Component List


201477

Component List (cont)


201478

Capture Console Dumps


201479

Capture IMS CF structures Dump


201480

z/OS perspective for IMS address spaces


201481

DBRC RECON datasets and VSAM stats

� Show RECON datasets placement (COPY1, COPY2, SPARE) for all

IMS systems

� Built-in LISTCAT for RECONs when drill down


201482

DBRC RECON VSAM stats


201483

Scenario 2 – Managing IMS System Parameters

�Many system run-time parameters – Sources: DFSPBxxx, overrides via Control Region PARM=

– Which ones are being used?

– Are the parameters the same across the Sysplex?

�System parameter display


201484

�System parameter display – Real-time scrollable display of “resolved” values

– Parameter values across all IMS systems for easy comparison

– New – System Parameter Tutor for instant description

IMS System Parameters


201485

IMS System Parameters


201486

IMS System Parameters – Showing Unequal Parms


201487

IMS System Parameters – Tutor


201488

Scenario 3 – Verifying IMS Resource Definitions

� Resource definitions – Transactions, Programs, Data Bases, Nodes, LTERMS, etc. – Are the definitions the same across the Sysplex?– What is the resource status across the Sysplex?– How do you alter the status across the Sysplex?

� Resource definition display – Query resource by status or attribute– Real-time scrollable display of resources


201489

– Real-time scrollable display of resources– Resource attributes and status across all IMS systems for easy comparison– Integrated IMS Type 1 / Type 2 command interface to alter resource status– Ability to drill-down to related resource (from transaction to PSB, from PSB to

databases… )

IMS Resource Definitions – Filter Specification


201490

IMS Resource Definitions – Drill-down to PSB


201491

IMS Resource Definitions – Drill-down to Database


201492

IMS Resource Definitions – Databases for a PSB


201493

IMS Resource Definitions – Manage Transactions


201494

IMS Resource Definitions – Manage Transactions


201495

Scenario 4 – Issuing IMS Commands

� Issue Type 1 command to all IMS systems – Display same resource type across Sysplex

– Alter resource status across Sysplex

�Command issued to each IMS system– Output recorded to command file and logged history file


201496

– Output recorded to command file and logged history file

– ISPF Browse(view) used to display result

– Scrollable, Primary and line commands (find, exclude, etc.)

– Retrievable list of previously entered commands

IMS Commands


201497

IMS Commands – Type-1


201498

IMS Commands – Type-2


201499

Scenario 5 – Maintaining command Audit Trail

�Keeping track of operations of your IMS systems– Need for audit trail? – Easy way of viewing IMS commands & system messages?– Automatic archive of audit log?

�Sysplex Manager with its history datasets– Log most type-1 IMS commands, command responses and MTO messages

from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI


2014100

from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI programs using CMD or ICMD interface)

– Log OM processed command and output– Searchable, filterable online viewing of log data using ISPF dialog– Automatic archive for full history dataset– Archive Directory to assist locating archived data– Issue type-1 and type-2 IMS commands while viewing log data (similar to z/OS

SDSF)– Check SM History datasets status

Command Audit Trail


2014101

Command Audit Trail


2014102

SM History database status and usage

� Up to 8 VSAM Linear datasets make up SM history database

� Used in a circular fashion similar to IMS OLDS

� Automatic archive process to save full datasets to user-defined output location

� Directory to help users keep track of archive output

� Real-time information on status of these history datasets


2014103

View and retrieve archive datasets

� Locate which archive dataset has the info you need

� Browse the archive dataset instantly


2014104

Scenario 6 – Managing Dependent Regions

�Transaction workload back-logged– What dependent regions are available?

– What transaction classes do they handle?

– Are they occupied with work?

– What are they doing?


2014105

�Scrollable list of all dependent regions across Sysplex– Enhanced IMS /DISPLAY ACTIVE REG

– Displays overall DL/I DB/TM call counts

– Sort the lock held column to see which reg holds the most locks

– Resequenced by primary or secondary classes

– Drill down to lower level of detail



2014106



2014107



2014108



2014109



2014110



2014111

Scenarios

� IMS– Scenario 1 – Taking Inventory and capture diagnostic data

– Scenario 2 – Managing IMS System Parameters

– Scenario 3 – Verifying IMS Resource Definitions

– Scenario 4– Issuing IMS Commands

– Scenario 5 – Maintaining Command Audit Trail

– Scenario 6 – Managing Dependent Regions

– Scenario 7 – Viewing IMS CF Structures

� IMSplex


2014112


� Shared Queue and CQS– Scenario 9 - View local buffer utilization

– Scenario 10 – Managing Destination Queue Depth

– Scenario 11 – Viewing CQS Structures


Scenario 8 – Aggregated SCI, RM, OM Statistics

�Managing the well being of Common Service Layer (CSL) address

spaces– Lack of tools to obtain CSL statistical information

– Multiple instances to check

� Information gathered from CSL address spaces across Sysplex


2014113

� Information gathered from CSL address spaces across Sysplex– Aggregated into single system image

– Drill down for information from individual address space

Aggregated RM Statistics


2014114



2014115



2014116

Scenarios








� IMSplex


2014117






Scenario 9 - SQ local buffer utilization


2014118

Scenarios








� IMSplex


2014119






Scenario 12 – Dashboard Sysplex at a Glance

�Managing the IMS Sysplex through key system indicators – Determine overall Sysplex health at a Glance

– Alert when indicators violate predefined thresholds

– Set multiple monitoring thresholds depending on workloads, time of day…

�Critical data from key areas summarized on one or more screens


2014120

�Critical data from key areas summarized on one or more screens– User customized content, positioning, threshold value

– Threshold exceptions are logged for future review

– One or more per user, default supplied

– Automatic screen refresh capability for operational ease

– Drill down for additional detail

Configure Your Dashboard

� Create and customize a new dashboard


2014121

When to monitor

� Define monitoring periods


2014122

Low and High Thresholds

� Define threshold values


2014123

Threshold Processing Interval

� Set dashboard data collection interval


2014124

Dashboard


2014125

Dashboard – Drill Down


2014126

References

�For publications and demo, visit IMS Sysplex Manager for z/OS at– http://www-306.ibm.com/software/data/db2imstools/imstools-library.html


2014127

Thank You!

Your Feedback is Important to Us


2014128

128

Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

Technology

Transcript of Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne