Online reorganization facility

1 © 2011 IBM Corporation

IMS Online Reorganization Facility

Dennis EichelbergerIMS [email protected]

2011 IBM Corporation2

IMS Online Reorganization FacilityAgenda

• IMS ORF Overview

• Database Types supported

• Other required IMS Tools

• Phases


IMS Online Reorganization FacilityOverview

• A one-step reorganization of IMS databases with minimal impact to its availability

• Reduces database downtime from hours to seconds

• Eliminates the need for manual intervention before, during and after a reorganization

• Supports:

• Internal logical relationships

• Secondary indexes

• Full Function Databases

• BMP Pause and Resume


IMS Online Reorganization Facility Why Reorganize a Database?

■ Performance– Fragmentation– Overflow– CI/CA Splits

■ Space– Nearing Limitations– Current Definitions– Reclaim– IMS Limitations

■ DBD Changes


IMS Online Reorganization Facility All Full Function Databases

■ HDAM

■ HIDAM

■ SHISAM

■ HALDB - PHDAM/PHIDAM – Entire database – all partitions– Single partition

■ Partitioned secondary indexes– Entire index

– Single partition

■ Secondary indexes

– When primary DB reorganized

– Index only

IBM IMS Online Reorganization

Facility


IMS Online Reorganization Facility Other Required IMS Tools

• Unload– IMS Basic Unload– IMS High Speed Sequential Retrieval (HSSR/HPUNLOAD)

• Load– IMS Basic Reload

– IMS High Performance Load (HPLOAD)

• Prefix Resolution/Update– IMS Basic Prefix Resolution & IMS Basic Prefix Update

– IMS High Performance Prefix Resolution/Update

• Image Copy– IMS Basic Image Copy

– IMS High Performance Image Copy

Note: Secondary Index tools are not used; indexes are built by ORF

The Offline Reorg Process

Unload

Load

Build Sec Indexes

Image Copy (w/Ptr Check)

Prefix Res/Upd

/DBR DB /STA DB

Data Unavailable

The Online Reorg Process – with shadow data sets

Allocate Shadow data sets

HP Unload

HP Load

Start Change Capture

Build Sec Indexes

Image Copy (w/Ptr Check)

Prefix Res/Upd

/DBR DB /STA DB

Update DBRC & Rename Data Sets

Copy Primary DB

Finish Change Capture

Apply Changes


IMS Online Reorganization Facility Phases

• ORF phases occur in the following sequence:

–Verification

–Copy

–Reorganization

–Apply –Takeover � The only phase allowing restart processing

–Completion


IMS Online Reorganization FacilityData Flow


IMS Online Reorganization Facility Reorganization Process

• ORF can be run while:– IMS system(s) are up

– IMS system(s) are down

– IMS databases are online– IMS databases are offline

• No JCL changes or manual intervention needed for any of the processes above


IMS Online Reorganization Facility VERIFICATION Phase

If Normal Execution (not Restart): • IMS Release and component requirements are verified• Operating system requirements are verified

• DBRC registration/database state requirements are verified

• Database DBD requirements are verified

• Database data set requirements are met (cannot exceed 42 characters in length and are cataloged)

• ORF level for STEPLIB in batch job/IMS control regions is verified

• JCL requirements are met

• Control statement syntax is verified

• Shadow data sets (.S data sets) are constructed/verified/allocated

• Temporary (.T data sets) data sets deleted and deletions verified

• Other miscellaneous checking is performed


IMS Online Reorganization Facility COPY Phase

• BMPs are paused (if usermod is installed) • Databases are closed/reopened using TOSI to flush buffers

across all IMS subsystems and to update the catalog/VTOC• BMPs are resumed (if usermod is installed)• DBRC status remains unchanged - allocated and authorized• Waits for database syncpoint, then online component of ORF

is notified to:� Begins capturing all changes made to the primary database data sets – type 50

records

• Copies the original primary database only to shadow copy• Near the end of the Copy phase, another close/reopen is

done and:� Captured changes are sent to ORF batch utility & applied to shadow primary

database datasets .� The shadow primary index (for HIDAM or PHIDAM) is created from the shadow

primary database� The ILDS (for HALDB) is created for the shadow ILDS


IMS Online Reorganization Facility REORGANIZATION Phase

• Shadow database is Unloaded

• Shadow database is Reloaded

• Shadow Secondary indexes are rebuilt except for HALDBPSINDEX databases

• Shadow internal logical relationships are rebuilt by Prefix Resolution/Update (if any)

• Shadow databases are Image Copied (and optionally Pointer Checked)

• Online component of ORF continues to capture update calls to the original databases and are written to a temporary data set for use in the subsequent Apply phase.


IMS Online Reorganization Facility APPLY Phase

– Captured changes are applied to the shadows

– ORF pauses any BMPs and issues a /DBR FEOV command to stop original databases

– DBRC flag is set to PROHIBIT AUTH

– Apply phase ends after all remaining captured-updates have been applied to shadows

» If heavy activity, the apply Is done in iterations (continues to capture while applying)

– Original database and its indexes are unavailable until after the Takeover phase (swapping the shadow database to the original database)


IMS Online Reorganization Facility TAKEOVER Phase

• TAKEOVER(Y)• Verifies the database and its indexes are unavailable

• Notifies DBRC of the REORG with a timestamp after the /DBR command

• Notifies DBRC of the image copy with a timestamp that is greaterthan the REORG timestamp

• Notifies DBRC of a DB ALLOC that is needed for log data sets • Notifies DBRC of logs that were created during the Apply phase• Swaps the reorganized shadow data sets with the original

database data sets• Performs the ACBGEN and ACBLIB or DMB replacements (if

DBD changes have occurred) and copies the NEWDBD to the current DBDLIB if DBDCOPY(Y) is specified

• Authorizes the databases in DBRC (turns off REORG INTENT and PROHIBIT AUTH flags)

• Starts the databases by using a /START command• Continues to recover the log marked in error• Notifies DBRC of the log with the new timestamp


IMS Online Reorganization Facility TAKEOVER (with DBD changes)

– NEWDBD and ONLINECHANGE keywords are ignored for HALDB

– If NEWDBD(ddname) was specified with ONLINECHANGE(Y) and DBDCOPY(Y), ORF automates the replacement of DBDs and ACBs/DMBs to eliminate manual intervention and increase databaseavailability

» Automatically generates an ACB» Copies changed DMB to ACBLIBA and ACBLIBB» Activates new DMB» Performs a DMB replacement in the staging and active ACBLIBs» Starts database(s) and resets DBRC flags

– If NEWDBD(Y) was specified with ONLINECHANGE(N), ORF does not automate the replacement of DBDs and ACBs/DMBs; manual intervention is required

» ORF does not automatically generate an ACB » Database(s) remain offline and DBRC flags are not reset» Manual intervention is required


IMS Online Reorganization Facility TAKEOVER Restart

If ORF detects that there is a restart pending by reading the data that was saved in the restart data set:

– Takeover can be completed by resubmitting your job with TAKEOVER(YES) and RESTART(AUTO) keywords

– Databases remain in PROHIBIT AUTH and REORG INTENT status until restart is complete

– Databases remain stopped until restart is complete– ORF Takeover Restart utility is invoked

– Automatically restarts from where it stopped in previous Takeover phase

− DBRC flags are reset when TAKEOVER restart has successfully completed

− Databases are started


IMS Online Reorganization Facility TAKEOVER (scheduled)

� Scheduled Takeover phase� Allows the reorganization Takeover process to be scheduled

within a window that you specify � Allows you to manually perform steps after the reorganization

process and before the databases are brought online, if needed.� Use the TAKEOVER.WINDOW(time) keyword � Must specify TAKEOVER(Y) on REORG command� The database is left in DB recovery needed state with PROHIBIT

AUTH and REORG status set in DBRC. � Restart information is saved in a restart data set. � Takeover can be completed by resubmitting your job with

TAKEOVER(YES) and RESTART(AUTO) keywords specified

� Format of Scheduled Time:� (begHH:MM[,endHH:MM[,endaction]])� Format '00:00' to '23:59'. Times are in 24–hour time


IMS Online Reorganization Facility COMPLETION Phase

Final reports and messages written

Shadow data sets deleted, if DELETE(Y)

Data sets are deallocated

Miscellaneous cleanup tasks performed

ORF job completes


IMS Online Reorganization Facility TAKEOVER (delayed)

� Allows you to delay the Takeover phase before the original database datasets are affected

� Allows you to manually perform steps after the reorganization process and before the databases are brought online, if needed

� Original job must have TAKEOVER(DELAY)

� Databases are left in DB recovery needed state with PROHIBIT AUTH and REORGI status set in DBRC

� Restart information is saved in a restart data set

� Takeover can be completed by resubmitting your job with TAKEOVER(YES) and RESTART(AUTO) keywords


IMS Online Reorganization Facility BMP Management

• The Pause/Resume:

– ORF sets a flag in a control block allocated in the CTL address space

– ORF verify on a CHKP call if the flag has been set– The BMP checks during the suspend state if the flag has cleared– ORF uses logger exit to communicate– It cleans up if a failure occurs (the XCF connection fails)– Failure is posted from XCF that the ORF job went away– ORF then clean up– Once the BMP determine the suspend flag has been cleared,

The BMP resumes

Note: DLIBATCH is not Supported by ORF:• If there is a DLI Batch running, IMS ORF will fail – Unable to

obtain authorization from DBRC.• Vice Versa – If ORF is active, DLI batch will fail to initiate as

the database is marked REORGI is active


IMS Online Reorganization Facility Online Processing

IMS • New transaction arrives when the database is DBR’d

– Transaction placed on suspend queue• Exit/Automation to process suspend queue and reissue

transaction• /STA DB will requeue message

CICS

• SCHEDULE PSB request when the database is DBR’d– ORF detects that it has DBR’d the database

• The Thread is put into temporary wait– Typically short lived


IMS Online Reorganization Facility Online Processing

ODBA

• APSB request when DB is DBR’d– ORF detects that it has DBR’d the DB

• Application TCB is put into temporary wait

– Typically short lived


IMS Online Reorganization Facility Installation Considerations

• IBM IMS Generic Exits is required and part of the IMS Tools Base

– Partner Exit (DFSPPUE0)– Logger Exit (DFSFLGX0)

• IBM IMS Tools Online System Interface (TOSI) is required

• IBM IMS BMP Pauser Interface – DFSRRC99

– No JCL Changes

• By aliasing DFSRRC99 for DFSRRC00 in the running reslib

– With JCL Changes

• By Steplib with DFSRRC99 aliased as DFSRRC00 and invoking HRFRRC00 as the region controller in the BMP

• IMS Control Regions (STEPLIB) modification and recycle needed


IMS Online Reorganization Facility Restrictions

• IMS Online Reorganization Facility does not support the following: – Batch jobs running as DLI, DBB, or ULU types during reorganization– HSAM databases – HIDAM databases with compressed root keys – IMS Utility Control Facility (UCF) – Databases and data sets that are not registered to DBRC – External logical relationships – Fast Path databases – Reorganization of multiple HALDB partitions in one step (you can do

this type of reorganization in parallel with separate IMS OnlineReorganization jobs)

– HALDB DBD changes – Reorganization of HALDB Multi-Volume data sets – Uncataloged or corrupted databases


IMS Online Reorganization Facility DBD Changes not allowed

• IMS Online Reorganization Facility does not allow the following DBD changes: – DBD name or type– Access method – Randomizer name– Segment changes

• type• variable to fixed or fixed to variable lengths• length decrease

– Conversion from Full Function to HALDB– Heirarchical structure change– Logical relationships added, modified or deleted– VSAM blocksize– Secondary index

• delete• conversion from or to shared secondary index


IMS Online Reorganization Facility Common Questions/Answers

1. How many times do the databases get stopped and a FEOV is done? They are temporarily paused twice (close and reopen the database(to get the control blocks updated in the catalog/VTOC) and stopped (/DBR’d) once with one force-end-of-volume unless FEOV(N) is specified.

1. What does ORF use to communicate between the ORF batch job and the IMS online control region(s)? ORF uses XCF to communicate between shared IMS systems. Two XCFGROUP names need to be setupduring ORF installation (one for TOSI (if none exists) and one for ORF online capturing).

1. How does ORF handle CICS/ODBA applications when ORF needs to pause/stop the databases? When a CICS or ODBA APSB request detects that ORF needs to momentarily stop or issue a /DBR on a database, the thread in which the APSB request is made is put into a temporary wait state until ORF restarts the database. No special implementation is needed. This feature can be disabled, if desired.

2. Do I need to recycle IMS to apply ORF maintenance?Not always, depends on APAR.


IMS Online Reorganization Facility Common Questions/Answers (continued)

5. Do I need to preallocate the shadow datasets?Depends on the keyword specified:

– You must preallocate the shadows if SHADOWS(E) for Existing was specified

– You do not need to preallocate the shadows if SHADOW(A) was specified. ORF will use the same attributes of the production database datasets.

6. How does the BMP Pauser Interface work during an ORF job with concurrent BMP(s)?

– If an ORF job needs to do a /DBR for a HALDB database, the PSB is automatically paused (no abend) after the next checkpoint and waits for ORF to restart the database and the BMP wait is released.

– If an ORF job needs to do a /DBR for a non-HALDB database, the PSB is automatically stopped (U3303 abend) after the next checkpoint. After the database(s) are restarted, ORF restarts the BMP at the last checkpoint.

– If a BMP attempts to start in the middle of an ORF job, a check is performed to see if a /DBR needs to be done or has not yet completed. If so, the job step TCB waits until ORF restarts the database.


IMS Online Reorganization Facility Common Questions/Answers (continued)

7. Can I run an ORF job even if no online systems are up? Yes, ORF relies on DBRC subsystem records and if none exist, it can run without active IMS system(s).

8. Can I restart a failed ORF job? Yes, if the original job which failed had RESTART(Y) and it failed in the TAKEOVER Phase. Failures in any other phases are not restartable; however, those phases do not affect the production database so no special action is needed.

9. How does ORF prevent any other utilities (reorg/recovery, etc.) from running during an ORF reorganization?ORF sets the REORG INTENT flag in DBRC which prevents any other utilities from running against the databases being reorganized by ORF.


IMS Online Reorganization Facility Benefits

� Reduced database downtime

� One step database reorganization

� Eliminates manual intervention during reorganization

� Automated replacement of DBDs and ACBs

� Improved DBA productivity

� Flexible reorganization schedule

� Integration with IBM IMS High Performance Tools


IMS Online Reorganization Facility

QUESTIONS AND COMMENTS

Online reorganization facility

Documents

Transcript of Online reorganization facility