DevOps: Automate Db2 Schema Management
Transcript of DevOps: Automate Db2 Schema Management
DevOps: Automate Db2 Schema ManagementTIM WELTZER – SOFTWARE ENGINEER – [email protected]
JAE CHUNG – SOFTWARE ENGINEER – [email protected]
1.19.2020
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Disclaimer
Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of June 2020 and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.
Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any contract.
Copyright © 2020 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies.
THIS PRESENTATION IS FOR YOUR INFORMATIONAL PURPOSES ONLY. CA assumes no responsibility for the accuracy or completeness of the information. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENT “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. In no event will CA be liable for any loss or damage, direct or indirect, in connection with this presentation, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised in advance of the possibility of suchdamages.
2
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Db2 DevOps Using JCL Scripting
3
Problem:
• Automate the management of Db2 object changes across multiple Db2 environments including
development, quality assurance, and production subsystems. Enable a self-service capability
for developers (Dev) and minimize the need for DBA (Ops) involvement.
Solution SO13807 is a DevOps enhancement to CA RC/Migrator and CA RC/Compare that helps
to guide you in the building of your own DevOps pipeline to address and automate the needs of
development and operations. The enhancement includes:
• A set of JCL templates
• A sample DevOps JCL workflow using sample JCL and PROCs derived from the templates
• Documentation that describes using JCL and/or open source DevOps tools such as Zowe CLI
and Jenkins to help you build your own workflow.
The set of JCL templates can be used for two different DevOps scripting contexts: JCL only and
open source tooling.
This presentation is focused on a JCL only scripting context.
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
JCL Creation Templates
4
Four templates are used as building blocks to create a DevOps workflow. The templates can be
used as input to an Apache Velocity compatible template engine. They can also be used as
models to build a pure JCL pipeline. The table below provides a summary of each template.
Template and Purpose CA Product Usage Input Output
RMATM001
Generate DDL
CA RC/Migrator is used to
generate DDL for selected
objects
-Source SSID
-Db2 object selections
-Options
-DDL file
RMATM002
Syntax Check DDL
CA RC/Migrator Syntax
Checker
-DDL File
-target SSID (for object verification)
-Validated DDL file
-Report of invalid DDL
RMATM003
Create an SSID update script
Three step process:
• Create strategy
• Analyze strategy
• Generate Incremental
Change Language (ICL)
CA RC/Compare is used to:
• Create a strategy
• Analyze a strategy
• Generate ICL
• For create strategy step
-DDL, -Target SSID, -Options
• For analyze strategy step
-Strategy, -Target SSID, -Options
• For generate ICL step
-Strategy, -Target SSID, -Options
• For create strategy step
-Compare strategy
• For analyze strategy step
-CA Batch Processor script
• For generate ICL step
-ICL script used as a report
RMATM004
Execute an update script
CA Batch Processor (BP) -Target SSID
-BP script
- The target SSID is updated
according to script
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Template RMATM001 – Generate DDL for Selected Objects
5
JCL variables ‘&’
Template engine variables ‘$’
Quick Migrate Db2 object selection
Invoke RC/Migrator
Template engine statements ‘#’
Generate DDL option
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.6
DEV QA PROD
Example DevOps Workflow Based on Templates
Changes and movement of Db2 Objects are managed in DDL format.
1. Provision DEV with Db2 objects extracted from PRD – uses RMATM001, RMATM003, and RMATM004
2. Iterate: edit, update, and test Db2 object changes – uses RMATM002, RMATM003, and RMATM004
3. Promote Db2 object changes from DEV to QA – uses RMATM001, RMATM003, and RMATM004
4. Perform QA testing
5. Promote Db2 object changes from QA to PROD - Uses RMATM001, RMATM003, and RMATM004
1
53
2
4
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Sample DevOps PROC and JCL to Promote Db2 Object Changes
7
Sample DevOps PROC and JCL are provided as an example pipeline which promotes Db2 object
changes from a source environment to a target environment. The table below provides a brief
description of the PROC and JCLs.
JCL/PROC CA Product Used Input Output
RMAPROMP - PROC
Multi-step process to:
• Generate DDL
• Create a strategy
• Analyze a strategy
• Generate ICL
• Send email (via SMTP)
• CA RC/Migrator
• CA RC/Compare
• source SSID
• target SSID
• submitter email address
• Db2 object selection
• BP update script
• ICL report
• Email notifications
RMAPROMJ - JCL
Invoke RMAPROMP PROC
• N/A • source SSID
• target SSID
• submitter email address
• Db2 object selection
• BP update script
• ICL report
• Email notifications
RMAPROMX - JCL
Execute a BP Script
• CA Batch Processor • submitter email address
• LOGSSID
• BP script
• Target SSID is updated
with changes
• Recovery BP script
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
One-time configuration parameters
Values supplied by RMAPROMJ
SYSUT1 overridden by RMAPROMJ to specify Db2 Objects
RMAPROMJ JCL Invokes RMAPROMP PROC
8
RMAPROMX is an instream
PROC to execute the update
script produced by RMAPROMJ
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
The Anatomy of the RMAPROMP Procedure
9
RMAPROMP is a sample PROC that shows how to shield a developer from implementation
details while minimizing the inputs needed to promote Db2 object changes from one environment
to another. The steps contained inside RMAPROMP are:
1. .
2. If copy failed then email submitter and exit
3. Generate DDL for selected objects
4. Delete analysis data sets created from a prior execution
5. If DDL generation failed then email submitter and exit
6. Create compare strategy between DDL and subsystem
7. If strategy creation failed then email submitter and exit
8. Analyze compare strategy
9. If analyze failed then email submitter and exit
10. Generate Incremental Change Language (ICL) for reporting
11. Email submitter with success notification
12. Why are you looking under here? Congratulations! You have found an Easter egg!
Input:
object selection
source ssid, target ssid, submitter email
Output:
update script
ICL report
One-time configuration values:
Db2 data sets DSNLOAD and DSNEXIT
HLQ of CA products
Product options
Site domain
Approver email address
Copy object selection to a temporary file
Generate DDL for selected objects
Create compare strategy between DDL and subsystem
Generate Incremental Change Language (ICL) report
Email request-for-approval to approver with attached details: update script and ICL report
Analyze compare strategy to create update script
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
DevOps Use Case: Promote Db2 Changes to Production
10
The diagram below depicts changes made to a Db2 application’s object hierarchy. It includes object additions
and changes to existing objects.
Developer Initiates Db2 Object Promotion
Execute RMAPROMJ to create an update script specifying:
• SRCSSID=QA ssid
• TRGSSID=PROD ssid
• Object selection specification: DATABASE CHUJAA5 DBA
RMAPROMJ invokes RMAPROMP which is configured to use the
QUICKME keyword to include all dependent objects.
Administrator Completes Db2 Object Promotion
Execute RMAPROMX to execute an update script specifying:
• [email protected] (to notify developer)
• LOGSSID=PROD ssid
• INPRPRT=update script generated by RMAPROMJ
The result is emailed to the administrator and developer.
The administrator also receives a recovery script which can be
executed to undo the changes if needed.
Execution of Two Jobs Will Promote the
Changes from QA to Production
+TS1B
+TG1B
+TB1B
IX1AAdd column
DBA
TB1AAdd column
Add constraint
SGA
+FK1B1A
+SQ
TS1A
Data Structure Changes
• Solid lines are existing
objects on QA and
production subsystems
• Dashed lines are new
objects on QA
subsystem
• Red text are attributes
added to existing
production objects and
are only on the QA
subsystem
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
RMAPROMJ Creates Update Script and Emails an Approver
11
+TS1B
+TG1B
+TB1B
IX1AAdd column
DBA
TB1AAdd column
Add constraint
SGA
+FK1B1A
+SQ
TS1A
RMAPROMJ generates an update script.
This snippet shows some of the changes
to be made. Utility calls are not shown.
The DATABASE CHUJAA5 DBA
is the only object that is specified
in RMAPROMJ because the
invoked RMAPROMP PROC is
configured to include all
dependent objects.
RMAPROMJ
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Results of RMAPROMJ Emailed to Approver for Approval
12
Snippets of ICL_REPORT.TXTApproval request emailed to approverThis pause in the flow allows a DBA to
approve or reject the changes.
These parameters are used when
submitting RMAPROMX to deploy the
changes by executing the update script
in z/OS data set WELTI01.ANALYSIS.
The ICL report provides a terse report of the
Db2 object changes being made compared
to the update script.
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Results of RMAPROMX Emailed to Submitter and Approver
13
Results emailed to submitter and approver
Recovery Script and execution output
Also saved in a z/OS data set
Snippet of recovery script
Snippet of execution output
RMAPROMX
Objects
successfully
promoted!
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Other Things You can Include in a DevOps Pipeline Using JCL
14
A DevOps pipeline is not constrained to only Db2 schema management. Some ideas for
other processes (JCL steps) to consider adding to your pipeline include:
• Provisioning environment(s) with test data
• Provisioning environment(s) with production Db2 statistics to tune SQL queries using Explain
• Maintaining a Db2 application in sync with Db2 object changes
• Recording and Reporting Db2 Explain results
• Comparing Explain results between development and production
• Backing-up and building a changed Db2 application i.e. pre-compiling, compiling, linking, Db2
binding
• Enforcing standards
• Tracking DevOps activity
• Archiving Db2 DDL, application source, build processes, etc.
• Etc.
Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.15
Automate Db2 Schema Management
Live demonstration of an open source tools approach to Db2 schema automation using python, Zowe, Git,
and Jenkins.