Wrangling With Oracle Ray Murray Sirius Users Group June 2001.
-
Upload
cassidy-ripley -
Category
Documents
-
view
216 -
download
1
Transcript of Wrangling With Oracle Ray Murray Sirius Users Group June 2001.
Introduction
How can OS/390 Oracle Benefit Wrangler Supplement Model 204 Complement Model 204 Replace Model 204
Tangible Benefits? Organizational Benefits? Intangible Benefits?
Complementing Model 204
How can OS/390 Oracle Complement M204 “Standard” DBMS Interfacing with “Power” User tools
MS Office MS Query Access, Excel VBA Applications
Maintain a subset of Wrangler Data Interfacing with common Web environments
Cold Fusion ...
Supplementing Model 204
Scaleable to Other Environments UNIX (Most Flavors) Linux NT
Enhance Wrangler Integration with other related applications Replicate related Oracle database on Mainframe
True Replication Export from UNIX - Import to OS/390
Replacing Model 204
Can OS/390 Oracle Host the entire Wrangler database? Does living on the mainframe have
advantages not found in the our other Oracle environments?
Database Conversion approach and effort Ability to re-use parts of Application
Software
Questions
Can Wrangler data be ported to Oracle? What are the storage usage differences? What are the performance differences? What is life with OS 390 Oracle like?
Tuning, Administration, Support In house Vendor
What other hurdles might there be?
“Try it out” Approach - II
Re-Attempt to “Port” Database to Oracle Second iteration using lessons learned Database Design Issues
More De-normalization Database Loading Retrievals
Coding Performance
Current Database
Single Major Record type (avg.. 500 bytes) Unique System Generated Primary Key (PK) 35 “OCC 1” mandatory fields 58 Mult Occ fields holding discrete values
Color, Make, Model … 26 Mult Occ fields of “range” values
Cost, Premium, Weight … 137 Optional single occ fields - descriptive info
Current Database
Substantial indexing (68 fields) Two NR fields 41 ORD CHAR
PK is ORD CHAR 11 of 58 Mult Occ discrete fields are ORD
CHAR 25 ORD NUM
19 of the Mult occ range fields are ORD NUM 2/3 Billion records - across 75 files
Oracle Test Database
Minimally “Normalized” Port 2 Tables
Base table (I) contains 152 columns 63 columns are indexed
Child table (O) contains 19 columns replaces many Mult Occ M204 fields 15 columns are indexed
No Reference or Join tables defined Didn’t define columns not used in M204 file sample
Sirius FUNLOAD FSTATS report very useful here First try 7-table port too inefficient
Database Porting Issues
Dates/Time pairs Combined in Oracle
Numeric values Varchar or Number?
Text Concatenated a multiply occ M204 field
Invisible fields Oracle indexing is very flexible, but ..
Database Preparation
Model 204 test database Extracted 3 million production DB records
Funload UAI job ran in 3 minutes Created 3234 Cyl M204 file in 10.1 minutes Sirius LAI job loaded M204 file in 46
minutes Loaded data and built all indexes
Oracle Database Preparation Extraction of data from M204 database
Prepared Sequential files to load “I” and “O” M204 User Language extract job took 67 minutes
FUEL would have been much faster Prepared SQL*Loader Positional scripts M204-Centric approach to preparing data for
Oracle is most straightforward due to power of User Language and Funload/FUEL tools
Added 3234 Cyl dataset to Oracle database VSAM job took 28.5 minutes to format it
Loading Oracle Test Database Loaded “I” table in 27 minutes (direct=true)
3 Million rows Loaded “O” table in 23 minutes(direct=true)
5,449,864 rows Created 63 “I” table indexes in 540 minutes Created 15 “O” table indexes in 66 minutes Total 656 minutes to build this heavily
indexed Oracle test database ‘Best’ numbers from several runs
Space Usage
M204 file Asize=4, Bhighpg=230769, Csize=2300,
Dpgsused=56102 Total: 1788.4 M space used
Oracle “I” table 1228.8 M “O” table 358.4 M “I” indexes 1867.1 M “O” indexes 685.7 M Total: 4140 M space used (231% x M204)
Query Comparisons
Oracle Queries Run via SQL*Plus running under TSO Statistics reported from SET TIMING ON Oracle Expert product guidelines used to tune Oracle
M204 Queries Run in an M204 online test region - large buffer pool Statistics reported from TIME REQUEST Both M204 and Oracle running on same machine
using same type of disk drives Identical query results required
Timing Comparisons
Run time Oracle Elapsed time of 0.02.15.55 means 0 hours,
2 minutes, 15 seconds, 550 milliseconds M204 CNCT (connect time) is reported in
seconds. Above Oracle time would be the same as CNCT=136
CPU time Oracle CPU time is reported in seconds. CPU of
20.25 is twenty and 1/4 second. M204 CPU is reported in milli-seconds. 20.25
Oracle seconds is the same as CPU=20250
Query Scenario 1
How many claims were filed having collision damage (CD) between 9000 and
9150 dollars? liability damage (LD) between 3000 and
4000 dollars? injury amount (INJ) between 200 and 300
dollars?
Query 1 - M204
User Language:
BEGINCLAIMS.FD: IN CLAIMS FPC
RECTYPE = CLAIMCD IS IN RANGE FROM 9000 TO 9150LD IS IN RANGE FROM 3000 TO 4000INJ IS IN RANGE FROM 200 TO 300
END
Answer: 9155CNCT=4, CPU=922, DKRD=381, RQTM=1681
Query 1 - Oracle
SQL*Plus (O table only, no join)Select Count(*) from O where CD between 9000 and 9150 and LD between 3000 and 4000 and INJ between 200 and 300;
Answer: 11004Elapsed=0:00:03:58, CPU=2.14 remember: Oracle reported CPU=2.14 is
like M204 reported CPU of 2140
Query 1 - completed
Join added to match M204 answer Select count(distinct I.PK) from I, O where
CD between 9000 and 9150 andLD between 3000 and 4000 andINJ between 200 and 300 and
I.PK = O.PK; Answer: 9155 (matches M204)
Elapsed=0.00:42:02 and CPU=5.65
Query 1 - narrowed ranges
LD between 9100 and 9110CD between 3500 and 3600INJ between 200 and 300
Answer: 38 from both systems Oracle Elapsed=0:00:00.53 CPU=.25
secs M204 CNCT=1, RQTM=416, CPU=340
ms
Query 1 - Widened Ranges
LD between 5000 and 12000CD between 2000 and 5000INJ between 200 and 300
Answer: 215959 “I” rows Oracle
Elapsed 0:07:17.06 Cpu 146.20 M204
Cnct=30, Cpu=16446, Dkrd=4754, Rqtm=25252
Query 2 - Simple Count
Oracle - Answer: 2,999,995 Select count(*) from I; Elapsed 0.02:15.55 CPU 20.25
M204 Begin
FPCEnd
CNCT=2, DKRD=127, RQTM=411, DKPR=251
Query 3 - Range Query
Oracle - Answer: 2,410,583 Select count(*) where mydate
between 970705 and 970725; Elapsed: 0.00.26.02 CPU 8.82
M204 Begin
FPC mydate is in range from 970705 to 970725end
M204 - cnct=1, cpu=25, dkpr=816, rqtm=543
Query 4 - Multiple criteria
Oracle Select count(*) from I where mydate between ‘07-JUL-97’ and ‘25-JUL-97’ and make = ‘AQ’ and model = ‘BC’ and ratinga between -600 and 1500 and ratingb between -10000 and 4000;
Answer: 861Elapsed 0:01:10.87 CPU 22.28
Query 4 - M204 version
M204BeginFPC mydate is in range from 970705 to 970725 make = ‘AQ’ and model = ‘BC’ ratinga is in range from -600 to +1500 ratingb is in range from -10000 to 4000End Answer: 861cnct=3, cpu=303, dkrd=1171, rqtm=3391
Query 5 - Scanning rows
Oracle - Answer: 685Select count(*) from I where make = ‘AQ’ and mydate between ‘07-JUL-97’ and ‘25-JUL-97’ andratingc like ‘235%’ Elapsed: 0:02:33.48 Cpu 26.51 Scanned through 445904 rows
M204 equivalent (ratingc like ‘235*’) CNCT=10, CPU=2637, RQTM=3094,
DIRRCD=445904, DKPR=127506 (many pages in buffer pool) CNCT=100, CPU=8072, DKRD=62201, RQTM=100528,
DKPR=127506 (no pages in buffer pool)
Query 6 - ‘Is Present’ Scan
Select same 445904 rows/records to scan Look for presence of a column/field
Answer: 235997 Oracle - mynamefield is not null
Elapsed 0.02.39.31 CPU 25.47 M204 - mynamefield is present
CPU=8785, CNCT=10 (yes - it’s true), RQTM=9997, DKPR=563314 (reused buffers)
CPU=14306, CNCT=129, DKRD=62201, RQTM=129608, DKPR=563314 (fresh pages)
Query 7 - Extracting data
Output 14 fields from 445904 rows in a columnar format
Both extracts output 1100 tracks of data Identical DCB for output files
Oracle (Using SQL*Plus) Elapsed: 0:06:20.81 Cpu: 118.54
M204 User LanguageCNCT=222, CPU=45228, DKRD=62196, RQTM=201628
Sirius FUEL via $FUNLOAD Total Time: 57.557 CPU=9.875
Query 8 - Count of each value Oracle
Select make,count(make) from Igroup by make;
Elapsed 0:02:39.76 cpu=118.54 M204
Make.frv: frv makeprint value in make.frv tab ...FPC make = value in make.frv
cpu=112, dkrd=485, finds=11, rqtm=996, cnct=1
Changes since Wrangling I
Simplified Oracle Database design Handles 98% of “I” type Wrangler data Need to handle other 2% with another
design “O” table design less generic
An intermediate table between I and O was eliminated
Changes to reduce Joins
Three multiply occurring M204 fields rigidified in Oracle to reduce joins Eliminated PK repetition in 3 tables Field A became A1, A2, A3, A4, A5, A6 Field M became M1, M2, M3, M4 Field R became R1, R2, R3, R4, R5, R6,
R7, R8 (Field “R” is used in every Wrangler Find)
Changes - continued
Index tuning Reduced segments used for indices Several major “O” table indices created
with the BITMAP option Matched M204 column indexing Refined column attributes Combined three date/time pairs
Observations: Space Mgmt
Oracle - More pools of Space to work with Flexibility Can become overdone
Additional Job Security for tuners
M204 Need to tune indexes ‘together’
Share same pools (C, D)
Observations: Programming
Simple Ad-hocs (1 table) Much easier with SQL Our end users don’t do such requests Would have to limit using Views
Complex queries SQL appears delusionally simple Short code can be very complex Pgmr has to learn about paths through database M204 Procedural cross referencing approach is
easier to follow.
Living With Oracle vs M204
Retrofitting Performance De-normalization Non-Standard SQL (Hints) Learning how optimizer works Moving Indices around / Eliminating Indices
Enqueueing Less enqueueing issues in Oracle
Read-consistent views
Backup/Recovery
Oracle Recovery Rollback Segment oriented can set up App specific RBS Each database’s recovery is independent Closer to TBO than Chkp/Rf
Backup/Exports Nothing like Fast Dump or UAI/LAI Dumps DDL and data as Inserts ...
Losing Oracle Control table is disastrous
System Administration
Oracle - Everything is in a bunch of tables Tools use those tables
Experts can be helpful but ... M204 Information in system stats,
journal .. Special commands and tools
Monitor Sirmon
DBMS Data Integrity Enforcement Oracle has “Standard” DBMS features
Triggers Referential Integrity Field constraints Major ongoing effort to setup, tune, maintain Programs still have to handle exceptions
“Decoupled” approach used by Wrangler Updates limited to controlled processes that
validate data
Should Wrangler be Ported?
Porting the Wrangler Database to Oracle is not economically feasible Substantially more design, prototyping, and
analysis is necessary Special Techniques necessary
The effort to develop and support an Oracle database grows exponentially as size and scope of database increases
Complementing M204
The OS 390 Version of Oracle works well and can effectively support hot backup of a Wrangler adjunct Oracle database that is currently under development. No need to setup and administer another Unix or
NT hardware/software environment for this. System Administration, Storage management,
Security processes already in place
Supplementing M204
Oracle makes it possible to “port” subsets of Wrangler data to Unix and NT machines that run Oracle for special purpose applications. Ported Wrangler subsets should be de-indexed Port can be rehearsed effectively on mainframe
Oracle integrates effectively with the rest of the World OS Environments Java (COBOL of the 00’s) Standards
Wrangler - Currently with Model 204 Current M204 Offerings (CCA/Sirius)
Leverage M204’s unique strengths Leverage knowledge of User Language Leverage Mainframe instruction set and
other features Wrangler performance stays ahead
Sirtune Evaluated MP/204
Clever product truly helps - easy to misunderstand
Wrangler - relies on M204
Wrangler survives in the M204 world because of “Current” features Reorganized entire database for structural
changes in one week. Sirius UAI/LAI
Dump operational M204 files while transactions are in-flight.
Sirius Fast Dump Simple monitoring and problem chasing
Sirscan Sirmon
Connecting Wrangler
Wrangler survives in M204 because it is becoming more and more interoperable Janus Web Server
Supports majority of retrieval user base now Software for selected updating users being tested
Janus Sockets Proof of Concept Wrangler Retrieval API
API for talking to Java, Perl, VB, C/C++ ... Tested with Perl on Unix
Connect * Evaluated too
Tangible Benefits ?
$$$ Performance
Not obvious Database Organization
Major effort to define Database and Views that would make “Standard” simple SQL programming work with Wrangler data.
Programmability - need to use proprietary Oracle features
Organizational Benefits?
Less database systems to support? Major effort to convert Extra effort necessary to port Wrangler
applications to Oracle (or DB2) will more than offset cost of continuing to support M204 on mainframe
Inherent RDBMS complexity Oracle “Knowledge” is much more
common - bookstore & libraries.
Intangible Benefits?
Career Fear Issue M204 - Should we call it M X’CC’ instead?
“USER” part of language name Mainframe?
Remaining “Current” call PAI the “self-describing data” statement?
Ongoing Supportability What is “Current” now anyway There are antique Oracle databases around too
OS/390 Oracle and Wrangler
Serve as a replicated site for a smaller (150M) Wrangler related database.
Use Janus Open Client to extract from Oracle database
Extract subsets of Wrangler data, then load them into Oracle form to be exported to offsite Oracle databases.
“Mainframe” Oracle
Observations about OS 390 Oracle 8i SQL*Plus works well with TSO and Batch SQL*Loader works well in Batch JCL env Works with NT based administration tools Truly indistinguishable from Unix and NT versions
via Net 8 connections Very stable Leverages Mainframe behind the scenes
VSAM, Media Manager ...
Oracle for Ad-hoc support?
Use of generalized “Power User” tools to query Wrangler data not feasible. Just too much data User Language oriented Selection best Specialized $FUNLOAD for high speed
extracts - limited purpose Ongoing improvements to UL Extract
speed
M204 - Alive and well
Evolutionary approach benefited Wrangler Incremental system upgrades Incremental database upgrades Incremental app performance upgrades
Our team is effective at combining knowledge of legacy system processes with ongoing incremental M204 refinements
M204 - Alive and Well
Modernized User Interface Ever-expanding Janus Web Server Application
suite Providing Wrangler services to external
apps Specialized FUEL extracts
Store and Forward Specialized API
Janus Sockets based Wrangler data server
Porting to Oracle?
“Porting” a big M204 database to Oracle will require iteratively trying many different design ideas.
In the first port attempt, 1 million Wrangler M204 records became 14 million Oracle rows.
In the second port attempt, 3 million Wrangler M204 records became 8.5 million Oracle rows. This design is less general.
Technical Observations
Only one hiccup (OC4) was encountered with OS/390 Oracle during this exercise.
Oracle Bitmap indexes used 30-50% less space than Normal indexes. 3 “O” table indexes were created as Bitmapped
Once formatted, space assigned to Oracle is easily re-used.
Living with Oracle - 1
World of M204 is less subject to “Press 1 for …” support.
Most Oracle support requests require entering request via Web and waiting for a formal reply.
As is the case with M204 the DBA for a large Oracle database must be familiar with the application.
Oracle documentation is readily available. Useful for OS 390 Oracle OS 390 based examples not readily available
Living with Oracle - 2
Ongoing effort to maintain performance of Oracle database is more involved than with M204. Oracle recommends rebuilds of indexes
periodically Programmers have to learn how
optimizer reacts to their database. Not feasible for end users
Living with Oracle - 3
World of Oracle encourages “Technique-ing”
M204 puzzling to those inclined to “Technique-ing”
Development of Wall Poster database designs
Many parts to Oracle What do you choose?
Pro C, SQLJ, JSQL, ???
Living with Oracle - 4
Upgrading an Oracle database can be cumbersome Ex: Changing length of a field that is used
in referential integrity constraints. Knowledge Base
Lots of basic knowledge duplicated Enough to run in circles forever
Unexpected findings
CCA and Sirius are not burdened with keeping M204 features working on multiple platforms.
Oracle helps give new respect to the M204 Coded field attribute.