IMS Database Performance - IMS UG Singapore 8-2013
-
Upload
ibm-ims -
Category
Technology
-
view
414 -
download
5
Transcript of IMS Database Performance - IMS UG Singapore 8-2013
@Copyright IBM Corporation 2013 5.2
Welcome to:
IMS Database Performance
Rick Long, IMS Level 2 Database Support Team
Silicon Valley Lab [email protected]
Disclaimer
© Copyright IBM Corporation 2009. All rights reserved.U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE.
IBM, the IBM logo, ibm.com, IMS, IBM High Performance Pointer Checker for z/OS and CICS are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml
Other company, product, or service names may be trademarks or service marks of others.
© Copyright IBM Corporation 2013
It is every DBA's nightmare – Does the database need to be reorganized or tuned?
This session describes the key performance indicators and how to interpret the large volume of information provided by the IBM HPPC utility.
Abstract
© Copyright IBM Corporation 2013
Example
HALDB stats from 2005
Partition name Roots Segments
ECDBP01 100,467 574,731
ECDBP02 81,687 581,307
ECDBP03 49,113 293,427
ECDBP04 249 2,140,708
ECDBP05 142 1,335,117
ECDBP06 180 1,300,517
ECDBP07 192 1,780,635
ECDBP08 123 1,479,159
ECDBP09 186 1,378,110
ECDBP10 257 1,875,931
ECDBP11 581 1,583,505
ECDBP12 415 382,060
ECDBP13 74 481,514
ECDBP14 1,476 978,823
ECDBP15 2,119 1,090,954
ECDBP16 1,992 331,247
ECDBP17 378 29,498
ECDBP18 149 2,767
TOTAL 239,780 17,620,010
HALDB stats from 2008
Roots Segments Additions
100,467 576,357 1,626
81,687 583,377 2,070
124,261 684,172 390,745
309 6,662,891 4,522,183
161 4,301,836 2,966,719
182 3,745,620 2,445,103
194 5,129,972 3,349,337
123 4,616,059 3,136,900
194 4,071,069 2,692,959
258 6,850,596 4,974,665
779 6,406,064 4,822,559
1,418 1,641,717 1,259,657
113 1,301,952 820,438
1,604 2,482,972 1,504,149
2,305 2,915,532 1,824,578
2,251 1,047,040 715,793
840 332,103 302,605
149 2,767 0
317,295 53,343,096 35,723,086© Copyright IBM Corporation 2013
• Review of Database types–Full Function Database
• HDAM,HIDAM–HALDB
• PHDAM, PHIDAM,PSINDEX
• Reorganization Needs–What factors cause a need for Reorganization
Agenda
© Copyright IBM Corporation 2013
PRIMARY
DATA
KSDS ESDS / OSAMKEY
INDEX
• Hierarchic Indexed Direct Access Method
• 2 databases• Index• Data
– 2 access method combinations• KSDS/ESDS (VSAM/VSAM)• KSDS/OSAM (VSAM/OSAM)
– 'Native-mode' index– Unique root keys required– Pointers– Space reusable
HIDAM
© Copyright IBM Corporation 2013
KEY
KEY
ESDS / OSAM
RANDOMIZER
• Hierarchic Direct Access Method
– 1 database• Data
– 2 access methods• OSAM• ESDS (VSAM)
– Randomizer– Non-unique root keys allowed– Synonyms – Pointers– Space reusable
HDAM
© Copyright IBM Corporation 2013
1 .. 100
101 .. 200
201 .. 300301 .. 400
• High Availability Large Data Base–Access methods
• PHDAM Partitioned HDAM• PHIDAM Partitioned HIDAM
–Secondary Index• PSINDEX Partitioned Secondary Index
–Partitions• Key Range• Partition Selection Exit
–Partition Selection beforerandomize – each partition can be different
HALDB
© Copyright IBM Corporation 2013
• Record Structure
–Prefix–Segment
• Variable length• Fixed length
Common Record Structure
ROOT
DEP DEP DEP DEP
DEP
© Copyright IBM Corporation 2013
Current Structure for R1CUST
ADJSMNTSPAYMENTSCUSTINVCUSTLOCNDISTRICT
CUSTOMER
ADDRLINE CUSTORDN
© Copyright IBM Corporation 2013
Data set group 2
Data set group 3
5.2
IMS Database Pointers
© Copyright IBM Corporation 2013
IMS Pointers
DELETE BYTE(FF), Prefix Descriptor(FP)
SEGMENT CODE
SC DB PTF DATA (BYTES=)
SEGMENT
PREFIX
LCFPTB PCF LCF
• HALDB
© Copyright IBM Corporation 2013
PHYSICAL TWIN FORWARD(4 BYTES) PHYSICAL TWIN BACKWARD(4 BYTES)
PHYSICAL CHILD FIRST(4 BYTES)
LOGICAL CHILD FIRST(28 BYTES)
IMS Pointers
DELETE BYTE(FF), Prefix Descriptor(FP)
SEGMENT CODE
SC DB PTF DATA (BYTES=)
SEGMENT
PREFIX
ILKLCFPTB PCF LCF
• HALDB
© Copyright IBM Corporation 2013
PHYSICAL TWIN FORWARD(4 BYTES) PHYSICAL TWIN BACKWARD(4 BYTES)
PHYSICAL CHILD FIRST(4 BYTES)
LOGICAL CHILD FIRST(28 BYTES)
INDIRECT LIST KEY
PTR=T
PTR=TB
PTR=NT
COURSE COURSECOURSE
CLASS
INSTRUCT STUDENT
EDUC EXPR
PTF PointersPTB Pointers
© Copyright IBM Corporation 2013
COURSE COURSECOURSE
CLASS
INSTRUCT STUDENT
EDUC EXPR
Physical child pointers are stored on the parent, but definedon the SEGM for the child:
Physical Child Pointer Specification
PCF ONLY: SEGM NAME=. . . ,PARENT=. . .
PCF ONLY: SEGM NAME=. . . ,PARENT=((. . . ,SNGL))
PCF, PCL: SEGM NAME=. . . ,PARENT=((. . . ,DBLE))
PCF PointersTwin Forward Pointers
© Copyright IBM Corporation 2013
5.2
Reorganization Objectives
© Copyright IBM Corporation 2013
• Access segments –Fewest I/Os–Segment placement
• Database records –optimal number of blocks
• Blocks/ Cis–Free space– In RAA (Root Addressable Area)–Room in the data set
Reorganization Objectives
© Copyright IBM Corporation 2013
• Information about the current state of the DB–Regular Statically Information gathering
• HPPC (or equalivant)• Unload/Reload stats• Historical information for Trend analysis
• Information on how the DB is used–Sequential processing vs. Random access–Mass insert or delete processing–Heavy sequential BMP processing –Which DBs are most frequently accessed–Which DBs are most frequently updated
• Splitting of segments –Variable length segments –Compressed segments
Reorganization Requirements
© Copyright IBM Corporation 2013
• Getting Segments into the Most Desirable block
• Re-sequence Segments – Root segments into Sequence– Child segment twin chains into a block
• HDAM Tuning– Getting Root segments into HOME block– Reducing Rap chain lengths
• Larger RAA (Root Addressable Area)• Rap chain out of the block
• Unspliting Split segments
• Refreshing Free space in the data set
• “Mean time” to reorganization
Reorganization Reasons
© Copyright IBM Corporation 2013
• How long do I want to go between reorganizations
• What are the reasons I need to reorg? –Re-sequencing segments–Split segments–Long twin chains–Rap chains in too many blocks
“Mean time” to Reorganization
© Copyright IBM Corporation 2013
• HDAM/PHDAM Root
–Block it Randomizes to
–Rap Chain stays within the block
– Ideal length of Rap chain is the number of DB records which will fit in the block
• All roots in RAA
Most Desirable Block
© Copyright IBM Corporation 2013
• HIDAM/PHIDAM Root
– PTR=TB - In the block with the next highest key value
– PTR=NT - In the block with the next lowest key value
– PTR=T - HIDAM Root (T) • SHOULD NEVER be used as RAPs are used and causes
DELETE performance issues
– Overflow not relevant
Most Desirable Block
© Copyright IBM Corporation 2013
COURSE COURSECOURSE
HIDAM ROOT Physical Twin PTR
PTR=TB KEY POINTER100000 RBA
200000 RBA
200100 RBA
400000 RBA
999999 RBACALL :ISRT
MDB
COURSE
In the block with the next highest key value
© Copyright IBM Corporation 2013
COURSE COURSECOURSE
HIDAM ROOT Physical Twin PTR
PTR=NOTWIN KEY POINTER100000 RBA
200000 RBA
200020 RBA
400000 RBA
999999 RBACALL :ISRT
COURSE
MDB
In the block with the next lowest key value
© Copyright IBM Corporation 2013
• Child Segment – Sequential key – Lowest key value
• Block with its parent
– Sequential key - not lowest value • In the block with the next lowest key value
– NOTWIN - Block with parent
– Rules • FIRST – Block with parent• LAST - Block with previous LAST twin• HERE - Block positioned on
Most Desirable Block
© Copyright IBM Corporation 2013
CLASS
• Sequential key defined on child segment– ISRT for child segment
• Inserting the lowest key value• Child PTF set to the next highest key value (taken from
PCF)• MDB – block with parent
Physical Child Pointer Specification
INSTRUCT STUDENT
© Copyright IBM Corporation 2013
CLASS
• Sequential key defined on child segment– ISRT for child segment
• Inserting the key that is not the lowest key value• Inserted in sequence and PTF set to next highest key
(taken from PTF) • MDB – block with next lowest key value
Physical Child Pointer Specification
INSTRUCT STUDENT
© Copyright IBM Corporation 2013
1. In the most desirable block 2. In the second-most desirable block or CI.3. In any block or CI in the buffer pool on the same cylinder.4. In any block or CI on the same track, as determined by consulting the bitmap.
(longest segment type defined. Uncompressed from DBD LENGTH=)5. In any block or CI on the same cylinder, as determined by consulting the bitmap.6. In any block or CI in the buffer pool within plus or minus n cylinders. Specify n in
the SCAN= keyword in the DATASET statement in the DBD. For HALDB databases, the value of the SCAN= keyword is always 0.
7. In any block or CI plus or minus n cylinders, as determined by consulting the bitmap.
8. In any block or CI in the buffer pool at the end of the data set.9. In any block or CI at the end of the data set, as determined by consulting the
bitmap. The data sets will be extended as far as possible before going to the next step.
HD space search algorithm
© Copyright IBM Corporation 2013
5.2
Reorganization “How Tos”
© Copyright IBM Corporation 2013
• Size of the RAA
• Packing Density – Packing Factor = bytes of data / bytes in RAA
• General usage guidelines was 70%• High Insert Less then 70%• Mostly read - 80%
• Put a “P” in front – Make it PHDAM– Reduce Packing density to suit how long between reorgs. – If data comes and goes, you can reorg very infrequently
How Do I Tune a: HDAM
© Copyright IBM Corporation 2013
• Keep Free space in it.–Don’t let heavy insert activity all end up at the end of
the data set.
• CI free space for dependent segments
• CA free space for Root and dependents
How Do I Tune a: HIDAM
© Copyright IBM Corporation 2013
• NEVER
• Tune those PARTITIONS which need to be.
• Change of mind set from a database to a PARTITION– Review partition information– Reorg partitions independent of the others
• Exceptions– Re-partitioning– STILL ONLY Deal with PARTITIONS and as few a possible
How Do I Tune a: HALDB
© Copyright IBM Corporation 2013
• Change the V to an O - Make it OSAM
• Faster access method
• OSAM Sequential buffering –BMP reading a HIDAM data base in root sequence
How Do I Tune : VSAM
© Copyright IBM Corporation 2013
• CI free space in the block
• Full length is used for comparison with bitmap data for insert
• HDAM – lower packing density
How Do I Tune : Split Segments
© Copyright IBM Corporation 2013
How Do I Tune : Split Segments
0VL SEGMENT SPLIT STATISTICS ---------------------------
SEGMENT <----------- PREFIX AND DATA -----------> SC NAME TYPE OCCURS NOT SPLIT SPLIT IN SAME BLOCK NOT IN SAME BLOCK -- -------- ---- ------ ------------ ---------------- --------------- --------------------03 CUSTORDN FC 400 0 0.0 % 200 050.0 % 100 50.0 % 100 50.0 % -- -------- ---- ------ ------------ ---------------- --------------- --------------------TOTALS 400 0 0.0 % 200 050.0 % 100 50.0 % 100 50.0 %
0VL SEGMENT SPLIT STATISTICS ---------------------------
SEGMENT <----------- PREFIX AND DATA -----------> SC NAME TYPE OCCURS NOT SPLIT SPLIT IN SAME BLOCK NOT IN SAME BLOCK -- -------- ---- ------ ------------ ---------------- --------------- --------------------03 CUSTORDN FC 202 0 0.0 % 202 100.0 % 6 3.0 % 196 97.0 % -- -------- ---- ------ ------------ ---------------- --------------- --------------------TOTALS 202 0 0.0 % 202 100.0 % 6 3.0 % 196 97.0 %
© Copyright IBM Corporation 2013
• CI free space in the block
• Data Set Group– Long segment– Left in Hierarchy– How many other segments does it displace for their MDB?– After reorg, Twins might be in a single block
• Large block size
How Do I Tune : Long twin chains
© Copyright IBM Corporation 2013
Current Structure for TxCUST
ADJSMNTSPAYMENTSCUSTINVCUSTLOCNDISTRICT
CUSTOMER
ADDRLINE CUSTORDN
© Copyright IBM Corporation 2013
How Do I Tune Data Set Groups
DBD S1CNTPY O1CNTPY U1CNTPY P1CNTPYPHIDAM 1DSG 5PARTS
PHIDAM 1DSG 1PARTS
PHIDAM 3DSG 5PARTS
PHIDAM 1DSG 1PART
Elapse time in Mins 2.24 2.25 0.37 2.24CPU SECS 15.19 14 7.49 14.03
© Copyright IBM Corporation 2013
How Do I Tune : Processing Sec Index (whole DB)
JOB name
Elaspe time in mins CPU SEC
No of Order segms No of GN
S1ORDLS1 33.19 484.15 5,668,280 5,338,196S1ORDLS2 4.23 83.04 5,668,280 5,338,196U1ORDLS1 37.13 575.09 5,668,060 5,668,281U1ORDLS2 4.23 97.27 5,668,060 5,668,281P1ORDLS1 33.43 497.83 5,668,060 5,668,061P1ORDLS2 4.48 87.46 5,668,060 5,668,061O1ORDLS1 35.49 523.12 5,668,145 5,668,146O1ORDLS2 3.48 94.66 5,668,145 5,668,146
© Copyright IBM Corporation 2013
ORDER
PROD
Segment BPROD
PROD
INVOICEPRODINVOICEPROD
INVOICEPRODINVOICE
INVOICE
ORDER
ORDER
CUST
PRODinvoice
PROD
ORDER
INVOICEPRODINVOICEPPOD
PRODINVOICEPROD
PROD
PRODINVOICEPROD
INVOICEPROD
Long Twin Chains
© Copyright IBM Corporation 2013
ORDER
PROD
Segment BPROD
PROD
INVOICEPROD
INVOICE
PROD INVOICEPROD
INVOICE
INVOICE
ORDER
ORDER
CUST
PROD
invoicePROD
ORDER
INVOICE
PROD
INVOICE
PPOD
PROD
INVOICE
PROD
PROD
PROD
INVOICE
PROD
INVOICE
PROD
PAYMENT
SHIP
PAYMENT PAYMENTPAYMENT
SHIPSHIPSHIP
ORDER PROD PROD
Long Twin Chains
© Copyright IBM Corporation 2013
ORDER
PROD
Segment BPROD
PROD
PRODPRODPROD
PROD
CUST
PROD
PROD
PROD PROD
PPOD
PRODPROD
PROD
PROD PRODPROD
PROD PROD PROD
PROD
PROD PROD
PPOD
PRODPROD
PROD PROD PROD
Long Twin Chains
© Copyright IBM Corporation 2013
ORDER Segment BPROD
PROD
PROD
INVOICE
PRODPROD
INVOICEORDERORDER
CUST
PROD
PROD
ORDER
INVOICE
PROD
PPOD
PROD
INVOICE PROD
PRODINVOICE PROD
INVOICE
PROD
PAYMENT
SHIP
PAYMENT PAYMENTPAYMENT
SHIPSHIPSHIP
ORDER
PROD PROD
Long Twin Chains
© Copyright IBM Corporation 2013
@Copyright IBM Corporation 2013 5.2
Welcome to:
HPPC Reports
Text Slide0DBNAME: C1CUST DB#: 001 PARTNAME: C1CUST7 PART ID: 00007 REORG#:00002 DSG#:A DDNAME:C1CUST7ADSNAME: RLONG.IMS10.C1CUST.C1CUST7.A00007 DBORG: PHIDAM
INTERVAL: START BLOCK 1 END BLOCK 359
--------------- BASE -------------- ------------ OVERFLOW ------------- SC SEGNAME OCCURRENCES SPLIT-PRFX SPLIT-DATA OCCURRENCES SPLIT-PRFX SPLIT-DATA
01 CUSTOMER 2000 0 0 0 0 0 02 DISTRICT 3997 0 0 0 0 0 03 CUSTORDN 34 34 0 374 374 408 04 CUSTINVN 34 0 0 782 0 0 05 PAYMENTS 0 0 0 0 0 0 06 ADJUSTMT 0 0 0 0 0 0
---------- ---------- ---------- ---------- ---------- ----------TOTALS 6065 34 0 1156 374 408 -NOTE : - VIRTUAL SEGMENTS ARE NOT SHOWN
© Copyright IBM Corporation 2013
Text Slide
DSNAME: RLONG.IMS10.C1CUST.C1CUST7.A00007 BLK# 0........1.........2.........3.........4.........5.........6.........7.........8.........9.........0
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1 0 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200 0000000000001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111300 11111111111111111111111111111111111111111111111111111111111
-BLOCKS W/O SPACE = 212 BLOCKS WITH SPACE = 147 NO OF BIT MAPS = 1
© Copyright IBM Corporation 2013
Text Slide
0DISTRIBUTION OF DEPENDENT SEGMENTS IN ROOT BLOCK #ROOTS WITH 00 DEPS = 2 #ROOTS WITH 12 DEPS = 0 #ROOTS WITH 01 DEPS = 4 #ROOTS WITH 13 DEPS = 0 #ROOTS WITH 02 DEPS = 1960 #ROOTS WITH 14 DEPS = 0#ROOTS WITH 03 DEPS = 0 #ROOTS WITH 15 DEPS = 0 #ROOTS WITH 04 DEPS = 34 #ROOTS WITH 16 DEPS = 0 #ROOTS WITH 05 DEPS = 0 #ROOTS WITH 17 DEPS = 0 #ROOTS WITH 06 DEPS = 0 #ROOTS WITH 18 DEPS = 0 #ROOTS WITH 07 DEPS = 0 #ROOTS WITH 19 DEPS = 0 #ROOTS WITH 08 DEPS = 0 #ROOTS WITH 20 DEPS = 0#ROOTS WITH 09 DEPS = 0 #ROOTS WITH 21 DEPS = 0 #ROOTS WITH 10 DEPS = 0 #ROOTS WITH 22 DEPS = 0#ROOTS WITH 11 DEPS = 0 #ROOTS WITH 23 DEPS = 0
#ROOTS WITH 24+ DEPS = 0 ----------------------------------------------------------------------------------#ROOTS IN THE DATA SET = 2000
© Copyright IBM Corporation 2013
Text Slide
0DISTRIBUTION OF DEPENDENT SEGMENTS BY SEGMENT CODE --------------------------------------------------0SEGM CODE #DEPIN #DEPIN PERCENTAGE SEGMENT CODE #DEPS IN PERCENTAGE
ROOTBLK ALLBLK (ROOT/ALL) ROOT BLOCK ALL BLOCKS (ROOT/ALL)------- ----- ----- ----- ------------- ------ ----- -----(SEGCODE 02 ) 3992 3997 99.87 (SEGCODE 13 ) N / A N / A N / A (SEGCODE 03 ) 34 408 8.33 (SEGCODE 14 ) N / A N / A N / A (SEGCODE 04 ) 34 816 4.16 (SEGCODE 15 ) N / A N / A N / A (SEGCODE 05 ) N / A N / A N / A (SEGCODE 16 ) N / A N / A N / A (SEGCODE 06 ) N / A N / A N / A (SEGCODE 17 ) N / A N / A N / A(SEGCODE 07 ) N / A N / A N / A (SEGCODE 18 ) N / A N / A N / A (SEGCODE 08 ) N / A N / A N / A (SEGCODE 19 ) N / A N / A N / A(SEGCODE 09 ) N / A N / A N / A (SEGCODE 20 ) N / A N / A N / A(SEGCODE 10 ) N / A N / A N / A (SEGCODE 21 ) N / A N / A N / A(SEGCODE 11 ) N / A N / A N / A (SEGCODE 22 ) N / A N / A N / A(SEGCODE 12 ) N / A N / A N / A (SEGCODE 23 ) N / A N / A N / A
(SEGCODE 24+) N / A N / A N / A ---------- ---- ----- ------ ------------- ------ ------ ----
#DEPS IN THE DS 4060 5221 77.76
© Copyright IBM Corporation 2013
DATABASE RECORD STATISTICS
SEGMENT <---- SEGMENT LENGTH ---> <-AVERAGE OCCUR-> AVG LGTH CUMULATIVE SC LV DG NAME OCCUR PRFX + DATA = TOTAL P/ROOT PER/PRNT DB RECORD LENGTH-- -- -- -------- ----- --- ------- ------ ----- ----- ------- ----------01 01 A CUSTOMER 2,000 38 130.0 168.0 1.0 168.0 168.0 02 02 A DISTRICT 3,997 18 62.0 80.0 2.0 2.0 159.9 327.9 03 02 A CUSTORDN 408 18 220.2V 238.2 0.2 0.2 48.6 376.5 04 02 A CUSTINVN 816 18 55.0 73.0 0.4 0.4 29.8 406.3 05 02 A PAYMENTS 0 18 25.0 43.0 0.0 0.0 0.0 406.3 06 02 A ADJUSTMT 0 18 56.0 74.0 0.0 0.0 0.0 406.3 -- -- -- -------- ----- --- ------- ----- ----- ----- ------- ----------TOTALS 7,221 AVERAGE DB RECORD LENGTH = 406.3
AVERAGE DB RECORD PREFIX LENGTH = 85.0
© Copyright IBM Corporation 2013
VL SEGMENT SPLIT STATISTICS
SC NAME TYPE OCCUR NOT SPLIT SPLIT SAME BLOCK NOT IN BLOCK -- -------- ---- ---- ---- ---- ------------- ---------------- -------------03 CUSTORDN FC 408 0 0.0% 408 100.0 % 14 3.4 % 394 96.6 % -- -------- ---- --- ----- ---- ------------ ---------------- --------- -----TOTALS 408 0 0.0% 408 100.0 % 14 3.4 % 394 96.6 %
© Copyright IBM Corporation 2013