Post on 31-Jul-2015
“Looking for LOB in all the wrong places”
or “Tainted LOB”
Experiences with Large Objects in DB2 for z/OS
Marcus Davage
GSE 2014
Session: CC
This presentation is based on my personal views and are not in any way endorsed by Lloyds Banking Group.
None of the information contained within presentation can be used for reference or for any other purpose by another company or organisation.
• Introduction – What LOBs are and how they work
• Experiences – Naming conventions
– Using utilities
– Space, the final frontier
– Performance
• DDL parameters
• Inline LOBs
• Conclusions – Prepare
– Plan
Syn
op
sis
• Large data objects that cannot be
contained in conventional data types
• VARCHAR 32K increasingly became
not big enough
• Appeared in DB2 v6
• Each new release brings new
improvements (and new headaches)
Wh
at a
re L
OB
s?
• One LOB table space per LOB column
per partition
• One auxiliary table and index per table
space
• SELECT LOBCOLUMN FROM T1
– Done under the covers via base row and
index
• Separate from base table space
– AUX YES in IBM utilities
Wh
at a
re L
OB
s?
• One LOB table space per LOB column
per partition
• One auxiliary table and index per table
space
• SELECT LOBCOLUMN FROM T1
– Done under the covers via base row and
index
• Separate from base table space
– AUX YES in IBM utilities
• CHECK LOB utility very handy!
Wh
at a
re L
OB
s?
• Implicitly created ROWID column
appended to end of row
• Table is incomplete until additional
objects created:
– TABLESPACE for LOB
– AUXILIARY TABLE for LOB
– INDEX for auxiliary table
• CURRENT RULES special register
Wh
at a
re L
OB
s?
Ta
ble
s w
ith 1
LO
B
Base table space
Aux Index
LOB TS
Row 1
Row 2 LOB1
LOB2
Ta
ble
s w
ith >
1 L
OB
Col1
Table
Aux 1 Index
LOB1
TS
Col2
LOBCol1
LOBCol2
LOB1 LOB2
LOB2
TS
Aux 2 Index
Experiences
Your mileage may vary…
• Many major projects now using LOBs
– Huge demand for DASD
• BLOBs
– Generally already-compressed and/or
encrypted data
• CLOBs
– True character data
– XML storage (for those whom XML data
types are yet arcane)
Ub
i su
mu
s?
PROJECT TABLESPACE MB GB
TYPE
DB2PDB NORMAL 23376 22
DB2PDB PBR 137148 133
OTHER NORMAL 251338 245
OTHER PBG 665 0
OTHER LOB 3444 3
OTHER XML 25 0
OTHER PBR 137242 134
SYS1 NORMAL 69098 67
SYS1 PBG 71509 69
SYS1 LOB 1522069 1486
SYS2 NORMAL 1877 1
SYS2 PBG 285 0
SYS2 LOB 296548 289
SYS2 PBR 19802 19
SYS3 NORMAL 14 0
SYS3 PBG 0 0
SYS3 LOB 7126 6
SYS3 PBR 2138411 2088
SYS4 NORMAL 2753 2
SYS4 LOB 381968 373
Curren
t usag
e
SELECT TYPE, COUNT(*)
FROM SYSIBM.SYSTABLESPACE
WHERE YEAR(CREATEDTS) >='2006'
GROUP BY TYPE ;
---------+---------+---------+-
TYPE
---------+---------+---------+-
1652
G 323
O 1110
P 6
R 70
---------+---------+---------+-
Cu
rren
t usa
ge
WITH TS (CREATED) AS (
SELECT YEAR(CREATEDTS)
FROM SYSIBM.SYSTABLESPACE
WHERE TYPE = 'O'
AND YEAR(CREATEDTS) >='2006'
)
SELECT CREATED, COUNT(*)
FROM TS
GROUP BY CREATED
WITH UR ;
---------+---------+---------+-
CREATED
---------+---------+---------+-
2006 1
2010 15
2012 180
2013 201
2014 713
---------+---------+---------+-
Cu
rren
t usa
ge
0
100
200
300
400
500
600
700
800
2006 2010 2012 2013 2014
No. of LOBs
No. of LOBs
Naming Conventions
• DB2 Object Naming Standards
– Generally a Good Thing™
– Ease of Control / Administration
– Housekeeping
– Diagnosis
• CURRENT RULES special register
– STD – let DB2 do it for you
– DB2 – do it yourself
Na
min
g C
on
ve
ntio
ns
• The Good
– More control
– Complies with local standards
– Great for limited growth
• The Bad
– Must be created beforehand
– Constant monitoring
– Not for Partition By Growth tablespaces
• The Ugly
– Can cause outages if not tightly controlled
CU
RR
EN
T R
UL
ES
DB
2
• The Good
– No monitoring/maintenance/administration
– Complete automation (TABLESPACE ALL)
– Great for Partition by Growth tablespaces
• The Bad
– No control over object names
– Contravenes local standards
• The Ugly
– Needs careful consideration of
housekeeping routines (TABLESPACE ALL)
CU
RR
EN
T R
UL
ES
ST
D
• Which CURRENT RULES currently
rules?
– STD (My opinions, YMMV)
• Gotchas
– Work our your TS naming standards
• Easily identify LOBs from names
– Plan your housekeeping
• ISV tools
– AUX YES
– Watch out for EXCLUDEs/INCLUDEs
• IBM – use LISTDEFs with ALL keyword
CU
RR
EN
T R
UL
ES
??
?
CREATE TABLE USER.TEST1
(INT1 INTEGER NOT NULL
GENERATED ALWAYS AS IDENTITY (
START WITH 1 INCREMENT BY 1
MINVALUE 1 MAXVALUE 2147483647 NO
CYCLE CACHE 20 NO ORDER )
,FLAG1 CHAR(1) NOT NULL WITH DEFAULT
,VARCHAR1 VARCHAR(128) NOT NULL WITH DEFAULT USER
,SMALLINT1 SMALLINT NOT NULL WITH DEFAULT
,DECIMAL1 DECIMAL(15, 2) NOT NULL WITH DEFAULT
,TS1 TIMESTAMP NOT NULL WITH DEFAULT
,DATE1 DATE NOT NULL WITH DEFAULT
,TIME1 TIME NOT NULL WITH DEFAULT
,BLOB1 BLOB(1 M) INLINE LENGTH 1024
,CLOB1 BLOB(1 M) INLINE LENGTH 1024
,XML1 XML
,BLOB2 BLOB(1 M) INLINE LENGTH 1024
,CLOB2 BLOB(1 M) INLINE LENGTH 1024
,XML2 XML
,CONSTRAINT INT1 PRIMARY KEY ( INT1 ) )
IN DATABASE DUSER
CCSID UNICODE NOT VOLATILE APPEND NO
PARTITION BY SIZE EVERY 4 G;
Na
min
g e
xa
mp
les
Table Name Database Tblspace
v----1----v----2----v----3----v----4----v----5--
USER.TEST1 DUSER TEST1
USER.TEST1BLOB1K8JVZOFT DUSER LK8JVP4A
USER.TEST1BLOB2K8JWV1TJ DUSER LK8JV9XB
USER.TEST1CLOB1K8JVCUGV DUSER LK8JV7XN
USER.TEST1CLOB2K8JW3K8V DUSER LK8JWTMK
USER.XTEST1 DUSER XTES0000
USER.XTEST1000 DUSER XTES0001
Index Name Table Name
--v----1----v----2----v----3----v----4----v----5----v-
USER.I_DOCIDTEST1 USER.TEST1
USER.I_NODEIDXTEST1 USER.XTEST1
USER.I_NODEIDXTEST1000 USER.XTEST1000
USER.ITEST1BLOB1K8JV6BC USER.TEST1BLOB1K8JVZOFT
USER.ITEST1BLOB2K8JW190 USER.TEST1BLOB2K8JWV1TJ
USER.ITEST1CLOB1K8JVISA USER.TEST1CLOB1K8JVCUGV
USER.ITEST1CLOB2K8JW919 USER.TEST1CLOB2K8JW3K8V
USER.TEST1ABC_#_YJU USER.TEST1
Na
min
g e
xa
mp
les
Using Utilities
• Separate from base table space
– CHECK, LOAD, REORG, COPY,
RECOVER, RUNSTATS
• Keep base + aux in sync
• Be aware of differences between
LOGGED and NOT LOGGED
• CHECK
– LOB
– INDEX
– DATA
Usin
g U
tilities
• RECOVER base and LOBs together
(simple, best option, need good copies)
• REPAIR (if LOB can be restored from
elsewhere)
• SQL (INSERT INTO…SELECT FROM)
• DSN1COPY (TSMYOYO)
• Re-LOAD
• Correcting LOB errors
LO
B R
eco
ve
ry
• KEYCARD keyword doesn’t work
• ISVs slow off the mark collecting LOB
stats
RU
NS
TA
TS
• Only REFERENCE (V9) / CHANGE
(V10) reclaims space
• SHRLEVEL CHANGE recommended
• Use AUX YES
– Rebalance PBR partitions
– Consolidate PBG partitions
• Issues with DISCARD
• Excluded from our automated
housekeeping due to size of LOBs
RE
OR
G
• Try and use LISTDEFs
• INCLUDE / EXCLUDE TABLESPACES
TABLESPACE database.space ALL
• ISVs have issues with INCLUDES and
EXCLUDES with AUX YES
CO
PY
• Job 1 COPY TABLESPACE DLBGP01.*
EXCLUDE DLBGP01.LJ6KYEZR, Aux TS for LOB part 2
DLBGP01.LJ7Z83KR, Aux TS for LOB part 3
DLBGP01.LJ762HEZ, Aux TS for LOB part 4
DLBGP01.LJ79T9ME, Aux TS for LOB part 5
DLBGP01.SPF0063, Base TS for Base table
DLBGP01.SPF0279 Aux TS for LOB part 1
AUX YES
• Job 2 COPY TABLESPACE DLBGP01.SPF0279
AUX YES
• Job 3 LISTDEF DPEGP01_TS01
INCLUDE TABLESPACES TABLESPACE DLBGP01.SPF0279 ALL
LISTDEF DPEGP01_TS02
INCLUDE TABLESPACES TABLESPACE DLBGP01.* ALL
EXCLUDE TABLESPACES TABLESPACE DLBGP01.SPF0279 ALL
CO
PY
• Cannot unload from image copies
– Would be a nice enhancement
• Unload with File Reference Variables
– Simpler to code
– Uses PDSE, one member per LOB
– Takes ages...and ages…and ages…
• Unload/LOAD with SPANNED
– Much faster
– A pain to code
– Generated LOAD cards in SYSPUNCH
would be handy for spanned records
UN
LO
AD
/LO
AD
Space – The Final Frontier
• WebSphere Message Broker
– Write-only application (mostly)
– c. 1 million inserts per day
– 4.5M tracks per day/30M tracks per week
– 1.5 TB per week
– Base Table had 1 BLOB and 1 CLOB
– LOBs defined to BP32K
– Max BLOB =3772, max CLOB=23368
– Wasted space!!!
• Broker logging reduced from 4M to 1.8M
tracks per day through app changes…
Sp
ace
– T
he
Fin
al F
ron
tier
Sp
ace
Exam
ple
Table Mon Tue Wed Thu Fri Sat Sun
Bufferpool BP32K2 BP21 BP21 BP21 BP32K2 BP32K2 BP32K2
Status 22/8 truncated and
reorged before use.
2/9 reorged by auto
reorg.
1/9 truncated and
reorged before use
1/9 truncated and
reorged before use,
in use now
1/9 truncated and
reorged before use
??/? truncated and
reorged before use.
2/9 reorged by auto
reorg.
22/8 truncated and
reorged before use.
2/9 reorged by auto
reorg.
22/8 truncated and
reorged before use.
2/9 reorged by auto
reorg.
AUDIT_LOG
Base 9765 8745 840 240 8355 4305 1965
LOB1 749430 88980 7665 690 829350 331155 148590
LOB2 748425 94245 7665 690 827655 331155 148590
1,507,620 191,970 16,170 1,620 1,665,360 666,615 299,145
rowcount 1,094,706 985,872 in use 0 983,277 485,007 215,940
Sp
ace
Exam
ple
0
200000
400000
600000
800000
1000000
1200000
1400000
1600000
1800000
1 2 3 4 5 6 7
CLOB
BLOB
Base
Sp
ace
Exam
ple
Class 1 Class 2 Class 3 Comments
0.075636 0.00762 0.005501 Base(BP16K) LOBS(BP32K)
0.039277 0.006351 0.004304 Base(16K) LOBS(BP8)
0.173326 0.042717 0.024962 base table in BP16K0 with inline BLOB and inline CLOB - LOBS in BP8
0.258535 0.071942 0.049692 base table in BP16K0 with max inline CLOB
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
1
2
3
4
Class 1
Class 2
Class 3
• Answer?
– Change bufferpool from 32K to 4K
• Tracks reduced from 1.8M to <200K per day
• Saving of 87% in DASD footprint!
• CPU went down 50%
– INLINE LOBs and compression
• Saved a further 10% in DASD
• CPU and elapsed time went up considerably
• Not really worth considering
• As I said, a mostly-write-only application, so
SELECTs not benefiting.
Sp
ace
– T
he
Fin
al F
ron
tier
Performance
• GBPCACHE
– CHANGED
– ALL
– SYSTEM
• SYSTEM is the default for LOBs
• Do not use CHANGED for a LOB table space
defined with LOG NO
– NONE
• Saves data transfer in GBP write and GBP
access for castout
• But updated pages must be written to DASD at
or prior to commit
• Useful when an updated page is rarely re-read
Pe
rform
an
ce
CREATE TABLE …. clobcol CLOB(x) INLINE LENGTH(y)
– Differentiates between SLOBs and FLOBs
– Allows LOBs to be stored in the base table
– Max 32680 bytes
– If a LOB cannot be fully inline, the LOB is
“split”. The first “y” bytes are stored inline
and the remaining bytes are stored in the
LOB table space.
Inlin
e L
OB
s
• Advantages
– Saves disk space
– Fewer synchronous I/Os
– Less CPU time to access LOB
– Prefetch can access multiple LOBs
– Improve FETCH CONTINUE
– Index on expression for LOB data
Inlin
e L
OB
s
• Disadvantages
– If most LOBs are split, performance may be
hurt
– If LOBs aren’t accessed, inline LOBs may
hurt performance
– If LOBs don’t compress well, CPU time
increases
– Buffer hit ratio for base table may decrease
– Larger base table image copies
– More logging, if LOBs are not logged
Inlin
e L
OB
s
• Considerations
– Requires Universal Table Spaces
– Required Re-ordered Row Format
– May require multiple reorg steps to
implement
– Know your LOB size distribution
• SQL histogram queries available
– May require new page size for base table
– May not compress well
• DSN1COMP doesn’t work for LOBs
• CLOBs better than BLOBs
Inlin
e L
OB
s
• Considerations
– May need to retune buffer pools
– Don’t bother Inlining if fewer than 50% of
LOBs can be fully inlined
– Increasing to next page size will reduce the
number of split LOBs
– If a LOB is not fully inlined, base and
auxiliary objects should be reorganized
in the same step. Use AUX YES or the
ALL keyword and SHRLEVEL
REFERENCE
Inlin
e L
OB
s
• Considerations
– Preparation is key
• “Before” performance stats reports
• “After” performance stats reports
– Iterative development
• Try several incremental changes
– INLINE length
– Page size
– Compression
– Storage
• Local and Group buffer pool sized correctly?
• Backed up by real storage?
Inlin
e L
OB
s
Conclusions
“If you go down to the LOBs today, you’d better not go alone”
• Prepare
– Read IBM redbooks
– IDUG/GSE/IBM presentations
– Read The Friendly Manuals
“It’s lovely down in the LOBs today, but it’s safer to stay at home”
• Plan
– DASD – make sure you have enough
– Real Storage – make sure you have enough
– Housekeeping
– Naming conventions – get them right first
Co
nclu
sio
n
http://db2forz.blogspot.co.uk/2013/07/is-length-of-your-inline-lobs-optimal.html
http://www.redbooks.ibm.com/redbooks/pdfs/sg247270.pdf
http://www-01.ibm.com/support/docview.wss?uid=swg1PM73034
http://www.gsebelux.com/system/files/Inline-LOB-BMC-GSEBEmar2013.pdf
http://robertsdb2blog.blogspot.co.uk/2012/01/got-lobs-get-db2-10-for-zos-part-1.html
http://robertsdb2blog.blogspot.co.uk/2013/07/db2-for-zos-clearing-up-some-
matters.html
http://db2forz.blogspot.co.uk/2013/07/is-length-of-your-inline-lobs-optimal.html
Re
fere
nce
s
Th
an
k y
ou
!
This presentation is based on my personal views and are not in any way endorsed by Lloyds Banking Group.
None of the information contained within presentation can be used for reference or for any other purpose by another company or organisation.
Marcus Davage @spufidoo
http://about.me/spufidoo
GSE 2014
Session: CC