Post on 01-Aug-2022
Unlocking the Mysteries of the DB2 Log
Paul TurpinS1 Corporationpaul.turpin@s1.com
Session Code: C12
Thursday, May 13, 2010 02:45 PM – 03:45 PMPlatform: DB2 for Linux, UNIX, Windows
Logging is a critical part of DB2, but often not understood. This presentation will help you understand the log and how toconfigure it for good performance and recoverability. We will also explore log record layouts and how to use log APIs. Inorder to use the log APIs, you need a C program, so we will cover what is required to setup a C environment.
1
Agenda• What is a log used for? • Understanding log configuration
parameters• What's in a log - log file layouts• Log APIs and how to use them• How to setup a C environment to use
log APIs
ObjectivesObjective 1: What is a log used for? Objective 2: Understanding log configuration parametersObjective 3: What's in a log - log file layoutsObjective 4: Log APIs and how to use themObjective 5: How to setup a C environment to use log APIs
2
3
Sources of Information
• Information Center for DB2 9.5
• Administrative API Reference (SC23-5842-02)• Appendix B. DB2 log records
• Developing Embedded SQL Applications (SC23-5852-02)
Product manualshttp://www-01.ibm.com/support/docview.wss?rs=71&uid=swg27009727 (DB2 9.5 for Linux, UNIX, Windows Product Manuals)
DB2 9.5 Information Centerhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp
Administrative API Reference (SC23-5842-02)http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.common.doc/doc/r0008264.html
What is a log used for?• To record changes within the database
• Used to replay changes in the event of certain types of system failures
Chris Eaton has an excellent section on logging in his book “High Availability Guide for DB2”. Algorithm Chapter 2 has a lot of detail about how DB2 logging works and the algorithm DB2 uses for logging.
4
Understanding Log Configuration Parameters
20 + Configuration parameters related to logging
Log Configuration Parameters
db2 get db cfg for <dbname> | grep –i log
For instance,
db2 get db cfg for testdb | grep –i log
6
Default Configuration ParametersLog sizes
Log buffer size (4KB) (LOGBUFSZ) = 51Log file size (4KB) (LOGFILSIZ) = 1024
Number of primary log files (LOGPRIMARY) = 13Number of secondary log files (LOGSECOND) = 4
The output has been re-formatted a bit to make it easier to read.
7
How much log space do you need?
Depends on application characteristics
How many updates – inserts, updates, deletes does the application do?
Always need more than you guessed
BAD things happen if you run out
Information on calculating log spacehttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.dbobj.doc/doc/c0004930.html
8
Adjust space allocationsAdjust with:
Log file size (LOGFILSIZ)Number of primary log files (LOGPRIMARY)Number of secondary log files (LOGSECOND)
The output has been re-formatted a bit to make it easier to read.
9
Log File SizeLog file size (LOGFILSIZ) = 1024
In 4 KB unitsDefault is ONLY 1024
Change withdb2 update db cfg for <dbname> using LOGFILSZ <For exampledb2 update db cfg for <dbname> using LOGFILSZ 25000
Approximately 4 MB
Approximately 100 MB
Default is only 4,194,304 bytes! Approximately 4 MB
25000 * 4KB = 100,000 KB or 100 MB
10
Log File SizeRange is 1,000 – 1,048,572 (1,024 GB) 9.5 FP3
Needs to be larger if there are many insert, delete and/or update operations against the database
Too small - affects performance due to frequent archiving
Too large – might not fit on archive media
50,000 reasonable
50000 * 4KB = 200,000 KB / 1024 is approximately 200 MB
11
Number of log filesHow many logs?
Primary vs secondary
Total (primary + secondary) <= 256, unless logsecond set to -1 (infinite logging)
Could have logprimary = 100 and logsecond = 156, for instance
Logprimary = 150 and logsecond = 156 would not be possible, though.
12
Primary log files
Maximum number controlled by LOGPRIMARY
When allocated is determined by archive mode
Will affect database “start up” time
The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.
13
Secondary log files
Maximum number controlled by LOGSECOND
When allocated is determined by archive mode and behavior of application
The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.
14
Change number of log files
Primary logs are pre-allocated at database creation and when the database is “started”
Can affect database “start up” timedb2 update db cfg for <dbname> using LOGPRIMARY 150
Secondary logs are allocated when neededdb2 update db cfg for <dbname> using LOGSECOND 106
The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.logprimaryhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0000240.html
15
Maximum number of log files
Number of primary and secondary (logprimary + logsecond) log files <= 256,
unless logsecond = -1.
If you exceed 256 you get:
SQL5153N The update cannot be completed because the following relationship would be violated: "logprimary + logsecond <= 256".
The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.logprimaryhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0000240.html
SQL5153N http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.messages.sql.doc/doc/msql05153n.html
16
Infinite LoggingLOGSECOND = -1
Is this a good or bad idea?
Roll back and recovery can be quite slow
If the database is enabled for infinite logging , archive logging MUST be enabled.
During recovery it may be necessary to retrieve archive logs and bring them back to the active log path.
17
Check log space usageSELECTsubstr(DB_NAME,1,10) as DB_NAME,LOG_UTILIZATION_PERCENT,TOTAL_LOG_USED_KB,TOTAL_LOG_AVAILABLE_KB,TOTAL_LOG_USED_TOP_KB FROM SYSIBMADM.LOG_UTILIZATION
Using the Log_Utilization Administrative view to retrieve log utilization information
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.sql.rtn.doc/doc/r0022019.html
If using this with DPF select the DBPARTITIONNUM to indicate the partition that the log utilization applies to. The example above is for a single partition database.
18
Log Space Usage
Small log files in log directory?logs truncated when all applications disconnect from
database
Activate database to prevent this behavior
Prevent small files & speed up connections to database
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.sql.ref.doc/doc/r0024842.html
20
Default Configuration ParametersLog Paths
Changed path to log files (NEWLOGPATH) =Path to log files =
/local/db/db2inst1/db2inst1/NODE0000/SQL00002/SQLOGDIR/
Overflow log path (OVERFLOWLOGPATH) =Mirror log path (MIRRORLOGPATH) =First active log file =
The output has been re-formatted a bit to make it easier to read.
21
Log pathsVery common to change put logs in their own file
systemThat generally would involve changing the path
to the log files.Path to log files =
/local/db/db2inst1/db2inst1/NODE0000/SQL00002/SQLOGDIR
db2 update db cfg for <dbname> using NEWLOGPATH /local/db2inst1/db2inst1/logs
The output has been re-formatted a bit to make it easier to read.
22
Default Configuration ParametersLog Paths
Overflow log path (OVERFLOWLOGPATH)location for DB2 to find log files needed for a rollforward
operation. Prevents having to specify it in Rollforward command
Mirror log path (MIRRORLOGPATH)Contains a duplicate of the active log files
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0005826.htmlOverflow log paths can also be used with the db2ReadLog API to indicate where log files are stored.
Mirrorlogpathhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0005781.html
23
Default Configuration ParametersMiscellaneous
First active log file =Block log on disk full (BLK_LOG_DSK_FUL) = NOPercent max primary log space by transaction (MAX_LOG) = 0Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0Percent log file reclaimed before soft chckpt (SOFTMAX) = 520HADR log write synchronization mode (HADR_SYNCMODE) =
NEARSYNCLog pages during index build (LOGINDEXBUILD) = OFF
The output has been re-formatted a bit to make it easier to read.
24
Block log on disk fullBLK_LOG_DSK_FUL=YES
Configurable online (immediate)
Prevent log full condition – retry every 5 minutes to create new log
Will cause application to hang on log full condition – can resolve issue
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0005787.htmlhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.ha.doc/doc/c0006131.html
If set to NO, then the transaction will be rolled back and sometimes will cause a DB2 crash
25
Maximum log per transactionMAX_LOG=0 to 100
Configurable online (immediate)
0 means no limit
1-100 tell how much log space a transaction is allowed to use
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0006018.html
26
Soft checkpoint frequencySOFTMAX=1 to 100 * logprimary
Configurable
Default is 100
Percentage of the size of one primary log file
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0000242.html
27
Maximum log per transactionMAX_LOG=0 to 100
Configurable online (immediate)
0 means no limit
1-100 tell how much log space a transaction is allowed to use
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0006018.html
28
LOGINDEXBUILDIf set to YESthen
index creation, index recreation and reorg operations are logged.
The logs can be used during rollforward or HADR replay procedures.
The output has been re-formatted a bit to make it easier to read.
29
Default Configuration ParametersArchiving parameters
Log retain for recovery status = NOUser exit for logging status = NOFirst log archive method (LOGARCHMETH1) = OFFOptions for logarchmeth1 (LOGARCHOPT1) =Second log archive method (LOGARCHMETH2) = OFFOptions for logarchmeth2 (LOGARCHOPT2) =Failover log archive path (FAILARCHPATH) =Number of log archive retries on error (NUMARCHRETRY) = 5Log archive retry Delay (secs) (ARCHRETRYDELAY) = 20
Deprecated in 9.5
The output has been re-formatted a bit to make it easier to read.The Log Retain and User exit information are deprecated for DB2 9.5 servers, but is still used for pre 9.5 servers.
NUMARCHRETRY – The number of times that DB2 will try to archive a log file to the primary or secondary archive directory, before it attempts to archive to failover directory.
ARCHRETRYDELAY – number of seconds to wait after a failed archive attempt, before retrying the archive operation
30
Log Archive Modes• Circular• Archive logging
If logarchmeth1 and logarchmeth2 are set to OFF, then circular logging will be used. Rollforward recovery is NOT supported with circular logging, while crash recovery is.
31
Circular logging
• Rollforward logging is NOT supported
1 2 3 4
32
Archive logging
• Rollforward logging IS supported
Logarchmeth1 and Logarchmeth2 are set to OFF Rollforward recovery is supported with archive logging, as well as, crash recovery.
33
Default Configuration ParametersArchiving parameters
Log retain for recovery status = NOUser exit for logging status = NOFirst log archive method (LOGARCHMETH1) = OFFOptions for logarchmeth1 (LOGARCHOPT1) =Second log archive method (LOGARCHMETH2) = OFFOptions for logarchmeth2 (LOGARCHOPT2) =Failover log archive path (FAILARCHPATH) =Number of log archive retries on error (NUMARCHRETRY) = 5Log archive retry Delay (secs) (ARCHRETRYDELAY) = 20
Deprecated in 9.5
The Log Retain and User exit information are deprecated for DB2 9.5 servers, but is still used for pre 9.5 servers.
34
LOGARCHMETHnLOGARCHMETH1 and LOGARCHMETH2
What Archive Mode Am I Using?db2 get db cfg for <dbname> | grep –i logarchmeth
In pre 9.5 servers you could also look at LOGRETAIN and USEREXIT parameters
LOGRETAIN and USEREXIT–deprecated in 9.5, but still listed in db cfg
Logarchmeth1 and Logarchmeth2 are set to OFF Circular logging will be used. Rollforward recovery is not supported with circular logging, while crash recovery is.
36
What Archive Mode Am I Using?db2 get db cfg for <dbname> | grep –i logarchmeth
If Logarchmeth1 and Logarchmeth2 are set to OFF,then
Circular logging is used
Rollforward recovery is not supported with circular logging, only crash recovery.
37
Change Archive Log Modedb2 update db cfg for <dbname> using logarchmeth1
for example
db2 update db cfg for <dbname> using logarchmeth1DISK:/logs/archive/
The Log Retain and User exit information are deprecated for DB2 9.5 servers, but is still used for pre 9.5 servers.
db2 update db cfg for <dbname> using logarchmeth1TSM
38
Turn Archiving OffIn DB2 9.5db2 update db cfg for <dbname> using logarchmeth1OFF
If you getSQL5153N The update cannot be completed because the
following relationship would be violated: "logsecond can only be set to -1 if log archiving is enabled".
db2 update db cfg for <dbname> using logsecond 10
You cannot have infinite logging (logsecond = -1) if you are using circular logging, so you must set logsecond to a number before you can turn off archive logging.
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.messages.sql.doc/doc/msql05153n.html
39
Where’s My Userexit?
Deprecated in 9.5
Still in pre-9.5 servers
If setarchive logging ON regardless of LOGRETAIN
TSM logging• db2 update db cfg for <dbname> using logarchmeth1
TSM
Carolyn Sanders “TSM is not just a black boxヤ - Commands a DBA Should Know” gives a wealth of information about using TSM with DB2. Her presentation can be found at:https://www.idug.org/conferences/NA2008/data/NA08I03.pdf
41
Log layouts (What is in a log)
Log record types
• Log manager header• Long field manager log records• Transaction manager log records• Utility manager log records• Data manager log records
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0001910.html
43
Log record types
• Log manager header
• All log records begin with log manager header• Contains information like length of log, type
of log record, log sequence number (LSN)
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0024864.html
44
Log record types
• Long field manager log records
• Written whenever long field data is inserted, deleted, or updated
• Only written is archive logging
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0024865.html
45
Log record types
• Transaction manager log records
• Written when transaction complete• Timestamps are Coordinated Universal Time
(CUT)
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0024866.html
46
Log record types
• Utility manager log records• Log records are associated with :
• Migration• Load• Backup• Table space rollforward
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0024867.html
47
Log record types
• Data manager log records
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.sql.ref.doc/doc/r0024842.html
Log manager header layouthttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.sql.ref.doc/doc/r0024842.html
48
Log APIs and how to use them
49
Log APIs• db2ReadLogNoConnInit - Initialize reading
the database logs without a database connection
• db2ReadLogNoConn - Read the database logs without a database connection
• db2ReadLog - Extracts log records
50
Log APIs• db2ReadLog - Extracts log records
• db2ReadLogNoConnTerm - Terminate reading the database logs without a database connection
• db2ArchiveLog - Archive the active log file
51
SQLCA data structureSQL Communication area (SQLCA)
• Used by database manager to return error information
• Updated after every API call and SQL statement issued
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.apdv.api.doc/doc/r0001673.html
52
db2ReadLogNoConnInit • Initialize reading the database logs without a
database connection
• Allocate memory needed by db2ReadLogNoConn
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0009369.html
53
db2ReadLogNoConn• Read the database logs without a database
connection
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0009368.html
54
db2ReadLogExtracts log records orqueries Log Manager for current log state
information
Works with recoverable database logs ONLY
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0001673.html
55
db2ReadLogNoConnTermTerminate reading database logs without a
database connection
Deallocate memory used by db2ReadLogNoConn
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0009370.html
56
db2ArchiveLogArchive the active log file
• Closes and truncates the active log file
• If archiving enabled it also issues archive request
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0008957.html
ARCHIVE command (CLP)db2 archive log for database <dbname> for single partition databasesOrdb2 archive log for database <dbname> on dbpartitionnum <partition_number> for DPF databases
57
Compiling C programsCopy sample C programs
cd <instance owner home directory>/sqllib/samples/c/tar -cvf /tmp/myfile.tar *
To expand tar file
tar –xvf myfile.tar
It is safest to make a copy of the sample programs before starting to work with them.
By cd’ing to directory first, the tar file created will NOT try to expand several directories
tar -tf /tmp/myfile.tar Will list all the files in the tar file name myfile.tar
C++<instance owner home directory>/sqllib/samples/cpp directory
58
Setup the environment• To experiment with the samples it is best to create the
SAMPLE database
• The majority of the sample programs work with the SAMPLE database by default
Use make to prepare programs• For example
make tsinfo
• Creates an executable called tsinfo
• Execute (if in directory where executable stored)• ./tsinfo
“Compile” dblognoconn• make dblognoconn• Creates an executable called dblognoconn
DB2 APIs USED:db2CfgGet -- Get Configurationdb2CfgSet -- Set Configurationdb2ReadLog -- Asynchronous Read Logdb2ReadLogNoConnInit -- Read log without a db
connectiondb2ReadLogNoConnTerm -- Terminate read log without a
database connection
61
dblognoconn setup• make dblognoconn
• out of make you get • dblognoconn.o• dblognoconn.bnd• Dblognoconn
dblognoconn setup
• Make uses a file called makefile to control process• with dblognoconn, it runs
• bldapp dblognoconn sample
• because in the makefile ALIAS is the database name and sample is the default value assigned to ALIAS
• If you do• make dblognoconn.sqc
Agenda• What is a log used for? • Understanding log configuration
parameters• What's in a log - log file layouts• Log APIs and how to use them• How to setup a C environment to use
log APIs
ObjectivesObjective 1: What is a log used for? Objective 2: Understanding log configuration parametersObjective 3: What's in a log - log file layoutsObjective 4: Log APIs and how to use themObjective 5: How to setup a C environment to use log APIs
64
Paul Turpinpaul.turpin@s1.com
Session Code: C12
Paul Turpin is a database architect and database administrator at a software development company in Charlotte, NC. He specializes in DB2 for LUW on large systems. He enjoys exploring new features and functionality in DB2. He has spoken at IDUG North America, IDUG Europe, SHARE, IBM’s Information on Demand conference, and several RUGs. Paul currently serves as chairperson on the IDUG 2010 North America Conference Planning Committee.
65
Extra Material
Setting up CygwinCompiling DB2 UDB Stored Procedures with GCC
on Windows
http://www.ibm.com/developerworks/data/library/techarticle/0306haungs/0306haungs.html
67