B1: After Imaging New features. Greater flexibility. Richard Banville Fellow.
-
Upload
timothy-hamilton -
Category
Documents
-
view
239 -
download
6
Transcript of B1: After Imaging New features. Greater flexibility. Richard Banville Fellow.
B1: After Imaging
New features. Greater flexibility.
Richard BanvilleFellow
© 2008 Progress Software Corporation2
Agenda
Basics New features Online implementation Archival and Maintenance
© 2008 Progress Software Corporation3
Agenda
Basics• What is after imaging?
• Why use after imaging?
• How does it work?
• What is the AI archiver?
© 2008 Progress Software Corporation4
What is after imaging?
A journal of transaction data that can replay changes to a database
Referred to as a redo log Granularity of recovery
• To last completed transaction
• To a point in time
• To a specific transaction number.
Space is not reused without interaction• (as it is in the before image journal files)
© 2008 Progress Software Corporation5
Why use after imaging?
Protection• Removes single point of failure
• Media loss
• Human “error”– Table drop– File deletion
No data loss High availability & Business Continuity For replication
• Manual or OpenEdge Replication
© 2008 Progress Software Corporation6
How does it work?
Transaction data synchronously written to an after image (AI) journal
AI journal “replayed” in recovery scenarios Not automatically enabled Automation
• Some automatic maintenance features
• Recovery not automated
© 2008 Progress Software Corporation7
How the AI File Works
FOR EACH customer:
UPDATE customer.
END.
BI Note Written
AI Note Written
Transaction Begin
BI and AI Transaction End Notes Written
*** Data updated “sometime” later
© 2008 Progress Software Corporation8
Forward Processing - Physical View
DatabaseBefore Image
Shared Memory
After Image
After Image
After Image
12
3 + t
Full AI files must be managed
© 2008 Progress Software Corporation9
Forward Processing - Physical View
DatabaseBefore Image
Shared Memory
After Image
After Image
After Image
12
3 + t
© 2008 Progress Software Corporation10
The Recovery Process
Database Before Image
After Image
After Image
After Image
BackupRemote copy
1a1b
3 thru nroll forward
2Redo
© 2008 Progress Software Corporation11
Agenda
Online AI functionality Automatic AI file management
• AI archiver
New Features
© 2008 Progress Software Corporation12
New after imaging features (10.1A thru 10.1C)
Online AI functionality• Add extents
• Start/Stop (enable/disable)
• Automatic archiving– Archive extents– Change archive directory– Change archive interval
• Manage AI buffers
Guarded recovery process
Online features are a huge advance in AI’s flexibility
© 2008 Progress Software Corporation13
What is the AI archiver?
Daemon (AIMD) to manage/move “full” AI files• Timer mode (every <n> seconds)• On-demand mode (when full)• Archives & marks AI extents for reuse
Enablement• Offline via rfutil• Online via the probkup utility
Additional online features• Switch archive locations• Change archive interval
No support for archiving to a second host No support for single user activity
Introduced in 10.1A
© 2008 Progress Software Corporation14
Empty
AI Archiver
Shared Memory
DB BI A1
AI ArchiveDaemon
Archive 1
Archive 2
Archive 3
Knows when andwhere to archive
A2
Busy
Busy
FullEmpty
© 2008 Progress Software Corporation15
Agenda
Creating AI files Enabling AI Starting and managing the AI archiver
Implementing After Imaging Online
© 2008 Progress Software Corporation16
Creating AI files
New Syntax
Notes:• Ensure structure file correct before adding
• Defaults to 8K AI block size• Ensure default user permissions are correct
prior to adding extents
prostrct addonline <db> <st-file-name>
prostrct addonline <db> <st-file-name> -validate
Adding extents online
© 2008 Progress Software Corporation17
Creating AI files
Q: How many AI files to create?• A: More than one
• A2: Everyone has a different opinion
• 10 is a good starting point
Extent type: variable, fixed or vixed?• Depends on archive mechanism
• Variable – “easier” to maintain
• Fixed for “ultimate” performance– Makes some sense with fixed amount of work
© 2008 Progress Software Corporation18
Need to have:• AI files must reside on a different disk/LUN
than any other aspect of the database (db, dn, bn)
Nice to have:• AI archived to yet another disk/physical locale
– not always possible/plausible
Avoid I/O bottlenecks• Add more spindles if need be
Location, location, location
After Image File and Archive Locations
© 2008 Progress Software Corporation19
Enabling After Imaging Online
New option to the probkup utility
Notes:• Make sure your AI files are in place
• Common uses:– Oops, “forgot” to enable after imaging– AI off Recoverable maintenance AI on
probkup online <db> <backup-file> enableai
© 2008 Progress Software Corporation20
Managing AI buffers online (10.1C)
New option
Notes:• For performance, set aibufs >= bi buf pool size
– Online default same as -bibufs (8K AI buffers)
• Permissions for new shared memory
proutil <db> increase -aibufs <n>
© 2008 Progress Software Corporation21
Offline:
• AI must already be enabled ;(
Online:
• AI can be enabled in conjunction ;)
Enabling the AI archiver
rfutil <db> -C aiarchiver enable
probkup online <db> <backup> [ enableai ] enableaiarchiver -aiarcdir <dir> -aiarcinterval <n-seconds> [ -aiarcdircreate ]
© 2008 Progress Software Corporation22
New Syntax:
Can optionally create directories on the fly
Again, watch permissions
Changing the Archive Directory Online
rfutil <db> -C aiarchiver setarcdir <dir-list>
rfutil <db> -C aiarchiver setarcdir <dir-list>/ -aiarcdircreate
© 2008 Progress Software Corporation23
Changing Archive Interval Online
New Syntax:
Valid range 120 to 86,400 (in seconds)• 2 minutes to 1 day
• Suggest 15 to 60 minutes based on activity
Common uses• Correct mistake of minutes vs seconds
• Change in business constraints (lost data)
Archiver awakes every 5 seconds
rfutil <db> -C aiarchiver setinterval <#>
Forcing an extent switch automatically
© 2008 Progress Software Corporation24
Changing Archive Mode Online
On-demand Mode• No interval or interval 0
• Use with fixed length extents or manual switch
Timed Mode• Interval set 120 to 86,400 (in seconds)
• Archives regardless of activity
Notes:• Change time and/or on-demand during off hours
• Timed must be set at each invocation
• Archiver still awakes every 5 seconds
rfutil <db> -C aiarchiver setinterval <#>
On-demand vs timed archival
© 2008 Progress Software Corporation25
Agenda
Monitoring your AI files Archive log naming Backing up AI archive files Cleaning up AI archive files
Archiving and Maintenance
© 2008 Progress Software Corporation26
Manual AI Archival
Manually archive AI files while db is online Check for full extents Force extent switch Find next full extent
to archive Rinse, lather, repeat Example code: http://
wss.com/products/downloads.html
No archiver? No problem.
rfutil <db> -C aimage new
rfutil <db> -C aimage full
rfutil <db> -C aimage list
© 2008 Progress Software Corporation27
Not user definable (it should be) Naming convention
Recovery easily managed• Files relate to a particular backup
• Date and time
• Extent sequence number in the name
Directory separator “/” replaced with “~” Example:
Archive Log Naming
<db> . date . time . backup-seq . ai-seq . ext-name
usr1~richb~x.20080806.151638.00003.00000001.x.a1
© 2008 Progress Software Corporation28
Backup of Archives
Increase types of recoverable disasters How often
• Throughout the day
Where to• Separate media for media failure protection
• Secondary machine as quickly as possible
• Best if archival system is “very” far away
NOT included in the OpenEdge database backup
© 2008 Progress Software Corporation29
Cleaning Up Archives
Eventual disk space issue Retention dependant on business needs Clean up
• Create an automated process to clean up archives after a period of time (days)
• Keep at least 7 days of archives online• How often do you backup?
– Archives without a backup is useless• Where are they stored?• Maintaining a hot standby?
Archiver retains archives forever!
© 2008 Progress Software Corporation30
Cleaning Up Archives - Unix/Linux
The find command is very useful for this task
# Archive Cleaner# Should be run from cron daily# Test yourself with echo before implementingARCDIR=<your-archive-directory>KEEPDAYS=+30
find $ARCDIR -mtime $KEEPDAYS -exec \rm {} \ 2>&1 > /dev/null
• Beware of changes in $ARCDIR
• Consider e-mail rather than /dev/null
• Incorporate remote copy
• Add automatic validation
© 2008 Progress Software Corporation31
Cleaning Up Archives - Windows
Uses file from the free windows resource kit File name: %WINDOWS%\system32\forfiles.exe
REM ** Archive cleanup batch fileREM ** Have this run as a scheduled taskREM ** Test yourself with echo before implementingSet ARCDIR=your-archive-directorySet KEEP=-30
forfiles /P %ARCDIR% /m *.a* /D %KEEP% /C “cmd /C del /q @FILE”
REM OR
forfiles -p%ARCDIR% -m*.a* -d%KEEP% -c“cmd /c del /q @FILE"
//technet.microsoft.com/en-us/library/cc755872.aspx
© 2008 Progress Software Corporation32
Disaster Scenarios
Floods Fire
Bad stuff happens
© 2008 Progress Software Corporation33
Using auto AI archives to recover
In the AI archiver log “<db>.archival.log”• Find most recent backup to restore
– Look for “BACKUP_SET”
• Identify AI extent to start roll forward with– Match “aiseq” value to archived AI extent
Ignore AI extent’s backup sequence value
– Roll forward it and all later archives
Log file layout documented in DB Admin Guide Example: …
It is quite easy
© 2008 Progress Software Corporation34
AI Example
# OpenEdge RDBMS After-image Extent Manager Archival Log File
# Do not edit this file with an editor as it is automatically
# maintained by the OpenEdge RDBMS after-image extent manager.
0032,/usr1/db/x,20080807,100553,1,2,20080807,100551,1,1,x.bak,BACKUP_SET_20080807_100551
#
# After-image Extent Manager started on : Thursday August 07, 2008 at 10:05:56
#
0255,20080807,100556,100100
0001,/usr1/db/x,20080807,100853,20080807,100552,2,1,/usr1/db/x.a1,/usr2/arcdir1,/usr2arcdir1/usr1~db~x.20080807.100552.00002.00000001.x.a1
0001,/usr1/dbx,20080807,100908,20080807,100552,2,2,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00002.00000002.x.a2
0032,/usr1/db/x,20080807,100947,1,3,20080807,100946,4,1,../x.bak2,BACKUP_SET_20080807_100946
0001,/usr1/db/x,20080807,100949,20080807,100552,3,3,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000003.x.a3
0001,/usr1/db/x,20080807,101029,20080807,100552,3,4,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000004.x.a4
0001,/usr1/db/x,20080807,101355,20080807,100552,3,5,/usr1/db/x.a1,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000005.x.a1
0001,/usr1/db/x,20080807,101441,20080807,100552,3,6,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000006.x.a2
0001,/usr1/db/x,20080807,101831,20080807,100552,3,7,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000007.x.a3
0001,/usr1/db/x,20080807,102037,20080807,100552,3,8,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000008.x.a4
© 2008 Progress Software Corporation35
A Eye Exam(ple)
# OpenEdge RDBMS After-image Extent Manager Archival Log File
# Do not edit this file with an editor as it is automatically
# maintained by the OpenEdge RDBMS after-image extent manager.
0032,/usr1/db/x,20080807,100553,1,2,20080807,100551,1,1,x.bak,BACKUP_SET_20080807_100551
#
# After-image Extent Manager started on : Thursday August 07, 2008 at 10:05:56
#
0255,20080807,100556,100100
0001,/usr1/db/x,20080807,100853,20080807,100552,2,1,/usr1/db/x.a1,/usr2/arcdir1,/usr2arcdir1/usr1~db~x.20080807.100552.00002.00000001.x.a1
0001,/usr1/dbx,20080807,100908,20080807,100552,2,2,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00002.00000002.x.a2
0032,/usr1/db/x,20080807,100947,1,3,20080807,100946,4,1,../x.bak2,BACKUP_SET_20080807_100946
0001,/usr1/db/x,20080807,100949,20080807,100552,3,3,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000003.x.a3
0001,/usr1/db/x,20080807,101029,20080807,100552,3,4,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000004.x.a4
0001,/usr1/db/x,20080807,101355,20080807,100552,3,5,/usr1/db/x.a1,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000005.x.a1
0001,/usr1/db/x,20080807,101441,20080807,100552,3,6,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000006.x.a2
0001,/usr1/db/x,20080807,101831,20080807,100552,3,7,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000007.x.a3
0001,/usr1/db/x,20080807,102037,20080807,100552,3,8,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000008.x.a4
© 2008 Progress Software Corporation36
A Eye Exam(ple)
# OpenEdge RDBMS After-image Extent Manager Archival Log File
# Do not edit this file with an editor as it is automatically
# maintained by the OpenEdge RDBMS after-image extent manager.
0032,/usr1/db/x,20080807,100553,1,2,20080807,100551,1,1,x.bak,BACKUP_SET_20080807_100551
#
# After-image Extent Manager started on : Thursday August 07, 2008 at 10:05:56
#
0255,20080807,100556,100100
0001,/usr1/db/x,20080807,100853,20080807,100552,2,1,/usr1/db/x.a1,/usr2/arcdir1,/usr2arcdir1/usr1~db~x.20080807.100552.00002.00000001.x.a1
0001,/usr1/dbx,20080807,100908,20080807,100552,2,2,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00002.00000002.x.a2
0032,/usr1/db/x,20080807,100947,1,3,20080807,100946,4,1,../x.bak2,BACKUP_SET_20080807_100946
0001,/usr1/db/x,20080807,100949,20080807,100552,3,3,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000003.x.a3
0001,/usr1/db/x,20080807,101029,20080807,100552,3,4,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000004.x.a4
0001,/usr1/db/x,20080807,101355,20080807,100552,3,5,/usr1/db/x.a1,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000005.x.a1
0001,/usr1/db/x,20080807,101441,20080807,100552,3,6,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000006.x.a2
0001,/usr1/db/x,20080807,101831,20080807,100552,3,7,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000007.x.a3
0001,/usr1/db/x,20080807,102037,20080807,100552,3,8,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000008.x.a4
Backup file-name, BACKUP_SET
AI Seq #4
AI Seq #4
AI file-name to start rollforward
AI Seq #4
© 2008 Progress Software Corporation37
Using auto AI archives to recover
Documentation issue: buseq aiseq Wrong AI file? No problem.
• Error message reported
• Retry with the correct one
Roll forward last active/not archived AI extent Always use AI sequence value
• Archive location within .log file is “relatively” meaningless w/respect to its BACKUP_SET
• WHY? …
Some notes on the AI archive log
© 2008 Progress Software Corporation39
Why AI Log is NOT Sequential
A1A1
DB BI
AI ArchiveDaemon
Archive 1Archive 2
Archive 3A2A2Users
Backup
DB.bakDB.bak
Archive Log
Extent S
witch
…with respect to the BACKUP_SET
No sync on
write order
BusyFullEmpty
EmptyBusy
© 2008 Progress Software Corporation40
Recovery Process Overview
Get everyone off the system (crons too!) Backup what you have (if anything)
• Database (even just parts)
• Recovery files (AI, BI,TL)
• AI Archives
Delete broken database Restore most recent backup Apply after image archives
rfutil <db> -C roll forward oplock –a <aifile> (10.1c)
© 2008 Progress Software Corporation41
Extending Archival Usefulness
Log-based replication
© 2008 Progress Software Corporation42
Log-Based Replication
Hot standby database• Good to use for backups
– Eliminates backup I/O on production– Use -norecover option with probkup
• Less recovery downtime
• Cannot be updated (subsequent rollf. will fail)
Hot standby copy (#2)• Provide for reporting (semi-stale)
• “Midnight” copy of hot standby #1
Extending archival usefulness
© 2008 Progress Software Corporation43
Log-Based Replication
Auto archived files• Deal w/awkward naming convention
• Create sub-directory under archive directory
Process looks for new archives• Applies new archives to hot-standby database
• Moves applied archives
Best if on a remote host• Less contention w/production
• Improved disaster recovery strategy
Extending archival usefulness
© 2008 Progress Software Corporation44
Summary
All OpenEdge systems should use AI Many new online AI capabilities AI management tool provided Use archives to maintain a hot-standby
• Improved high availability
© 2008 Progress Software Corporation45
Questions?
© 2008 Progress Software Corporation46
Outstanding Questions Answered
Does ai archiver use ai extract or simply copy?• It uses Ai Extract.
Is ai archiver available for Workgroup licenses?• Yes. It is functionality and not performance related.
You can specify several archive directories. Is there a way to tell which is the current archive directory?
• There is a message in the log when a dir switch is performed. There is no way to programmatically determine this. I will add an enhancement request.
When ai is locked due to replication, is the file archived and later marked empty or does the archive process stall waiting for the locked extent to become unlocked?
• The extent is archived. Once it is “unlocked” it is emptied by the watch dog. The aimgt deamon also attempts to empty locked extents.
Is there any documentation on using ai archiver with replication?• Yes, there is a callout in the ai mgt doc when using it with replication. The doc
explains what happens to an extent with ai mgt and repl running. Is it safe to change permissions of a file after add online completes?
• No. Although add online has completed, all users connected to the database have not opened the new file. They open it the next time they access it. Changing its permissions while the database is still running may cause the database to shutdown when a user attempts to open a file it does not have permission to.
© 2008 Progress Software Corporation47
Thank You
© 2008 Progress Software Corporation48