Preparing for High Availability Adam Backman [email protected] V.P. of Technology White Star Software.

Preparing for High AvailabilityPreparing for High Availability

Adam BackmanAdam Backman

[email protected]@wss.com

V.P. of TechnologyV.P. of Technology

White Star SoftwareWhite Star Software

What We Will CoverWhat We Will Cover

Planning considerationsPlanning considerations Installation issuesInstallation issues Maintenance issuesMaintenance issues

Planning Phase - PeoplePlanning Phase - People

Who “owns” the dataWho “owns” the data Be inclusive Be inclusive This is not solely an IT decisionThis is not solely an IT decision Eliminate surprisesEliminate surprises

Planning ConsiderationsPlanning Considerations

Budget – high availability is not freeBudget – high availability is not free Hardware – fault tolerant, redundancy, …Hardware – fault tolerant, redundancy, … Software – Progress is good but how is your Software – Progress is good but how is your

“other” software?“other” software? Knowledge – buy or rentKnowledge – buy or rent Time – schedule and outage timeTime – schedule and outage time Personnel constraints – Who is on call?Personnel constraints – Who is on call?

Goals During OutageGoals During Outage

Do no additional damageDo no additional damage Shortest amount of timeShortest amount of time Reduce/Eliminate impact to customerReduce/Eliminate impact to customer

The Cost of DowntimeThe Cost of Downtime

WagesWages Idle workersIdle workers Cost to replace dataCost to replace data

ProductionProduction Lost productionLost production

Impact to the customerImpact to the customer Can’t click websiteCan’t click website Can’t place orderCan’t place order

How Much Downtime Can You Afford?How Much Downtime Can You Afford?

For maintenance For maintenance ApplicationApplication DatabaseDatabase

For failuresFor failures HardwareHardware SoftwareSoftware Natural disasterNatural disaster

Planning Phase - BudgetPlanning Phase - Budget

Less downtime = additional costLess downtime = additional cost Better disks (RAID, Mirrors, EMC, …)Better disks (RAID, Mirrors, EMC, …) Redundant systemRedundant system Remote siteRemote site

More money does not equal less downtimeMore money does not equal less downtime PrioritizePrioritize Look for most likely scenariosLook for most likely scenarios Look beyond coolLook beyond cool

Planning Phase - HardwarePlanning Phase - Hardware

Disks – The only moving partDisks – The only moving part RAID – RAID – RRedundant edundant AArray rray IInexpensive nexpensive DDisksisks Avoid software mirroringAvoid software mirroring Use multiple controllersUse multiple controllers Try to stick with a 1 vendor solutionTry to stick with a 1 vendor solution

What RAID really meansWhat RAID really means

RAID has many levels, here are the most RAID has many levels, here are the most commoncommon

RAID 0: This level is also called RAID 0: This level is also called striping.striping. RAID 1: This is referred to as RAID 1: This is referred to as mirroringmirroring.. RAID 5: Poor performance RAID levelRAID 5: Poor performance RAID level RAID 10: This is mirroring and striping. RAID 10: This is mirroring and striping.

Also known as RAID 0 + 1Also known as RAID 0 + 1


CPUCPUCheck with vendor to ensure fault toleranceCheck with vendor to ensure fault tolerance

MemoryMemoryDo not interleave memoryDo not interleave memory

VendorVendorChoose a reliable vendor (IBM, HP, Sun, Choose a reliable vendor (IBM, HP, Sun,

Compaq, …)Compaq, …)


Other hardwareOther hardware File serversFile servers Network stuff (LAN & WAN)Network stuff (LAN & WAN) Phone/Internet connectionsPhone/Internet connections

Planning Phase - SoftwarePlanning Phase - Software

Inventory all software (client and server) Inventory all software (client and server) and make sure it is current and supportedand make sure it is current and supported

Determine what software is needed all of Determine what software is needed all of the time (Production control – Yes, the time (Production control – Yes, Reporting software – No)Reporting software – No)

Planning Phase - ProgressPlanning Phase - Progress

Version of Progress (look for patches)Version of Progress (look for patches) Layout of databaseLayout of database

Single database or Multi-databaseSingle database or Multi-database Storage area layout (logical and physical Storage area layout (logical and physical

layout)layout) Application issuesApplication issues

Client/Server, N-Tier or Host basedClient/Server, N-Tier or Host based Where does the application code reside?Where does the application code reside?

Planning Database LayoutPlanning Database Layout

Single databaseSingle database Easy to maintainEasy to maintain Still have storage areas to spread dataStill have storage areas to spread data Single point of failureSingle point of failure

Multi-databaseMulti-database More to maintainMore to maintain Allows application partitioning Allows application partitioning Maintenance flexibility Maintenance flexibility Two phase commitTwo phase commit

After ImagingAfter Imaging

Before image files keep information about Before image files keep information about records giving you the ability to undo a records giving you the ability to undo a transactiontransaction

After image files keep information about After image files keep information about records that allows you to redo a transaction records that allows you to redo a transaction in the event of media failurein the event of media failure

After imaging is only part of a high After imaging is only part of a high availability strategyavailability strategy


Every high availability system should have Every high availability system should have after imaging enabledafter imaging enabled

Multiple after image areas are required for Multiple after image areas are required for high availabilityhigh availability

Only enable after imaging after you have a Only enable after imaging after you have a comprehensive backup and recovery plan in comprehensive backup and recovery plan in placeplace

How Does Journaling Work?How Does Journaling Work?

Here is an logical over-simplification of how Here is an logical over-simplification of how journaling worksjournaling works

FOR EACH customer: FOR EACH customer:

BI Note writtenBI Note written

UPDATE customer.UPDATE customer.

AI Note writtenAI Note written

END. END.

Planning Phase - KnowledgePlanning Phase - Knowledge

OwnOwnOur people have the knowledge to do the projectOur people have the knowledge to do the project

BuyBuyWe can train our people to do this projectWe can train our people to do this project

RentRentWe will hire consultants to implement this for usWe will hire consultants to implement this for us

(Insert shameless plug here)(Insert shameless plug here)

Planning Phase - TimePlanning Phase - Time

Schedule for projectSchedule for project Machine purchase and deliveryMachine purchase and delivery Software availabilitySoftware availability Resource availabilityResource availability Do we need a long weekend for Do we need a long weekend for

implementation?implementation?

Timings determined later may determine Timings determined later may determine implementation schedule itemsimplementation schedule items

Planning Phase - PersonnelPlanning Phase - Personnel

24 hr. Operators24 hr. OperatorsIf you don’t have operators you will need to If you don’t have operators you will need to

develop monitoring routines with paging abilitydevelop monitoring routines with paging ability Database Administrator(s) Database Administrator(s) System Administrator(s)System Administrator(s)

Develop an escalation plan with “on call” Develop an escalation plan with “on call” schedule for off hours issuesschedule for off hours issues

Installation PhaseInstallation Phase

All items should have been already All items should have been already developed and tested prior to this stagedeveloped and tested prior to this stage



Get the point?Get the point?

Installation StepsInstallation Steps

Develop a schedule with timings and leave Develop a schedule with timings and leave room for error as there WILL be errorsroom for error as there WILL be errors

Write scripts to do tasks where possible to Write scripts to do tasks where possible to eliminate the human factoreliminate the human factor

Have a master checklist with the person/ Have a master checklist with the person/ people responsible for each itempeople responsible for each item

Maintenance GoalsMaintenance Goals

Provide consistent performanceProvide consistent performance Allow to advanced planningAllow to advanced planning Avoid unscheduled outagesAvoid unscheduled outages

MaintenanceMaintenance

Don’t design something you cannot supportDon’t design something you cannot support Scripting should be flexible but bulletproofScripting should be flexible but bulletproof

Example: www.peg.com/utilities.htmlExample: www.peg.com/utilities.html Monitoring and trending are very important Monitoring and trending are very important

to maintain high availability systemsto maintain high availability systems

MonitoringMonitoring

Areas of concern for high availabilityAreas of concern for high availability ProgressProgress

Database areas fillingDatabase areas filling BI not being reusedBI not being reused AI space depleted AI space depleted Running out of licensesRunning out of licenses

SystemSystem Disk spaceDisk space Resources (memory, CPU, tunables, …)Resources (memory, CPU, tunables, …)

Monitoring Progress - DBMonitoring Progress - DB

/* Storage Area fill rate program *//* Storage Area fill rate program */

DEF VAR percent-free as DEC FORMAT ">9.99".DEF VAR percent-free as DEC FORMAT ">9.99".

FOR EACH _AreaStatus:FOR EACH _AreaStatus:

percent-free = 100 - ((_AreaStatus-HiWater / percent-free = 100 - ((_AreaStatus-HiWater /

_AreaStatus-TotBlocks * 100))._AreaStatus-TotBlocks * 100)).

DISPLAY _AreaStatus-areaname "Percent Free:" percent-free . DISPLAY _AreaStatus-areaname "Percent Free:" percent-free .

Monitoring Progress - BIMonitoring Progress - BI

/* Last BI file growth program *//* Last BI file growth program */

DEF VAR t_filename AS c FORMAT "x(40)".DEF VAR t_filename AS c FORMAT "x(40)".

t_filename = pdbname(1) + ".b". t_filename = pdbname(1) + ".b".

FIND LAST _ActIOFile WHERE _IOFile-filename BEGINS t_filename.FIND LAST _ActIOFile WHERE _IOFile-filename BEGINS t_filename.

IF _IOfile-Extends = 0 THEN IF _IOfile-Extends = 0 THEN

DISPLAY "ALL IS WELL".DISPLAY "ALL IS WELL".

ELSE ELSE

DISPLAY "The Sky is Falling !!!".DISPLAY "The Sky is Falling !!!".

Monitoring Progress - AIMonitoring Progress - AI

# Program: After image extent full checker# Program: After image extent full checker

FULL_EXT=`rfutil $DB -C aimage extent list | grep -i full | wc -l`FULL_EXT=`rfutil $DB -C aimage extent list | grep -i full | wc -l`

if [ $FULL_EXT -lt 9 ]if [ $FULL_EXT -lt 9 ]

thenthen

echo “$DB has $FULL_EXT full extents STATUS – OK”echo “$DB has $FULL_EXT full extents STATUS – OK”

elseelse

echo “WARNING - $DB has $FULL_EXT full extents” echo “WARNING - $DB has $FULL_EXT full extents”

fifi

Monitoring Progress - UsersMonitoring Progress - Users

/* License count tester *//* License count tester */

DEF VAR remaining-licenses AS INT.DEF VAR remaining-licenses AS INT.

FIND _license.FIND _license.

remaining-licenses = _Lic-ValidUsers - _Lic-MaxActive.remaining-licenses = _Lic-ValidUsers - _Lic-MaxActive.

/* You may want to use _Lic-ActiveConns instead of _Lic-MaxActive *//* You may want to use _Lic-ActiveConns instead of _Lic-MaxActive */

IF .10 > (remaining-licenses / _Lic-ValidUsers) THENIF .10 > (remaining-licenses / _Lic-ValidUsers) THEN

DISPLAY "Less than 10% of licenses remaining" DISPLAY "Less than 10% of licenses remaining"

WITH FRAME X.WITH FRAME X.

ELSEELSE

DISPLAY "More than 10% of licenses remaining" DISPLAY "More than 10% of licenses remaining"

WITH FRAME Y.WITH FRAME Y.

System MonitoringSystem Monitoring

Disk SpaceDisk Space How much disk available for growthHow much disk available for growth Also look at throughput capacity (average wait)Also look at throughput capacity (average wait)

Memory capacityMemory capacity Free memory is not a good indicatorFree memory is not a good indicator I focus on the scan rateI focus on the scan rate

CPU CapacityCPU Capacity How much idle timeHow much idle time

Maintenance TasksMaintenance Tasks

Backup and restoreBackup and restore After imagingAfter imaging Log based replicationLog based replication Data maintenanceData maintenance

Backup and RestoreBackup and Restore

Progress online backupProgress online backup Quiet point backupQuiet point backup Warm standby backupWarm standby backup

Backup and RestoreBackup and Restore

Why can’t I just backup the database and Why can’t I just backup the database and before image files while the database is at a before image files while the database is at a slow point?slow point?

Answer: The database consists of three Answer: The database consists of three portions while it is up and those are: The portions while it is up and those are: The database files, the before image file(s) and database files, the before image file(s) and memorymemory

Portions of an Active DBPortions of an Active DB

Shared memory

DB BI

Shared memory holds the most volatile data

The database contains older committed data

The before image holds transaction information

All three are needed for a

complete backup

Online BackupOnline Backup

What happens during an online backup?What happens during an online backup?1.1. Grab a db latchGrab a db latch

2.2. Do a pseudo-checkpoint (this synchs memory Do a pseudo-checkpoint (this synchs memory to disk)to disk)

3.3. Switch AI file (if necessary)Switch AI file (if necessary)

4.4. Backup the before image fileBackup the before image file

5.5. Release the db latchRelease the db latch

6.6. Backup the database (starting at the end)Backup the database (starting at the end)

Quiet PointsQuiet Points

Very little impact to system availabilityVery little impact to system availability Allows for integration with hardware Allows for integration with hardware

utilities utilities Only way to get an online backup with an Only way to get an online backup with an

operating system utility without shutting operating system utility without shutting down the brokerdown the broker

How quiet points work.How quiet points work.

Get database latchGet database latch do pseudo checkpoint do pseudo checkpoint wait for quiet point to be removedwait for quiet point to be removed

NOTE: All processing will wait for the quiet NOTE: All processing will wait for the quiet point to be removed point to be removed

Quiet Point BackupQuiet Point Backup

How to do a quiet point backupHow to do a quiet point backup1.1. Enable the quiet point (This synchs memory Enable the quiet point (This synchs memory

to disk)to disk)2.2. Synchronize your disk mirrorsSynchronize your disk mirrors3.3. Split your disk mirrorsSplit your disk mirrors4.4. Disable the quiet pointDisable the quiet point5.5. Mount the mirrors as different file systemsMount the mirrors as different file systems6.6. Backup your mounted mirrors with an OS Backup your mounted mirrors with an OS

utility (tar, cpio, fdumputility (tar, cpio, fdump, …), …)


Every high availability system should have Every high availability system should have after imaging enabledafter imaging enabled

Only enable after imaging after you have a Only enable after imaging after you have a comprehensive backup and recovery plan in comprehensive backup and recovery plan in placeplace

AI is sometimes referred to as the redo logAI is sometimes referred to as the redo log

Multi-volume after image filesMulti-volume after image files

Not a backup but a journal of completed Not a backup but a journal of completed transactionstransactions

Can be used to keep a copy of the database Can be used to keep a copy of the database up to dateup to date

Can be switched with no interruption to Can be switched with no interruption to user processinguser processing

Should part of every high availability Should part of every high availability environmentenvironment

How to integrate after imagingHow to integrate after imaging

In conjunction with a backup siteIn conjunction with a backup site To update a report serverTo update a report server As a means of backupAs a means of backup

AI to update a backup siteAI to update a backup site

Poor man’s replicationPoor man’s replication Allows for periodic update of a copy of the Allows for periodic update of a copy of the

databasedatabase The copy can then be backed up with a The copy can then be backed up with a

conventional backup mechanismconventional backup mechanism

Log Based ReplicationLog Based Replication

Log based replication is another way to say Log based replication is another way to say applying AI files to a copy of your databaseapplying AI files to a copy of your database

Excellent way to maintain a warm copy of Excellent way to maintain a warm copy of your database for fail overyour database for fail over

Can be used on the same machine or on a Can be used on the same machine or on a remote machine for additional protectionremote machine for additional protection

Log Based Replication RulesLog Based Replication Rules

The standby database can only be accessed The standby database can only be accessed read-only (-RO) which means no remote read-only (-RO) which means no remote (client/server) connections to the standby (client/server) connections to the standby datadata

You must have a multi-volume AI. This is You must have a multi-volume AI. This is a must for high availability in any casea must for high availability in any case

The standby database can have a different The standby database can have a different structure than the primary datastructure than the primary data

AI as a Means of BackupAI as a Means of Backup

Not generally a good ideaNot generally a good idea• Increased recovery timeIncreased recovery time• Reduced reliability Reduced reliability

Backup the database each weekendBackup the database each weekend Backup the AI file(s) each weeknight Backup the AI file(s) each weeknight

Backup – Points to RememberBackup – Points to Remember

Simplicity and minimizing user interaction Simplicity and minimizing user interaction will increase backup reliabilitywill increase backup reliability

You are only as good as your last tested You are only as good as your last tested backupbackup

Archiving off site is essentialArchiving off site is essential

Database MaintenanceDatabase Maintenance

Data StuffData Stuff Table moveTable move Database analysisDatabase analysis

Index StuffIndex Stuff Index rebuild (offline)Index rebuild (offline) Index CompressIndex Compress Index FixIndex Fix

Table MoveTable Move

ProsPros SimpleSimple Bullet proofBullet proof

ConsCons SlowSlow Table is read only for the duration of the moveTable is read only for the duration of the move Uses tons of logging space Uses tons of logging space

Table MoveTable Move

Syntax:Syntax:

proutil dbname –C tablemove tablename proutil dbname –C tablemove tablename table-area [index-area]table-area [index-area]

Table-area = The target application data area into which the Table-area = The target application data area into which the table is to be movedtable is to be moved

Index-area = The name of the target index area, if not Index-area = The name of the target index area, if not specified the indexes will be left in there existing locationspecified the indexes will be left in there existing location

Database AnalysisDatabase Analysis

Useful tool for determining low level Useful tool for determining low level storage informationstorage information

Helpful for determining records per block Helpful for determining records per block in storage areasin storage areas

Help determine when to compress/rebuild Help determine when to compress/rebuild indexes indexes

Database AnalysisRecord LevelDatabase AnalysisRecord Level

RECORD BLOCK SUMMARY FOR AREA "Schema Area" : 6RECORD BLOCK SUMMARY FOR AREA "Schema Area" : 6-------------------------------------------------------------------------------------------------------------- -Record Size (B)- ---Fragments--- Scatter-Record Size (B)- ---Fragments--- ScatterTable Table Records Size Records Size Min Max Mean Count Factor FactorMin Max Mean Count Factor FactorPUB.agedar 26 PUB.agedar 26 871.0B 31 41 33 871.0B 31 41 33 26 1.0 1.726 1.0 1.7PUB.customer 33 PUB.customer 33 5.7K 5.7K 159 196 175 33 1.0 0.9159 196 175 33 1.0 0.9PUB.item 55 PUB.item 55 4.5K 4.5K 73 95 83 73 95 83 55 1.0 1.155 1.0 1.1PUB.monthly 20 PUB.monthly 20 798.0B 798.0B 37 42 39 37 42 39 20 1.0 1.020 1.0 1.0PUB.order 20 PUB.order 20 2.2K 2.2K 98 138 113 98 138 113 20 1.0 1.220 1.0 1.2PUB.order-line 71 PUB.order-line 71 2.1K 2.1K 29 31 30 29 31 30 71 1.0 1.071 1.0 1.0PUB.salesrep 3 PUB.salesrep 3 219.0B 71 75 73 219.0B 71 75 73 3 1.0 1.03 1.0 1.0PUB.shipping 250 PUB.shipping 250 5.7K 5.7K 18 24 23 18 24 23 250 1.0 1.0250 1.0 1.0PUB.state 51 PUB.state 51 1.7K 1.7K 29 40 34 29 40 34 51 1.0 1.151 1.0 1.1PUB.syscontrol 1 PUB.syscontrol 1 134.0B 134.0B 134 134 134 1 1.0 1.0134 134 134 1 1.0 1.0

Database AnalysisIndex LevelDatabase AnalysisIndex Level

INDEX BLOCK SUMMARY FOR AREA "Schema Area" : 6INDEX BLOCK SUMMARY FOR AREA "Schema Area" : 6--------------------------------------------------------------------------------------------------------------Table Index Fields Levels Blocks Size % Util FactorTable Index Fields Levels Blocks Size % Util FactorPUB.agedar PUB.agedar ar_cust 8 ar_cust 8 1 1 1 194.0B 4.8 1.01 1 1 194.0B 4.8 1.0 ar_inv 9 ar_inv 9 1 1 1 242.0B 5.9 1.01 1 1 242.0B 5.9 1.0 ar_invdat 10 ar_invdat 10 1 1 1 55.0B 1.4 1.01 1 1 55.0B 1.4 1.0PUB.customer PUB.customer cust-num 11 cust-num 11 1 1 1 305.0B 7.5 1.01 1 1 305.0B 7.5 1.0 name 12 name 12 1 1 1 809.0B 19.9 1.01 1 1 809.0B 19.9 1.0 zip 13 zip 13 1 1 1 326.0B 8.0 1.01 1 1 326.0B 8.0 1.0PUB.item PUB.item idesc 14 idesc 14 1 1 1 879.0B 21.6 1.01 1 1 879.0B 21.6 1.0 item-num 15 item-num 15 1 1 1 503.0B 12.4 1.01 1 1 503.0B 12.4 1.0

Index RebuildIndex Rebuild

Used to be the only way to repair and Used to be the only way to repair and reorganize indexesreorganize indexes

Provides a positive performance impact in Provides a positive performance impact in many casesmany cases

Off line – database only available in read Off line – database only available in read only modeonly mode

Index CompressIndex Compress

The best new feature in version 9The best new feature in version 9 Reorganizes indexes while the database is Reorganizes indexes while the database is

onlineonline Allows you to specify compaction levelAllows you to specify compaction level Possible performance gain with no Possible performance gain with no

downtimedowntime

Index fixIndex fix

Combine with index compress to get all of Combine with index compress to get all of the benefits of an index rebuildthe benefits of an index rebuild

Corrects problems with the index (missing Corrects problems with the index (missing and extra entries)and extra entries)

Works online with little interruption to user Works online with little interruption to user activityactivity

Database Maintenance Points to RememberDatabase Maintenance Points to Remember

Progress provides many utilities, more are Progress provides many utilities, more are moving online with each release moving online with each release

Table and index move WILL interrupt Table and index move WILL interrupt operationsoperations

Periodically analyze your databasePeriodically analyze your database Use Index compress to optimize indexesUse Index compress to optimize indexes

Non-Stop ProgressPoints to RememberNon-Stop ProgressPoints to Remember

PlanPlan Timing is everythingTiming is everything Script to avoid mistakesScript to avoid mistakes

ImplementImplement Checklist, Checklist, ChecklistChecklist, Checklist, Checklist

MaintainMaintain TrendingTrending Low impact is paramountLow impact is paramount

QuestionsQuestions

Preparing for High Availability Adam Backman [email protected] V.P. of Technology White Star Software.

Documents

Transcript of Preparing for High Availability Adam Backman [email protected] V.P. of Technology White Star Software.