Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software...

85
Progress Database Setup, Maintenance and Tuning Adam Backman Adam Backman V.P. of Technology V.P. of Technology White Star Software White Star Software [email protected] [email protected]

Transcript of Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software...

Page 1: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Progress Database Setup, Maintenance and TuningProgress Database Setup, Maintenance and Tuning

Adam BackmanAdam Backman

V.P. of TechnologyV.P. of Technology

White Star SoftwareWhite Star Software

[email protected]@wss.com

Page 2: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

AgendaAgenda

Progress architectureProgress architecture Hardware configuration discussionHardware configuration discussion Database maintenanceDatabase maintenance Performance tuningPerformance tuning Database future directionDatabase future direction

Page 3: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Database InternalsDatabase Internals

Internal BlocksInternal Blocks Data blocksData blocks Index blocksIndex blocks Other block types Other block types

Physical LayoutPhysical Layout Data storage areasData storage areas Primary recovery areaPrimary recovery area After image journalAfter image journal Other storageOther storage

Page 4: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Master BlocksMaster Blocks Data Block (RM Block)Data Block (RM Block) Index Block (Ix Block)Index Block (Ix Block) Index Anchor BlockIndex Anchor Block Free BlocksFree Blocks Empty BlocksEmpty Blocks

Database BlocksDatabase Blocks

Page 5: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

This stores the “master” information for the This stores the “master” information for the database including:database including:

Area Status (opened, closed, crashed)Area Status (opened, closed, crashed) Last opened date & timeLast opened date & time High water mark for the areaHigh water mark for the area Last backup date & timeLast backup date & time

This information and more can be retrieved This information and more can be retrieved through the virtual system tables (VSTs)through the virtual system tables (VSTs)

Master BlockMaster Block

Page 6: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

These are also known as RM (Record These are also known as RM (Record Manager) blocksManager) blocks

Can contain information from one or more Can contain information from one or more tablestables

They can be “full” (RM Blocks)They can be “full” (RM Blocks) Partially full (RM Chain blocks)Partially full (RM Chain blocks)

Data BlocksData Blocks

Page 7: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

In most environments, records are mixed from In most environments, records are mixed from different tables in the same blockdifferent tables in the same block

Progress can store from 1 to 256 records per block Progress can store from 1 to 256 records per block per storage areaper storage area

All areas for a database must have the same block All areas for a database must have the same block size (1 – 8kb) size (1 – 8kb)

Total records per area is fixedTotal records per area is fixed More records per block equals lower total blocksMore records per block equals lower total blocks

Record StorageRecord Storage

Page 8: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Also known as IX blocksAlso known as IX blocks Only contain information from one indexOnly contain information from one index Always considered partially full, blocks Always considered partially full, blocks

will split to accommodate growthwill split to accommodate growth Each block contains the address of itself, Each block contains the address of itself,

the next and previous blocks to support the next and previous blocks to support forward and reverse searchesforward and reverse searches

Index BlocksIndex Blocks

Page 9: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Balanced B-treeBalanced B-tree Compressed dataCompressed data All data access* through indexAll data access* through index

*Except rowid access*Except rowid access

Index StructureIndex Structure

Page 10: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

43-54 55-6532-41 86-9866-85

32-65 66-982-31

22-312-11 12-21

Record 65

Record 55

.

.

.

Record 41

Record 32

.

.

.

Record 54

Record 43

.

.

.

Record 98

Record 86

.

.

.

Record 85

Record 66

.

.

.

Record 31

Record 22

.

.

.

Record 11

Record 2

.

.

.

Record 21

Record 12

.

.

.

Progress Index Layout ExampleProgress Index Layout Example

Page 11: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Contain address information Contain address information No affiliation with IX or RM until utilized No affiliation with IX or RM until utilized Under high water mark of the databaseUnder high water mark of the database

Free BlocksFree Blocks

Page 12: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

White spaceWhite space No addressesNo addresses Under total blocks of the database (area)Under total blocks of the database (area) Above high water markAbove high water mark

Empty BlocksEmpty Blocks

Page 13: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Data objectsData objects Control AreaControl Area Schema/Default AreaSchema/Default Area Primary recovery areaPrimary recovery area Application Data area(s)Application Data area(s)

Storage AreasStorage Areas

Page 14: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Index objectIndex object Table objectTable object Schema objectSchema object SequencesSequences

Storage Areas – Data ObjectsStorage Areas – Data Objects

Page 15: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Always has a .db extensionAlways has a .db extension Describes the “physical schema” or layout Describes the “physical schema” or layout

of the databaseof the database Lists storage areas and extents associated Lists storage areas and extents associated

with a databasewith a database Also known as the structure fileAlso known as the structure file

Control AreaControl Area

Page 16: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Also known as the “schema” areaAlso known as the “schema” area This is always area 6This is always area 6 All information that is not assigned a All information that is not assigned a

storage area will be stored herestorage area will be stored here All information is stored in this area if the All information is stored in this area if the

data is converted from version 8 with a data is converted from version 8 with a conv89conv89

Default Storage AreaDefault Storage Area

Page 17: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

An area can contain 1 or more data objectsAn area can contain 1 or more data objects An area can have 1 or more extentsAn area can have 1 or more extents Tables and indexes can share areasTables and indexes can share areas These use area numbers 7-1000These use area numbers 7-1000

Application Data AreasApplication Data Areas

Page 18: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

General informationGeneral information FormatFormat Reuse Reuse Same as bi file(s) in version 8 and earlierSame as bi file(s) in version 8 and earlier This is always area number 3This is always area number 3

Primary Recovery AreaPrimary Recovery Area

Page 19: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

This process is automatic and can’t be This process is automatic and can’t be turned offturned off

This file is vital to database integrity both This file is vital to database integrity both physical and logicalphysical and logical

This file is generally sequentialThis file is generally sequential

Primary Recovery AreaGeneral InfoPrimary Recovery AreaGeneral Info

Page 20: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Each block is a clusterEach block is a cluster Each cluster contains information regarding Each cluster contains information regarding

transactions to allow transaction undo and transactions to allow transaction undo and redoredo

The transaction information is called notesThe transaction information is called notes Each note contains a transaction idEach note contains a transaction id

Primary Recovery AreaFormatPrimary Recovery AreaFormat

Page 21: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Clusters fill sequentiallyClusters fill sequentially When the last formatted cluster is reached When the last formatted cluster is reached

there is a reuse decision pointthere is a reuse decision point The “oldest” cluster is examined to The “oldest” cluster is examined to

determine if it can be reuseddetermine if it can be reused Then, either the oldest cluster is reused or Then, either the oldest cluster is reused or

another cluster is added, formatted and usedanother cluster is added, formatted and used

Primary Recovery AreaReusePrimary Recovery AreaReuse

Page 22: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

The database engine can be serverlessThe database engine can be serverless The database engine can be multi-serverThe database engine can be multi-server Progress applications can be host-basedProgress applications can be host-based Progress applications can be 2-tier Progress applications can be 2-tier

client/serverclient/server Progress applications can be n-tier Progress applications can be n-tier

client/serverclient/server

Progress Memory ArchitectureProgress Memory Architecture

Page 23: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Buffers (-B)

Record locks (-L)

Index Cursors (-c)

After Image Buffers

Before Image Buffers

Hash Table

UserControlTable

ServerControlTable

Latch ControlTable

Other Stuff

Shared MemoryHost-based Configuration Shared MemoryHost-based Configuration

Page 24: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

What are Latches?What are Latches?

Concurrency control mechanismConcurrency control mechanism Very course in old versions of ProgressVery course in old versions of Progress More granular in current versions of More granular in current versions of

ProgressProgress

Page 25: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Buffers (-B)

Record locks (-L)

Index Cursors (-c)

After Image Buffers

Before Image Buffers

Hash Table

UserControlTable

ServerControlTable

Latch ControlTable

Other Stuff

ListenSocket

Servers

Shared MemoryClient/Server ConfigurationShared MemoryClient/Server Configuration

Page 26: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Shared MemoryClient/Server ConfigurationShared MemoryClient/Server Configuration

ListenSocket

AppServer

DatabaseBrokerMemory

Servers

AppServer

Page 27: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Hardware ConfigurationsHardware Configurations

Disk ConsiderationsDisk Considerations Memory AllocationMemory Allocation CPU ConsiderationsCPU Considerations

Page 28: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

In most environments disks are the largest In most environments disks are the largest area for improvement. All of the data flows area for improvement. All of the data flows from the disks to the other resources so this from the disks to the other resources so this effects both local and networked userseffects both local and networked users

Disk ContentionDisk Contention

Page 29: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Balancing disk I/O is the process of making Balancing disk I/O is the process of making sure you are using all of the available disk sure you are using all of the available disk resources (filesystems, disks and controllers) resources (filesystems, disks and controllers) are working equally as hard at load. This is are working equally as hard at load. This is also called eliminating also called eliminating variance. variance. A well A well tuned system will have less than a 15% tuned system will have less than a 15% variancevariance

Balancing Disk I/OBalancing Disk I/O

Page 30: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Operating system (swapping and paging)Operating system (swapping and paging) ProgressProgress

Database (DB and BI)Database (DB and BI) Application (code and temp files)Application (code and temp files)

Other applicationsOther applications

What Causes Disk I/O?What Causes Disk I/O?

Page 31: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

RAID has many levels. I will only cover a RAID has many levels. I will only cover a fewfew

RAID 0: This level is also called RAID 0: This level is also called striping.striping. RAID 1: This is referred to as RAID 1: This is referred to as mirroringmirroring.. RAID 5: Most common RAID levelRAID 5: Most common RAID level RAID 10: This is mirroring and striping. RAID 10: This is mirroring and striping.

Also known as RAID 0 + 1Also known as RAID 0 + 1

What RAID Really MeansWhat RAID Really Means

Page 32: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Stripe 1Stripe 1Stripe 2Stripe 2Stripe 3Stripe 3Stripe 4 ...Stripe 4 ...

Disk 1Disk 1 Disk 2Disk 2 Disk 3Disk 3

Volume SetVolume Set

Disk Disk ArrayArray

Raid 0: StripingRaid 0: Striping

Page 33: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Good for read and write I/O performanceGood for read and write I/O performance No failover protectionNo failover protection Lower data reliability (1 fails they all fail)Lower data reliability (1 fails they all fail)

Raid 0: Striping (continued)Raid 0: Striping (continued)

Page 34: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

PrimaryPrimary

ParityParity

Disk 1Disk 1 Disk 2Disk 2

Parity 1Parity 1 Parity 2Parity 2

Raid 1: MirroringRaid 1: Mirroring

Page 35: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

OK for read and write applicationsOK for read and write applications Good failover protectionGood failover protection High data reliabilityHigh data reliability Most expensive in terms of hardwareMost expensive in terms of hardware

Raid 1: Mirroring (continued)Raid 1: Mirroring (continued)

Page 36: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

This is the kiss of death for OLTP This is the kiss of death for OLTP performanceperformance

User information is stripedUser information is striped Parity information is striped WITH user Parity information is striped WITH user

informationinformation OK for 100% read only applicationsOK for 100% read only applications Poor performance for writesPoor performance for writes

Raid 5: Poor Man’s MirroringRaid 5: Poor Man’s Mirroring

Page 37: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Good for read and write applicationsGood for read and write applications High level of data reliability though not as High level of data reliability though not as

high as RAID 1 due to stripinghigh as RAID 1 due to striping Just as expensive as RAID 1Just as expensive as RAID 1

Raid 10: Mirroring and StripingRaid 10: Mirroring and Striping

Page 38: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Manual spread of data across non-striped Manual spread of data across non-striped disksdisks

Better control as you can see where the I/O Better control as you can see where the I/O is goingis going

More attention by system administrator is More attention by system administrator is neededneeded

Software Methods for I/O DistributionSoftware Methods for I/O Distribution

Page 39: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Progress multi-volumeProgress multi-volume Progress storage areasProgress storage areas 8K database block size8K database block size BI Cluster sizeBI Cluster size Use page writersUse page writers Move the temp-file I/O with -TMove the temp-file I/O with -T Location of application filesLocation of application files Use of program libraries to reduce I/OUse of program libraries to reduce I/O

OptionsOptions

Page 40: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

• Progress-specific way to distribute I/OProgress-specific way to distribute I/O• Only way to eliminate I/O indirection in a Only way to eliminate I/O indirection in a

Progress environmentProgress environment• Only way to pre-allocate database blocksOnly way to pre-allocate database blocks• Every database is multi-volume in Progress Every database is multi-volume in Progress

version 9version 9

Multi-Volume DatabaseMulti-Volume Database

Page 41: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

BenefitsBenefits Greater control of location of dataGreater control of location of data Minimize downtime for utilitiesMinimize downtime for utilities Stripe some, leave some on straight disksStripe some, leave some on straight disks

DrawbacksDrawbacks More things to breakMore things to break More complex to monitorMore complex to monitor

Storage AreasStorage Areas

Page 42: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

A storage area can hold 1 or more data A storage area can hold 1 or more data objects (index, table, schema, …)objects (index, table, schema, …)

Separate schema from data if possibleSeparate schema from data if possible Try to keep the number of areas Try to keep the number of areas

manageable, only add more areas for valid manageable, only add more areas for valid business reasonsbusiness reasons

Storage Areas - ControlStorage Areas - Control

Page 43: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Smaller data areas allow utilities, such as off Smaller data areas allow utilities, such as off line index rebuild, to run faster as they have line index rebuild, to run faster as they have less blocks to scanless blocks to scan

Minimize DowntimeMinimize Downtime

Page 44: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Database Administration ToolsDatabase Administration Tools

Backup and restoreBackup and restore After image journalingAfter image journaling Other UtilitiesOther Utilities

Page 45: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

ProsPros Progress awareProgress aware Supports online backupSupports online backup easyeasy

ConsCons Slower than OS methodsSlower than OS methods Does not backup more than the databaseDoes not backup more than the database

probkupprobkup

Page 46: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Utility to restore a Progress backupUtility to restore a Progress backup Can restore to a different structure provided Can restore to a different structure provided

there are enough storage areasthere are enough storage areas

Syntax:Syntax:

prorest prorest dbnamedbname device_or_filename device_or_filename [-list | [-list | -vp | -vf]-vp | -vf]

prorestprorest

Page 47: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

ProsPros Allows you to recover to present Allows you to recover to present Recover from media failureRecover from media failure Only way to “repair” catastrophic user errorOnly way to “repair” catastrophic user error

ConsCons Additional point of failureAdditional point of failure Adds complexity to the systemAdds complexity to the system Performance impactPerformance impact

After ImagingAfter Imaging

Page 48: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

FOR EACH CUSTOMER: FOR EACH CUSTOMER:

UPDATE CUSTOMER.UPDATE CUSTOMER.

END. END. After image note written

Before image note written

How After Imaging WorksHow After Imaging Works

Page 49: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

In conjunction with a backup siteIn conjunction with a backup site To update a report serverTo update a report server As a means of backupAs a means of backup

How to Integrate After ImagingHow to Integrate After Imaging

Page 50: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Poor man’s replicationPoor man’s replication Allows for periodic update of a copy of the Allows for periodic update of a copy of the

databasedatabase The copy can then be backed up with a The copy can then be backed up with a

conventional backup mechanismconventional backup mechanism

AI to Update a Backup SiteAI to Update a Backup Site

Page 51: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

AI to Update a Report ServerAI to Update a Report Server

Similar to keeping a backup siteSimilar to keeping a backup site Requires two copies of the database in Requires two copies of the database in

addition to the original (one for update and addition to the original (one for update and a second for reporting)a second for reporting)

The reporting database is a copy of the The reporting database is a copy of the backup that is done periodically to keep the backup that is done periodically to keep the data synchronizeddata synchronized

Page 52: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Not generally a good ideaNot generally a good idea Increased recovery timeIncreased recovery time Reduced reliability Reduced reliability

Backup the database each weekendBackup the database each weekend Backup the AI file(s) each weeknight Backup the AI file(s) each weeknight

AI as a Means of BackupAI as a Means of Backup

Page 53: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Index rebuildIndex rebuild idxbuildidxbuild idxfixidxfix idxcompactidxcompact

DB analysisDB analysis

Progress® UtilitiesProgress® Utilities

Page 54: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Truncate BITruncate BI BI GrowBI Grow Table moveTable move Index moveIndex move Database log truncationDatabase log truncation

Progress® Utilities (continued) Progress® Utilities (continued)

Page 55: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Can only be run on a database that has been Can only be run on a database that has been shutdownshutdown

Can be run on 1 or more indexesCan be run on 1 or more indexes

Syntax: Syntax:

proutil <dbname> -C idxbuild [-TB proutil <dbname> -C idxbuild [-TB nn] ] [-TM [-TM nn] [-T ] [-T dirnamedirname]]

idxbuildidxbuild

Page 56: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Verifies index to record linkageVerifies index to record linkage Verifies index block to index block linkageVerifies index block to index block linkage Works online while in multi-user modeWorks online while in multi-user mode

Syntax: Syntax:

proutil <dbname> -C idxfixproutil <dbname> -C idxfix

idxfixidxfix

Page 57: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Fast way to compress (reorganize) indexes onlineFast way to compress (reorganize) indexes online Utility will pass through the index several times Utility will pass through the index several times

(number of index levels + 1)(number of index levels + 1) Runs onlineRuns online

Syntax: Syntax:

proutil <dbname> -C idxcompact [owner-proutil <dbname> -C idxcompact [owner-name.]table-name.index-name [n] name.]table-name.index-name [n]

idxcompactidxcompact

Page 58: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

ixanalys – analysis of indexes ixanalys – analysis of indexes chanalys – analysis of record chainschanalys – analysis of record chains dbanalys – analysis of records and indexesdbanalys – analysis of records and indexes

Syntax: Syntax:

proutil <dbname> -C XXanalysproutil <dbname> -C XXanalys

Database AnalysisDatabase Analysis

Page 59: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Reduce for size of the BI fileReduce for size of the BI file Change the cluster size of the BI fileChange the cluster size of the BI file Change the block size of the BI fileChange the block size of the BI file

Syntax: Syntax:

proutil <dbname> -C truncate bi [-bi proutil <dbname> -C truncate bi [-bi nn] ]

[-biblocksize [-biblocksize nn] [-G ] [-G nn]]

Truncate BITruncate BI

Page 60: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

After truncation it is best to pre-grow your After truncation it is best to pre-grow your BI file to it’s anticipated sizeBI file to it’s anticipated size

Keeps BI sequential (good for performance)Keeps BI sequential (good for performance) Database must be shutdownDatabase must be shutdown

Syntax: Syntax:

proutil <dbname> -C bigrow proutil <dbname> -C bigrow nn

BI GrowBI Grow

Page 61: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Allows the movement from one storage Allows the movement from one storage area to anotherarea to another

Works “online”Works “online” Uses 4-times the amount of BI space as is Uses 4-times the amount of BI space as is

taken up by the tabletaken up by the tableSyntax: Syntax:

proutil <dbname> -C tablemove [owner-proutil <dbname> -C tablemove [owner-name.]table-name table-area [index-area]name.]table-name table-area [index-area]

Table MoveTable Move

Page 62: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Allows movement of indexes from one storage Allows movement of indexes from one storage area to anotherarea to another

Works “online”Works “online” Uses a significant amount of BI spaceUses a significant amount of BI space

Syntax: Syntax:

proutil db-name -C indexmove [owner-proutil db-name -C indexmove [owner-name.]table-name.index-name area-name name.]table-name.index-name area-name

Index MoveIndex Move

Page 63: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Reduces the size of the log fileReduces the size of the log file Database must be down for it to work Database must be down for it to work

Syntax: Syntax:

prolog <dbname>prolog <dbname>

Database Log TruncationDatabase Log Truncation

Page 64: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Performance Tuning - BasicsPerformance Tuning - Basics

Before Image cluster sizeBefore Image cluster size Database block sizeDatabase block size Tuning APWsTuning APWs Memory tipsMemory tips Increasing CPU efficecyIncreasing CPU efficecy

Page 65: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Keep things localKeep things local No temp files on network drivesNo temp files on network drives Move the application “close” to the userMove the application “close” to the user

Use -cache to speed initial connectionUse -cache to speed initial connection Use -pls if you are using program libraries Use -pls if you are using program libraries

over the networkover the network Application issues are magnified over a Application issues are magnified over a

network (field-lists, no-lock, indexes, …) network (field-lists, no-lock, indexes, …)

Networking TipsNetworking Tips

Page 66: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

-Mm 8192 to increase the tcp packet size -Mm 8192 to increase the tcp packet size from 1k to 8kfrom 1k to 8k

-Ma Increase the number of servers to -Ma Increase the number of servers to reduce or eliminate server contentionreduce or eliminate server contention

Networking Tips (Continued)Networking Tips (Continued)

Page 67: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Tables that are accessed sequentially may Tables that are accessed sequentially may benefit from being isolated to their own table benefit from being isolated to their own table spacespace

Randomly accessed tables will generally Randomly accessed tables will generally perform better on striped volumesperform better on striped volumes

Disk systems that have read ahead algorithms Disk systems that have read ahead algorithms will help sequential access most when placed will help sequential access most when placed on a single disk (or mirror)on a single disk (or mirror)

Stripe Some, Leave Others FlatStripe Some, Leave Others Flat

Page 68: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Most systems will benefit from using 8k Most systems will benefit from using 8k block size (NT should use 4k)block size (NT should use 4k)

You will retrieve more information per You will retrieve more information per physical I/O especially on index readsphysical I/O especially on index reads

I/O is done how the operating likes it to be I/O is done how the operating likes it to be donedone

8k Block Size8k Block Size

Page 69: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Somewhere between 1MB and 4MB works for Somewhere between 1MB and 4MB works for most peoplemost people

If you are checkpointing every 2 minutes or more If you are checkpointing every 2 minutes or more often during peak periods increase the cluster sizeoften during peak periods increase the cluster size

If you a “workgroup” version of Progress leave If you a “workgroup” version of Progress leave your cluster size alone (512kb)your cluster size alone (512kb)

Don’t forget to use bigrow to avoid allocating Don’t forget to use bigrow to avoid allocating clusters one at a timeclusters one at a time

BI Cluster SizeBI Cluster Size

Page 70: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Every database that does updates should Every database that does updates should have a before image writer (BIW)have a before image writer (BIW)

Every database that does updates should Every database that does updates should have at least 1 asynchronous page writer have at least 1 asynchronous page writer (APW)(APW)

Every database that is using after imaging Every database that is using after imaging should have a after image writer (AIW) should have a after image writer (AIW)

Progress® Page WritersProgress® Page Writers

Page 71: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Start with 1 APWStart with 1 APW Monitor buffers flushed at checkpoint on Monitor buffers flushed at checkpoint on

the activity screen (option 5) in promonthe activity screen (option 5) in promon If buffers flushed increases during the If buffers flushed increases during the

“important” hours of the day add 1 APW“important” hours of the day add 1 APW

Tuning APWsTuning APWs

Page 72: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Local (host based) users and batch jobs Local (host based) users and batch jobs should use the -T parameter to place their should use the -T parameter to place their temporary file (.srt, .pge, .lbi, …) I/O on a temporary file (.srt, .pge, .lbi, …) I/O on a drive that is not working as hard as the drive that is not working as hard as the other drives on the systemother drives on the system

Note: -T should never point to a network Note: -T should never point to a network drivedrive

Use -T to Level Disk I/OUse -T to Level Disk I/O

Page 73: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Keep paths shortKeep paths short say run <subdir>/program to eliminate say run <subdir>/program to eliminate

unnecessary searchesunnecessary searches Put programs into libraries (prolib) to Put programs into libraries (prolib) to

reduce I/O to temp filesreduce I/O to temp files Libraries use a hashed search mechanism Libraries use a hashed search mechanism

for better performancefor better performance

Application FilesApplication Files

Page 74: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Memory should be used to reduce disk I/O. Memory should be used to reduce disk I/O. Broker (server) side parameters should be Broker (server) side parameters should be tuned first and then user parameters can be tuned first and then user parameters can be modified. In a memory lean situation, modified. In a memory lean situation, memory should be taken away from memory should be taken away from individual users before reducing broker individual users before reducing broker parametersparameters

Memory ContentionMemory Contention

Page 75: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Swapping is bad, buy more memory or Swapping is bad, buy more memory or reduce parameters to avoid itreduce parameters to avoid it

Increase -B in 10% increments until the Increase -B in 10% increments until the point of diminishing returns or swapping, point of diminishing returns or swapping, whichever comes firstwhichever comes first

Use V9 private buffers (-Bp) for reportingUse V9 private buffers (-Bp) for reporting Do not use private buffers (-I) prior to V9Do not use private buffers (-I) prior to V9

Memory HintsMemory Hints

Page 76: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Use memory for the users closest to the Use memory for the users closest to the customer first (developers increase last)customer first (developers increase last)

Use -Bt for large temp tablesUse -Bt for large temp tables Set -bibufs between 50 and 120. Look at Set -bibufs between 50 and 120. Look at

the activity screen in promon (BI buffer the activity screen in promon (BI buffer waits) to see if additional tuning is waits) to see if additional tuning is necessary. Start with 50 as this will work necessary. Start with 50 as this will work for the vast majority of peoplefor the vast majority of people

Memory Hints(continued)Memory Hints(continued)

Page 77: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

High CPU activity is not bad in and of itself High CPU activity is not bad in and of itself but high system CPU activity is bad and but high system CPU activity is bad and should be correctedshould be corrected

CPU ContentionCPU Contention

Page 78: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

USER - This is what you paid forUSER - This is what you paid for SYSTEM - This is overheadSYSTEM - This is overhead WAIT - This is wasteWAIT - This is waste IDLE - This is nothing ;-)IDLE - This is nothing ;-)

Components of CPU ActivityComponents of CPU Activity

Page 79: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

The goal is to have as much USER time as The goal is to have as much USER time as possible with as little SYSTEM and WAITpossible with as little SYSTEM and WAIT

A practical split is USER: A practical split is USER: 70%70%

SYSTEM:SYSTEM: 20%20%

WAIT:WAIT: 0% 0%

IDLE: IDLE: 10%10%

CPU Activity GoalsCPU Activity Goals

Page 80: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Always use -spinAlways use -spin Use a setting of 1 for single CPU systemsUse a setting of 1 for single CPU systems Use a higher setting for multiple CPU systemsUse a higher setting for multiple CPU systems

Testing has shown that the optimal setting for -Testing has shown that the optimal setting for -spin is somewhere between 2000 and 10000. First spin is somewhere between 2000 and 10000. First try 2000try 2000

-napmax should default to 5000 but in some late 7 -napmax should default to 5000 but in some late 7 and early 8 versions of Progress it is set to 100 and early 8 versions of Progress it is set to 100 which is way too lowwhich is way too low

Eliminating High SYSTEMCPU ActivityEliminating High SYSTEMCPU Activity

Page 81: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

WAIT = Waiting on I/OWAIT = Waiting on I/O If you still have IDLE time it generally is If you still have IDLE time it generally is

not a big problemnot a big problem Look at paging/swapping firstLook at paging/swapping first Next look at your disk I/ONext look at your disk I/O

Eliminating High WAITCPU ActivityEliminating High WAITCPU Activity

Page 82: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Progress Database Future DirectionsProgress Database Future Directions

Increased uptime through online utilitiesIncreased uptime through online utilities Increased speed of utilities to maintain the Increased speed of utilities to maintain the

databasedatabase Support for clusters to increase reliabilitySupport for clusters to increase reliability Open standards supportOpen standards support

Page 83: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

ReplicationReplication

New feature in 9.1D of ProgressNew feature in 9.1D of Progress Fathom High AvailabilityFathom High Availability Allows for single or bi-directional Allows for single or bi-directional

replicationreplication Target database can be used for update or Target database can be used for update or

reportingreporting

Page 84: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

Replication (continued)Replication (continued)

Source database has an agent that forwards Source database has an agent that forwards changes to the target database(s)changes to the target database(s)

Only one agent per databaseOnly one agent per database One or more targets per agentOne or more targets per agent Raw record format is used to increase Raw record format is used to increase

performance and reduce overheadperformance and reduce overhead

Page 85: Progress Database Setup, Maintenance and Tuning Adam Backman V.P. of Technology White Star Software adam@wss.com.

QuestionsQuestions