Understanding Firebird Transactions Pavel Císař IBPhoenix Firebird Conference 2003.
Resolving Firebird performance problems
-
Upload
alexey-kovyazin -
Category
Software
-
view
57.285 -
download
2
Transcript of Resolving Firebird performance problems
![Page 2: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/2.jpg)
Usually big production database runs at• Firebird 2.5• Classic or SuperClassic architecture
• They can scale accross multiple CPU/cores!
• In Firebird 3 SuperServer is the better choice… • but now let’s talk about 2.5 Classic and SuperClassic
![Page 3: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/3.jpg)
What are performance indicators to monitor in Firebird 2.5?1. Lock table parameters2. Monitoring record versions and garbage collection3. Transactions
1. Long running transactions2. Rolled back transactions
4. Temporary files – quantity and size5. Connections, SQL queries and transactions
![Page 4: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/4.jpg)
LOCK TABLE MONITORING
![Page 5: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/5.jpg)
fb_inet_serverfb_inet_server
How lock table works
fb_inet_server
Firebird
Page cache Page cache Page cache
Page X
How to resolve a write conflict when changing the same page?
![Page 6: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/6.jpg)
How lock table works
fb_inet_server fb_inet_server fb_inet_server
mapped
Lock table
mapped mapped
shared memoryAll requests to access internal database objects go through lock table
![Page 7: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/7.jpg)
Why should we monitor lock table parameters?• Lock table is a critical part of Classic and SuperClassic
architectures• Access to shared objects is implemented through locks in
Lock table…
Analysis of lock table parameters is the easiest way to reveal problems with concurrent access
![Page 8: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/8.jpg)
Lock table analysis - raw• Fb_lock_print –d <database_name>• LOCK_HEADER BLOCK• Version: 17, Active owner: 0, Length: 6291456, Used: 5517236• Flags: 0x0001• Enqs: 10906251, Converts: 58907, Rejects: 22373, Blocks: 210859• Deadlock scans: 5841, Deadlocks: 0, Scan interval: 10• Acquires: 13636997, Acquire blocks: 558879, Spin count: 0• Mutex wait: 4.1%• Hash slots: 2003, Hash lengths (min/avg/max): 2/ 11/ 26• Remove node: 0, Insert queue: 0, Insert prior: 0• Owners (107): forward: 26696, backward: 5517140• Free owners: *empty*• Free locks (1630): forward: 3878196, backward: 2264580• Free requests (793): forward: 5412916, backward: 1906516• Lock Ordering: Enabled
![Page 9: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/9.jpg)
Lock table analysis – where to look• Fb_lock_print –d <database_name>• LOCK_HEADER BLOCK• Version: 17, Active owner: 0, Length: 6291456, Used: 5517236• Flags: 0x0001• Enqs: 10906251, Converts: 58907, Rejects: 22373, Blocks: 210859• Deadlock scans: 5841, Deadlocks: 0, Scan interval: 10• Acquires: 13636997, Acquire blocks: 558879, Spin count: 0• Mutex wait: 4.1%• Hash slots: 2003, Hash lengths (min/avg/max): 2/ 11/ 26• Remove node: 0, Insert queue: 0, Insert prior: 0• Owners (107): forward: 26696, backward: 5517140• Free owners: *empty*• Free locks (1630): forward: 3878196, backward: 2264580• Free requests (793): forward: 5412916, backward: 1906516• Lock Ordering: Enabled
![Page 10: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/10.jpg)
Thresholds and recommendationsEssential values: Firebird.conf params to tune:
Length: 6291456 LockMemSize - default is 1MbSet at least:Cache_pages * max_connections_count * 100
Hash slots: 2003, Hash lengths (min/avg/max): 2/ 11/ 26
LockHashSlots = 1009 Prime number. Set x20, like 20011
Mutex wait: 4.1% No parameter to adjust. More than 10% could be a problem, check hardware and concurrency settings
Owners (107) Number of connections for server
Use optimized Firebird configuration files from IBSurgeon:http://ib-aid.com/en/optimized-firebird-configuration/
![Page 11: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/11.jpg)
How to monitor lock table• 1) Command prompt (on the server only), run every N min
fb_lock_print -d E:\OLTP-EMUL\oltp30.fdb
• 2) Alerts and automatic recommendations
![Page 12: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/12.jpg)
Monitoring record versions and garbage collection in Firebird
![Page 13: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/13.jpg)
What is a record version?
TR50 TR60
UPDATE set data =200read
N Tx Data1 10 100
60 200
...
readwrite
Different transactions can see different versions of the record.
![Page 14: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/14.jpg)
Why many record versions are bad
The more versions record has, the more scan operations server does to find necessary version
After data load or restore – no versions
![Page 15: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/15.jpg)
• When UPDATE changes indexed fields, indices also must be updated, and - UPDATE does not update keys in the index, it adds new keys for new record version!
• DELETE does not delete index keys!
N Tx Data1 10 100
60 200
...
Index key = 100
Index key = 200
Why many record versions are bad
![Page 16: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/16.jpg)
What to monitor?
Database statistics• Database statistics shows information about Record
Versions and Max Versions• Taking database statistics can be long!
![Page 17: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/17.jpg)
How to monitor1. With gstat command line tool2. With HQbird Firebird Database Analyst
More details: https://www.youtube.com/watch?v=_KZqXcgwN98
![Page 18: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/18.jpg)
Goal of every Firebird developer is
to avoid creating versions for records!
Not only because server slowly reads multiple record versions, but also because of
GARBAGE COLLECTION
![Page 19: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/19.jpg)
What is garbage and how it is collected
• When record versions become obsolete and non-interested by any transaction, it is considered as garbage and need to be cleaned.
• Garbage collection is a process of removing unnecessary records versions• It can be cooperative (Classic or SuperClassic) or
background (SuperServer)
![Page 20: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/20.jpg)
Why should we monitor garbage collection?
• It consumes resources.• We should locate and fix those parts of the application(s) which produce many record versions which increase an intensity of GC
![Page 21: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/21.jpg)
How to monitor GC?• 1) Manual SQL queries to MON$ tables• 2) HQbird Firebird MonLogger
![Page 22: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/22.jpg)
How to decrease number of garbage versions?
1. Use as short write transactions as possible2. For read-only operations use (if possible)
transactions READ COMMITTED READ-ONLY
If you see a lot of record versions, and you are not sure how to fix it, consider to order our Database Optimization service:http://ib-aid.com/en/firebird-interbase-performance-optimization-service/
![Page 23: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/23.jpg)
Monitoring transactions in Firebird
![Page 24: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/24.jpg)
Why to monitor transactions?
Transactions are key indicators of versioning processes in Firebird database.
2 main problems1. Long-running [writeable] transactions2. Rolled back transactions
![Page 25: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/25.jpg)
Gstat –h gives us snaphot of transactionsDatabase header page information: Flags 0 Checksum 12345 Generation 1564 Page size 4096 ODS version 10.1
Oldest transaction 10009 Oldest active 20001 Oldest snapshot 20001 Next transaction 25007 Bumped transaction 1 Sequence number 0 Next attachment ID 1 Implementation ID 16 Shadow count 0 Page buffers 3000 Next header page 0 Database dialect 1 Creation date Nov 1, 2015 15:55:55
![Page 26: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/26.jpg)
Long-running transactions• All transactions have sequential numbers from 1 to…• Oldest Active Transaction – currently active transaction
with the minimal numberOldest transaction 10009Oldest active 20001Oldest snapshot 20001Next transaction 25007
Interval NEXT – OAT shows number of potentially active transactions – server cannot clean versions created by transactions with numbers > OAT
![Page 27: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/27.jpg)
How to track long running active transactions in Firebird?
• 1. Manual tracking of gstat –h output (check every N minutes)
• 2. Alerts from HQbird FBDataGuard
![Page 28: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/28.jpg)
How to fix long running active transactions in Firebird?
It is possible to find who is running long-running transaction and CANCEL it
1. Manual query to MON$ tables2. HQbird Firebird MON$Logger (GUI)
![Page 29: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/29.jpg)
Rolled back transactions• When some transaction is marked in transaction inventory
as rolled back, it prevents record versions beeing cleaned by collective or background garbage collection
Oldest transaction 10009Oldest active 20001Oldest snapshot 20001Next transaction 25007
Interval Oldest Snapshot – Oldest Interesting determines the need for sweep
![Page 30: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/30.jpg)
Sweep• Sweep reads whole database from the beginning to the end and cleans garbage record versions
• Sweep is necessary when OST-OIT interval becomes big
![Page 31: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/31.jpg)
How to make sweep• Autosweep
• by default 20000• Starts immediately when interval > threshold• It causes slowness at unpredicted moments
• Manual sweep• Scheduled sweep during the inactivity period of time
• Can be run manually with gfix –sweep or scheduled in HQbird FBDataGuard
![Page 32: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/32.jpg)
Sweep must be controlled!• If sweep did not succeed to align transaction markers, it can happen due to the serious problem/corruption!
• HQbird FBDataGuard checks sweep status
![Page 33: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/33.jpg)
TEMP FILES
![Page 34: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/34.jpg)
Temp files• Temp files are created in the default temp folder or in folders specified by TempDirectories in firebird.conf
• Actually they are written to the disk only if size of all sort files is more than TempCacheLimit parameter
• It is better to have sorting in memory
![Page 35: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/35.jpg)
How to track Temp files1. Manually check size and quantity of temp files
(fb_sortxxx) in all temp folders2. HQbird FBDataGuard monitors temp files size and
quantity, and sends warnings if necessary
![Page 36: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/36.jpg)
How to move temp files to memory• Increase TempCacheLimit parameter• Warnings:
• At Classic TempCacheLimit is allocated for each process• 32bit processes have 2Gb limitation of memory to
address
Use optimized configuration files from IBSurgeon and HQbird!http://ib-aid.com/en/optimized-firebird-configuration/
![Page 37: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/37.jpg)
Monitor connections, transactions, and SQL queries
![Page 38: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/38.jpg)
Why to monitor Firebird connections?• Number of connections should be constantly monitored:
• Peaks of connections• Firebird Classic/SuperClassic consume RAM as
Number_Of_Connections x Page Buffers• In case of peaks can consume all RAM and go to swap – slowness!
• Application can do more than 1 connection to the database• By mistake• By design
• In case of connection pool – adjust the size of the pool and its increment
• Suspicious connections• Long running connections and connections from developer tools• Connections from non-standard applications
![Page 39: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/39.jpg)
How to monitor connections1. Manually run SELECT count(*) FROM
MON$ATTACHMENTS2. Use HQbird
1. FBDataGuard constantly monitors number of connections and send alerts if their number grows too much
2. MonLogger can be used to go deep and get detailed information about connections: memory, IO, garbage collection
![Page 40: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/40.jpg)
Transactions• Yes, again!• Besides monitoring of intervals between transactions
markers (Next-OAT, OIT-OST), also we should check these important things, which can affect Firebird performance:
1. Transaction’s parameters (isolation level, wait/nowait) 2. Transactions flows and their relation with business
logic3. Transaction-level statistics (IO, time to live)
![Page 41: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/41.jpg)
Transaction’s parameters• Transactions parameters must correspond to business
logic.• Several often mistakes
• Non-modyfing queries (SELECT) are started in read-write mode• Should be in READ ONLY
• Snapshot are used for operational queries, instead of read-commited• It should be used only for queries where consistency is really required
(snapshots)
![Page 42: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/42.jpg)
Transactions flows mistakes• Transaction (other then read-committed read only) never
ends, expires due to connection closing, and leads to ROLLBACK
• Only 1 sentence in the transaction• For example, refreshing query with timer:
• Quickly exhausts 2 billion limit of transactions• it should be run in permanent read committed read only transaction
• Using the only read-write transaction for all operations• Better split read-only and write operations
![Page 43: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/43.jpg)
How to track transactions flows and transactions parameters? - FBScanner
HQbird FBScanner allows to track transactions flows: parameters, containing queries, plans, etc.
![Page 44: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/44.jpg)
How to track transactions flows and transactions parameters? - PerfMon• HQbird Firebird Performance Monitor (PerfMon) uses
TraceAPI to capture all engine events – connections, queries, transactions, stored procedures, triggers, execution plans, execution statistics
• Works correctly only at Firebird 2.5.3+ versions
![Page 45: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/45.jpg)
Quick check of problems with transactions
• HQbird MonLogger gives a fast and detailed overview of active transactions in the system
![Page 46: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/46.jpg)
HQbird tools for SQL and transactions monitoring and optimizationHQbird FBScanner
+ works as a proxy, non-invasive tool, can be installed on separate server and track only selected workstations— cannot track stored procedures and triggersHQbird PerfMon
+ works with TraceAPI, can monitor everything— TraceAPI significantly supresses performanceHQbird MonLogger
+ Great ad-hoc tool, fast— Captures only snapshots, lack of important information
![Page 47: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/47.jpg)
IBSurgeon Firebird Optimization service
We can increase the performance of your databasehttp://ib-aid.com/en/firebird-interbase-performance-optimization-service/
![Page 48: Resolving Firebird performance problems](https://reader030.fdocuments.in/reader030/viewer/2022033016/589cddee1a28abf86d8b493d/html5/thumbnails/48.jpg)
Thank you and don’t forget these links:
IBSurgeon optimized configuration fileshttp://ib-aid.com/en/optimized-firebird-configuration/
Firebird Hardware Guidehttp://ib-aid.com/download/docs/Firebird_Hardware_Guide_IBSurgeon.pdf
Contact us: [email protected]