Understanding And Resolving Free Buffer Wait Contention · Resolving Free Buffer Wait Contention...

(c)2015 OraPub, Inc. Free Buffer Waits

UUnnddee rr ss tt aanndd ii nngg AAnndd RReessoo ll vv ii nngg FF rreeee BBuu ff ff ee rr

WWaa ii tt CCoonn tteenn tt ii oonn

Craig A. Shallahamer [email protected]


Redo Buffer

System Global Area (SGA)

LGWR

Redo Log 1 Redo Log 2

DBWR

DBFs

Block Buffer

Shared Pool

Always check orapub.com for an updated version of this presentation. This presentation was given by Craig Shallahamer ([email protected])

at the November 2015 DOAG conference in Nuremberg, Germany.


About Me...   Long time Oracle DBA  Specialize in Oracle Database

performance and predictive analysis  Performance researcher  Blogger: A Wider View About Oracle

Performance Tuning  Author: Oracle Performance Firefighting

and Forecasting Oracle Performance.  Conference speaker  Teacher and mentor  Oracle ACE Director


EEmmaaiill:: [email protected]

TTwwiitttteerr:: @CShallahamer

OOrraaPPuubb..CCoomm:: Everything starts here!

LLiinnkkeeddIInn:: Connect and network with Craig and the OraPub Group.

Connect with Craig and OraPub.





Outline

  Key buffer cache memory structures

  Least recently used internals

  How a free buffer wait occurs

  Solutions to resolve the issue

  Going deeper




WWhhaatt’’ss tthhee ppooiinntt ooff tthhiiss??

Get the visualization tool: OP search “visual”


``

Key

Buffer Cache

Memory Structures




There are three key lists.   HHaasshh//CCaacchhee bbuuffffeerr cchhaaiinnss are used to quickly

determine if a block is in the cache.

  LLRRUU ((LLRRUU)) cchhaaiinnss are used to keep popular buffers cached and to find free buffers.

  WWrriittee ((LLRRUU--WW)) lliissttss are used by the its DBWR to batch write dirty buffers. Also called the dirty list.

  BBuuffffeerr hheeaaddeerrss contain pointers to the physical block, the buffer cache location, and its position on its LRU, write, and hash chain list.


Buffer headers are on many lists.

CBCs

LRUs

WLs




The buffer head view, v$bh in 12c.

SQL> desc v$bh Name Null? Type ------------------------------------------------------------------ -------- ------------------- FILE# NUMBER BLOCK# NUMBER CLASS# NUMBER STATUS VARCHAR2(10) XNC NUMBER FORCED_READS NUMBER FORCED_WRITES NUMBER LOCK_ELEMENT_ADDR RAW(8) LOCK_ELEMENT_NAME NUMBER LOCK_ELEMENT_CLASS NUMBER DIRTY VARCHAR2(1) TEMP VARCHAR2(1) PING VARCHAR2(1) STALE VARCHAR2(1) DIRECT VARCHAR2(1) NEW CHAR(1) OBJD NUMBER TS# NUMBER LOBID NUMBER CACHEHINT NUMBER FLASH_CACHE VARCHAR2(7) CELL_FLASH_CACHE VARCHAR2(7) CON_ID NUMBER

No changes in x$bh. TCH column still only in x$bh.


The “real” stars of the buffer cache!

400 BHs350 CBCs 2 LRUs20% dirty No cloning




It’s the lists and the buffer headers.


Visualizing 400 BH fully connected.

400 BHs350 CBCs 2 LRUs20% dirty No cloning





Free, Dirty, and Pinned...   FFrreeee//mmiirrrroorreedd.. When a block's cached image mirrors the actual

block on disk. The cached block (I.e., the buffer) can be replaced by another block and it can be referenced by any Oracle process. When the contents of a dirty buffer has been written to disk, it is once again a mirror of the actual block and placed back on the LRU end of its LRU list. An empty buffer is also called a free buffer. There are multiple buffer states (x$bh.state) related to a free/mirrored buffer.   DDiirrttyy. When a block's cached image is not the same as its database

file image. After a block has been changed (I.e., was pinned) and has not yet been written back to its database file, it is deemed “dirty.” (v$bh.dirty=‘‘Y’’)  PPiinnnneedd. A buffer is pinned to keep it from being replaced. You

don’t want a block you’re referencing to be replaced! Pinning can also be used to ensure serial access.




``

Least Recently Used

Lists Internals


Least recently used chains; LRUs.

  LRUs are used to cache hot buffers and to quickly supply free buffers to server processes.  Cached buffers are divided into working sets.  Each working set has an associated LRU chain or list.  LRUs are made up of cached buffer headers ordered by

popularity, descending. (not really…)  Each LRU can contain free, pinned, and also dirty buffer

headers.  Each DBWR is associated with one or more LRUs.  There appears to be one LRU latch

(_db_block_lru_latches) for each LRU chain (v$latch_children, x$kcbwds). A typical value is 32 LRUs.




The LRUs

SQL> select count(*) from x$kcbwds; COUNT(*) ---------- 48 SQL> @ipx %lru%latch% Instance Parameter and Value Description Dflt? -------------------------------------------------- --------------------- ----- _db_block_lru_latches = 48 number of lru latches TRUE SQL> select count(*) from v$latch_children where lower(name) like '%lru%'; COUNT(*) ---------- 48

o12.1


Oracle’s caching struggle…

  Standard least recently used algorithm.  Modified least recently used algorithm.  Touch-Count algorithm




The key TC components.   Midpoint insertion

  Touch count incrementation

  Buffer promotion

  Parameters for everything...


Does the 12c touch count still oscillate?

truncate table op_tch_results; alter session set commit_wait=nowait; declare looper_var number; begin for looper_var in 1..1000 loop insert into op_tch_results select looper_var, current_timestamp, dbarfil, dbablk, tch from x$bh where dbarfil = 5 and dbablk = 30000 ; commit; dbms_lock.sleep(1.0); end loop; end; /

The touch count instance parameters appear to be the same number, name, and default... no change. But does the algorithm work the same?...




SQL> l 1* select to_char(the_time,'HH24:MI:SS'), tc from op_tch_results where fl_num=5 and blk_num=30000 order by 1 SQL> / 10:06:56 220 10:06:58 220 10:06:59 220 10:07:00 221 10:07:01 221 10:07:03 222 10:07:04 222 10:07:05 223 10:07:06 223 10:07:08 223 10:07:09 224 10:07:10 224 10:07:12 225 10:07:13 225 10:07:14 225 10:07:15 226 ... 10:08:41 254 10:08:42 255 10:08:44 255 10:08:45 0 10:08:46 0 10:08:47 1 10:08:49 1 ... 10:21:28 254 10:21:29 255 10:21:30 255 10:21:32 0 10:21:33 0

Note: This buffer was constantly being touched. The workload was only LIO, so there was never any need to find a non-popular free buffer. Therefore no buffer replacement occurred or no promotion and therefore the TCs where not reset to zero. Notice: 1.  The TC never exceeds 255.

2.  The TC never increases more than one every three seconds.

o12.1


The TCH never exceeded 255. SQL> l 1 select dbarfil, dbablk, tch 2 from x$bh 3* where tch = ( select max(tch) from x$bh ) SQL> / DBARFIL DBABLK TCH ---------- ---------- ---------- 3 4707 255 1 81461 255 1 81453 255 1 81492 255 1 17943 255 1 337 255 1 81408 255 1 245 255 3 4674 255 1 81412 255 1 17965 255 1 15114 255 1 36516 255 1 81409 255 6 13700 255 o12.1




truncate table op_tch_results; alter session set commit_wait=nowait; declare looper_var number; dbarfil_var number := 6 ; dbablk_var number := 502360 ; tch_var number ; begin for looper_var in 1..1000 loop begin select tch into tch_var from x$bh where dbarfil = dbarfil_var and dbablk = dbablk_var; EXCEPTION WHEN OTHERS THEN tch_var := -1; end; insert into op_tch_results values ( looper_var, current_timestamp, dbarfil_var, dbablk_var, tch_var ); end loop; end; /

An active buffer in a

FBW workload

mix.


>0 it has been touched =0 just brought into the cache -1 not in the cache, replaced

This block is constantly being brought into the cache and then replaced! What I would expect with a free buffer wait load! There also lots of promotion because server processes are on the look out!

o12.1




12c Touch Count Algorithm

  Based on what I previously showed:– Three second rule is still the default– Promotion still occurs when a server process or

the DBWR is looking for a buffer AND the buffer’s TC is >= 2.

– When promotion occurs, the TC is still reset to 0.

  Summary: operation remains the same.  I did not test if the max 255 tch “rule” was in affect in 11g.


``

How A

Free Buffer Wait

Occurs

1.  Dirty buffers - DML 2.  Replacement required -PIOR 3.  DBWR falling behind




  The DBWR can’t ppuullll dirty blocks out of the buffer cache fast enough to make them free. It’s not a ppuusshh to disk issue, it’s a ppuullll from cache issue.

  The server process is assigned a working set (and the related LRU chain/list) based upon its session ID. This can change.

  Two scenarios are common; when a server process is scanning its LRU looking for free buffers...

–  It has scanned “too many” buffers (pinned or dirty) before it finds a free buffer. This can be caused by intense PIO reads or too many CR buffers are needed.

The wait event “free buffer wait” is posted, the DBWR is told to write the related LRU-W list, and the server process waits no more than one second. Upon wake-up the server process starts looking for a free buffer again at the LRU end of its LRU.

–  Stumbles upon a dirty buffer and tries to move a buffer to the write list (LRU-W) but can’t (perhaps the DBWR is writing it to disk). The server process posts a free buffer wait event and waits until the DBWR is finished.

What is free buffer wait contention?


A typical FBW situation. SQL> @rtpctx Remember: This report must be run twice so both the initial and final values are available. If no output, press ENTER about nine times. Database: prod18 04-MAY 11:22am Report: rtpctx.sql OSM by OraPub, Inc. Page 1 Response Time Activity (467 sec interval) Time Avg Time Wait RT Component (sec) % RT Waited (ms) Count(k) -------------------------------------- ---------- ------- ----------- -------- free buffer waits 2663.010 37.82 40.218 66 direct path read 2055.780 29.20 4.881 421 CPU consumption: Oracle SP + BG procs 461.302 6.55 0.000 0 db file async I/O submit 406.800 5.78 2748.649 0 db file sequential read 216.220 3.07 5.772 37 read by other session 178.170 2.53 65.818 3 log file parallel write 110.440 1.57 244.336 0 . . . Instance Workload Statistic Interval Work Interval Work/Sec ---------------------------------------- ------------- ----------------- user calls 59 0.126 session logical reads 21132973 45252.619 physical reads 4716617 10099.822 physical read IO requests 327596 701.490 execute count 8520 18.244

Source: OSM Toolkit




``

Solutions

To Resolve The Issue


  The DBWR can not get dirty buffers to disk fast enough, so increase DBWR pull power (e.g., more DBWRs) and perhaps some I/O write throughput improvement…

  Increase the buffer cache to increase the likelihood of finding a non-popular free buffer.

  Tune SQL to reduce PIO. By reducing PIO, you reduce the number of times a free buffer must be found.

  Find DML SQL to understand why there are so many dirty buffers. This may be an application issue or perhaps a workload balancing issue. If possible, reduce the number of changed buffers.

  There may be lots of clone (CR) buffer building, which requires a free buffer and also possibly additional free buffers for non-cached undo segment blocks. Look for SQL statements query a changing table, then tune the CR SQL to reduce CR buffers.

  Encourage the DBWR to write more often, but with smaller batches, by decreasing _db_large_dirty_queue.

  Be more patient, search longer, and give the DBWR more time to write its batch by increasing _db_writer_max_scan_pct. Not in 11g.

Resolving free buffer contention.




`` Going Deeper


lru,free

buffer, visual




lru,free

buffer,visual






UUnnddee rr ss tt aanndd ii nngg AAnndd RReessoo ll vv ii nngg FF rreeee BBuu ff ff ee rr

WWaa ii tt CCoonn tteenn tt ii oonn

Craig A. Shallahamer [email protected]



Understanding And Resolving Free Buffer Wait Contention · Resolving Free Buffer Wait Contention...

Documents

Transcript of Understanding And Resolving Free Buffer Wait Contention · Resolving Free Buffer Wait Contention...