Post on 21-Jan-2018
100500 способовкэширования в Oracle Database или как достичьмаксимальной скоростиобработки запросовминимальной ценой
Токарев Александр
DataArt
Agenda• Database caches
• Result cache
• Result cache in DBMSs different from Oracle
• Hand-made Oracle result cache implementation
• Embedded Oracle result cache implementation
• Performance tests
• Limitations and caveats
• Cases
• Conclusion
Agenda• Retailer case
• Finance case
• Limitations and caveats
• Loyalty case
• Conclusion
Database caches
• Buffer cache – cache for data pages/data blocks
• Statement cache – cache of queries plan
• Result cache – rows from queries
• OS cache
Retailer case
DWH report
Oracle 1120 Tb300 users
20 min350 distinct SKU5000 rows
Select sku_id,Shop_id,sku_detail(sku_id),…..
from dim_saleswhere ….Order by shop_id……..
Create or replace function sku_detail(sku_idnumber) return number isSelect 1If Select 2Else Select 3………Select 30End;
400 lines of SQL+PL/SQL
0.2 second per SKU
5000 * 0.2 = 1000 seconds
Retailer case Hand-made cache
DWHreport
Oracle 1120 Tb300 users
4 min350 distinct SKU5000 rows
Select sku_id,Shop_id,sku_detail(sku_id),…..
from dim_saleswhere ….Order by shop_id……..
Create or replace function
sku_full(sku_id number)
return number isSelect 1If Select 2Else Select 3………Select 30End;
400 lines of SQL+PL/SQL
0.2 second per SKU
350* 0.2 = 70 secondsCREATE PACKAGE BODY cache_sku AS
TYPE sku_cache_aat IS TABLE OF number INDEX BY PLS_INTEGER;
cache sku_cache_aat;
end cache_sku;
cache
FUNCTION sku_detail(skunumber) RETURN number ISBEGINIF NOT cache.EXISTS(sku) THEN
cache(sku) := sku_full(sku);END IF;RETURN cache(sku);
END sku_detail;
Hand-made cache Memory
12X
Hand-made cachePros:- Very fast- Easy to implement- No configuration efforts- No intra-process sync logic burden
Cons:- Cache consumes expensive memory from DB- Memory is allocated per-session basis- PL/SQL or other DB stored logic is required- Vendor specific- No automatic invalidation
Hand-made cache Not OracleNo cache
Hand-made cache Not OracleWith propopulated cache
Case 2 Recommendation engine
Oracle main
OracleDG
Application server
Application server
Load balancer
Client browser
4000users10000 requests per second
In-memory cluster
Case 2 Recommendation engine
Recommendation rules
1. 10 best recommendations by text match 2. Multilanguage capabilities3. Should be taken from 12 previous recognized documents of the client4. If there is no documents – from all clients of the same industry5. If no in same industry – from clients similar by margin and e.t.c
max 100 rows
2-3 columns
max 100 users
Case 2 Recommendation engine
1 week before the Release
1. Recommendations work slow – 5 minutes for 1 document
2. Code freeze
Case 2 Recommendation engine
Solution
1. Use database to cache queries
2. Use Oracle Database Result Cache
Why
1. SQL to get recommendation works 0.5 sec, no options for query tuning – Oracle full text search engine + it is really heavy SQL
2. Same parameters appear at least 5-10 times – cache will be used
3. Data to get recommendations is refreshed 1 hour basis
4. PL/SQL is prohibited
Oracle Result Cache
Oracle result cache
1. Memory area to share query result sets
2. Read consistent – auto invalidation during DML
3. Automatic dependency tracking
4. Minimal changes in the application
5. There is an option how not to change application
6. Could cache PL/SQL logic as well
Oracle Result Cache Way 1
ID of result
First execution Second execution
Oracle Result Cache Table annotations
No query changes but RC works
Oracle Result Cache Table annotations
No annotation on JOBS table – no result cache
Oracle Result Cache Table annotations
2 annotations –result cache works
Oracle Result Cache Dependency TrackingNo in dependency list
Oracle Result Cache Dependency Tracking
Oracle Result Cache Inline Views
Oracle Result Cache Inline Views
Oracle Result Cache Simple Views
Oracle Result Cache Invalidation
Oracle Result Cache Invalidation
Cache is ignored for current session
Good for others sessions
Oracle Result Cache Invalidation
Invalidated after commit for others sessions!
Oracle Result Cache Invalidation
Unexpected cache invalidation
1. SELECT FOR UPDATE statement even there were no changes at all
2. an unindexed foreign key + delete/update/insert a record from parent table
3. Update/delete statements in main table with no records affected + an update to any table where rows were affected.
P.S. Result Cache doesn’t track partitions even if a result cache query works with only 1 partition. Table level tracking always.
Case 2 Recommendation engine
Final solution
1. Do not use annotations – not all queries should be cached
2. Use /*+ result_cache*/ for long-running query
3. Performance is tested. Document recognition –30 seconds.
Time for production
Case 2 Recommendation engine Early morning
Level 3 support
Production incident
Severity 1
Users can’t provide document recognition. Recognition takes 20 minutes
at least. Sessions hangs.
Regards, L2 support team.
Case 2 Recommendation engine
• Active user count: 400
• Database active session count: 1200 = 400* 3
• Row count: 500
• Columns count: 5-8
X5 more!
Monitoring features
View Name DescriptionV$RESULT_CACHE_STATISTICS Lists cache settings and memory usage statisticsV$RESULT_CACHE_MEMORY Lists all the memory blocks and corresponding
statisticsV$RESULT_CACHE_OBJECTS Lists all the objects(cached results and
dependencies) along with their attributesV$RESULT_CACHE_DEPENDENCY Lists the dependency details between the cached
results and dependenciesV$SQLAREA Lists SQL statements issued inside Oracle database
Management features
Procedure Name DescriptionBYPASS Instruction to ignore result cache: for current
session or for all DBFLUSH Clean cacheMEMORY_REPORT Memory detail report
STATUS Checks the status
INVALIDATE Invalidates the specified result-set object
Package: DBMS_RESULT_CACHE
V$RESULT_CACHE_MEMORY
V$RESULT_CACHE_OBJECTS
V$RESULT_CACHE_STATISTICS
DBMS_RESULT_CACHE.MEMORY_REPORT
Locks Result cache prior 12.2 – use very careful!!!
Case 2 Recommendation engine Investigations
Strange queries for 40 small tables each minute:
ETL
Case 2 Recommendation engine Investigations
Result cache annotation
Still 20 minutes per document
Case 2 Recommendation engine
We have received very positive feedback about Oracle Adaptive Statistic feature from customer with respect to adaptive plans. It has proved to be very able at improving system performance for a huge range of workloads. (c) Oracle
20000 queries
10 minutes per document!!!
Via bug? WTF!!!
Result cache latches
Latches are Oracle-internal low-level locks that protect the memory structures of the system global area (SGA) against simultaneous accesses.
Result cache latches
Result cache latches Type 1When setsFirst row of dataset is placed in Result Cache
When releaseLast row of dataset is placed in Result Cache
Who waits
Sessions with same SQL which requested the latch
How much
_RESULT_CACHE_TIMEOUT – 10 seconds. Next - result cache bypassed.
Result cache latches Type 2When setsFirst row of dataset is requested from Result Cache
When releaseLast row of dataset is read from Result Cache
Who waits
Sessions with same SQL which requested the latch
How much
It depends
Result cache latchesLatches not only makes SQL to wait but consumes CPU.
There is no options to get rid of result cache latches – slow for concurrent environment..
Be ready to convince DBA latches wait time saves DB time.
Result cache statistics
NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096Result Size Maximum (Blocks) 204Create Count Success 500
Create Count Failure 0Find Count 20000Invalidation Count 10000Delete Count Invalid 155Delete Count Valid 14000Hash Chain Length 1Find Copy Count 1770Latch (Share) 0
They are equal – the cache is full!!!
Proper results are deleted
Memory estimate
Result Cache Size = row width (bytes)* expected row count
NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096
Memory allocated by blocks!!!
Result Cache Size = block size (if fits in 1024) * expected row count
Administration23 undocumented parameters6 documented parameters
AdministrationParameter Purpose
RESULT_CACHE_MAX_SIZE memory allocated to the server result cache in bytes. default – 0 bytes
RESULT_CACHE_MAX_RESULT maximum amount of server result cache memory (in percent) that can be used for a single result. The default value is 5%.
RESULT_CACHE_MODE Default is MANUAL which means that the cache should be requested explicitly via the RESULT_CACHE hint
_RESULT_CACHE_TIMEOUT (undocumented)
Maximum time a session request for a latch. Default 10 sec.
6 minutes per document!!!
Case 2 Recommendation engineNAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096Result Size Maximum (Blocks) 204Create Count Success 500
Create Count Failure 0Find Count 20000Invalidation Count 10000Delete Count Invalid 155Delete Count Valid 14000Hash Chain Length 1Find Copy Count 1770Latch (Share) 0
A lot of updates on source tables
Case 2 Recommendation engine
Final statistics for result cache
40 seconds per document!!!
NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096Result Size Maximum (Blocks) 204Create Count Success 500
Create Count Failure 0Find Count 20000Invalidation Count 10000Delete Count Invalid 155Delete Count Valid 14000Hash Chain Length 1Find Copy Count 1770Latch (Share) 0
NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 8192Block Count Current 6000Result Size Maximum (Blocks) 204Create Count Success 1000
Create Count Failure 0Find Count 20000Invalidation Count 30Delete Count Invalid 155Delete Count Valid 0Hash Chain Length 1Find Copy Count 1770Latch (Share) 0
Case 2 Recommendation engine Auto-expiring
SHELFLIFE = read-consistent result life time in seconds
SNAPSHOT = NON-read-consistent result life time in seconds
Restrictions• Dictionary tables/views (sys. schema)
• Temporary and external tables
• Sequences (nextval and curval columns)
• Non-deterministic SQL functions:
current_date, current_time, local_timestamp, sys_guid…
• Non-deterministic PL/SQL function: dbms_random, hand-written, …
• Pipelined functions (returning rowsets)
• Only IN parameter with simple data types: no CLOB, BLOB, records, objects, collections, ref cursors
• The same for return result
Result cache inside OracleWhere in Oracle
Jobs related stuffSELECT /*+ NO_STATEMENT_QUEUING RESULT_CACHE (SYSOBJ=TRUE) */
OBJ#,SCHEDULE_LMT,PRIO,JOB_WEIGHT FROM "SYS"."SCHEDULER$_PROGRAM" WHERE bla-bla-bla
APEXSELECT /*+result_cache*/ NAME, VALUE FROM WWV_FLOW_PLATFORM_PREFS
WHERE NAME IN ( 'QOS_MAX_WORKSPACE_REQUESTS', 'QOS_MAX_SESSION_REQUESTS', bla-bla-bla
select *
from v$sqlarea
where upper(sql_fulltext) like
'%RESULT_CACHE%‘
Result cache inside OracleDynamic samplingSELECT /* OPT_DYN_SAMP */ /*+ ALL_ROWS RESULT_CACHE(SNAPSHOT=3600)
opt_param('parallel_execution_enabled', 'false') NO_PARALLEL(SAMPLESUB)
NO_PARALLEL_INDEX(SAMPLESUB) NO_SQL_TUNE */ NVL(SUM(C1),:"SYS_B_0"),
NVL(SUM(C2),:"SYS_B_1") FROM (bla-bla-bla
Adaptive statisticSELECT /* DS_SVC */ /*+ dynamic_sampling(0) no_sql_tune no_monitoring
optimizer_features_enable(default) no_parallel RESULT_CACHE(SNAPSHOT=3600) */
NVL(SUM(C1),0) FROM (SELECT /*+ qb_name("innerQuery") NO_INDEX_FFS bla-bla-bla
Oracle internals for result cache
1
2
3
Database result cache pros&cons
Pros:- Minimal or no intervention at all into application code- No DB stored logic required- Read consistency - Fast in certain scenarios
Cons:- Cache consumes expensive memory from database- Should be properly set up- Sometimes could lead even to performance degradation- Vendor specific
We are not alone
1. Не расчитан размер кэша Troubleshooting Latch Free (Result Cache: RC Latch) Issues When The Result Cache is Full (Doc ID 2143739.1)
2. Блокировки Patch 14665745: DBMS_RESULT_CACHE.MEMORY_REPORT LOCKS OUT WRITERS TO THE RESULT CACHE
Bug 19846066 : LATCH FREE IN RESULT CACHE WHEN QUERYING V$RESULT_CACHE_OBJECTSPatch 14665745: DBMS_RESULT_CACHE.MEMORY_REPORT LOCKS OUT WRITERS TO THE RESULT CACHE
We are not alone
Result_cache_max_size /*+ result_cache*/ removed ordbms_result_cache.add_to_black_list or/*+ no_result_cache*/
We are not alone: Lessons learned
Best approach to roll out updates:
1. Adjust result cache memory
2. Disable cache before bulk loading
dbms_result_cache.bypass;
data ingestion scripts;
Issue dbms_result_cache.bypass(false);
Client side result cache
DBClient cache is ON
Client driver2. Configuration message
Connection thread 1
Connection thread 2
Result cache
3. SQL
Statistics messages
1. connect
1. connect 3. Cached SQL 1
4. Results
4. Results
5. Cached SQL 1
6. Results
CACHE SIZE
Client side result cache Invalidation Case 1
DBClient cache is ON
Client driver
Connection thread 1
Result cache
1. non-cached SQL
2. Invalid resultset list
2. Results
t last cached SQL 1 < Invalidation lag
Invalidation rules = Invalidation rules for Server Side Result Cache
Invalidation
Client side result cache Invalidation Case 1
DBClient cache is ON
Client driver
Result cache
1. Get invalid result set list
2. Invalid result set list
Current time = t Invalidation message + Invalidation lag
Invalidation rules = Invalidation rules for Server Side Result Cache
Invalidation
Client side result cache Configuration
Parameter Purpose
CLIENT_RESULT_CACHE_LAG maximum time in milliseconds that client result cache can lag behind changes in the database that affect its result sets. Default 3000 milliseconds
CLIENT_RESULT_CACHE_SIZE the maximum size of the client result set cache for each client process. Default 0 – not active, min - 32KB, max – 2G
Client side result cache
Client side result cache
Client side result cache
NAME VALUEBlock Size (Bytes) 256Block Count Maximum 256
Block Count Current 3
Create Count Success 1
Create Count Failure 0Find Count 9Invalidation Count 0Delete Count Invalid 0Delete Count Valid 0
= 10
Client side Result cache pros&consPros:
- Cheap client memory
- JDBC and .NET drivers
- Minimal or no intervention at all into application code
- Significant CPU, I/O, network roundtrip reduction
- No extra caching layer/API is required
- No latches
Cons:
- Eventual read consistency with delay
- Oracle OCI client should be installed
- Vendor specific
- 2 Gb per client limitation
- Not enough information about production
Hand-made cache bad scenario• Cache invalidation in case of data changes is a must
• Database stored logic isn’t in favor
• There is strong database developers team
• PL/SQL business logic is already in place
• There are limitations which don’t permit others caching techniques
Hand-made cache good scenario
Server side Result cache bad scenario• SQL populates a large number of distinct result sets
• SQL statement takes more than _RESULT_CACHE_TIMEOUT
• Cached results are requested very often from many sessions
Result cache good scenario• Queries have a limited number of possible result sets
• Result sets are relatively small (200-300 rows)
• SQL statements are relatively expensive
• Queries run against relatively static tables
• There is a strong DBA
Client side Result cache bad scenario• Instant cache invalidation in case of data changes is a must
• Thin drivers are required
• There is fine middle-tier developers team
• Middle tier uses a lot of SQL without any caching layer
• There are DB server hardware limitations
Hand-made cache good scenario
Conclusion1. Estimate memory size properly:
volume (Mb) = (
number of result rows * block size+
avg number of apex usage +
avg number adaptive statistic usage
)/1024
2. Add auto-cleaning capabilities with (snapshot + shelflife) options
3. Bypass the cache during bulk data changes
4. Adjust _result_cache_timeout to expected queries duration
5. Never use FORCE mode for all database
6. Check does FORCE used as expected in table annotations
7. Decide about adaptive statistics: _optimizer_ads_use_result_cache = false
Q&A
Alexander Tokarev
Database expert
DataArt
Alexander.Tokarev@dataart.com
THANK YOU.WE ARE HIRING!