Application Performance Tuning @ KBC Dirk Beauson.

Application Performance Tuning @ KBC

Click icon to add picture

Dirk Beauson

2

Agenda

Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC Application Performance Tuning Organization @ KBC New Challenges Current optimizations Conclusions

3

Agenda

Why Application Performance Tuning @ KBC Tools Used @ KBC Kinds of Application Performance Tuning @ KBC

Reactive Proactive Ad Hoc initiative Appl DBA’s

Application Performance Tuning Organization @ KBC History

New Challenges Current optimizations Conclusions

4

Why Application Performance Tuning @ KBC

RECOGNIZE ?

DB2 online - Poor Response Time- Timeouts/Deadlocks

Long Running DB2 jobs

Locking problems

http://www.google.be/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.conversationagent.com/2010/09/page/2/&ei=vm5cVYjBGKmt7Aas6oGAAw&bvm=bv.93756505,d.bGQ&psig=AFQjCNFo259WkB4xSxPIogFUuoAL33Coiw&ust=1432207279029268

http://www.google.be/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.conversationagent.com/2010/09/page/2/&ei=vm5cVYjBGKmt7Aas6oGAAw&bvm=bv.93756505,d.bGQ&psig=AFQjCNFo259WkB4xSxPIogFUuoAL33Coiw&ust=1432207279029268

http://www.google.be/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.toonvectors.com/clip-art/cartoon-angry-customer/10370&ei=5W5cVb7BJ6_Y7AbOtIDYDA&psig=AFQjCNFo259WkB4xSxPIogFUuoAL33Coiw&ust=1432207279029268

5


80% of the performance problems are related to inefficient application and/or database implementation !!

Building

Cost

Design Implementation

Problems can cost up to 50 times more to fix during implementation than in design !!!

6


Performance Players Applications (Design/Build)

Databases

DB2 subsystem

z/OS

Network

Performance Factors Workload

Throughput

Resources

Contention

7

Why Application Performance Tuning @ KBC Using more CPU

results in using more MSU Can result in higher 4 hour rolling average

Common Performance Issues @ KBC Index design/usage SQL (please know your tables/indexes and explain) Application Design Lock waits Database design

So we want to run applications fast and as cheap as possible

8

Agenda





9

EXPLAIN Tools Used

Simply Explain and PLAN_TABLE Need to look @ extra explain tables !!! (Study DB2 11)

Data Studio (visual explain) SQL Explorer Catalog Manager (to view Current Access Path)

COST*RATE QB PL MIX QTYPE METH ACC MTCH IX TBNAME IXNAME 11.083908 1 1 0 SELECT 0 N 2 N CBTXXX22 CBX7XXX 1.118052 1 2 0 SELECT 1 I 1 N CBTXXX03 CBX1XXX 1.118566 1 3 0 SELECT 1 I 2 N CBTXXX02 CBX3XXX 1.293676 1 4 0 SELECT 3 0 N

Access table CBTXXX22

using index CBX4XXX22 (1 COL)

Join using nested loop join with






Composite table is sorted for ORDER BY clause,

10

EVAL reporting tool

Everyday we report bad BMP’s & MPP’s @ DEV List under construction :

ABEND :ABENDs per BMP / DAY > x (dependent on USER/U0240/pseudo/system abends)

ABENDs per MPP / DAY > y (dependent on USER/U0240/pseudo/system abends)

LOCKING :DEADLOCKS & TIMEOUTS per BMP / DAY > x

DEADLOCKS & TIMEOUTS per MPP / DAY > x

LOCK ESCALATIONS

COMMIT FREQ :COMMIT FREQ in BMP > x sec and updates

ELP time MPP > x SEC and updates

Online Transaction running under IMS

Batch running under IMS

11

EVAL reporting tool

Number of SQL CALL : Number of SQL CALLS in BMP > x

Number of SQL CALLS in MPP > x

Number of GETPAGES per PLAN :Number of GETPAGES per BMP > x

Number of GETPAGES per MPP > x

Number of GETPAGES per PACKAGE/function :Number of GETPAGES per PACKAGE in BMP > x

Number of GETPAGES per PACKAGE in MPP > x

Number of GETPAGES per STMT :Number of GETPAGES per PLAN per STMT in BMP > x

Number of GETPAGES per PLAN per STMT in MPP > x

12

EVAL reporting tool

Long running SQL STMT in MPP :SQL STMT > x SEC DB2 CPU

13

Performance Monitor Tools

Mainview Locking problems

Date / Time Lockout Victim Victim Victim Victim Victim ----------------- Type Plan AuthID Connect CType Corr ID 22MAY-11:36:52.54 DEADLOCK PLAN1 USER001 IKC1 MPP 0037PLAN122MAY-11:29:13.49 TIMEOUT PLAN2 USER002 ITC2 MPP 0012PLAN2

Blocker Blocker Blocker Holder Plan Connect Corr ID AuthID PLAN1 IKC2 0111PLAN1 USER011DISTSERV SERVER db2jcc_appli USER012

14

Performance Analyse Tools

ITSV &EXTA-DB2-tables (SMF info in files & DB2-tables) day-1 : Performance information about every execution (MPP, BMP)

PLANNAME CORRID BEGINTIME DB2AUTH ELPTIME--------++------------++--------------------------++--------++------------PLAN1 0012PLAN1 2015-05-21-09.22.55.309341 USER1 0.077PLAN1 0008PLAN1 2015-05-21-09.23.01.482621 USER1 0.227

CPUTIME CPUTIMEDB2 EWAITIO EWAITLAL-------++------------++------------++------------ 0.011 0.002 0.060 0.000 0.023 0.005 0.167 0.000

BPGETPAGE BPSYNCRD BPPREFET BPDPF---------++-----------++-----------++----------- 37 17 0 0 93 44 0 0

SELECT OPEN FETCH UPDATE INSERT DELETE ------++-----------++-----------++-----------++-----------++----------- 8 1 3 0 0 0 18 2 2 0 1 0

…

15


RapitWeb Interface on ITSV data & extra info

- KBC made Reporting on performance information- Graphical presentation- Also known as

16


ITSV (files and derived DB2 tables) : RapitWeb

- A lot of system information for the system guys- Also lots of information to help DEV

17


ITSV (files and derived DB2 tables) : RapitWeb

- IMS info

- DB2 info

TRX1

18


ITSV (files and derived DB2 tables) : RapitWeb TOPPERS (overall, domain, subsystem, …)

- Toppers Online

- Toppers Batch

MPP1

MPP2

MPP3

BMP1

BMP2

BMP3

19


Apptune Performance problems

- PLAN- PACKAGE- STMT- OBJECT

20


DWH from Apptune PLAN PACKAGE STMT OBJECT STMT/OBJECT Daily and monthly base

- Daily, 2 hour interval (16 days)- Daily (15 months)- Monthly (15 months)

21

Compare Access paths & UOC

Changing access paths Test -> Acceptance :

- A list is published of changing access pathsLIMITATION : only if queryno doesn’t change

Acceptance -> Production :- Same list

No follow up because list is not complete- Search for better (100 %) solution.

TS SCAN instead of IX access

Wrong registration

22

DB2 Object Follow Up Tools

Daily reporting : Object not RW state 70% or more full

- Only TS/PT Not recoverable TS …

Weekly process : Runstats of selected objects

- Own selection <> DSNACCOX IC REORG of selected objects

- Own selection <> DSNACCOX

23

Agenda





24

Reactive Application Performance Tuning Triggers :

25

Reactive : INCIDENTS/PROBLEMS

CASE : REBIND query in function XXXXX is performing badly in acceptance QUERY

26


XXXXX – C200 op ACC: BAD










XXXXX – C200 op PRO: GOOD


scan index CBX7XXX22 (2 COLS)






using index CBX3XXX02 (2 COLS)


27


IX information QUALIFIER: TABLE=CBTXXX22 CmIndex Name Column Seq Num O Lth DatTyp ----v----1----v----2----v----3----v----4----v----5----v----6----v----7----v---- CBX1XXX22 OVK_NR 1 8 A 12 CHAR CBX2XXX22 LOG_BSD_REG_DS 1 1 A 10 TIMESTMP CBX2XXX22 VLG_NR 2 2 A 2 SMALLINT CBX3XXX22 OSP_REK_NR 1 10 A 34 CHAR CBX4XXX22 OLOG_BSD_REG_DS 1 3 A 10 TIMESTMP CBX5XXX22 OVK_NR 1 8 A 12 CHAR CBX5XXX22 LOG_BSD_REG_DS 2 1 A 10 TIMESTMP CBX5XXX22 OSP_REK_NR 3 10 A 34 CHAR CBX6XXX22 OSP_REK_NR 1 10 A 34 CHAR CBX6XXX22 LOG_BSD_REG_DS 2 1 A 10 TIMESTMP CBX6XXX22 OVK_NR 3 8 A 12 CHAR CBX7XXX22 OVK_NR 1 8 A 12 CHAR CBX7XXX22 OLOG_BSD_REG_DS 2 3 A 10 TIMESTMP

QUALIFIER: TABLE=CBTXXX22 C Index Name Table Name UT Cl Col Keys ----v----1----v----2----v----3----v----4----v----5----v----6----v----7----v--- CBX1XXX22 CBTXXX22 D2 NN 1 79K CBX2XXX22 CBTXXX22 P2 YY 2 6573K CBX3XXX22 CBTXXX22 D2 NN 1 95K CBX4XXX22 CBTXXX22 D2 NN 1 4104K CBX5XXX22 CBTXXX22 D2 NN 3 5544K CBX6XXX22 CBTXXX22 D2 NN 3 5544K CBX7XXX22 CBTXXX22 D2 NN 2 4310K

28


EXTA : after BIND Function used in online transaction YYYYY YYYYY info 19/05 DB2 related ? CPU -> GP ? ELP -> IO, wait ?

29


APPTUNE or DWH APPTUNE : after BIND PLAN YYYYY 12743 = open YYYYY info 19/05 08:00 hour interval PLAN ? PACKAGE ? STMT ?

30


EXPLAIN Tune query

Changed Query

COST*RATE QB PL MIX QTYPE METH ACC MTCH IX TBNAME IXNAME 11.083908 1 1 0 SELECT 0 N 2 N CBTXXX22 CBX7XXX 1.118052 1 2 0 SELECT 1 I 1 N CBTXXX03 CBX1XXX 1.118566 1 3 0 SELECT 1 I 2 N CBTXXX02 CBX3XXX 1.293676 1 4 0 SELECT 3 0 N

OK

31


EXTA after fix: Function used in online transaction YYYYY YYYYY info 20/05

32


APPTUNE or DWH APPTUNE after fix PLAN YYYYY 12741 = open YYYYY info 20/05 08:00 hour interval

33


CONCLUSION Change Query :

FROM CBVXXX02 B, CBVXXX03 C, CBVXXX22 D

WHERE C.LOG_BSD_REG_DS = D.LOG_BSD_REG_DS

AND B.OLOG_BSD_REG_DS = C.OLOG_BSD_REG_DS

AND D.OLOG_BSD_REG_DS BETWEEN :WS-WV.WWV-END-DT-LZN AND :WS-WV.

WWV-END-DT-ITV

AND D.OLOG_BSD_REG_DS <= :WS-WV.WWV-HPOS-KEY-REG-DS

AND D.OVK_NR IN(:WTA-OVK-NR-STR-ZN.WTA-VMT-OVK-NR-1, :

In

AND D.OLOG_BSD_REG_DS >= :WS-WV.WWV-END-DT-LZN AND D.OLOG_BSD_REG_DS <=:WS-WV.WWV-END-DT-ITV

I also changed cardinality of PRD_TYPE_KD from 15 to 1005, to force DB2 to use the correct index on CBTXXX02.

34


Next slides hidden : Describes steps to investigate :

- Batch incidents- Online incidents

42


Conclusion Access path problems Much more difficult to force access path in DB2 10 and DB2 11

- We (THINK) know what DB2 has to do- So we try to let DB2 choose that access path- Let DB2 know all you know !!! (extra predicates can help)

Statistics more and more important to DB2- Cardinality columns in indexes -> very important (1e column)- Index Levels

Query Rewrite process !!! (DB2 11 study) Even order of tables in stmt can matter >= and <= better than BETWEEN …

43

Agenda





44

Proactive Application Performance Tuning Sometimes we (applicative DBAs) are really asked to participate

in a new projects Not that often

Reason : Very important project where performance is an issue by design Large Workload Large Throughput needed Locking can kill system Large DB system Very fast respons expected

45

Proactive Application Performance Tuning We give advise to the DEV team how to design the application,

taking into account the previous issues We design the database model together with the DEV team

Lots of rows (> 1.000.000.000) High volume insert Active/Passive data (archive) Complex overview queries Index design (larger index pages, include columns, …)

We co-write SQL, or at least screen the access paths We monitor the application’s behaviour from test over

acceptance to production And are co-responsible for the performance of the application

46

Proactive Application Performance Tuning We set up performance test together Follow up these tests

Test Acceptance Production

47

Agenda





48

Ad Hoc Initiatives Applicative DBA-group

Performance days : On regular base lookup bad processes via EVAL Reporting

- Analyse - Solution- Fix

If possible without touching application -> budget !!!

But sometimes redesign is really needed -> negotiate with DEV

DB2 10 optimizations : Look at index Usage/Design

- Lots of random access via indexes with number of levels > 3Check benefit of larger index pages

Fix

49


Performance days Look at index Usage/Design

- Tables with lots of indexesIndexes used ?

Possible to delete indexes ?

Use include columns in existing indexes

100 Cases done :- Conclusion

Saved a lot of CPU/ELP TIME

Difficult to predict what the effective benefit will be

Predicted value always > the real value

In some cases good estimated values (look per STMT instead of PACKAGE or PLAN)

50


Performance days What’s in for ourselves (Applicative DBA’s) ?

- We share knowledge- Train each other- Try to get everyone on a higher DB2 knowledge level- …

51

Agenda





52

Application Performance Tuning Organization @ KBC

Only DBA’s (both system & applicative) + system guys

1

1

53


KIT Quality Program Performance becomes very important LOOKUP & SOLVE current performance problems Training DEV-people by applicative DBA’s

- Performance Course :Why Performance

Working of DB2

Explain/STAGE 1/STAGE 2/Access Paths/Joins

Performance Tools

1

2

2

54


KIT Quality Program Birth of :

- EVAL Reporting & Incident Proces- RapitWeb- PFA (Performance Aanspreekpunt) (Performance Contact)

Performance specialists @DEV

1

2

2

55


Role PFA : Has to become a performance specialist @ DEV

- Education by applicative DBA’s- 1 per department

Knowlegde -> DEV Proactive Perf role in every new project (obliged !!!) Follow up EVAL reporting They do lots of very good work

ONION model

DBA (S&A)SYST

PFA

DEV

56


PFA become PFM Also reporting role about Performance per department Less problems to solve No fulltime PFM need anymore So have to do other DEV tasks Performance less important

1 32

3 anagerM

57


Less PFM New PFM PFM not a real role anymore

1 32 4

4

58


Performance back to DBA Reactive Triggered (no continu follow up) EVAL Reporting (no incidents process)

More performance issues

1 324

5

5

59


Performance becomes more and more important Agile Mobile More need for PFA

1 324 6

5

6

60

Agenda





61

New Performance Challenges

AGILE New way of Development

- Not always started with Global Technical Design- Build on build on build on … repetitive

Not always first Database Design in mind

- If needed first Database Design not corrected- Misuse DB2 objects (columns, number of rows,…)- Less communication with DBA’s

Only little things applied at a time

No big changes

- Performance not always in mind during repetitive DEV process

62


Mobile/Online workload Much more sessions Much more executions Workload Throughput Locking

63


More Data More tables More rows, lots of rows !!! Data has to be active much longer Sizing objects much more difficult Follow up usage % object Problems table-PT full

- Acute increase in the number of rows- Other hotspots

64


More Data Problems index-PT full

- Usage larger columns in IX (CHAR 50 and > 50)- In combination with no compression- IX larger than TS/PT

Only follw up TS/PT space usage !!!

65


BIG DATA DWH-environment Link IDAA Data Analists

- Play in DB2- Own tables- Keep all data

Forever ???

66

Agenda





67

Current Optimizations

Redesign EVAL process : Production problems

- Visable in EVAL since acceptance- Not followed up

DEV wants better EVAL process :- INCIDENTs ?

NO : Applicatieve DBA group cannot afford 1 person fulltime being busy with that

- Better EVAL process, designed together with DEV : OK ;o)- Exception reporting :

Problem parked till date xx/xx/xxxx

- Only reporting per SUBSYSTEMNot all problems to everyone

68

Current Optimizations

Quality Assurance DB2 objects : Better follow up process

- Not only follow up TS/PT- Also IX

Usage of large columns 50 and > 50

Better weekly process :- Runstats

Better follow up exceptions

- Reorgs- Evaluate DSNACCOX rules versus our own rules based on RTS

Lots of Differences !!!

69

Agenda





70

Conclusions

We are going back in the right direction. Performance becomes important again

Is Jan (System Engineer) satisfied with our current Performance Setup ? Jan ?

Is our current Performance Setup Bulletproof ? Jan ?

71

Questions ???

Application Performance Tuning @ KBC Dirk Beauson.

Documents

Transcript of Application Performance Tuning @ KBC Dirk Beauson.