Databases in Databases in ContextContext

Wendy MoncurWendy Moncur

Department of Computing Science, Department of Computing Science,

University of AberdeenUniversity of Aberdeen

Databases in ContextDatabases in Context

Database design in a major bankDatabase design in a major bank

Database managementDatabase management

6000-table Personnel database6000-table Personnel database

Who am I ?Who am I ?

Wendy MoncurWendy Moncur

DataBase AdministratorDataBase Administrator (DBA) at one of UK’s (DBA) at one of UK’s largest banks.largest banks.

Designed databases for high performance & Designed databases for high performance & availability. availability.

19 years industry experience.19 years industry experience.

Platform: DB2 & SQLPlatform: DB2 & SQL

Largest database: 6000 tablesLargest database: 6000 tables

Why listen?Why listen?

DBA Average Minimum Salary DBA Average Minimum Salary £41,896£41,896

DBA Average Maximum Salary DBA Average Maximum Salary £47,147£47,147

Source: http://www.itjobswatch.co.uk Source: http://www.itjobswatch.co.uk

What does a What does a DBADBA do? do?

Database design & optimisationDatabase design & optimisation

Quality assurance of SQLQuality assurance of SQL

Performance managementPerformance management

Database administrationDatabase administration

Case study: the monster database

Case study: the monster database

• 6000+ tables• 18000+ indexes

Part1: ChallengesPart1: Challenges

““One size fits all”One size fits all” External supplierExternal supplier 6000+ tables6000+ tables 18000+ indexes18000+ indexes 1 tablespace1 tablespace Short timescaleShort timescale

Challenges: “one size fits all”?Challenges: “one size fits all”?

One size does One size does notnot fit all. fit all.

Performance of SQL statements Performance of SQL statements dependent on:dependent on:

Database designDatabase design Index designIndex design

The The DATADATA

Challenges: “one size fits all”?Challenges: “one size fits all”?

Every company has different requirements.Every company has different requirements.

Customers Customers demand demand high performance... and high performance... and control the budget. control the budget.

Service Level Agreements (Service Level Agreements (SLAsSLAs) dictate … ) dictate … Minimum transaction speedMinimum transaction speed Number of concurrent usersNumber of concurrent users Number of remote locationsNumber of remote locations Daily system availabilityDaily system availability

Database must be Database must be tailored tailored to achieve site-specific to achieve site-specific SLAs.SLAs.

Challenges: external Challenges: external suppliersupplier

Software package & database from Software package & database from external supplier.external supplier.

CannotCannot change this. change this.

Challenges: 6,000+ tablesChallenges: 6,000+ tables

Cannot Cannot change tables: no denormalisation change tables: no denormalisation allowed.allowed.

Supplied program code demands these tables Supplied program code demands these tables exist. exist.

Cannot change supplied program code unless Cannot change supplied program code unless essentialessential..

Challenges: 18,000+ indexesChallenges: 18,000+ indexes

Can Can change indexes:change indexes:

Unique indexesUnique indexes

Clustering indexesClustering indexes

Secondary indexesSecondary indexes

Unique indexUnique index

Defines what makes a row unique. Defines what makes a row unique.

Components of the index Components of the index cannotcannot be be changed. changed.

Order of componentsOrder of components cancan be be changed.changed.

Unique indexUnique index

E.g. – for Table “E.g. – for Table “EMPLOYEEEMPLOYEE” ”

Unique index =Unique index = DateOfBirth, Firstname, Surname.DateOfBirth, Firstname, Surname.

Most queries ask for data where only Most queries ask for data where only Surname, Surname, FirstnameFirstname are known.are known.

SELECT Surname, Firstname, DateOfBirthSELECT Surname, Firstname, DateOfBirthFrom EmployeeFrom EmployeeWhere Where SurnameSurname = “Jenkins” And= “Jenkins” And FirstnameFirstname = “Malcolm” ;= “Malcolm” ;

Recommendation: Change order of unique index to Recommendation: Change order of unique index to Surname, Firstname, DateOfBirth. Surname, Firstname, DateOfBirth.

Clustering indexesClustering indexes

Defines the physical order in which rows Defines the physical order in which rows of data should be stored.of data should be stored.

Components of the index Components of the index cancan be be changed. changed.

Order of components Order of components cancan be changed. be changed.

Clustering indexesClustering indexes

E.g. – Table “EMPLOYEE” E.g. – Table “EMPLOYEE”

Clustering index = Clustering index = DateOfBirthDateOfBirth

Yet most queries order by Yet most queries order by EmploymentStartDateEmploymentStartDate

SELECT EmploymentStartDate, Surname, FirstnameSELECT EmploymentStartDate, Surname, Firstname

From EmployeeFrom Employee

Where Surname = “Jenkins” And Firstname = “Malcolm” ;Where Surname = “Jenkins” And Firstname = “Malcolm” ;

Order by Order by EmploymentStartDate;EmploymentStartDate;

Recommendation: Change clustering index to use Recommendation: Change clustering index to use

EmploymentStartDate.EmploymentStartDate.

Secondary indexesSecondary indexes

Not unique.Not unique.

Do not dictate how the data is to be held.Do not dictate how the data is to be held.

Created to improve performance of queries and updates.Created to improve performance of queries and updates.

Increases cost of insert and update, as must be created and Increases cost of insert and update, as must be created and maintained along with the table.maintained along with the table.

Recommendation: Recommendation: Drop superfluous secondary Drop superfluous secondary indexes. indexes.

Challenges: One Challenges: One tablespacetablespace

Many tablespacesMany tablespaces

Create many new tablespaces. Create many new tablespaces. Split the tables between them, according to table Split the tables between them, according to table

function. function.

At least 4 test environments:At least 4 test environments:

96,000 objects!96,000 objects! ((6,000 tables + 18,000 indexes) * 4 environments)((6,000 tables + 18,000 indexes) * 4 environments)

3 months3 months

Challenge: Short Challenge: Short timescaletimescale

Vanilla Unit test System test Pre-live

ToolsTools

Use tools to…Use tools to…

Check performance of each SQL Check performance of each SQL statementstatement

Manage change processManage change process

Check performanceCheck performance

““EXPLAIN”EXPLAIN”

Evaluates route to data for every SQL statement.Evaluates route to data for every SQL statement.

Identifies what indexes are used Identifies what indexes are used

Doesn’tDoesn’t identify redundant indexes identify redundant indexes

Doesn’tDoesn’t identify indexes that need to be changed. identify indexes that need to be changed.

Manage change process Manage change process

Rigorous control neededRigorous control needed

Achieved through…Achieved through… Consistent naming standards Consistent naming standards Detailed record of every changeDetailed record of every change Consistent route through environments, no short cutsConsistent route through environments, no short cuts DBA tools DBA tools

Part1: Recap of Part1: Recap of challenges challenges

Can’t change:Can’t change:

““One size fits all”One size fits all”

External supplierExternal supplier

6000+ tables6000+ tables

CanCan change: change:

18000+ indexes18000+ indexes

1 tablespace1 tablespace

Short timescaleShort timescale

Part2: The Production Part2: The Production DatabaseDatabase

Does it perform?Does it perform?

Can the Can the rightright people use it? people use it?

If disaster strikes, can the data be recovered?If disaster strikes, can the data be recovered?

Does the database perform?Does the database perform?

Database performance monitored against Service Database performance monitored against Service Level Agreements (SLAs).Level Agreements (SLAs).

Regular health checks carried out:Regular health checks carried out: Data stored in sequence?Data stored in sequence? Enough space? Enough space?

If sub-standard performance, further database If sub-standard performance, further database design work done. design work done.

Can the right people access the data?Can the right people access the data?

PERSONNEL database

Can the right people access the data?Can the right people access the data?

Personnel team

Query & update data at individual or regional level

PERSONNEL database

Can the right people access the Can the right people access the data?data?

Personnel team

Query & update data at individual or regional level

PERSONNEL database

DBA

Backup/ restore data

Reorganise data

Change database definitions

Update statistics on data

Can the right people access the Can the right people access the data?data?

Personnel team

Query & update data at individual or regional level

PERSONNEL database

DBA

Backup/ restore data

Reorganise data

Change database definitions

Update statistics on data

Chief executive

Employee statistics

Can the right people access the Can the right people access the data?data?

Personnel team

Query & update data at individual or regional level

PERSONNEL database

DBA

Backup/ restore data

Reorganise data

Change database definitions

Update statistics on data

Chief executive

Employee statistics

Staff member

Their own data

Can the right people use the Can the right people use the database?database?

Different people, different information needs.Different people, different information needs.

Sensitive data – salary, health, discipline…Sensitive data – salary, health, discipline…

Solution Solution VIEWSVIEWS Transaction ManagementTransaction Management

If disaster strikes, If disaster strikes, can the data be recovered?can the data be recovered?

Robust backup & recovery strategies for:Robust backup & recovery strategies for: Hardware failureHardware failure Software failureSoftware failure

Part2: Recap of Part2: Recap of Production Database Production Database

issuesissues Database must perform to acceptable level.Database must perform to acceptable level.

Only the Only the rightright people should have access to any people should have access to any data item.data item.

No matter what, the data must be recoverable.No matter what, the data must be recoverable.