Post on 18-Jan-2018
description
1
CS 540 Database Management Systems
Lecture 5: DBMS Architecture, storage, and access methods
2
Database System Implementation
Conceptual Design
Physical Storage Schema
Entity Relationship(ER)
Model
Relational Model Files and Indexes
User Requirements
3
The advantage of RDBMS• It separates logical level (schema) from physical
level (implementation). • Physical data independence– Users do not worry about how their data is stored and
processes on the physical devices.– It is all SQL!– Their queries work over (almost) all RDBMS
deployments.
4
Challenges in physical level• Processor: 10000 – 100000 MIPS• Main memory: around 10 Gb/ sec.• Secondary storage: higher capacity and durability• Disk random access – Seek time + rotational latency + transfer time– Seek time: 4 ms - 15 ms!– Rotational latency: 2 ms – 7 ms!– Transfer time: at most 1000 Mb/ sec– Read, write in blocks.
5
Gloomy future: Moor’s law• Speed of processors and cost and maximum
capacity of storage increase exponentially over time.
• But storage (main and secondary) access time grows much more slowly.
6
Random access versus sequential access
• Disk random access : Seek time + rotational latency + transfer time.
• Disk sequential access: reading blocks next to each other
• No seek time or rotational latency • Much faster than random access
7
DBMS Architecture
Query Executor
Buffer Manager
Storage Manager
Storage
Transaction Manager
Logging & Recovery
Lock Manager
Buffers Lock Tables
Main Memory
User/Web Forms/Applications/DBA
query transaction
Query Optimizer
Query Rewriter
Query Parser
Files & Access Methods
Process manager
8
DBMS Architecture
Query Executor
Buffer Manager
Storage Manager
Storage
Transaction Manager
Logging & Recovery
Lock Manager
Buffers Lock Tables
Main Memory
User/Web Forms/Applications/DBA
query transaction
Query Optimizer
Query Rewriter
Query Parser
Files & Access Methods
Process manager
This lecture
9
A Design Dilemma
• To what extent should we reuse OS services? • Reuse as much as we can – Performance problem (inefficient)– Lack of control (incorrect crash recovery)
• Replicating some OS functions (“mini OS”) – Have its own buffer pool – Directly manage record structures with files –…
10
OS vs. DBMS Similarities?
• What do they manage?• What do they provide?
11
OS vs. DBMS: Similarities
• Purpose of an OS: – managing hardware– presenting interface abstraction to applications
• DBMS is in some sense an OS?– DBMS manages data– presenting interface abstraction to applications
• Both as API for application development!
12
OS vs. DBMS: Related Concepts• Process Management What DB concepts? – process synchronization– deadlock handling
• Storage management What DB concepts?– virtual memory– file system
13
OS vs. DBMS: Differences?
14
OS vs. DBMS: Differences• DBMS: Top-down to encapsulate high-level semantics!– Data• data with particular logical structures
– Queries• query language with well defined operations
– Transactions• transactions with ACID properties
• OS: Bottom-up to present low-level hardware
15
Problems with DBMS on top of OS
• Buffer pool management
• File system
• Process management
• Consistency control
• Paged virtual memory
16
Buffer Pool Management
• Performance of system calls• LRU replacement – Query-aware replacement needed for performance– Circular access: 1, 2, …, n, 1, 2, ..
• Prefetching– DBMS knows exactly which block is to be fetched next
• Crash recovery – Need “selected force out”
17
Relations vs. File system
• Data object abstraction– file: array of characters– relation: set of tuples
• Physical contiguity: – large DB files want clustering of blocks• sol1: managing raw disks by DBMS• sol2: simulate by managing free spaces in DBMS
• Multiple trees (access methods)– file access: directory hierarchy (user access method)– block access: inodes– tuple access: DBMS indexes
18
Process management• Reuse OS process management
– One process for each user• Problem: DB processes are large
– long time to switch between processes• Problem: critical sections
– Processes may have to wait for a descheduled process that has locks.
– n server processes that handle users’ requests• duplication of OS multi-tasking inside servers! • communication between processes:
– Message passing is not efficient
• Solutions: OS implements – favored processes
• not forced out, relinquish the control voluntarily. – faster message passing methods.
19
Consistency control
• OS provides some support for locking and recovery.– OS provides lock on files– DB requires lock on smaller units like tuples
• Commit point – Buffer manager ensures all changes are flushed on disk.– Buffer manager must know the inside of transactions.
20
State of the art• DBMSs duplicate some OS functionalities.• OS customized for DBMS
21
Access methods• The methods that RDBMS uses to retrieve the
data.• Attribute value(s) Tuple(s)
22
Types of search queries• Point query over Product(name, price) Select *
From ProductWhere name = ‘IPad-Pro’;
• Range query over Product(name, price) Select *
From ProductWhere price > 2 AND price <
10;
23
Types of access methods• Full table scan– Inefficient for both point and range queries.
• Sequential access– Efficient for both point and range queries. – Should keep the file sorted. • Inefficient to maintain
• Middle ground?
24
Indexing• An old idea
25
Index• A data structure that speeds up selecting tuples in
a relation based on some search keys.• Search key– A subset of the attributes in a relation–May not be the same as the (primary) key
• Entries in an index– (k, r)– k is the search key.– r is the pointer to a record (record id).
26
Index• Data file stores the table data. • Index file stores the index data structure.
• Index file is smaller than the data file. • Ideally, the index should fit in the main memory.
10
20
30
40
50
60
70
80
10
20
30
40
50
60
Data File Index File
27
Well known index structures• B+ trees:– very popular
• Hash tables: – Not frequently used
28
B+ trees• The index of a very large data file gets too large.
• How about building an index for the index file?
• A multi-level index, or a tree
29
B+ trees• Degree of the tree: d• Each node (except root) stores [d, 2d] keys:
10 32 94
[A , 10) [10, 32) [32, 94) [94, B)
Non-leaf nodes
12 28 32
12 28 32
39 41 65Leaf nodes
Records
30
Example
60
19 50 80 90 110
12 13 17 19 21 30 40 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
d = 2
31
Retrieving tuples using B+ tree • Point queries– Start from the root and follow the links to the leaf.
• Range queries– Find the lowest point in the range.– Then, follow the links between the nodes.
• The top levels are kept in the buffer pool.
32
Inserting a new key• Pick the proper leaf node and insert the key.• If the node contains more than 2d keys, split the
node and insert the extra node in the parent.
– If leaf level, add K3 to the right node
K1 K2 K3 K4 K5
R0 R1 R2 R3 R4 R5
K1 K2
R0 R1 R2
K4 K5
R3 R4 R5
(K3, ) parent
33
Example
60
19 50 80 90 110
12 13 17 19 21 30 40 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Insert K = 18
34
Insertion
60
19 50 80 90 110
12 13 17 18 19 21 30 40 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Insert K = 18
18
35
Insertion
60
19 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Insert K= 20
19 20 21 30 40
2018
36
Insertion
60
19 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Need to split the node
19 20 21 30 40
2018
37
Insertion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Split and update the parent node.What if we need to split the root?
20
19 20 21 30 40
18
38
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Delete K = 21
20
19 20 21 30 40
18
39
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
Note: K = 21 may still remain in the internal levels
20
19 20 30 40
18
40
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
Delete K = 20
20
19 20 30 40
18
41
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
We need to update the number of keys on the node: Borrow from siblings: rotate
19 30 40
18
42
Deletion
60
19 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
We need to update the number of keys on the node: Borrow from siblings: rotate
18 19 30 40
18
43
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
We need to update the number of keys on the node: Borrow from siblings: rotate
18 19 30 40
18
44
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
What if we cannot borrow from siblings?Example: delete K = 30
18 19 30 40
18
45
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
What if we cannot borrow from siblings?Merge with a sibling.
18 19 40
18
46
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
What if we cannot borrow from siblings?Merge siblings!
18 19 40
18
47
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
What to do with the dangling key and pointer? simply remove them
18 19 40
18
48
Deletion
60
18 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
Final tree
18 19 40
18
49
What You Should Know
• What are some major limitations of services provided by an OS in supporting a DBMS?
• In response to such limitations, what does a DBMS do?
• B+ tree indexing