CS317 File and Database Systemsmercury.pr.erau.edu/~siewerts/cs317/documents/Lectures/Lecture...–...
-
Upload
truongcong -
Category
Documents
-
view
230 -
download
0
Transcript of CS317 File and Database Systemsmercury.pr.erau.edu/~siewerts/cs317/documents/Lectures/Lecture...–...
October 23, 2017 Sam Siewert
CS317File and Database Systems
Lecture 10 – Physical DBMS Designhttp://commons.wikimedia.org/wiki/Category:R-tree#mediaviewer/File:R-tree_with_Guttman%27s_quadratic_split.png
RemindersAssignment #4 – Grading in Progress
Assignment #5 (Available on Canvas)– Physical DB Design Practice– Propose your Final DBMS Project
Assignment #6, DBMS Project of Your Interest – FINAL ORAL PRESENTATION– Design Schema for DBMS project in a small team
Logical design focusNormalizationPhysical is MySQL on PRClab
– Combine Network Applications with DBMS in C/C++, JDBC, or Python - http://www.mysql.com/products/connector/
– Add Stored Programs and Triggers– Add Views– Create Transactions where needed
Sam Siewert 2
Goals for This WeekWrap-Up DBMS Conceptual Design
Wrap-Up DBMS Logical Design
Introduce DBMS Physical Design
DBMS Indexing [B-Tree & B+-Tree, Hash, R-Tree]– Indexing Fundamentals– ISAM – Indexed Sequential Access Method [Records in Files]
MyISAM – Index, Format, Data Files for Each Table– InnoDB – Cached Indexes and Data, Several Files or Raw Disk Partitions– Memory (Heap) – Unordered Set of Records, Hash Index, B-Tree Options– Introduction to B and B+ Trees (Bayer or perhaps Boeing or Balance Tree)– Hash - Fast Table Lookup, Collisions handled with List or Tree (Similar, but
different than data digest or cryptographic - e.g. MD5 Hash)– Exploration beyond B-Trees - R-Tree, X-Tree (Not covered)
Mapping to DBMS to Storage– Memory– Nand Flash and SSD– HDD– File Systems and Block Storage
Sam Siewert 3
DBMS Design in 10 StepsDBMS Design - Decomposition1. Conceptual Design – Analysis, Centralized or View Integration2. Build and Validate Logical Data Model – ER/EER,
Normalization, Schema Prototype (Workbench)3. Translate Logical Data Model for Target DBMS (MySQL)4. Design File Organization (or Block) and Indexing5. Triggers and SQL/PSM for Referential Integrity6. User View Design (Lossless Join Verification)7. Security8. Transactions for Complex Updates9. Redundancy (RTO/RPO and RAID)10. Connectors for App Interfaces
Covered, To be covered
Sam Siewert 4
Logical DBMS EngineeringIdeally use a CASE tool like Workbench – EER DB
Sam Siewert 5
ForwardReverse
SQL
EER
Workbench ManualModel Editor – Name your Model to match DB deployment Target name – e.g. siewertsDHv1_3, siewertsdb, etc.Test and Interaction Tab (Queries, Schemas,
Sam Siewert 6
Workbench ModelHigh Level View of Logical Model and Test SQL Scripts
Sam Siewert 7
Deployed DB from EER Logical Design
Logical to Physical MappingsBranch Table Logical Design Details– Primary Key
Designation, Not-NULL, Default, etc.
– Foreign Keys, Triggers, etc.
Branch Table Physical Design Details– Indexes – Engine
[InnoDB, MyISAM, MEMORY], Columns to Index, Storage Type [BTREE, HASH]
Sam Siewert 8
Engine = MEMORY
Index Type = BTREE
Indexing for Optimal Query
9
Physical Design - Objectives
Purpose of physical database design.
How to map the logical database design to a physical database design.
How to design base relations for target DBMS.
How to design general constraints for target DBMS.
10
Physical Design - Objectives
How to select appropriate file organizations based on analysis of transactions.
When to use secondary indexes to improve performance.
How to estimate the size of the database.
How to design user views.
How to design security mechanisms to satisfy user requirements.
11
Logical v. Physical Database Design
Sources of information for physical design process includes logical data model and documentation that describes model.
Logical database design is concerned with the what, physical database design is concerned with the how.
12
Physical Database Design
• Process of producing a description of the implementation of the database on secondary storage.
• It describes the base relations, file organizations, and indexes used to achieve efficient access to the data, and any associated integrity constraints and security measures.
13
Overview of Physical Database Design Methodology
Step 1 Conceptual Design (Chapters 10-12, Lifecycle, Analysis – Centralized, View Integration)
Step 2 Build and Validate Logical Data Model (Chapters 12 – 14 – ER/EER, Normalization)
Step 3 Translate logical data model for target DBMS– Step 3.1 Design base relations– Step 3.2 Design representation of derived data – Step 3.3 Design general constraints
14
Overview of Physical Database Design Methodology
Step 4 Design file organizations and indexes– Step 4.1 Analyze transactions– Step 4.2 Choose file organizations– Step 4.3 Choose indexes– Step 4.4 Estimate disk space requirements
15
Overview of Physical Database Design Methodology
Step 5 Design user views (Lossless Join)Step 6 Design security mechanismsStep 7 Consider the introduction of controlled redundancy Step 8 Monitor and tune operational system
Step 9 RAID and DR (Disaster Recovery), ScalingStep 10 Applications and Connectors
16
Step 3 Translate Logical Data Model for Target DBMS
To produce a relational database schema from the logical data model that can be implemented in the target DBMS.Need to know functionality of target DBMS such as how to create base relations and whether the system supports the definition of:– PKs, FKs, and AKs;– required data – i.e. whether system supports NOT
NULL;– domains;– relational integrity constraints;– general constraints.
17
Step 3.1 Design base relations
To decide how to represent base relations identified in logical model in target DBMS.
For each relation, need to define:– the name of the relation;– a list of simple attributes in brackets;– the PK and, where appropriate, AKs and FKs.– referential integrity constraints for any FKs identified.
18
Step 3.1 Design base relations
From data dictionary, we have for each attribute:– its domain, consisting of a data type, length,
and any constraints on the domain;– an optional default value for the attribute;– whether it can hold nulls;– whether it is derived, and if so, how it should
be computed.
19
DBDL for the PropertyForRent Relation
20
Step 3.2 Design representation of derived data
To decide how to represent any derived data present in logical data model in target DBMS.
Examine logical data model and data dictionary,and produce list of all derived attributes.
Derived attribute can be stored in database orcalculated every time it is needed.
21
Step 3.2 Design representation of derived data
Option selected is based on:– additional cost to store the derived data and keep it
consistent with operational data from which it isderived;
– cost to calculate it each time it is required.
Less expensive option is chosen subject to performance constraints.
22
PropertyforRent Relation and StaffRelation with Derived Attribute
noOfProperties
23
Step 3.3 Design general constraints
To design the general constraints for target DBMS.
Some DBMS provide more facilities than others for defining enterprise constraints. Example:
CONSTRAINT StaffNotHandlingTooMuchCHECK (NOT EXISTS (SELECT staffNo
FROM PropertyForRentGROUP BY staffNoHAVING COUNT(*) > 100))
24
Step 4 Design File Organizations and Indexes
To determine optimal file organizations to store the base relations and the indexes that are required to achieve acceptable performance; that is, the way in which relations and tuples will be held on secondary storage.
Must understand the typical workload that database must support.
25
Cross-referencing transactions and relations
26
Example Transaction Usage Map
27
Example Transaction Analysis Form
28
Step 4.2 Choose file organizations
To determine an efficient file organization for each base relation.
File organizations include Heap [Unordered Set of Records], Hash, Indexed Sequential Access Method (ISAM), B+-Tree, and Clusters.
Some DBMSs may not allow selection of file organizations.
29
Step 4.3 Choose indexes
To determine whether adding indexes will improve the performance of the system.
One approach is to keep tuples unordered andcreate as many secondary indexes as necessary.
30
Step 4.3 Choose indexes
Another approach is to order tuples in therelation by specifying a primary or clusteringindex.
In this case, choose the attribute for ordering orclustering the tuples as:– attribute that is used most often for join operations -
this makes join operation more efficient, or– attribute that is used most often to access the tuples in
a relation in order of that attribute.
31
Step 4.3 Choose indexes
If ordering attribute chosen is key of relation,index will be a primary index; otherwise, indexwill be a clustering index.
Each relation can only have either a primaryindex or a clustering index.
Secondary indexes provide a mechanism forspecifying an additional key for a base relationthat can be used to retrieve data more efficiently.
32
Step 4.3 Choose indexes
Have to balance overhead involved in maintenanceand use of secondary indexes against performanceimprovement gained when retrieving data.This includes:– adding an index record to every secondary index
whenever tuple is inserted;– updating secondary index when corresponding
tuple updated;– increase in disk space needed to store secondary
index;– possible performance degradation during query
optimization to consider all secondary indexes.
33
Step 4.3 Choose indexes –Guidelines for choosing ‘wish-list’
1. Do not index small relations. 2. Index PK of a relation if it is not a key of the file
organization. 3. Add secondary index to a FK if it is frequently
accessed. 4. Add secondary index to any attribute heavily used
as a secondary key.5. Add secondary index on attributes involved in:
selection or join criteria; ORDER BY; GROUP BY; and other operations involving sorting (such as UNION or DISTINCT).
34
Step 4.3 Choose indexes –Guidelines for choosing ‘wish-list’
6. Add secondary index on attributes involved in built-in functions.
7. Add secondary index on attributes that could result in an index-only plan.
8. Avoid indexing an attribute or relation that is frequently updated.
9. Avoid indexing an attribute if the query will retrieve a significant proportion of the relation.
10. Avoid indexing attributes that consist of long character strings.
Note on MySQL IndexesMySQL Create Index
MySQL 5.7 Provides Indexes to Speed Query for Query Engines as Follows
Spatial R-Tree Indexing Possible for MyISAM
Using Spatial Data Extensions
MEMORY Storage Engine –Hash Option Sam Siewert 35
Storage Engine Supported Index Types
MyISAM B-tree, R-Tree Spatial on POINT,GEOMETRY Columns
InnoDB B-tree
Memory (Heap) Hash, B-Tree
Defaults - Common
Options – Advanced Indexing
B-TreeRudolf Bayer, Edward McCreight – Boeing
Optimized for Read/Write Large Block Data – DBMS and File Systems
B+-Tree is Variant with Keys [Indexes] Only for Internal Nodes and all Keys and Data Found in Leaves– Keys [Indexes] Repeated in Tree– Keys [Indexes] + Data in Sequential Order in Leaves
Goal is to Increase the Size of Nodes to Match File System Block Size [E.g. 4K … 64K] with N Keys [Data] Stored in Each Tree Node [8x512 … 128x512 Sectors]
Sam Siewert 36
B-Tree ExamplestaffNo Key, staff tuples with multiple attributes in staffTableOrder N=4, 4 Pointers, 3 Keys per Node, 2 Pointers, 1 Key Minimum [Splits and Combines on Insert and Delete]Tuple Pointer for All Keys [Implicit]Insert Tuples for staffNo S005, S003, S021, S009, S001, S013
Sam Siewert 37
Ptr1 S003 Ptr2 S005 Ptr3 S0021 Ptr4
Ptr1 S001 Ptr2 S003 Ptr3 S005 Ptr4 Ptr1 S013 Ptr2 S021 Ptr3
Ptr1 S009 Ptr2
First 3 inserts, fill root node
S009 insert, splits root nodeS001 and S013 fill out children
B-Tree Inserts ContinuedSplits Continue on Staff Table InsertsInsert S002, …
Sam Siewert 38
Ptr1 S001 Ptr2 S003 Ptr3 S005 Ptr4 Ptr1 S013 Ptr2 S021 Ptr3
Ptr1 S009 Ptr2 S002 insert splits left child andpromotes S003 to root node
Ptr1 S001 Ptr2 S002 Ptr3 Ptr1 S005 Ptr2 Ptr1 S013 Ptr2 S021 Ptr3
Ptr1 S003 Ptr2 S009 Ptr3
B-TreeProvides Nice WHERE Clause Support
Index [Key] Appears ONCE in Each Node
Downside – SELECT * with Tuples in staffNo ASC or DSC Order is Time Consuming
Split and Combines Required on Insert/Delete
Key, Pointers, and Tuple Data in Nodes Limit “Bushiness”
B+-Tree – Stores Keys, Pointers ONLY on Internal Nodes, Increasing Bushiness [Key for File systems], Record/File Data ONLY on Leaf Nodes
Sam Siewert 39
B+-TreeReplication of Keys [Indexes]Tuple Data ONLY in LeavesUseful for Key [Index] Cache and Storage of Leaves
Sam Siewert 40
Ptr1 S001 Ptr2 S002 Ptr3
Ptr1 S003 Ptr2 S005 Ptr3
Ptr1 S009 Ptr2 S013 Ptr3 S021 Ptr4
Ptr1 S003 Ptr2 S009 Ptr3
DISK DRIVE
I/O CACHE
B-Tree DetailsCovered in Data Structures – B-tree Example in C
“The Ubiquitous B-Tree”, by Douglas Comer– Insertion Balancing– Deletion Balancing– Cost of Operations
Recall Disk Drive Performance– 100 to 200 Random I/Os per Second [Seek + Rotate is 5 to 10
msec]– Sequential Rates 100 to 200 Mbytes per Second
MySQL Memory (Heap) B-Tree for Small Tables – Fit in Memory Pages [Cache Lines] Sam Siewert 41
Hash TableIndex = Function (Candidate Key)E.g. Index = MD5(staffNo, staffName) % Table_Size
Perfect Hash Guarantees No 2 Staff will Map to Same Index
Hash Collision Requires Secondary Collision Structure –E.g. List
Great for Small Tables and Data Sets
Typically in Memory – Performs like LUT [Look Up Table or Array] Sam Siewert 42
Multi-Value QueryB-Tree Great for Selection and Theta-Join– Finding Tuples Satisfying WHERE clauses
What About Complex Query for Multiple Conditions?– E.g. WHERE clause and Distance from Current Location < 20 miles– Satisfy B-Tree WHERE clause– Check X,Y Distance Constraint as Post Filter?
Possible to Combine Multiple Dimensions for Query in the Tree Index?
R-Trees or Rectangular Trees, X-Tree, etc. [Not Covering, but Interesting]
Wrap-Up B-Trees and RAID Next Week …
Sam Siewert 43