PA State Police (PSP) Database Design

PA State Police (PSP)

Database Design

Data Modeling Standards

Database Team Version 1.4 Date: 03/19/2014

SECURITY WARNING

The information contained herein is proprietary to the Commonwealth of Pennsylvania and must not be

disclosed to un-authorized personnel. The recipient of this document, by its retention and use, agrees to

protect the information contained herein. Readers are advised that this document may be subject to the

terms of a non-disclosure agreement.

DO NOT DISCLOSE ANY OF THIS INFORMATION WITHOUT OBTAINING PERMISSION FROM

THE MANAGEMENT RESPONSIBLE FOR THIS DOCUMENT.

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

APPENDIX R 6100038971

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

janibrown

Typewritten Text

COMMONWEALTH OF PENNSYLVANIA PENSYLVANIA STATE POLICE

DATABASE DESIGN DATA MODELING STANDARDS PAGE 2 OF 8

Version History Date Version Modified By / Approved By Section(s) Comment

09/03/2009 1.0 S. Greer All Initial Version

01/20/2010 1.1 S. Greer 5.2 Removed model name as upper case

05/05/2010 1.2 S. Greer / C. Reber 2.0 / 5.3 Update links to point to web page instead of an individual document (2.0 intro / 5.3).

11/15/2011 1.3 C. Reber All Change ESF to EDC

03/19/2014 1.4 S. Greer All Change EDC to PSP



Table of Contents 1 DATA MODELING INTRODUCTION .............................................................................................. 4

1.1 PURPOSE........................................................................................................................................ 4 1.2 DATA MODELING OVERVIEW ......................................................................................................... 4

2 ROLES AND RESPONSIBILITIES .................................................................................................. 5

2.1 DBA ROLE AND RESPONSIBILITY ................................................................................................. 5 2.1.1 Logical Data Modeling ......................................................................................................... 5 2.1.2 Physical Data Modeling ....................................................................................................... 5

2.2 DEVELOPER ROLE AND RESPONSIBILITY ..................................................................................... 5

3 DATA MODELING COMPONENTS ................................................................................................ 6

4 LOGICAL DATA MODELING STANDARDS ................................................................................ 7

4.1 LOGICAL DATA MODELING CONVENTIONS ................................................................................... 7 4.2 LOGICAL DATA MODEL NAMING STANDARDS .............................................................................. 7

5 PHYSICAL DATA MODELING STANDARDS .............................................................................. 8

5.1 PHYSICAL DATA MODELING CONVENTIONS ................................................................................. 8 5.2 PHYSICAL DATA MODEL NAMING STANDARDS ............................................................................ 8 5.3 PHYSICAL DATA MODELING TABLE AND COLUMN NAMING STANDARDS ................................... 8



1 Data Modeling Introduction

1.1 PURPOSE

The Pennsylvania State Police (PSP) Bureau of Information Technology Database team provides

Database Design Services (DDS) for applications that require database development

This document focuses on data modeling standards related to the logical and physical design of a

database. This document lists and defines data model components and the data modeling naming

conventions provided as part of Database Design Services.

1.2 DATA MODELING OVERVIEW

Data modeling is a very vital part in the development process. Data models consist of three basic types:

conceptual data models, logical data models and physical data models. Major events in data modeling

include:

Identify entities, data requirements and processes

Define attributes of the data such as data types, sizes, defaults

Apply validation and business rules to ensure data integrity

Define data management and security processes

Specifying data archival and storage

ITB-INF003 Data Modeling Standard defines the policy that is followed for Database Design Services.



2 Roles and Responsibilities

Database Team requires the developer to complete and submit the Database Design Requirements

Questionnaire (available on the Bureau of Information Technology Database Sharepoint site).

The template requests a list of the data elements metadata to be used to develop the data models and data

dictionary.

2.1 DBA ROLE AND RESPONSIBILITY

2.1.1 Logical Data Modeling

The following are the roles and responsibilities Database Design Services provides for logical data

modeling:

Participate in project meetings related to data processing

Communicate with Application Developers to understand all aspects of the logical model

Construct, support and maintain development of the logical model

Ensure the logical model conforms to Database Design Database Naming Standards, Data

Modeling Standards, and OA/OIT Data Modeling Standards

2.1.2 Physical Data Modeling

The following are the roles and responsibilities Database Design Services provides for physical data

modeling:

Generate the physical model from the logical model and estimate the size of the database

Perform standards reviews of the model to ensure the physical model conforms to the Database

Design Database Naming Standards, Database Naming Standards Abbreviations list, Data

Modeling Standards, and OA/OIT Data Modeling Standards

Generate a data dictionary from the physical model

Provide a data dictionary (Deliverable)

Provide the physical model Entity Relationship Diagram (ERD) (Deliverable)

Generate DDL from the physical model and create the database (Deliverable)

Maintain and keep the physical model in synchronization with the database at all times

2.2 DEVELOPER ROLE AND RESPONSIBILITY

Complete and submit the Database Design Requirements Questionnaire Document (Deliverable)

Support standards reviews of the physical model

http://pspsites.state.pa.us/sites/bit/Database/Database%20Design%20Standards/Forms/AllItems.aspx

http://www.portal.state.pa.us/portal/server.pt?open=512&objID=416&PageID=210791&mode=2



3 Data Modeling Components

A data model is composed of the following components and may be illustrated using a data-modeling

tool:

Conceptual Data Model

A conceptual data model is a visual diagram from the business user’s perspective. Conceptual data

models are often created as the precursor to Logical Data Models or as alternatives to Logical Data

Models.

Logical Data Model

A logical data model is an abstract Database Management System (DBMS) independent representation of

a set of data entities and their relationships within the scope of a system’s business requirements. The data

model is a visual diagram of the database schema that contains all the information required to generate a

physical database on a specific server. The model should identify all the tables, columns, and

relationships related to the database. The logical data model should be constructed prior to the creation of

the database. The coding and database development should begin after the logical model has been

approved.

Physical Data Model

A physical data model is a representation of a data design which takes into account the facilities identified

in the physical data modeling table and column naming standards section and constraints of a given

database management system. In the lifecycle of a project it is typically derived from a logical data

model, though it may be reverse-engineered from a given database implementation.

A physical data model is translated into a data schema. A series of steps to normalize the data should be

applied at this phase to ensure efficiency and integrity of the data schema.

Data Dictionary

A data dictionary is a comprehensive document that outlines the business definition of all tables and

columns within the model. Each table is defined by its name, columns (including data type, length, fixed

or variable length, valid values, default values and nullable), primary key, foreign key, and business

definition for every table and column.

http://en.wikipedia.org/wiki/Entity-relationship_diagram

http://en.wikipedia.org/wiki/Logical_data_model

http://en.wikipedia.org/wiki/Logical_data_model



4 Logical Data Modeling Standards

4.1 LOGICAL DATA MODELING CONVENTIONS

A logical data model must comply with the following:

Use English words with no abbreviations separated by a space.

Do not use underscores in the logical model.

Specify all primary and foreign keys in the data model.

Specify relationship types as identifying or non-identifying and the cardinality for each

relationship.

Add comments for every table and column that describes the business definition including

valid values for the columns.

Enter table and column descriptions and other Metadata as required by the developer.

4.2 LOGICAL DATA MODEL NAMING STANDARDS

Since most data modeling software tools allow for both the logical and physical versions of the data

model to exist in one file, the data model name will follow the data model naming standards

outlined in the Physical Data Modeling Standards section.



5 Physical Data Modeling Standards

5.1 PHYSICAL DATA MODELING CONVENTIONS

A physical data model must comply with the following:

Specify all primary and foreign keys in the data model.

Define the primary key as the first data item in the table.

Specify relationship types as identifying or non-identifying and the cardinality for each

relationship.

Enter table and column descriptions and other Metadata as required by the developer.

Synchronize the physical data model and the physical database with each other in the

production environment.

5.2 PHYSICAL DATA MODEL NAMING STANDARDS

Standards are established for naming data models to easily identify the data model, data model

subsystem and version. Name data models using the following guidelines:

The data model name will be of the format “SPApplicationName”. This will be the same

name as the database.

Data-model subsystem names will be in the format “SPApplicationName_SubsystemName”,.

Versioning of the data model will be done as part of the modeling tool.

5.3 PHYSICAL DATA MODELING TABLE AND COLUMN NAMING STANDARDS

Consistent naming practices are vital. The use of a common methodology when naming database tables

and columns promote the uniform description and definition of the elements while, at the same time,

facilitates:

The reduction of redundant data elements by identifying like name elements

The reduction of element documentation by utilizing existing documents

The promotion of data integrity by ensuring data singularity and consistency

The simplification and standardization of system documentation by utilizing existing documents

Consistency in the use and meaning of abbreviations

Follow the standards defined in Database Design SQL Server Database Naming Standards (available on

the Bureau of Information Technology Database Sharepoint site).



Database Design

SQL Server Database Naming Standards


SECURITY WARNING







COMMONWEALTH OF PENNSYLVANIA PENNSYLVANIA STATE POLICE

DATABASE DESIGN SQL SERVER DATABASE NAMING STANDARDS PAGE 2 OF 10


8/11//2009 1.0 M. Forbes All Initial Version

9/30/2009 1.1 S. Greer 2.5 Changed indicatior datatype to CHAR

1/20/2010 1.2 S. Greer 2.4, 2.5 Added singular name, voitaile tables and logical delete. Added column not past tense.

05/05/2010 1.3 S. Greer / C. Reber 1.2 / 3.1 / App. B Update links to point to web page instead of an individual

documents (1.2 – App. B)

- Rework verbiage in section 3.1

08/12/2010 1.4 S Greer 2.5 Removed identiy column to Primary Key names for

convention to apply to all primary keys not just identy

columns. Added Primary key identity column data type.

12/09/2010 1.5 S. Greer 2.3, 2.7 Multiple databases naming convention, Added Login Section


03/14/2014 1.7 S. Greer All Changed EDC to PSP



Table of Contents 1 INTRODUCTION ................................................................................................................................. 4

1.1 PURPOSE........................................................................................................................................ 4 1.2 SQL SERVER DATABASE NAMING STANDARDS OVERVIEW ........................................................ 4

2 SQL SERVER NAMING CONVENTIONS ...................................................................................... 5

2.1 DATABASE OBJECTS...................................................................................................................... 5 2.2 SCHEMAS ....................................................................................................................................... 5 2.3 DATABASE NAMES ......................................................................................................................... 5 2.4 TABLES ........................................................................................................................................... 6 2.5 COLUMNS ....................................................................................................................................... 6 2.6 VIEWS ............................................................................................................................................. 7 2.7 APPLICATION LOGINS .................................................................................................................... 7

3 DATABASE STANDARD DATA ELEMENTS & ABBREVIATIONS ........................................ 8

3.1 STANDARD DATA ELEMENTS ........................................................................................................ 8 3.2 ABBREVIATIONS ............................................................................................................................. 8

4 APPENDIX A – RESERVED WORDS ............................................................................................ 9

5 APPENDIX B – RESOURCES & REFERENCES ....................................................................... 10



1 Introduction

1.1 PURPOSE


Database Design Services (DDS) for applications that require database development. This document

describes the SQL Server database naming standards that will be applied for Database Design.

This document focuses on defining the rules for database object and naming construction and the rules for

abbreviating the names to meet length constraints imposed by programming languages and database

management systems.

1.2 SQL SERVER DATABASE NAMING STANDARDS OVERVIEW

Standardizing naming conventions and abbreviations are defined in order to promote structure,

consistency, readability, reusability, and maintainability. Applying standards promotes data element

definition reuse between databases in an effort to improve data sharing, data mapping, and eliminate data

redundancy where possible.

Standards are important to developers for a number of reasons:

80% of the lifetime cost of a piece of software goes to maintenance.

Hardly any software is maintained for its life by the original author.

Code standards improve the readability of the software, allowing developers to understand new

code more quickly and thoroughly.

For the conventions to work, each person who writes software for SQL Server databases must conform to

these naming standards. It is also understood that there are situations where it is necessary to stray from

these guidelines. Commercial-Off-The-Shelf (COTS) or third party products will adhere to these

standards, if possible.

For additional SQL Server naming conventions and standards, reference the documents listed under the

Database Design Standards library on the Bureau of Information Technology Database Sharepoint site.

The SQL Server standards will only be enforced when PSP Database Design Services are engaged.

If any of these standards cannot be met, a justification must be submitted to PSP Data Administration

within the Database Section.




2 SQL Server Naming Conventions

Naming conventions provide a standard approach to naming different objects and help to troubleshoot and

locate objects. All objects also need a detailed description of use. The following naming conventions

must be adhered to for SQL Server databases. When reading the naming conventions, remember that

consistent naming is the most important rule.

2.1 DATABASE OBJECTS

A database object is a database component like a table, trigger, view, key, constraint, default, rule, user-

defined datatype, or stored procedure in a database. The following naming conventions apply for all database

objects:

Names for user-defined database objects shall be case sensitive (use upper and lower case) for

readability and programmability.

Names must be significant and meaningful.

Names must begin with a letter followed by letters, digits, or _ with a maximum length of 30

characters

Names for user defined database objects should not use reserved words, keywords (Appendix A),

quoted (or bracketed) names, spaces and special characters (such as $, #, etc.) except the

underscore “_”.

Example: Use the name „Account‟ for Accounts table.

Use the name „Emp_Num‟ for the column of employee number.

2.2 SCHEMAS

A schema is a collection of components and database objects under the control of a given database user.

Schema names will be suffixed with „dbo‟.

The schema name will be of the format “XXApplicationNamedbo”, where XX is the agency two-

digit code.

All user-defined database objects are owned by a schema. The Fully Qualified Domain Names

(FQDN) of these database objects must be unique. The FQDN for all database objects must be in

this form:

[Server_Name].[Database_Name].schema.[Object_Name].

Example:

Use the name „Server01.Sample. XXApplicationNamedbo.Account‟ for the Accounts table,

XXAccountdbo is the schema name in the Sample database residing on Server01.

2.3 DATABASE NAMES

A database in the relational context is organized by tables, rows and columns used to store information

organized in such a way that a computer program can quickly select desired pieces of data.

The database name will be of the format “XXApplicationName”, where XX is the agency two-

digit code.

For applications that require multiple databases all of the database names must start with the same

“XXApplicationName”.

Example: SPLTC, SPLTCWorkFlow, SPLTCAudit

The database name must not include the underscore “_”.



2.4 TABLES

A table is a two-dimensional object, consisting of rows and columns, used to store data in a relational

database.

The table name must be singular.

The table name must describe the contents of the table.

Avoid prepositions in table names.

If possible, table names should not be longer than 26 characters.

Do not use the prefix of „sys‟, „spt‟ and „sysremote‟. These are used for system tables.

Tables used to resolve a many-to-many relationship must include both the source table names and

a suffix of „_Xref‟.

Example: Permit_Type and Status_Type are combined as Permit_Status_Type_Xref

A lookup table or list of values must be suffixed with „_Lkp‟.

The fact table name in the data mart must be suffixed with a „_Fact‟.

The dimension table name in the data mart must be suffixed with a „_Dim‟.

Volatile tables (tables with business data that is changed on a frequent basis) should include

columns, which identify when the row was entered, who entered it, when it was last updated, and

who last updated it. These columns are named:

Entered_By VARCHAR(30) NOT NULL default suser_sname()

Entered_Date SMALLDATETIME NOT NULL default getdate()

Update_By VARCHAR(30) NULL

Update_Date SMALLDATETIME NULL

Tables that logically delete rows (i.e. Lookup tables) should include columns that identify the row

was deleted by using obsolete, when it was last updated, and who last updated it. These columns

are named:

Obsolete BIT NOT NULL default „0‟

Update_By VARCHAR(30) NULL

Update_Date SMALLDATETIME NULL

Xref tables must not include the Entered_By, Entered_Date, Update_by and Update_Date.

2.5 COLUMNS

A column is the area in each row that stores the data value for some attribute of the object modeled by

the table.

The column name must be singular.

The column name must not be past tense.

The column name must describe the contents of the column.

Column names that appear in multiple tables should be consistently named throughout the

Enterprise (across all agency tables). There may be cases where a column name would be

referenced twice. If so, add a meaningful prefix to the original column name.

Example: Color_Code would become Interior_Color_Code and Exterior_Color_Code.

Primary key columns must include the table name and suffixed with „_ID‟.

Example: Use the name Address_ID as the primary key column of the ADDRESS table.

Primary key columns must exclude the Xref, Lkp, Fact and Dim suffix of the table name and be

replaced with „_ID‟.

Example: Use the name State_ID as the primary key identify column of the STATE_Lkp table.



Primary key identity columns data type must be integer.

If the Primary key is also used as a Foreign key, the table name may be excluded to follow the

consistent naming of columns throughout the database.

Example: PFA table primary key would be Person_ID instead of PFA_ID since the Person_ID is

also a foreign key to the Person table.

Foreign-key columns should also have the same name in both parent (master, reference, or

lookup) and child (detail or data) tables including the suffix „_ID‟. There may be cases where a

foreign-key column requires a meaningful prefix to the parent name.

Example: Primary key State_ID foreign key would become Address_State_ID

The column defined as a date or to only store the date must be suffixed with „_Date‟.

Example: Activity_Date

The column defined as a time must be suffixed with „_Time‟.

Example: Activity_Time

The column defined as a datetime must be suffixed with „_DateTime‟.

The column defined as a timestamp must be suffixed with „_TimeStamp‟.

The indicator column must be suffixed with „_IND‟. The preferred method is using values „Y‟

for Yes and „N‟ for No, and the column datatype must be CHAR(1). Other values such as „0‟ for

False and „1‟ for True with a BIT or CHAR datatype may be used in justifiable cases.

Example: Active_IND CHAR(1)

2.6 VIEWS

Views are customized presentations of data in one or more tables or other views. A view can also be

considered a stored query. Views do not actually contain data. Rather, they derive their data from the

tables on which they are based.

The view name must be prefixed with a „v_‟.

If the view comes from a single table, then the name must describe the contents of the table.

Example: v_Address

If the view comes from more than one table, then the name must describe the combined name of

the underlying tables.

Example: v_Customer_Address

2.7 APPLICATION LOGINS

An application login is an account that provides the application permissions to access the SQL Server

databases. These conventions only apply to SQL authentication accounts.

The application owner login name must be prefixed with the database name

“XXApplicationName”, where XX is the agency two-digit code.

The application owner login name should be suffixed with its database purpose. This allows for

multiple logins created for an application.

Example: SPLTCuser, SPLTCAdmin

The login name must not include spaces and special characters (such as -, $, #, +, etc.) including

the underscore “_”.

The report login should be suffixed with _Report (only exception to underscore “_”).

Example: SPLTC_Report



3 Database Standard Data Elements & Abbreviations

3.1 STANDARD DATA ELEMENTS

Standard data elements are created to achieve consistency of names, datatype, lengths, etc. throughout the

agencies databases. This will benefit the sharing of data between agencies and applications. To set a

standard, all database elements are researched, and PSP‟s data dictionary is researched to determine the

most used or most accurate element name and attributes.

Standard elements take precedence over using any abbreviation.

Prefixes can be added to the standard element. Ex: FIRST_NAME could become

CLIENT_FIRST_NAME, CONTACT_FIRST_NAME, etc.

3.2 ABBREVIATIONS

The PSP Data Administration owns and maintains the PSP Database Naming Standards Abbreviations

list. Please contact the Data Administrator within the Database team for access to the list. General

guidelines are as follows:

Abbreviations should be avoided whenever possible but, if necessary, they must conform to the

standard abbreviations as set forth in the PSP Database Naming Standards Abbreviations list. An

attempt shall be made to keep this list to a minimum.

Abbreviations should not be used for words with six or less characters. If words over six

characters are used and an abbreviation is required, consult the PSP Database Naming Standards

Abbreviations list. If an abbreviation does not appear in the list, spell the word out and the PSP

Data Administrator will assign the official abbreviation and include it in the list.

Abbreviations and acronyms that are unique to the agency must be provided to the PSP Database

team. The PSP Database Naming Standards Abbreviations list only contains common

abbreviations to be used across the Commonwealth Enterprise.



4 Appendix A – Reserved Words

Microsoft SQL Server uses reserved keywords for defining, manipulating, and accessing databases.

Reserved keywords are part of the grammar of the Transact-SQL language used by SQL Server to parse

and understand Transact-SQL statements and batches.

Although it is syntactically possible to use SQL Server reserved keywords as identifiers and object names

in Transact-SQL scripts, this can be done only by using delimited identifiers.

The following Microsoft links list the SQL Server reserved keywords.

SQL Server 2008 R2 - http://msdn.microsoft.com/en-us/library/ms189822(v=sql.105).aspx

SQL Server 2012 - http://msdn.microsoft.com/en-us/library/ms189822.aspx

http://msdn.microsoft.com/en-us/library/ms189822(v=sql.105).aspx

http://msdn.microsoft.com/en-us/library/ms189822.aspx



5 Appendix B – Resources & References

For additional standards information, please reference the documents listed below which are available on

the Bureau of Information Technology Database Sharepoint site:

Database Design Data Modeling Standards defines the logical and physical design of a

database, listing of data model components, and data modeling naming conventions.

Database Design SQL Server Database Naming Standards defines naming standards,

construction, abbreviations, and rules for database objects.

Database Design SQL Server Stored Procedure Standards defines naming standards and

examples of how stored procedures should be coded and documented. A code review checklist is

also provided.

Database Design SQL Server SSIS Standards defines roles and responsibilities, naming

conventions, standards, and best practices for SSIS development.



Database Design

SQL Server Stored Procedures Standards


SECURITY WARNING








DATABASE DESIGN SQL SERVER STORED PROCEDURE STANDARDS PAGE 2 OF 9


8/11/2009 1.0 M.Forbes All Initial Version

05/05/2010 1.1 S. Greer / C. Reber 2 / 3 Update links to point to web page instead of individual

documents

06/17/2011 1.2 S. Greer 4.1, 5 Added programmability object paragraph.

Added TFS as a code repository.


03/14/2014 1.4 S. Greer All Change EDC to PSP




1.1 PURPOSE........................................................................................................................................ 4 1.2 SQL SERVER STORED PROCEDURE OVERVIEW ......................................................................... 4


2.1 EDC DBA ROLE AND RESPONSIBILITY ........................................................................................ 5 2.2 AGENCY ROLE AND RESPONSIBILITY ........................................................................................... 5

3 STORED PROCEDURE NAMING CONVENTIONS .................................................................... 6

4 DESIGNING & CODING STORED PROCEDURES ..................................................................... 7

4.1 STORED PROCEDURE HEADER STRUCTURE ............................................................................... 7 4.2 CODING STORED PROCEDURES ................................................................................................... 8

5 APPENDIX A – SQL SERVER CODE REVIEW CHECKLIST ................................................... 9



1 Introduction

1.1 PURPOSE


Database Design Services for applications that require database development.

This document describes the stored procedure standards that will be applied for Database Design and

focuses on defining the naming conventions for stored procedures. Examples of how stored procedures

are to be coded and documented is provided. A code review checklist is defined in Appendix A to be

used as a guideline for coding stored procedures.

1.2 SQL SERVER STORED PROCEDURE OVERVIEW

Stored procedures are server level procedures that are called within application programs. They are a

group of Transact-SQL statements compiled into a single execution plan. If stored procedures are utilized,

they must follow the standards set within this document.

Many activities in SQL Server are performed through system stored procedures. System stored

procedures are created and stored in the database and have the “sp_” prefix.

Developers will be granted the ability to create stored procedures within the development environment.

In all client applications, stored procedures should be used rather than sending T-SQL commands directly

to the SQL Server. The DBA will be responsible for the migration of the stored procedures to all other

environments. When coding applications, the use of stored procedures to separate application logic from

database code has been found beneficial. The advantages include:

Eased Maintenance – When supporting database structure or application logic changes, the DML

is easily accessible directly from the database, can be easily scanned using SQL. As such, it is

recommended that all web-based applications use stored procedures for the majority, if not all,

database access.

Enhanced Performance – Stored procedures run directly on the database server, which can be

optimally tuned to perform such operations. Stored procedures are pre-compiled and do not need

to be compiled each time they are executed.

Reusability – Stored procedures can be called from multiple applications, thereby reducing the

time required to design, code and test commonly used application functions.

Enhanced Security –The application users are restricted to execute only the specified stored

procedures within the database. There is no need to grant direct table access. Stored procedures

are compiled code and cannot be manipulated or executed as SQL code. Stored procedures help to

reduce SQL injection risks.




EDC Database Design requires the agency to complete and submit the EDC Database Design Process

Requirements document (available on the EDC Database Design Services website page) for complex

stored procedures. This document includes an overview of the stored procedure tasks, and the tasks

associated with those requirements, for stored procedures requiring EDC DBA design.

Database Design Services (DDS) will collaborate with the developer and provide the design for complex

stored procedure by updating the Database Design Process Requirements Document with a recommended

solution on how to develop the stored procedure.


Support the development of stored procedures with the Developer

Review the Database Design Process Requirements Document of a complex stored procedure

Collaborate on design for a complex stored procedure and provide recommended solution to the

Developer (Deliverable)

Ensure best practices (such as proper error-checking or error-raising from database to application)

are followed

Test all stored procedures to ensure best optimization choice

Deploy the stored procedures


Complete and submit the Database Design Process Requirements Document Template for

complex stored procedures to the DBA (Deliverable)

Collaborate on design for complex stored procedures

Write the stored procedures

Test and tune stored procedures to ensure best optimization choice

Submit stored procedure to DBA which contains the header section and process logic as defined in

Section 4.

http://www.portal.state.pa.us/portal/server.pt/community/technology_service_offerings/750/database_design_services/597270



3 Stored Procedure Naming Conventions

Naming conventions provide a standard approach to naming different objects and help to troubleshoot and

locate objects. Stored procedures should be named according to the business function they perform or

business rule they enforce. The following naming conventions must be adhered to for SQL Server stored

procedures.

A user-defined stored procedure name must include a prefix of „USP_‟ and a business description

of the action performed.

A user defined extended stored procedure name must include a prefix of „UXP_‟ and a business

description of the action performed.

SP_ and XP_ are not to be used as the prefix of user defined stored procedures and extended

stored procedures.

The stored procedure name should be appropriately abbreviated, as defined in the abbreviations

list, which is available from PSP Data Administration. An example of a procedure name would be

USP_Insert_New_Bldg. As the name implies, this procedure would insert a new Building row

where Bldg is the abbreviation for Building. For more details, reference the Database Design

SQL Server Database Naming Standards document (available on the Bureau of Information

Technology Database Sharepoint site.

The stored procedure name must contain a verb (such as Select, Insert, Update, Delete or

Combine) and the description that best describes the work the stored procedure will perform.

The stored procedure name should have a maximum length of 30 characters.





4 Designing & Coding Stored Procedures

This section provides examples of the standards that should be followed when designing a stored

procedure. A code review checklist is also used to help developers as a guideline in helping to insure that

standards are being met as they create or alter stored procedures. See Appendix A for the checklist.

4.1 STORED PROCEDURE HEADER STRUCTURE

All programmability objects should contain the header section described below. Use Properties

Extended Properties for the object description in place of the header section when not available for the

object (i.e. User-Defined Table Types).

Header section

This is to be placed at the beginning of the stored procedure.

/***********************************************************************

* Database: (name of database being used for the procedure goes here.)

* Procedure Name:

* Date:

* Author:

* Procedure Description:

*

* Parameters:

*

* Returns:

*

* Calling

* Mechanism: (How is it being called: DTS, Stored Procedure etc.)

*

* Tables / Alias

* Definitions: (what tables are being accessed)

*

* Procedures Used: (if any other stored procedures are called within this

* procedure.)

*

* Notes:

*

* Special Comments/Warnings:

*

************************************************************************

* Version: (current version number) Author: (current version author)

* Date:

*************************************************************************

* Description of Modifications:



* 1. See { } (what is modified)

* Special Comments:

*

* Other modules changed with this request:

*

***********************************************************************/

4.2 CODING STORED PROCEDURES

The following is an example of how a new procedure is to be coded and documented. Every table should

be accessed in the same order for every stored procedure that references it, to help prevent deadlocks.

CREATE PROCEDURE [schema name].usp_test

As

Set NOCOUNT ON /*this should be first line in code*/

Go

/***********************************************************************

** This area describes what the sql statement is doing.

DROP TEMP TABLE USED TO HOLD VALUES

*******************************************************************************/

DROP TABLE ##logfiles

GO



5 Appendix A – SQL Server Code Review Checklist

ITEM YES NO COMMENTS

1. CODE IS IN SOURCE SAFE or TFS X

2. PROPER HEADER COMMENT X See Section 4

3. PROPER MAINTENANCE COMMENT X See Section 4

4. CURSORS USED X

5. PROPER INDENTING OF FIELDS X

6. USE OF =* OR *= X

7. USE OF SELECT INTO X

8. PROPER COLUMN WIDTH(80) X

9. PROPER OVERALL LENGTH OF OBJECT(4 PAGE MAX) X Total number of lines (excluding

comment lines): 45

10. USE OF * X

11. PROPER USE OF COMPOUND STATEMENTS X

12. PROPER FORMATTING OF FIELDS X

13. VERBS ARE DIFFERENT CASE FROM VARIABLES X

14. OBJECT COMPILES X

15. PROPER USE OF ERROR HANDLING X

16. NUMBER OF NESTED IF‟S DOES NOT EXCEED 4 X

17. SPEED AND PERFORMANCE OF OBJECT IS

ACCEPTABLE X EXECUTION PLAN ANALYZED

18. OBJECT HAS ONE PURPOSE X

19. OBJECT CAN BE BROKEN DOWN INTO SMALLER

PARTS X

20. T_SQL VERBS ARE SPELLED OUT X EX. INNER JOIN VS INNER

21. DEAD OR UNUSED CODE REMOVED X

22. LEFT JOINS CAN BE REWRITTEN AS INNER JOINS X

23. ORDER OF JOINS RETURNS MINIMAL ROWS NEEDED X

24. FIELDS LISTED ON INSERT X

25. DYNAMIC SQL X

26. TABLE VARIABLES OR VIEWS INSTEAD OF TEMP

TABLES X

27. TABLE & INDEX SEEKS NOT SCANS X EXECUTION PLAN ANALYZED

28. TEXT DATA TYPE IF LESS THAN 8 KB USE

VARCHAR(8000) X


Database Design

SQL Server Integration Services (SSIS) Standards


SECURITY WARNING







COMMONWEALTH OF PENNSYLVANIA PENNSYLVAINA STATE POLICE

SQL SERVER INTEGRATION SERVICES (SSIS) STANDARDS PAGE 2 OF 15


8/11/2009 1.0 C.Hoffman All Initial Version

9/24/2009 1.1 C.Hoffman 3.3 New item added

4/23/2010 1.2 C.Hoffman 4.1, 4.2, 4.3, 5.1, 5.2, 3.1

SSIS File Structure, Error Log file location Added link to naming standards.

05/05/2010 1.3 S. Greer / C. Reber 2.0 / 3.1 Update links to point to web page instead of an individual

document. (2.0 intro / 3.1 – bullet 3 – rework verbiage)


08/10/2012 1.5 S. Greer 3.3, 4.1, 4.2 FTP guidelines added

03/04/2013 1.6 S. Greer 3.3 FTP guidelines updated

03/ 1.7 S. Greer All Change EDC to PSP




1.1 PURPOSE........................................................................................................................................ 4 1.2 SQL SERVER INTEGRATION SERVICES (SSIS) OVERVIEW ........................................................ 4


2.1 EDC DBA ROLE AND RESPONSIBILITY ........................................................................................ 5 2.2 AGENCY ROLE AND RESPONSIBILITY ........................................................................................... 5

3 SSIS NAMING AND CODING CONVENTIONS ............................................................................ 6

3.1 SSIS PACKAGE NAMING CONVENTIONS ...................................................................................... 6 3.2 SSIS PACKAGE OBJECT NAMES AND DESCRIPTIONS ................................................................. 7 3.3 DESIGNING AND CODING SSIS PACKAGES.................................................................................. 9

4 SSIS PACKAGE DEVELOPMENT STANDARDS ..................................................................... 11

4.1 SSIS FILE STRUCTURE ON DATABASE SERVER ........................................................................ 11 4.2 SSIS FILE STRUCTURE ON FILE SERVER ................................................................................... 11 4.3 SSIS PACKAGE ERRORS AND LOGGING .................................................................................... 11

4.3.1 Package Error Log ............................................................................................................. 11 4.3.2 Transform Data Task Exception Log ............................................................................... 12

4.4 SSIS PACKAGE DEPLOYMENT .................................................................................................... 14

5 SSIS PACKAGE TROUBLESHOOTING ..................................................................................... 15

5.1 REVIEWING THE PACKAGE LOGS ................................................................................................ 15 5.2 REVIEWING TRANSFORM DATA TASK EXCEPTION FILE ............................................................ 15



1 Introduction

1.1 PURPOSE


Database Design Services (DDS) for developing applications that require database development.

This document defines the Roles and Responsibilities, Naming Conventions, Standards, and Best

Practices for SSIS development provided by the Database team.

1.2 SQL SERVER INTEGRATION SERVICES (SSIS) OVERVIEW

SQL Server Integration Services (SSIS) is a component of SQL Server. In SSIS, the Data Flow Engine is

segregated from the Control Flow Engine or SSIS Runtime Engine, which improves the performance.

SSIS is „basically‟ an Extract, Transform, and Load (ETL) tool. ETL describes the processes that take

place in data warehousing environments for extracting data from source transaction systems;

transforming, cleaning and conforming the data; and finally loading it into cubes or other analysis

destinations.

SSIS is a platform for building high performance data integration and workflow solutions. SSIS packages

are made up of tasks that can move data from source to destination and if necessary alter it on the way.

Integration Services provides a Control Flow for performing work that is tangentially related to the actual

processing that happens in data flow, including downloading and renaming files, dropping and creating

tables, rebuilding indexes, performing backups, and any other number of tasks. Integration Services

provides a full-featured control flow to support such activities.

The Data Flow Task is a high-performance tool because you can use it to perform complex data

transformations on very large datasets without hindering the processing performance. The pipeline

concept means that you can process data from multiple heterogeneous data sources, through multiple

parallel sequential transformations, into multiple heterogeneous data destinations, making it possible to

process data found in differing formats and on differing media in one common "sandbox" location.




PSP Database Design requires the developer to complete and submit the Database Design Process

Requirements document (available on the Bureau of Information Technology Database Sharepoint site)

for all SSIS packages. This document includes an overview of tasks and the requirements associated with

those tasks.

Database Design Services (DDS) will collaborate with the developer and provide the design for complex

packages by updating the Database Design Process Requirements Document with recommended

solutions on how to develop the SSIS package. DDS will determine if a package is simple or complex

based on the requirements. Simple packages will be designed by the developer and noted in the Database

Design Process Requirements Document solution as “Designed by Developer”.


Support the development of SSIS package with the Developer

Review the Database Design Process Requirements Document of the SSIS package

Collaborate on design for complex SSIS packages

Provide the Database Design Process Requirements Document with recommended solutions to

the Developer (Deliverable)

Ensure SSIS Package conforms to the SSIS Standards Document

Setup and test SSIS Package job schedule using SQL Server Agent

Deploy SSIS packages with the Package Installation Wizard on the Development and User

Acceptance Testing environments

Test packages prior to implementing them on the production server so that there is no negative

impact to the server


Prior to coding, complete and submit the Database Design Process Requirements Document

Template of the SSIS package to the DBA (Deliverable)

Collaborate on design for complex SSIS packages

Design and code SSIS packages with Visual Studio

Test SSIS Packages by using Visual Studio

Create package configuration files (manifest, Config, DTSX) and provide to DBA

Notify the System Administrator of the appropriate rights to all the data sources for the SSIS

package




3 SSIS Naming and Coding Conventions

3.1 SSIS PACKAGE NAMING CONVENTIONS

SSIS Packages should be named according to the business function they perform. Naming

conventions are as follows:

Example: XXApplicationName_Personnel_Load

XXApplicationName_Transfer_Web_Appl

XXApplicationName_Data_Load_v2

The package name will include a prefix of „XXApplicationName‟ and a business description of

the action performed.

The “XXApplicationName” prefix will be the same name as the database, where XX is the

agency two-digit codes. The main purpose for this prefix is to keep all Agency database related

packages grouped together.

The package name should be appropriately abbreviated, as defined in the abbreviation list, which

is available from PSP Data Administration. For more details, reference the PSP Database Design

SQL Server Database Naming Standards document (available on the Bureau of Information

Technology Database Sharepoint site.

If developers are working on multiple versions of the package, any additional versions will be

suffixed with a lower case “v” followed by the version number. The first time a version number

is assigned, it will start with the number 2. The first version of the package will not contain a

version number.

Use an underscore (_) for blank spaces.

The Package name should be under 100 characters.





3.2 SSIS PACKAGE OBJECT NAMES AND DESCRIPTIONS

1. Rename all Name and Description properties from the default for all SSIS Objects.

The default description should be changed to a short but suitable name for the object. This

will greatly aid personnel in troubleshooting package errors.

If the default name is not known, then do the following:

Open the SSIS package. Right click on the object and select Properties.

Figure 1: Locating Default Names for Package Objects

2. Use acronyms at the start of the name to better identify the object.

The following acronyms should be used at the beginning of the names of tasks to identify the

type of task.

Task Prefix

For Loop Container FLC

Foreach Loop Container FELC

Sequence Container SEQC

ActiveX Script AXS

Analysis Services Execute DDL ASE

Analysis Services Processing ASP

Bulk Insert BLK

Data Flow DFT

Data Mining Query DMQ

Execute DTS 2000 Package EDPT

Execute Package EPT

Execute SQL SQL

File System FSYS

FTP FTP

Message Queue MSMQ

Script SCR

Send Mail SMT



Transfer Database TDB

Transfer Error Messages TEM

Transfer Jobs TJT

Transfer Logins TLT

Transfer Master Stored Procedures TSP

Transfer SQL Server Objects TSO

Web Service WST

WMI Data Reader WMID

WMI Event Watcher WMIE

XML XML

Figure 2: Task Acronyms

The following acronyms should be used at the beginning of the names of components to

identify the type of component.

Component Prefix

DataReader Source DR_SRC

Excel Source EX_SRC

Flat File Source FF_SRC

OLE DB Source OLE_SRC

Raw File Source RF_SRC

XML Source XML_SRC

Aggregate AGG

Audit AUD

Character Map CHM

Conditional Split CSPL

Copy Column CPYC

Data Conversion DCNV

Data Mining Query DMQ

Derived Column DER

Export Column EXPC

Fuzzy Grouping FZG

Fuzzy Lookup FZL

Import Column IMPC

Lookup LKP

Merge MRG

Merge Join MRGJ

Multicast MLT

OLE DB Command CMD

Percentage Sampling PSMP

Pivot PVT

Row Count CNT

Row Sampling RSMP

Script Component SCR

Slowly Changing Dimension SCD

Sort SRT

Term Extraction TEX

Term Lookup TEL

Union All ALL

Unpivot UPVT

Data Mining Model Training DMMT_DST

DataReader Destination DR_DST



Dimension Processing DP_DST

Excel Destination EX_DST

Flat File Destination FF_DST

OLE DB Destination OLE_DST

Partition Processing PP_DST

Raw File Destination RF_DST

Recordset Destination RS_DST

SQL Server Destination SS_DST

SQL Server Mobile Destination SSM_DST

Figure 3: Component Acronyms

3.3 DESIGNING AND CODING SSIS PACKAGES

The following should be followed when designing and coding SISS packages.

1. All SSIS packages must contain a Data related task that either reads data from the database,

extracts data from the database, transforms data, or loads data into the database. SSIS Packages

should not be used entirely for file manipulation, FTP, Email etc. since these processes can be

done within a .NET Application.

2. All File Paths and Connections should be setup as Variables and apart of the Config file so that it

can be changed within the Config file when moving the package from one environment to

another.

3. All SSIS packages will be called from SQL Agent Jobs.

4. Keep all SSIS packages as small as possible for efficiency, readability, and for rerunning after

fixing errors.

5. When using SSIS; use the native OLE DB provider instead of the ODBC provider when

importing and exporting data, as it provides better performance.

6. If data transforming (row by row) is not needed during a SSIS import in SQL Server, the Bulk

Insert task provides the fastest data loads into SQL Server

7. SSIS lookups slow down performance. Use a Transact-SQL statement (stored procedure is best)

to perform the same function within the SSIS package. Avoid using global variables or COM

objects for performing lookup type functions, as they are even slower than using a SSIS lookup.

8. The Data Pump Task is faster than a Data Driven Query within a SSIS package, if there is a one-

to-one mapping of the columns and no transformations are involved when moving data between

tables. If there are transformations involved, then a Data Driven Query will offer better

performance.

9. For transformations, setting an error threshold greater than 0 is NOT recommended unless the

package is designed to only load new records. Otherwise, there will be duplicate records loaded

when rerunning the package after a failure.

10. When calling an SQL Server stored procedure that will execute a BCP (Bulk Copy Process)

command, the SSIS package needs to commit all transactions before calling the procedure,

because a BCP command will control its own transactions and does not allow a calling procedure

to handle it. Therefore, the previous step before the BCP step must commit its transactions on

successful completion of package.

11. Do not use ActiveX scripts because they will not be supported in future versions of SSIS.

ActiveX scripts also do not migrate during migration from DTS Package to SSIS Package. They

should be replaced with .NET Scripts

12. Only select columns that you need in the pipeline, to reduce buffer size and reduce OnWarning

events at execution time. This helps in avoiding extra execution time overhead of package and in

turn improves overall performance of package execution.



13. Use Sequence Containers to organize package structure into logical units of work. This makes it

easier to identify what the package does and helps to control transactions if they are being

implemented.

14. Use caching in your LOOKUP components where possible. It makes them quicker. Watch that

you are not grabbing too many resources when you do this though.

15. If you want to conditionally, execute a task at runtime use expressions on your precedence

constraints. Do not use an expression on the “Disable” property of the task.

16. Don‟t put all configurations into a single XML configuration file. Instead, put each configuration

into a separate XML configuration file. This is a „modular‟ approach and means that

configuration files can be reused by different packages more easily.

17. Make sure to set the package protection level to „DontSaveSensitive‟ to avoid package

development errors when deploying to other environments. If you use the default package-

protection level, „EncryptSensitiveWithUserKey‟, the same package may not execute as expected

in other environments because the package was encrypted with your personal key.

18. While configuring any OLEDB connection manager as a source, avoid using „Table or view‟ as

data access mode, this is similar to „Select * From‟. Always try to use „SQL Command‟ data

access mode and only include required column names in your SELECT T-SQL statement.

19. Sorting of data is a time consuming operation, in SSIS you can sort data coming from upstream

using „Sort‟ transformation, however this is a memory intensive task and may degrade the overall

package execution performance. It‟s better to perform the sorting operation at the database level

where sorting can be performed within the query. SQL Server database sorting is much refined

and happens at the SQL Server level, which results in overall performance improvement in

package execution.

20. Use Flat File Connection Manager very carefully, creating Flat File connections with default

setting will use data type string [DT_STR] as a default for all the column values. This may not be

the right option because you might have some numeric, integer etc. in your source, passing them

as a string to downstream tasks would take unnecessary memory space and may cause errors at

tasks that follow in the package execution.

21. Use the SSIS feature, Checkpoint, to restart failed packages from the point of failure.

22. Do not use Execute Process Task.

23. Execute SQL Task is our best friend in SSIS; it can be used to run a single or multiple SQL

statement at a time. The results can be returned as a single row, full result set and XML.

24. Files created or FTP on the database server are to be deleted when the package is done processing

the file. If multiple packages need to process the same file the last package to process the file

should delete the file.



4 SSIS Package Development Standards

4.1 SSIS FILE STRUCTURE ON DATABASE SERVER

The SSIS File structure shall include the following folders:

1. A Main Folder named “SSIS”.

2. A subfolder named “FTPFiles”. This is used for files that are FTP to or from the database server.

Once the file is processed it should be deleted.

3. A subfolder name that follows the format “XXApplicationName”, where XX is the agency

two-digit code

4. Under each application subfolder, the following Subfolder names:

Subfolder Purpose

Archive_Files Used for file archiving SSIS packages

Logs Used for any SSIS package Transform Data

Tasks exception files

Packages Used to store the package deployment files

PKGConfig Used to store the Package Configuration

file, which allow us to make SSIS packages

portable so they can easily be migrated from

one environment to another

4.2 SSIS FILE STRUCTURE ON FILE SERVER

1. A Share folder name that follows the format “XXApplicationName”, where XX is the agency

two-digit code

2. The following Subfolder names:

Subfolder Purpose

Logs Used for the package error file.

Data_Files Used for storing SSIS data files

4.3 SSIS PACKAGE ERRORS AND LOGGING

Because you can‟t foresee all the conditions that will ultimately occur in a production environment, SQL

Server Integration Services (SSIS) includes logging features that write log entries when run-time events

occur and can write custom messages to display information about a package after it‟s been executed. The

challenge is to log enough information to help you quickly diagnose and minimize the impact of problems

that might occur in production.

Integration Services supports a diverse set of log providers, and gives you the ability to create custom log

providers. The Integration Services log providers can write log entries to text files, SQL Server Profiler,

SQL Server, Windows Event Log, or XML files. Logs are associated with packages and are configured at the package level. Each task or container in a package can log information to any package log.

4.3.1 Package Error Log

1. To add logging to a package, click Logging on the SSIS menu and select one or more error log

providers, which enable you to write to a target destination. Select the check box next to the

events that you want to log, then click the details tab and specify the events you want to log.

Next, click Advanced to specify the columns to be logged, otherwise all the columns will be

logged. Depending on the events you choose to log, the error log can grow rapidly. Be sure to

log only the events that you need and occasionally prune old log entries. Because SSIS doesn‟t



include a process to do this out of the box, you must manually prune logs if your error log

provider doesn‟t provide the functionality to do so.

2. The package error path will be located in the Logs subfolder of the Application folder structure

on the file server.

3. Name the file name according to the SSIS package name.

4. The package log file will be suffixed with “_Error.txt”.

4.3.2 Transform Data Task Exception Log

1. To record errors in Data Transformation, create an Exception File to log all errors in the

transformation. The exception file path will be located in the Logs subfolder of the Application

folder structure on the file server. Name the file name according to the SSIS package name.

2. The exception log files will be suffixed with “_Exception.txt”.

Figure 4: SSIS Process to save errors to an Exception File



Figure 5: Redirect Error Row Output



4.4 SSIS PACKAGE DEPLOYMENT

SSIS has a feature called "Package Configurations". Package configurations allow us to make SSIS

packages portable so they can easily be migrated from one environment to another. Configuration Files

contain the connection strings for various sources and destinations, package variables and the expressions.

We can modify these properties without even opening up the package. At the run time, the SSIS engine

looks for the Configuration File. If the file does not exist then it takes the configuration/variable

information contained in the SSIS package. If the file exists, then the information in the Configuration

File is made used. If any variable/configuration information is missing in the Configuration File, that

particular value of the configuration parameter/variable from the SSIS package is made used.

1. When your SSIS Package is complete, use the "Package Configuration Wizard" to create the

package configurations, which will create a Config file, a Manifest file for deploying, and the

actual SSIS Package file (DTSX).

2. Use the „XML configuration file‟ option because it is more flexible.

3. Provide the Config, Manifest, and SSIS Package files to the EDC DBA who will deploy them to

SQL Server Integration Services.



5 SSIS Package Troubleshooting

This section explains how to review a package log and exception files to assist with SSIS Package

troubleshooting.

5.1 REVIEWING THE PACKAGE LOGS

The Error Log file is a running result of each step in a SSIS package. It will report a success or

failure and the reason for the failure.

This file will also contain a start and completed time for the package and for each step

including the total execution time, Package Name, description, ID, Version, server name where

package was executed, and who executed the package.

The error file is located in the Logs subfolder of the Application folder structure on the file

server.

5.2 REVIEWING TRANSFORM DATA TASK EXCEPTION FILE

The exception file is located in the Logs subfolder of the Application folder structure on the

file server.

PA State Police (PSP) Database Design

Documents

Transcript of PA State Police (PSP) Database Design