© 2005 EMC Corporation. All rights reserved. SQL Server 2005 BI Cost Effective Information...

© 2005 EMC Corporation. All rights reserved.

SQL Server 2005 BICost Effective Information Lifecycle Management for Data Warehouses

Adam Waddell

© 2005 EMC Corporation. All rights reserved. 22

Agenda How Information Drives Business

Managing Growing Mountains of Data

New Features for Storage in SQL Server 2005

REAL Solutions to Real Problems

Applying Data Lifecycle Management


The Information Age

Information management drives business efficiency– Six Sigma– Total Quality Management– Total Customer Experience

Wal-Mart sells products to consumers and information to producers

Managing inventory, logistics, and supply chains requires information


Information Overload

Data we are REQUIRED to keep– Regulatory compliance– Legal obligations– Archive requirements

Data we DESIRE to keep– Sales and inventory records– Customer, supplier, and partner records – More detail is generally better


Information Overload

Data Analysis becoming increasingly critical to daily business operation

– Many business processes now depend on historical data analysis

– Availability requirements of data and data warehouses are increasing

Better tools and processes are needed to manage the growing stream of data

– Take advantage of changing access and usage patterns over the data lifecycle


Agenda

Increasing Importance of Information


Partitioning in SQL Server 2005




Managing Mountains of Data

Data reduction– Archiving unused data– Pruning unused dimensions– Replacing complete data with summary or aggregate

data

Tiered storage solutions– Consolidate back up and recovery operations – Age data off most expensive storage– Match business information needs with appropriate

storage capabilities


Backing Up Mountains of Data on SQL 2000

VDI, provided by Microsoft with MS SQL 2000– Uses “WITH SNAPSHOT” functionality

Creates a Meta-data file with locations to aid recovery processing

Checkpoint and write-hold to quiesce database The database remains online and accessible during

the replication process Minimal/no impact on the performance of the

database server


Backing Up Mountains of Data on SQL 2005 with VSS

Volume ShadowCopy Service

RR Requester(Replication Manager)

Writer(Exchange)

Writer(SQL)

Writer(Other Apps)

Writer(Other Apps)

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5

Provider(Symmetrix)

Provider(CLARiiON)


Replication Manager Family Portfolio

Replication Manager/SE (new) creates local Exchange, SQL Server, and filesystem replications on a CLARiiON

Replication Manager/Local manages local (within an array) application-focused replications for repurposing and rapid recovery

ReplicationManager/SE

ReplicationManager/Local


Hot Split Backup with CloneSQL Backup

1) Check Production database integrity

2) Synchronize clones (VSS/VDI)

3) Perform hot-split (VSS/VDI)

6) Post processing task (optional)5) Truncate logs (optional)4) Mount replicas on backup server

BackupServer

SQL Production Server

DMX orCLARiiON

F:\Logs-Clone

G:\DB-Clone

F:\Logs

G:\DB Truncatelogs

Completely automated


Rapid Recovery from a CloneExchange Recovery

1) DB gets corrupted

2) Reverse sync from Clone

3) Mount DB and roll log forward

BackupServer

SQLProduction Server

DMX orCLARiiON

F:\Logs-Clone

G:\DB-Clone

F:\Logs

G:\DBG:\DBRoll forward logs

Completely automated


Managing Mountains of Data

Three tier cost savings can be substantial

Partitioned Storage Savings

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

Capacity (GB)

3 Tier $

2 Tier $

1 Tier $


Managing Mountains of DataCost Effective Tiered Storage Solutions

FastFast

MediumMedium

Highest DensityHighest Density

FastFast FastFast FastFast

MediumMedium

Data ages from high performance storage needs to higher density storage

Over time data ages to highest density storage


Agenda







SQL Server 2005Partitioned Tables and Indexes

What are they?– Default unit of data organization for all tables and

indexes in SQL Server 2005– Single object for the relational engine– Multiple objects for the storage engine

Why do I care?– Increased database uptime– Improved performance– Foundation for leveraging tiered storage


Partitioned Tables vs Views

Partitioned Views– Query optimizations (select) in 7.0 & 2000– No efficiencies with data modification– Tables managed separately

Partitioned Tables– Introduced in SQL Server 2005– Lower complexity – managed as one table– Higher performance – reduced contention


Divide and Conquer

Why partition databases?– Scalability

• Easier support for very large databases• See sessions DBA416 and DBA311

– Performance• Improved query optimization• Increases parallelization and reduced contention

– Availability & Manageability• Backup and restore individual filegroups• Finer control over data ranges


Sliding Window Scenario

Aging ranges across storage platforms

Q2Q220042004

Q3Q320042004

Q4Q420042004

Partitions move from primary storage to secondary storage as usage decreases

Q2Q220032003

Q3Q320032003

Q4Q420032003

Q1Q120042004

Q1Q120052005

Q2Q220052005

Q3Q320052005

Q4Q420052005

Q2Q220042004


Agenda







Project REAL Overview

Joint effort to discover best practices for BI applications on SQL Server 2005

Project comprises– Schema design including partitioning– Data ETL processes– Management and maintenance of systems– Management of incremental data updates

Visit project REAL for best practices• www.microsoft.com/sql/bi/ProjectREAL


Project REAL Overview Based on a real customer scenario:

Barnes & Noble Booksellers– 800 stores in the United States– 40,000 employees

Data warehouse supports planners, buyers and managers in headquarters, stores and distribution centers

Sales and inventory data

Data is “masked” to protect privacy and business value


Project REAL Overview

One year of inventory data comprises over 8 billion rows

– Data loaded/processed weekly– Weekly partitions < 12 million rows

Partitions read only after 8 weeks– Actually, filegroups are marked read only– Another value in storage aligned partitions

Two data aging methodologies– Sliding window-style with data copy– Storage array based (simplified ETL)


REAL BI System Architecture

ETL system integrates and moves data from sources to relational data warehouse

– SQL Server Integration Services

Relational warehouse provides flexiblelong-term storage

– SQL Server relational database

OLAP cubes allow fast interactiveexploration of large amounts of data

– SQL Server Analysis Services

Reports provide consistent, familiar outputsfrom the system

– SQL Server Reporting Services

Data mining helps to find patterns in large amounts of data

– SQL Server Analysis Services

SSIS manages data movement

Relationaldata

warehouse

Reportsgenerated

by RS

AS cubesand

data mining

Inventorydata

source

Salesdata

source


REAL Data Aging

Host-based file copy–Leverage sliding window concept–Data retained in table, moves on disk

Storage array-based LUN migration–“Data in place” aging of existing partitions–No change to partition function or scheme

Independent of loading new partitions


Older data is Older data is aged to highest aged to highest density storage density storage after one yearafter one year

REAL Tiered Storage

Last quarter Last quarter data is aged to data is aged to higher density higher density

storagestorage

LUN Migration reduces ETL process complexity and

increases storage and database availability


Agenda







Service Offerings


International Network Services

Vendor-Independent, Global Provider of IT Infrastructure Services & Software

– 12 years providing business-centric, technology solutions

• Professional services spanning IT infrastructure lifecycle

• Software to manage, secure & operate multiple IP infrastructures

– Privately held by West Coast Venture Capital and INS employees

– 30+ offices across North America, Europe, and the Pacific Rim

– Fortune 1000 focus

• Experience with >75% of Fortune 500

• Conducting business with >50% of the Fortune 100


Consulting Portfolios


Best-of-Breed Partner Strategy

Sec

urity

Project M

anagement

IT Planning & Strategy

OS & Directory Services

NetworkStorage

IP Networks

ConvergedConvergedInfrastructureInfrastructure

CiscoCisco

MSFTMSFT EMCEMC22Tech

nolo

gy

Tech

nolo

gy

Part

ners

Par

tners

Local/Reg

ional P

artners

Local/Reg

ional P

artners


EMC SQL Server Database Layout Accelerator Overview

Accelerated implementation or migration of your SQL Server database onto EMC networked storage—Symmetrix or CLARiiON

– Storage design for database layout

Best-practices implementation from EMC – Reference architectures

• Symmetrix• CLARiiON

– Best-practice configurations


What Makes the SQL Server Database Layout Unique?

Leverage EMC’s expertise

Documented customer deliverables

The EMC / Microsoft / INS alliance

– Technology collaboration– Best-practices collaboration– Joint Customer Support process

33


How the Solution Works

Assessment / Planning Design /Recommendations

Implementation /Testing / Validation

Phase 1: Assessment and Planning Review current

infrastructure—server, operating systems, database

Review current database-I/O usage

Define requirements for new MS SQL environment

Determine recommended architecture to optimize performance, scalability, and availability

Phase 2: Database Layout and Design Recommendations Provide recommendations

for optimal deployment of SQL Server on EMC SAN infrastructure, based on EMC best practices

Phase 3: Joint Testing and Validation (optional) Assist your staff with the

implementation, testing, and validation of recommendations

Provide knowledge transfer to your staff


• Completed Configuration Guide • Completed Test and Acceptance Plan (optional)

• Assistance with implementing server and / or storage design recommendation

• Successful completion of Test and Acceptance Plan (optional)

• Project Plan• Microsoft SQL Server review• Final Report detailing recommendations• Draft Configuration Guide • Proposed Test and Acceptance Plan

Customer Deliverables—Ensuring Knowledge Transfer to Your IT Staff

Phase 1

Assessment / Planning

Phase 2

Design / Recommendations

Phase 3

Implementation /Testing / Validation

Deliverables


SQL Design Validation Service

– Pre-Site Planning• Review your IT infrastructure and business requirements.• Verify that your IT infrastructure documentation aligns with your objectives

for the project

– On-Site Reviews and Gap Analysis• Uses a storage analyzer tool to gather info on servers & storage• Reviews existing Windows/SQL architecture and requirements definition.• Perform an SQL 200x design review and gap analysis.• Design an SQL 200x high-level architecture.• Provide a storage and server consolidation design review and gap

analysis.

– Final Report• Present findings and gap analysis

SQL DesignSQL Design

Validation Validation ServiceService

SQL experts review your design against industry best practices

Final Report includes Gap Analysis and Recommendations

Backbone

IP IP


SQL Archiving Assessment Service

– Workshop that details the benefits of Data archiving

– Final report includes• Key business requirements• High-level design and requirements• TCO/ROI review of project• Roadmap of required projects to implement archiving

solution

SQL Archiving SQL Archiving Assessment Assessment

ServiceService

1-2 Week Engagement

SQL 2000/2003 Data Archive

Assessment Report

Backbone

IP IP


To Learn More…

Visit :

EMC.com

Visit:

www.microsoft.com/sql/bi/ProjectREAL

http://www.emc.com/solutions/microsoft/index.jsp?openfolder=ba

http://www.microsoft.com/sql/bi/ProjectREAL

http://www.microsoft.com/sql/bi/ProjectREAL


Questions?


We invite you to participate in our online evaluation on CommNet,

accessible Friday only

If you choose to complete the evaluation online, there is no need to complete the paper evaluation

Your Feedback is Important!

© 2005 EMC Corporation. All rights reserved. SQL Server 2005 BI Cost Effective Information...

Documents

Transcript of © 2005 EMC Corporation. All rights reserved. SQL Server 2005 BI Cost Effective Information...