INDEX REORGANIZATION AND REBUILDING FOR SUCCESSAn end-to-end Enterprise re-indexing process

ABSTRACTDean will introduce a process and accompanying tools to make Index reorganization/rebuilding successful. Applicable on small databases and multi-Tb databases with over 20,000 tables (including examples of what can go horribly wrong without the right preparation). Bonus: PowerShell monitoring script included.

AGENDA• Indexing - What can go wrong?• Process Overview• Pre-requisites (initial setup/configuration)• Pre-index preparation• Running Index optimization• Index progress monitoring (awesome PowerShell script)

WHAT COULD POSSIBLY GO WRONG?

WHAT COULD POSSIBLY GO WRONG?

Lose your job BAD• Disk out of space

• Tlog or Database files• Tlog backup files

• Exponential Tlog growthTemporarily pain• Tlog or database growth initialization causing application timeout

CONSIDERATIONS• Enterprise Edition

• Online Index rebuild• Tlog auto-growth settings

• % or size• Size planning

• Full backups during re-index• Tlog will not truncate during a full backup even if log backups are being taken• Big database (4 hr backup + recovery) use differentials during re-index?• TODO: Add log_reuse_wait_desc to Powershell monitoring tool when log used percent is >

warning threshold.• Re-Index vs. Re-Organize• MaxDOP?• TempDb

PROCESS OVERVIEW

PROCESS OVERVIEW

INITIAL SYSTEM SETUP/CONFIGURATIONPre-requisites to using the proposed (demo) process

OLA HALLENGREN’S INDEX MAINTENANCE SCRIPT

• https://ola.hallengren.com/ • Optional, but recommended• CommandLog table

OLA’S INSTALL SCRIPTS• Demo uses a DBA AdminUtility

database (instead of master)• @CreateJobs set to Yes• @LogToTable set to Yes

SQL AGENT JOBS• Ola’s scripts will add these jobs• The IndexOptimize –

USER_DATABASES job is the one of interest for indexing

ADMINISTRATIVE DATABASE OR TABLES

SP_WHOISACTIVE• Adam Machanic’s sp_WhoIsActive script is a new and improved version of

sp_who2• http://

sqlblog.com/blogs/adam_machanic/archive/tags/who+is+active/default.aspx

• Shows current connections, % progress of index re-organization, and much more

SCHEDULED TASK

SCHEDULED TASK - TRIGGER• How often to run during indexing

SCHEDULED TASK - ACTION• ExecutionPolicy Bypass

SCHEDULED TASK - RUN

PRE-INDEX PREPARATION

PRE-INDEX PREPARATION FOR VLDB

• Tlog backup schedule• Shorten duration

• Full backup schedule• Temporarily switch to differential backups

• Open monitoring tools or schedule the Sweet Powershell script

INDEX MAINTENANCE

INDEX MAINTENANCE• Ola Hallengren Index Maintenance Solution (recommended)

• Queryable maintenance database (read: Sweet PowerShell monitoring)• Post maintenance review (% fragmentation, re-org vs. rebuild, etc.)

• Maintenance Plans

PROGRESS MONITORING

INDEX MONITORING – OLD SCHOOL

• DMV – Disk Usage• Windows Explorer – .trn backup file sizes• Windows Explorer – tlog backup duration• sp_who2

INDEX MONITORING – LIKE A BOSS

• Powershell Script• sp_WhoIsActive (by Adam Mechanic)• Queries

• Completed indexes• Current index rebuild progress (%, rows remaining)• Current index reorg progress (%)• Disk space used (Database files and Tlog• Tlog backup size and backup duration

• Windows Scheduler• Email

TLOG ABOUT TO GROW• Transactions are paused while

TLOG file is growing• Truncation should prevent growth

depending on initial size• Autogrowth settings…

DATA FILE ABOUT TO GROW

TLOG ACTIVITY

Re-org (~1 Gb/minute) Re-build (~200Kb/minute)

TLOG BACKUP DURATION• Durations shown in the query

don’t include the –WithVerify time• I usually use a 3X multiplier for

estimating actual tlog backup time• Look at start time of next backup

INDIVIDUAL INDEX PROGRESS

Re-Org Re-Build

WARNING SIGNS• VLDB data file about to grow• TLOG file about to grow• TLOG backup duration + Verify

taking longer than backup schedule schedule

• Log file not truncating (excessive growth)

• Wait times on sp_WhoIsActive• File sizes growing larger than

available disk space

DEAN WILLSON• Director of Application Development, Aunt Millie’s Bakeries • Long time member of NUFW• Former President of fwPASS• Twitter: @deanwillson• Blogs: dean-o.blogspot.com, practicalhoshin.blogspot.com• LinkedIn: http://linkedin.com/in/deanwillson