Windows 2000 / .NET Server Windows 2000 / .NET Server Performance Tuning and OptimizationPerformance Tuning and Optimization

Created and presented by:Rand MorimotoConvergent Computingrand@cco.com

AgendaAgendaOptimum Configuration and SetupOptimum Configuration and Setup

Ram / Hard Drive / LAN / WANRam / Hard Drive / LAN / WANScaleScale--up up vsvs SaleSale--OutOut

Performance TuningPerformance TuningBaseliningBaseliningCapacity PlanningCapacity PlanningDatabase Size and ReplicationDatabase Size and Replication

Regular Maintenance PracticesRegular Maintenance PracticesBackup ProceduresBackup ProceduresDaily / Weekly / Monthly Maintenance Daily / Weekly / Monthly Maintenance RoutinesRoutines

Convergent Computing (CCO)Windows .NET ServerWindows .NET ServerUNLEASHEDUNLEASHED____________________________________

Rand MorimotoRand MorimotoMichael NoelMichael NoelOmar DroubiOmar Droubi

Rand Morimoto, CCO President: Books on Exchange v5.x, NT Tuning / Optimization, Win2k Design and Migration, Exchange 2000 / Conferencing Server / SharePoint Portal, Maximum Windows 2000 Security, BizTalk Server Unleashed, and Windows .NET Server Unleashed

Contributing Writer - Windows 2000 Performance Tuning and Optimization (Osborne McGraw)

Section 1

Optimum ConfigurationOptimum Configurationand Setupand Setup

The Entire Networking Environment

Hubs

App Software

WAN to Clients / Vendors

Internet

WAN toRemote Sites

Terminal ClientsPC ClientsLegacy Hosts

Roaming Profiles

Desktop Systems

Domain Controllers

Clustering / FaultTolerance

Switches

ApplicationServers

Routers

Firewalls

File / Print Servers

Remote -TerminalClients

Remote -PC Clients

Internet User

SecuritySystemPolicies

Domain

Common Configuration ProblemsWrong amount of RAM in a server (either too little OR too much)Mismatched components (fast server, slow workstations / fast LAN, slow WAN)Improperly configured replication causing database corruption (AD Corruption, Exchange Corruption, WINS push/pull, DNS forwarding)Improper design assumptions (wrong use of switches, incorrect assumption of 10/100 and duplexing on ports)Overuse of protocols (IP, IPX, NetBeui, etc)

Server Configuration (RAM)

Ram ConfigurationToo little can cause excessive pagingToo much can cause excessive CPU and disk management of the pagefile

Best off having just the right amount of RAM

Hard Drive Configuration

Boot Drive (NOS and Applications)hardware mirrored for boot integrity

Data Drive (AD Databases)hardware mirrored

Transaction Logs (LOG files)hardware mirrored

Common RAID Levels

DD FT DD FT

RAID 5DDFT DD

FTDD DD

FT FT

RAID 0+1 DD FT DD FT

DDFT

FT FT FT FTRAID 1 DD DD DD DD

RAID 0 DD DD DD DD

DD indicates dataindicates dataFTFT indicates FT infoindicates FT info

RAID Level PerformanceRAID 0 (striping)• Highest performance• Lowest cost• No fault tolerance

RAID 5• Lowest fault tolerance performance• Lowest fault tolerance cost

RAID 1 And RAID 0+1• Highest fault tolerance performance • Highest cost

LAN Communications

• 10mb Ethernet• 100mb Fast Ethernet• Half Duplex / Full Duplex• 10/100 Ports• Switched / Shared Ethernet• Gigabit Ethernet

PCI Bus BandwidthAssuming a standard 32-bit 33-MHz PCI bus / Transfer Rate ~133-MB/sec (32-bit x 33mhz)With 40 MB/sec throughput on a PCI Adpt, 3 controllers could saturate a PCI busMany large systems support two or more PCI buses to extend PCI throughput to devicesPCI bus rates are getting faster • 32 bit 66 MHz 266 MB/sec • 64 bit 66 MHz 533 MB/sec

Multiple Fast IO Pipes

PCI 64bit DualPCI 64bit Dual

Next Gen IONext Gen IO

EISAEISAPCI 32bitPCI 32bit

PCI 32bit DualPCI 32bit Dual

MB/secMB/sec 3333 ~800~800132132 264264 528528This graph is based on current expectations. Actual results mayThis graph is based on current expectations. Actual results may differ materially.differ materially.

WAN Communications• ISDN• DSL• Frame Relay

– 56kb (DS0)– 128kb-1.54mb (T1/DS1)– T3/DS3 (45mb)

• Point to Point• Committed Information Rate (CIR)• Virtual Private Network (VPN)

Looking for Thruput

Win2k Active Directory Built on JetTransacted Information Store

RPCRPC

Transacted Transacted Information Information

StoreStore

Secure Process LayerSecure Process Layer

TransactionTransactionLogLog

AD DBAD DBProcessingProcessing

DC UpdatesDC Updates

Data is always written to Transaction Log• Ensures complete transaction integrity• Fast automatic rollback after restarting server

Scaling-Up / Scaling-Out / Distribution of Tasks

Scaling-Up• Faster Server / More Processors

– Database Servers (SQL, Oracle)– Engineer / CAD / Animation Servers

Scaling-Out• More Servers / Distributed Processing

– Exchange Messaging Servers (Exchange, Notes)• Redundancy

– Web Farms

Distribution of Tasks• Front-End / Back-End Servers• Component Load Balancing (AppCenter)

Leveraging Front-End/Back-End Servers

Single URL – ex: http://mail.companyabc.com• Users do not need to know their server’s name• Flexibility to move users between servers

Offload Processing: • SSL encryption done by Front-Ends• Communication between Front-Ends and Back-

Ends is not encrypted (faster response times)• Client requests compressed data. Front-end can

compress results

Section 2

Performance TuningPerformance Tuning

Areas of InterestActive Directory Database SizingActive Directory Network Traffic• AD Replication Traffic• Client Logon Traffic• LDAP Traffic

Server Sizing• Cost of Operations • Server load

Exchange 2000

Capacity Planning Aspects

Performance = Analyze_Capacity (Workload, Configuration)

Activity Given Given Determine

Sizing Workload Performance Configuration

Prediction Workload Configuration Performance

Saturation Configuration Performance Workload

Database FilesNTDS.DITLog files• Circular versus non-circular logging• Defaults to circular logging• HKEYLocalMachine\CurrentControlSetServices\

NTDS\Parameters\CircularLogging = 1• 2 Res.log files

Edb.chkTemp.edb

RequestorRequestor

1

NTDS.DIT

4Checkpoint

LSASS.EXELSASS.EXETransaction

Commit

Memory Cache

Log Files

I/O Patterns for the ESENTDS.DIT• 8KB I/O Size• 70-90% Read• Async. Write• Multi-Thread• Random Access

Log Area• 8KB I/O Size• Sync. Write• Single Thread• Sequential Access

NTDS.DIT NTDS.DIT

LSASS.EXE LSASS.EXE

Log Files Log Files

Performance TestsWe simulated 16-million and 40-million users• Same response time whatever the size• Scaled up to 110-million users until we

started to see noticeable response time degradation

Hardware Platform Tested focused on:• Security• Performance (I/O/s, not MB/s)• Capacity (several dozens GB)

Keeping an eye on the storage16 Million Users createdLinear growth pattern• 10,000 users: 62.2 MB• 100,000 users: 454 MB• 1,000,000 users: 4.1 GB• 10,000,000 users: 41.9 GB• 16,000,000 users: 68.6 GB

Replicated to Second DC

Adding Users to a Fragmented vsDefragmented Database

Fragmented/Defragmented databaseFragmented/Defragmented database

0500,000

1,000,0001,500,0002,000,0002,500,0003,000,0003,500,0004,000,0004,500,0005,000,000

0 200,000 400,000 600,000 800,000 1,000,000

1 User = 4,366 Bytes1 User = 4,366 Bytes

When Tuning a System, you need to Create a Baseline

With a baseline, you have:• A measure of the capacity of the system.• A point of reference from which to

measure.• A way to discuss what it can do.• A measure of its reliability.

How Do You Baseline?Find the point of maximum throughput.• Identify the infrastructure for a baseline.• Choose frequency and timing for measurement.• Choose a baseline test.

Choose Frequency and Length of Measurement• Frequency of measurement• The longer the interval between measurements, the

higher the risk

Set a Duration of Measurement• Vary your step lengths.• Short tests lead to misleading results and wrong

conclusions.

What Can You do With a Baseline?

You can compare it to itself over time• Did something change?• Where did the change occur?• How much did it change?

You can compare it to something else• Another segment or infrastructure. • Another server or network technology.• A different configuration.

Your Ongoing Role

Reliability Reliability PlanPlan

AcceptanceAcceptanceTestTest

Change Change PlanningPlanning

BaselineBaseline

Fine Tuning ReplicationFast LAN speed between DCs (100Mb)By default a DC is optimized to reduce network utilizationFollowing can be optimized• Packet sizes• Number of packets• Priority to increase replication (Multi-

processing• Latency for notification of

– New modification– Notification of next replication partner

How To Measure Replication TrafficAdd desired objectsStart NetMonWait until replication happens• Notification and replication callsForce replication using repadmin• Replication calls only• Naming context has to be specifiedUse Sites and Services snap-in to force replication• Sources all naming contexts which replication

partners have in common• Slightly higher network traffic

Intra-Site ReplicationIntra-Site replication assumes great network connectivity• Domain controllers use CPU cycles

for logon/search operations, not for data compression

Replication traffic is very predictable

Inter-Site Replication

Inter-Site replication compresses data very efficientlyThis reduces traffic to ~10% or belowReplication can be fully scheduledClients use site information to log on to a DC within the same site when possibleIf there is a WAN connection, create a site!

Quality Of ServicePrioritizing traffic through links• Getting important traffic through

congested links• Allocating service for bursty, high-demand

applications

Why Implement QoS?• Efficient usage of network• Traffic prioritization• Voice/Video/Data integration• Policy Networking• Application aware networking

Active Directory Replication Tools

NetDom• Manage Domain Resources & Trusts

Replmon• View & verify replication topology

DSAStat• Check replication between DCs

Repadmin• Manage, force, and verify replication

Section 3

Regular Maintenance PracticesRegular Maintenance Practices

Online Backup Types

TypeType DataData LogsLogs RestoreRestoreBacked UpBacked Up PurgedPurged

Normal (full)Normal (full)

IncrementalIncremental

DifferentialDifferential

SnapshotSnapshot

CopyCopy

Database & Database & log fileslog files

Log files onlyLog files only

Log files onlyLog files only

ImplementationImplementation

DatabaseDatabase

YesYes

YesYes

NoNo

NoNo

NoNo

•• Last NormalLast Normal•• Start ServiceStart Service

•• Last NormalLast Normal•• Every IncrEvery Incr•• Start ServiceStart Service

•• Last NormalLast Normal•• Last DiffLast Diff•• Start ServiceStart Service

•• SnapRestoreSnapRestore•• Log RestoreLog Restore•• Soft RecoverSoft Recover

•• Database PITDatabase PIT

Windows 2000 Backup Utility

• Reparse points• Directory junctions• Volume mount points

2. New Storage-related Features:

• Sparse files• Change journal• Encrypted File

System (EFS)

Backing Everything up in Win2k1. New NTFS-related features

• Single Instance Store• System File Protection• Indexing Service

• Remote Storage (using Removable Storage Manager – RSM)

• Distributed File System3. System State

System State components:

– System boot files.– Registry.– Com+ Class

Registration database.

– Certificate Services database.

– Cluster database.– File Replication Service

(FRS).

Always backed up & restored as a set.

Domain controllers alsohave:• Active Directory• System Volume (SYSVOL)

What else is “in” the System State?

System Databases• Disk quota information• Event logs• RSM database• Terminal Server database• Content Indexing catalogs

Backed up as unit.Can be restored individually.

Windows 2000 “Backup”

Able to:• Backup selected user files and folders, including network

shares.• Backup local computers’ “System State”.• Restore files and folders to any accessible hard drive.

(some System State components restorable on local computer only).

• Schedule backups.• Make copy of Remote Storage data and data stored on

any mounted drives.• Backup and restore encrypted files and folders without

decrypting them.

Windows 2000 “Backup”

Unable to:

• Backup/restore remote computer “System State” or Registry.

• Backup open files without 3rd party tool.• Centrally manage multiple backup servers.• Initiate Pre- or Post- Backup/Restore commands.• Automatically retry failed backups/restores.• Search for specific files to restore. (Browse only).

Best Practices – Backup Procedures Checklist

Create and Verify Daily BackupsPerform Periodic File Based BackupsStandardize on Tape FormatsImplement Battery BackupPerform a Period FiredrillReview the Environment Where Servers are PlacedCheck Windows NT Event LogsCreate a Disaster Kit

Best Practices – Backup Procedures Checklist (con’t)

Publish a Maintenance WindowDetermine Downtime CostsMaintain Off-Site Tapes and EquipmentDedicate Recovery Equipment and Build a Recovery LabKeep Records on All Server ConfigurationsDevise an Archiving Plan

Compaction and Defragmentationof the Active Directory DBDirectory Information Defragmentation(periodically compact the directory)Periodically Compacting the Directory will speed up access to the Directory• Offline Compacting of the Directory is more

efficient than the automatic, online compaction

Online and Offline Maintenance Processes and Procedures

Online Maintenance• Automatically invoked• Does about 60% of the DB maintenance

System Reboot• Does about 70% of the DB maintenance

Offline Maintenance• Database Defragmentation

– Maintains sequential integrity of data– More efficiently stores data

• Database Compaction– Shrinks database after old data is deleted or removed

Windows 2000 Directory CompactionConduct a full System State Backup (and verify backup by doing a restoral to a non-production test server)Boot to Directory Restore Mode (F8 on bootup)Integrity Check: NTDSUtil• “Files” | “Integrity” (if okay, then - )• “Semantic Database Analysis” | “Verbose On” | “Go” (dsdit.dmp log is

created)AD DB Compaction: NTDSUtil | Files• “Compact to %s” (where %s is a subdirectory name for the new DB)• Archive the old NTDS.DIT• Move the compacted DB to the old DB directory

Network Maintenance ChecklistDaily• Backup the ENTIRE System State• Check event viewer for successful backupWeekly• Check event viewer for errored events• Check available disk space and RAM/CPU

utilization• Check for Service Pack and Security UpdatesMonthly• Start/Stop services or Reboot• Apply Service Packs and Updates accordinglyMonthly/Quarterly (based on size of org)• Run Offline maintenance routines• Validate system optimization

Questions?

Windows 2000 / .NET Server Windows 2000 / .NET Server Performance Tuning and OptimizationPerformance Tuning and Optimization

Rand MorimotoConvergent Computing

internet: rand@cco.comhttp://www.cco.com