Managing 2000+ Windows/NT Servers in a Wan/LAN Environment

Stan DylnickiRoyal Bank Financial GroupCMG OrlandoDecember 13, 2000stan.dylnicki@royalbank.com(416) 348-4469

AgendaEnvironmentBackgroundIssuesMethodologyCurrent StatusToolsFuture

Bank EnvironmentLargest Bank in Canada49,000 employees2,000 IT employees1,600 branches

Bank Environment4 Processing Centres across Canada3 Campus environmentsDistricts & Service DeliveryBusiness Banking CentresBranches

PC/LAN Environment30,000 Windows NT10,000 Windows/95, DOS, OS/2, PCs2,000 Windows NT servers100 OS/2 serversHead Office, Branch, Business Banking

Application EnvironmentMost applications ports from DOS & OS/2New applications developed in Visual BasicClient data is on mainframesMajority of applications are bought solutions & modified for multi-tier

BackgroundMy background is mainframeLessons learned from OS/2 deploymentProactive branch supportMove from OS/2 to NT

IssuesLarge number of servers & locationsDiverse types of serversMany configurations & hardwarePerformance data hard to getApplications written for functionality (not necessarily for performance)Bought solutions dont always scale across platformsSystems Management emerging

More IssuesHow to report on large number of serversHow to collect enough information without overloading networkExplosive growth of memory and disk on desktopsExplosive growth of File servers disk spaceExplosive growth of Internet usage

MethodologyUse alert tool for performance monitoringDont collect everythingSize servers & desktops for life of the boxManage & report by exceptionProvide tools to users so they can diagnose their own problemsProvide limited set of reports

Current StatusAll NT images include a performance agentClassify servers by typeProvide exception reports to area managersPilot disk quota monitoringProvide web access for management reportingBenchmark hardware to understand capacity limits

ToolsTivoli alert agent performance & availabilityNTSMF collectorsSAS ITSV repositoryWeb reportingDesktop Memory MonitorBenchmarks: Iometer, Netbench, Sysbench, SYSMARK32

NTSMF & SAS ITSV Data FlowNTSMFNTSMF

SAS ITSV EnvironmentRepository & IISNetfinity 5500 Server2 Pentium II 450 cpus512 MB RAMRaid-5 (9 GB X 6)FTP Server (shared)Dell 6100 Server4 Pentium Pro cpus512 MB RAMRaid-5 (9GB X 6)

SAS ITSV Processing Statistics

Sheet1

ITSV Processing Time

DatabaseServer CountTotal Time (HH:MM)Time per server (MM:SS)

APPLICATION4:145:234/12/001:094/12/00APPLICATION651:0901:03.7

BRANCH0:174:054/12/003:484/12/00BRANCH6493:4800:21.1

BRANCH10:171:394/12/001:224/12/00BRANCH13311:2200:14.9

CALLCENTRE3:225:064/12/001:444/12/00CALLCENTRE761:4401:22.1

COMM6:337:454/12/001:124/12/00COMM1051:1200:41.1

DATABASE6:146:524/12/000:384/12/00DATABASE290:3801:18.6

EXCHANGE5:236:334/12/001:104/12/00EXCHANGE301:1002:20.0

FILEPRINT5:065:424/12/000:364/12/00FILEPRINT310:3601:09.7

INTERNET6:527:384/12/000:464/12/00INTERNET280:4601:38.6

NOTES7:388:014/12/000:234/12/00NOTES190:2301:12.6

TERMSERVER5:426:144/12/000:324/12/00TERMSERVER150:3202:08.0

Totals:1,37813:2000:34.8

Actual Elapsed Time7:4400:20.2

Sheet2

Sheet3

SAS ITSV Processing by Database

Chart1

0.03472222220.25347222220.03055555560.02361111110.04930555560.01805555560.02222222220.02430555560.015972222200

0.03263888890.23750.05208333330.02013888890.05138888890.01805555560.018750.01597222220.015972222200

0.03472222220.28263888890.02916666670.03055555560.03819444440.01805555560.01458333330.02083333330.013888888900

0.031250.23611111110.03333333330.01736111110.05763888890.01805555560.01666666670.01736111110.016666666700

0.03194444440.25763888890.03958333330.02083333330.05277777780.01736111110.02291666670.02013888890.0187500

0.03333333330.29583333330.02847222220.02430555560.04583333330.01805555560.02222222220.02638888890.027777777800

0.03194444440.281250.03958333330.02569444440.06458333330.01805555560.01736111110.02361111110.022222222200

0.031250.26388888890.03888888890.0250.05763888890.01666666670.01944444440.01944444440.015972222200

0.031250.26180555560.04513888890.02152777780.05694444440.01805555560.01666666670.01805555560.0187500

0.03055555560.25763888890.04236111110.02222222220.05763888890.01736111110.018750.01944444440.0187500

0.03333333330.27916666670.05277777780.018750.05694444440.018750.02013888890.01736111110.009722222200

0.03333333330.30694444440.02916666670.02361111110.03888888890.01805555560.01736111110.01805555560.027777777800

0.03333333330.24097222220.02916666670.03194444440.04027777780.01736111110.01458333330.01805555560.000694444400

0.03194444440.22291666670.02847222220.02291666670.04861111110.01805555560.018750.01666666670.000694444400

0.03055555560.22083333330.02986111110.02152777780.04652777780.01666666670.02291666670.01736111110.020138888900

0.03055555560.21736111110.03472222220.02361111110.05486111110.01736111110.01736111110.01527777780.017361111100

0.03333333330.22986111110.03472222220.02222222220.05347222220.01666666670.018750.018750.017361111100

0.03194444440.22916666670.03194444440.02222222220.05555555560.01666666670.01666666670.02013888890.0187500

0.03611111110.26111111110.02430555560.018750.04513888890.01944444440.01597222220.01250.013194444400

0.03402777780.243750.031250.03055555560.03541666670.01597222220.01458333330.01944444440.016666666700

0.03333333330.23611111110.03055555560.02152777780.03958333330.01666666670.02013888890.01736111110.024305555600

0.03333333330.23680555560.03055555560.00138888890.04097222220.01597222220.01597222220.01805555560.02500

0.03263888890.24652777780.03680555560.00277777780.03611111110.01666666670.01736111110.01527777780.019444444400

0.07083333330.50277777780.06388888890.05972222220.07361111110.03194444440.03750.03611111110.036111111100

0.07083333330.52777777780.06666666670.06666666670.07916666670.03333333330.04166666670.04166666670.043055555600

0.03333333330.256250.031250.02916666670.03680555560.01666666670.01597222220.018750.0187500

0.04652777780.27986111110.03263888890.02222222220.05555555560.02569444440.02222222220.02847222220.015972222200

0.03680555560.26736111110.02847222220.03055555560.03611111110.01666666670.01944444440.02013888890.013888888900

0.03402777780.26319444440.02986111110.02222222220.04305555560.01666666670.01666666670.01944444440.028472222200

0.03611111110.27013888890.03055555560.03402777780.03611111110.01666666670.01805555560.02013888890.015972222200

0.03541666670.28888888890.03819444440.03680555560.03680555560.01597222220.01666666670.01944444440.012500

0.03402777780.27708333330.03263888890.02777777780.03958333330.01666666670.01944444440.02222222220.027777777800

0.03611111110.32708333330.06458333330.02013888890.03680555560.01666666670.018750.01944444440.010416666700

0.03541666670.30555555560.04930555560.02013888890.03402777780.01666666670.01736111110.01944444440.011111111100

0.03680555560.29583333330.03819444440.05416666670.03888888890.01597222220.018750.02569444440.019444444400

0.03680555560.28333333330.031250.03472222220.03680555560.01597222220.02152777780.02083333330.015277777800

0.03819444440.318750.05208333330.01736111110.03611111110.01527777780.02569444440.018750.011111111100

0.05486111110.37569444440.031250.02291666670.05972222220.02222222220.02152777780.01319444440.015277777800

0.03680555560.26805555560.03055555560.03888888890.03750.01597222220.02291666670.02013888890.002083333300

0.03750.27916666670.03263888890.03402777780.03680555560.01597222220.02222222220.02013888890.013194444400

0.03611111110.35694444440.02847222220.02152777780.04652777780.01805555560.02152777780.01250.014583333300

0.03611111110.45347222220.02291666670.01527777780.03680555560.01527777780.01597222220.01180555560.011111111100

0.03333333330.36041666670.02013888890.01527777780.03472222220.01597222220.01666666670.02222222220.011111111100

0.03402777780.37430555560.01944444440.01388888890.03611111110.01458333330.01597222220.01041666670.010416666700

0.03333333330.33472222220.02013888890.01527777780.03611111110.01527777780.01527777780.01527777780.011111111100

0.03402777780.34513888890.02083333330.01458333330.03888888890.01458333330.01597222220.01041666670.011111111100

0.03263888890.33541666670.02361111110.01388888890.03611111110.01597222220.01527777780.01180555560.011111111100

0.03402777780.34444444440.02222222220.01458333330.03611111110.01527777780.01597222220.01041666670.011111111100

0.03333333330.368750.02083333330.01527777780.03958333330.01527777780.01666666670.01458333330.011111111100

0.03472222220.39444444440.00138888890.01527777780.03958333330.01527777780.01736111110.01111111110.011111111100

0.03263888890.29791666670.03194444440.01527777780.04027777780.01597222220.01736111110.0250.011111111100

0.03194444440.12777777780.02083333330.01388888890.03333333330.01458333330.01527777780.00972222220.009027777800

0.03333333330.07916666670.02152777780.01458333330.03194444440.01527777780.01597222220.01111111110.009722222200

0.04027777780.34930555560.02916666670.01597222220.04652777780.01944444440.018750.01111111110.012500

0.04027777780.26944444440.02361111110.01597222220.03680555560.018750.018750.01111111110.011805555600

0.04166666670.28333333330.02638888890.01527777780.03888888890.01944444440.018750.01111111110.012500

0.04652777780.26111111110.02638888890.01666666670.03888888890.02083333330.018750.01180555560.011805555600

0.05069444440.27083333330.02777777780.01736111110.04097222220.01666666670.018750.01111111110.011805555600

0.04513888890.22777777780.02777777780.018750.03958333330.02083333330.02083333330.01319444440.012500

0.03194444440.22777777780.02916666670.01805555560.03750.01736111110.02152777780.01041666670.01319444440.1250

0.031944444400.02708333330.02013888890.03263888890.01458333330.02013888890.01041666670.01527777780.10972222220

0.03263888890.22152777780.03263888890.02083333330.04236111110.01666666670.02291666670.01319444440.01458333330.11180555560

0.03750.20416666670.02986111110.01805555560.04027777780.01805555560.02222222220.01111111110.01458333330.10555555560

0.03888888890.22152777780.02916666670.01944444440.03958333330.01944444440.02152777780.01041666670.01111111110.11180555560

0.03958333330.21944444440.03333333330.02013888890.03958333330.018750.01944444440.01319444440.01180555560.11527777780

0.04166666670.22291666670.03472222220.018750.04166666670.02013888890.02013888890.01319444440.01388888890.08472222220

0.04305555560.2250.03333333330.01944444440.043750.01944444440.02083333330.01041666670.01666666670.08750

0.04305555560.21458333330.03472222220.018750.043750.018750.02152777780.01111111110.01736111110.07777777780

0.04236111110.20555555560.03611111110.02152777780.04444444440.018750.02361111110.01111111110.01527777780.07013888890

0.04513888890.2250.03611111110.02152777780.04166666670.02013888890.02291666670.01388888890.01458333330.06944444440

0.04444444440.206250.03611111110.02013888890.04236111110.02152777780.02222222220.01180555560.01736111110.06944444440

0.04444444440.21180555560.03680555560.018750.04027777780.02083333330.02222222220.01319444440.01597222220.07222222220

0.04305555560.22152777780.03958333330.02152777780.04097222220.02083333330.02361111110.01458333330.01458333330.068750

0.04305555560.21736111110.03888888890.02222222220.04236111110.02222222220.02361111110.01666666670.01527777780.11388888890

0.04444444440.22916666670.03958333330.02083333330.04166666670.02222222220.02430555560.01319444440.01736111110.10763888890

0.03263888890.22152777780.031250.018750.03055555560.02083333330.01944444440.01250.01388888890.10069444440

0.043750.22291666670.03819444440.02152777780.04305555560.02291666670.02222222220.01597222220.01458333330.10972222220

0.04305555560.20833333330.03888888890.02222222220.04166666670.02361111110.02152777780.01666666670.01458333330.09930555560

0.04166666670.20555555560.04097222220.02152777780.04027777780.02291666670.02569444440.01666666670.01388888894.76388888890

0.04166666670.20902777780.03819444440.02013888890.04097222220.02291666670.02361111110.01597222220.01527777780.093750

0.04236111110.21250.03888888890.02083333330.04097222220.02361111110.02361111110.01666666670.015972222200

0.04652777780.21944444440.03819444440.02222222220.04166666670.02291666670.02222222220.01388888890.018055555600

0.04305555560.21527777780.03888888890.02083333330.04236111110.02569444440.02291666670.01736111110.016666666700

0.04444444440.22847222220.04027777780.01944444440.04305555560.02361111110.02291666670.01319444440.018055555600

0.04444444440.21388888890.03819444440.02013888890.04166666670.0250.02291666670.01458333330.018055555600

0.04305555560.20972222220.03958333330.02222222220.04236111110.02708333330.02291666670.01805555560.015972222200

0.04583333330.22083333330.04027777780.02152777780.04097222220.0250.02291666670.01319444440.019444444400

0.04513888890.22083333330.04097222220.02222222220.04166666670.02430555560.02291666670.01319444440.019444444400

0.04722222220.21944444440.04166666670.02291666670.04305555560.02361111110.02569444440.01319444440.019444444400

0.04861111110.22013888890.043750.02361111110.04236111110.02430555560.02430555560.01319444440.0187500

0.04861111110.24097222220.04513888890.02222222220.04305555560.02638888890.0250.01944444440.016666666700

0.04652777780.21527777780.04097222220.02083333330.043750.0250.02430555560.01319444440.018055555600

0.04652777780.21319444440.04166666670.02013888890.04722222220.0250.0250.01527777780.0187500

0.04652777780.22708333330.04236111110.02083333330.04444444440.02430555560.02430555560.01388888890.0187500

0.04513888890.22222222220.04791666670.02152777780.04305555560.02638888890.02638888890.02013888890.018055555600

0.04722222220.22222222220.04583333330.02222222220.04513888890.02847222220.02638888890.018750.0187500

0.04583333330.22222222220.04097222220.02222222220.04513888890.02638888890.02430555560.01736111110.0187500

0.050.243750.04513888890.02361111110.04513888890.0250.02361111110.018750.000694444400

0.04652777780.21805555560.04791666670.02152777780.04236111110.02638888890.02708333330.01527777780.000694444400

0.04583333330.21527777780.04583333330.02083333330.04097222220.02569444440.02847222220.01736111110.022222222200

0.03680555560.30972222220.03611111110.02083333330.04444444440.02569444440.02361111110.01527777780.015972222200

0.03888888890.22638888890.04444444440.02222222220.04236111110.02361111110.02430555560.01319444440.0187500

0.04236111110.22847222220.04305555560.02152777780.04236111110.0250.02638888890.01805555560.017361111100

0.04236111110.21805555560.04166666670.01944444440.04236111110.02569444440.02569444440.01805555560.016666666700

0.04791666670.26597222220.04444444440.02083333330.04236111110.02569444440.02708333330.01666666670.0187500

0.04513888890.22152777780.04444444440.02152777780.04236111110.02847222220.0250.01805555560.016666666700

0.04583333330.2250.04305555560.02222222220.04513888890.02847222220.02638888890.01736111110.016666666700

0.04861111110.24444444440.04097222220.02083333330.043750.02569444440.0250.01388888890.019444444400

0.04791666670.24166666670.04305555560.02152777780.04583333330.0250.02638888890.01458333330.0187500

0.03402777780.22569444440.03472222220.01944444440.02986111110.02361111110.02152777780.01319444440.015972222200

0.043750.22013888890.04583333330.02083333330.04305555560.02361111110.02708333330.01944444440.018055555600

0.04652777780.243750.04444444440.02222222220.04444444440.02291666670.02430555560.01388888890.019444444400

0.04583333330.22152777780.04444444440.02361111110.043750.02430555560.02569444440.018750.017361111100

0.04444444440.218750.043750.02361111110.04236111110.02430555560.02569444440.02013888890.016666666700

0.04513888890.21805555560.04583333330.02291666670.04236111110.02291666670.02708333330.018750.015972222200

0.056250.25902777780.03819444440.02222222220.04444444440.02777777780.02361111110.01388888890.015972222200

0.04722222220.2250.04722222220.02013888890.043750.02569444440.02777777780.01944444440.0187500

0.04444444440.22708333330.04236111110.02083333330.04305555560.0250.02638888890.018750.0187500

0.04861111110.24097222220.04513888890.02430555560.043750.02291666670.02638888890.01527777780.019444444400

0.04583333330.2250.04513888890.02361111110.04444444440.0250.02569444440.01805555560.015972222200

00.21180555560.05208333330.02291666670.04027777780.02222222220.02638888890.02013888890.019444444400

0.00069444440.06250000000.000694444400

0.06666666670.33611111110.05347222220.02430555560.04791666670.02916666670.02708333330.02152777780.020138888900

0.05555555560.34513888890.04791666670.02430555560.04861111110.02777777780.02708333330.01527777780.019444444400

0.04444444440.12222222220.050.02222222220.03611111110.02152777780.02847222220.02083333330.020138888900

0.05069444440.22847222220.04444444440.02222222220.04305555560.02638888890.02708333330.01458333330.0187500

0.0479166667-0.47916666670.05069444440.02430555560.04652777780.02638888890.031250.02361111110.0187500

0.05416666670.35694444440.04722222220.02222222220.04305555560.02777777780.02708333330.02083333330.019444444400

0.05486111110.31388888890.04861111110.02083333330.04444444440.02361111110.02777777780.02152777780.019444444400

0.05416666670.29861111110.04513888890.02222222220.04305555560.02291666670.02708333330.01388888890.01944444440.11666666670.0916666667

00.30694444440.04027777780.01805555560.03611111110.01944444440.02013888890.01458333330.01527777780.06250.0895833333

0.07152777780.1625-0.23194444440.02986111110.05763888890.031250.03750.03055555560.02708333330.07638888890.0847222222

0.0604166667-0.01041666670.06319444440.03055555560.06250.02986111110.03680555560.02291666670.02569444440.05972222220.08125

0.0493055556-0.01111111110.04722222220.02291666670.043750.02638888890.03055555560.01736111110.01805555560.06041666670.08125

0.06597222220.175-0.22986111110.02638888890.05694444440.03055555560.03472222220.02569444440.02569444440.07708333330.0875

0.043750.14513888890.04652777780.02152777780.04444444440.02430555560.02638888890.02083333330.01736111110.05833333330.0333333333

0.04861111110.13333333330.04652777780.02361111110.04722222220.02361111110.02638888890.02083333330.02222222220.04722222220.0166666667

0.050.13819444440.04930555560.02291666670.03680555560.02569444440.02638888890.01736111110.018750.04652777780.0173611111

0.04652777780.156250.04652777780.02430555560.04722222220.02361111110.02638888890.01527777780.02152777780.06944444440.0229166667

0.04513888890.131250.04444444440.02430555560.04791666670.02291666670.02638888890.01458333330.02083333330.05416666670.01875

0.04652777780.13263888890.04305555560.02291666670.04652777780.02361111110.0250.01388888890.02083333330.050.01875

0.04722222220.13750.04652777780.02291666670.04652777780.02361111110.02777777780.01597222220.02013888890.05138888890.01875

0.04652777780.13611111110.04444444440.02430555560.04583333330.02361111110.02569444440.01527777780.02013888890.04861111110.0180555556

0.04513888890.13611111110.04583333330.0250.04583333330.02222222220.02708333330.01458333330.02152777780.05486111110.01875

0.05069444440.13680555560.043750.02222222220.04583333330.02222222220.02777777780.01597222220.02013888890.05347222220.01875

0.05208333330.14583333330.04652777780.02291666670.04583333330.0250.02638888890.02013888890.02013888890.04861111110.0243055556

0.04722222220.13819444440.04861111110.0250.04583333330.02222222220.02638888890.01666666670.01944444440.05416666670.0201388889

0.04444444440.13472222220.043750.02361111110.04583333330.02222222220.02638888890.01458333330.018750.05416666670.0208333333

0.03611111110.13680555560.03541666670.02083333330.03402777780.02152777780.02222222220.01458333330.00069444440.03541666670.0194444444

00.13888888890.00069444440.01944444440.03472222220.0187500.000694444400.03402777780.0159722222

0.04236111110.13333333330.04166666670.02152777780.04305555560.02222222220.02638888890.01458333330.02013888890.05347222220.0152777778

0.04166666670.13819444440.04444444440.02152777780.04166666670.02291666670.02777777780.01527777780.02013888890.05347222220.0194444444

0.04513888890.14444444440.043750.02222222220.04513888890.02430555560.02430555560.01597222220.02013888890.05486111110.0256944444

0.04513888890.13680555560.04305555560.02152777780.04652777780.02361111110.02708333330.01527777780.01944444440.04791666670.0208333333

0.04444444440.13472222220.04305555560.02222222220.04444444440.02361111110.02708333330.01319444440.018750.05555555560.0201388889

0.04652777780.14027777780.04166666670.0250.04305555560.02222222220.02986111110.01597222220.01944444440.05555555560.01875

0.04305555560.13819444440.04097222220.02152777780.04583333330.02430555560.02777777780.01388888890.02083333330.04930555560.01875

0.04236111110.14027777780.04097222220.02083333330.04583333330.02361111110.02638888890.01388888890.02152777780.04930555560.01875

0.043750.14027777780.043750.02222222220.04861111110.02361111110.02777777780.01597222220.02291666670.04791666670.0194444444

0.04861111110.14930555560.04166666670.02430555560.04722222220.02569444440.02847222220.01805555560.02083333330.04861111110.0263888889

0.04652777780.13611111110.04166666670.02361111110.04652777780.02430555560.02777777780.01458333330.02013888890.04722222220.0215277778

0.04166666670.13402777780.043750.02361111110.043750.02361111110.02708333330.01458333330.01944444440.05347222220.0215277778

0.043750.14097222220.04236111110.02361111110.04444444440.02430555560.02777777780.01666666670.02222222220.05833333330.0201388889

0.04305555560.13888888890.043750.02291666670.04652777780.02361111110.02777777780.01527777780.02152777780.05486111110.0243055556

0.04513888890.14097222220.04305555560.02222222220.04791666670.02430555560.02916666670.01597222220.02291666670.06041666670.0243055556

0.04027777780.14097222220.043750.02152777780.04444444440.02430555560.02847222220.01458333330.02152777780.06180555560.025

0.04166666670.15069444440.04444444440.02430555560.04444444440.02291666670.02777777780.01527777780.02222222220.06250.0305555556

0.043750.13888888890.04166666670.02291666670.043750.02222222220.02777777780.01597222220.02152777780.06111111110.025

0.03888888890.13402777780.04236111110.02222222220.04861111110.02222222220.02777777780.018750.01944444440.06250.0243055556

0.04166666670.13958333330.04444444440.02152777780.04305555560.02222222220.02708333330.01527777780.02152777780.06111111110.025

0.04097222220.13958333330.04652777780.02152777780.04513888890.02222222220.02708333330.01597222220.02013888890.06180555560.0243055556

0.04166666670.14236111110.04236111110.02152777780.043750.02222222220.02916666670.01527777780.02013888890.06180555560.025

0.04305555560.14444444440.04166666670.02291666670.04652777780.02291666670.02777777780.01597222220.02013888890.06250.0243055556

0.04513888890.15486111110.04236111110.02083333330.04652777780.02291666670.02916666670.01597222220.02152777780.06319444440.0326388889

0.043750.14097222220.04166666670.02291666670.04513888890.02361111110.02708333330.01527777780.02013888890.06180555560.0243055556

0.04444444440.14097222220.04305555560.02083333330.04583333330.02222222220.02777777780.01458333330.02083333330.05833333330.0243055556

0.04513888890.14791666670.04444444440.02222222220.04444444440.02291666670.02638888890.01666666670.02222222220.05694444440.0243055556

0.04791666670.15833333330.04236111110.02152777780.04444444440.0250.02916666670.01805555560.01944444440.06250.0229166667

0.04236111110.14513888890.04097222220.02291666670.04444444440.02361111110.02847222220.01736111110.02222222220.06458333330.025

0.04027777780.143750.04305555560.02152777780.043750.02222222220.02708333330.01527777780.02222222220.06319444440.025

0.04652777780.16944444440.04236111110.02222222220.04513888890.0250.02847222220.01736111110.02013888890.06388888890.0277777778

0.04305555560.14444444440.043750.0250.043750.02361111110.02847222220.01666666670.02013888890.06111111110.025

0.039583333300.04305555560.02291666670.04305555560.02291666670.02777777780.01805555560.01736111110.06180555560

0.05486111110.27777777780.04236111110.02569444440.04513888890.02708333330.02916666670.018750.01805555560.07291666670.0270833333

0.043750.14513888890.04166666670.02291666670.04583333330.02569444440.02847222220.01597222220.02013888890.06319444440.0263888889

0.043750.14444444440.04236111110.02430555560.04444444440.02291666670.02916666670.01805555560.02083333330.06388888890.0263888889

0.04444444440.14652777780.04722222220.02430555560.050.02291666670.02777777780.01597222220.02152777780.06250.0270833333

0.04722222220.16597222220.04027777780.02430555560.04791666670.0250.02708333330.01736111110.02222222220.06250.0340277778

0.04444444440.14513888890.04444444440.02361111110.04861111110.0250.02569444440.01527777780.02083333330.06180555560.0256944444

0.04236111110.13958333330.04444444440.02638888890.04652777780.02291666670.02777777780.01527777780.02013888890.05833333330.0256944444

0.04722222220.14652777780.04305555560.02291666670.04722222220.02291666670.02847222220.01805555560.02013888890.05555555560.0263888889

0.04861111110.156250.04583333330.02361111110.04513888890.02638888890.02777777780.01944444440.02013888890.056250.0333333333

0.04027777780.156250.04305555560.02291666670.04513888890.02222222220.02638888890.018750.01944444440.06041666670.0347222222

0.04513888890.15972222220.04513888890.02430555560.04791666670.0250.02916666670.018750.02013888890.05416666670.0375

0.00069444440.177083333300000.000694444400.00069444440.04305555560.0326388889

0.00069444440.14444444440.00069444440.016666666700000.00069444440.03541666670.0256944444

00.140972222200.017361111100000.00069444440.04027777780.0256944444

0.04236111110.16805555560.043750.02152777780.04930555560.02430555560.02916666670.01666666670.018750.06180555560.0236111111

0.04166666670.19444444440.03611111110.02222222220.04722222220.02083333330.02708333330.01319444440.02222222220.05555555560.03125

0.04166666670.43402777780.04583333330.0250.04722222220.02083333330.02847222220.01944444440.02777777780.05347222220.0270833333

0.043750.16180555560.03680555560.02222222220.04305555560.02569444440.02638888890.01666666670.018750.05416666670.03125

0.04444444440.16944444440.03888888890.02152777780.043750.0250.02638888890.01388888890.02361111110.056250.0388888889

0.04236111110.150.03819444440.02222222220.04583333330.02361111110.02777777780.01666666670.02013888890.050.0319444444

0.04444444440.15972222220.03750.02361111110.04513888890.02361111110.02708333330.01666666670.01805555560.06319444440.0305555556

0.043750.16944444440.03888888890.02152777780.04583333330.0250.02986111110.01458333330.02013888890.05972222220.0326388889

0.04097222220.14027777780.04166666670.02291666670.04861111110.02361111110.02986111110.01527777780.02361111110.06111111110.05

0.04930555560.15555555560.04652777780.02569444440.05069444440.02638888890.02777777780.01944444440.02222222220.05902777780.0458333333

0.05138888890.15347222220.04444444440.0250.050.02777777780.031250.01805555560.02152777780.05694444440.05

0.05069444440.17569444440.04444444440.02569444440.04791666670.02569444440.02986111110.01805555560.02430555560.05277777780.0576388889

0.04444444440.19236111110.04722222220.02569444440.04722222220.02638888890.031250.01527777780.02222222220.06180555560.04375

0.05138888890.18333333330.04930555560.02777777780.04930555560.0250.03333333330.01805555560.02291666670.06250.0479166667

0.043750.13750.04722222220.0250.04930555560.00902777780.03333333330.01597222220.02291666670.05902777780.0458333333

0.050.15069444440.05069444440.02708333330.04861111110.02708333330.03055555560.01805555560.02222222220.05833333330.0520833333

0.04652777780.14861111110.04861111110.02569444440.04722222220.02569444440.02986111110.01597222220.02361111110.06597222220.0527777778

0.04722222220.150.04583333330.02777777780.04722222220.02569444440.02916666670.01527777780.02291666670.05972222220.0534722222

0.050.17430555560.04861111110.02777777780.04652777780.02291666670.02916666670.01736111110.02361111110.05902777780.0625

0.04444444440.13958333330.04583333330.02777777780.04513888890.02430555560.02986111110.01736111110.02152777780.05902777780.05

0.04305555560.13750.04791666670.02777777780.04513888890.02430555560.03194444440.01805555560.02013888890.05763888890.0493055556

0.04861111110.16111111110.043750.02638888890.04930555560.02777777780.031250.01458333330.02222222220.04930555560.0597222222

0.04791666670.15833333330.050.02638888890.04861111110.0250.03194444440.01597222220.02222222220.07222222220.0569444444

Application

Branch

Comm

Database

Exchange

FilePrint

Internet

Notes

TermServer

CALLCENTRE

BRANCH1

Elapsed Time

Sheet1

DateApplicationBranchCommDatabaseExchangeFilePrintInternetNotesTermServerCALLCENTREBRANCH1

8/30/990:506:050:440:341:110:260:320:350:230:000:00

8/31/990:475:421:150:291:140:260:270:230:230:000:00

9/1/990:506:470:420:440:550:260:210:300:200:000:00

9/2/990:455:400:480:251:230:260:240:250:240:000:00

9/3/990:466:110:570:301:160:250:330:290:270:000:00

9/4/990:487:060:410:351:060:260:320:380:400:000:00

9/5/990:466:450:570:371:330:260:250:340:320:000:00

9/6/990:456:200:560:361:230:240:280:280:230:000:00

9/7/990:456:171:050:311:220:260:240:260:270:000:00

9/8/990:446:111:010:321:230:250:270:280:270:000:00

9/9/990:486:421:160:271:220:270:290:250:140:000:00

9/10/990:487:220:420:340:560:260:250:260:400:000:00

9/11/990:485:470:420:460:580:250:210:260:010:000:00

9/12/990:465:210:410:331:100:260:270:240:010:000:00

9/13/990:445:180:430:311:070:240:330:250:290:000:00

9/14/990:445:130:500:341:190:250:250:220:250:000:00

9/15/990:485:310:500:321:170:240:270:270:250:000:00

9/16/990:465:300:460:321:200:240:240:290:270:000:00

9/17/990:526:160:350:271:050:280:230:180:190:000:00

9/18/990:495:510:450:440:510:230:210:280:240:000:00

9/19/990:485:400:440:310:570:240:290:250:350:000:00

9/20/990:485:410:440:020:590:230:230:260:360:000:00

9/21/990:475:550:530:040:520:240:250:220:280:000:00

9/23/991:4212:041:321:261:460:460:540:520:520:000:00

9/24/991:4212:401:361:361:540:481:001:001:020:000:00

9/27/990:486:090:450:420:530:240:230:270:270:000:00

9/28/991:076:430:470:321:200:370:320:410:230:000:00

9/29/990:536:250:410:440:520:240:280:290:200:000:00

9/30/990:496:190:430:321:020:240:240:280:410:000:00

10/1/990:526:290:440:490:520:240:260:290:230:000:00

10/2/990:516:560:550:530:530:230:240:280:180:000:00

10/3/990:496:390:470:400:570:240:280:320:400:000:00

10/4/990:527:511:330:290:530:240:270:280:150:000:00

10/5/990:517:201:110:290:490:240:250:280:160:000:00

10/6/990:537:060:551:180:560:230:270:370:280:000:00

10/7/990:536:480:450:500:530:230:310:300:220:000:00

10/8/990:557:391:150:250:520:220:370:270:160:000:00

10/12/991:199:010:450:331:260:320:310:190:220:000:00

10/13/990:536:260:440:560:540:230:330:290:030:000:00

10/14/990:546:420:470:490:530:230:320:290:190:000:00

10/15/990:528:340:410:311:070:260:310:180:210:000:00

10/16/990:5210:530:330:220:530:220:230:170:160:000:00

10/17/990:488:390:290:220:500:230:240:320:160:000:00

10/18/990:498:590:280:200:520:210:230:150:150:000:00

10/19/990:488:020:290:220:520:220:220:220:160:000:00

10/20/990:498:170:300:210:560:210:230:150:160:000:00

10/21/990:478:030:340:200:520:230:220:170:160:000:00

10/22/990:498:160:320:210:520:220:230:150:160:000:00

10/23/990:488:510:300:220:570:220:240:210:160:000:00

10/24/990:509:280:020:220:570:220:250:160:160:000:00

10/25/990:477:090:460:220:580:230:250:360:160:000:00

10/26/990:463:040:300:200:480:210:220:140:130:000:00

10/27/990:481:540:310:210:460:220:230:160:140:000:00

10/28/990:588:230:420:231:070:280:270:160:180:000:00

10/29/990:586:280:340:230:530:270:270:160:170:000:00

10/30/991:006:480:380:220:560:280:270:160:180:000:00

10/31/991:076:160:380:240:560:300:270:170:170:000:00

11/1/991:136:300:400:250:590:240:270:160:170:000:00

11/2/991:055:280:400:270:570:300:300:190:180:000:00

11/3/990:465:280:420:260:540:250:310:150:193:000:00

11/4/990:460:000:390:290:470:210:290:150:222:380:00

11/5/990:475:190:470:301:010:240:330:190:212:410:00

11/6/990:544:540:430:260:580:260:320:160:212:320:00

11/7/990:565:190:420:280:570:280:310:150:162:410:00

11/8/990:575:160:480:290:570:270:280:190:172:460:00

11/9/991:005:210:500:271:000:290:290:190:202:020:00

11/10/991:025:240:480:281:030:280:300:150:242:060:00

11/11/991:025:090:500:271:030:270:310:160:251:520:00

11/12/991:014:560:520:311:040:270:340:160:221:410:00

11/13/991:055:240:520:311:000:290:330:200:211:400:00

11/14/991:044:570:520:291:010:310:320:170:251:400:00

11/15/991:045:050:530:270:580:300:320:190:231:440:00

11/16/991:025:190:570:310:590:300:340:210:211:390:00

11/17/991:025:130:560:321:010:320:340:240:222:440:00

11/18/991:045:300:570:301:000:320:350:190:252:350:00

11/19/990:475:190:450:270:440:300:280:180:202:250:00

11/20/991:035:210:550:311:020:330:320:230:212:380:00

11/21/991:025:000:560:321:000:340:310:240:212:230:00

11/22/991:004:560:590:310:580:330:370:240:2018:200:00

11/23/991:005:010:550:290:590:330:340:230:222:150:00

11/24/991:015:060:560:300:590:340:340:240:230:000:00

11/25/991:075:160:550:321:000:330:320:200:260:000:00

11/26/991:025:100:560:301:010:370:330:250:240:000:00

11/27/991:045:290:580:281:020:340:330:190:260:000:00

11/28/991:045:080:550:291:000:360:330:210:260:000:00

11/29/991:025:020:570:321:010:390:330:260:230:000:00

11/30/991:065:180:580:310:590:360:330:190:280:000:00

12/1/991:055:180:590:321:000:350:330:190:280:000:00

12/2/991:085:161:000:331:020:340:370:190:280:000:00

12/3/991:105:171:030:341:010:350:350:190:270:000:00

12/4/991:105:471:050:321:020:380:360:280:240:000:00

12/5/991:075:100:590:301:030:360:350:190:260:000:00

12/6/991:075:071:000:291:080:360:360:220:270:000:00

12/7/991:075:271:010:301:040:350:350:200:270:000:00

12/8/991:055:201:090:311:020:380:380:290:260:000:00

12/9/991:085:201:060:321:050:410:380:270:270:000:00

12/10/991:065:200:590:321:050:380:350:250:270:000:00

12/11/991:125:511:050:341:050:360:340:270:010:000:00

12/12/991:075:141:090:311:010:380:390:220:010:000:00

12/13/991:065:101:060:300:590:370:410:250:320:000:00

12/14/990:537:260:520:301:040:370:340:220:230:000:00

12/15/990:565:261:040:321:010:340:350:190:270:000:00

12/16/991:015:291:020:311:010:360:380:260:250:000:00

12/17/991:015:141:000:281:010:370:370:260:240:000:00

12/18/991:096:231:040:301:010:370:390:240:270:000:00

12/19/991:055:191:040:311:010:410:360:260:240:000:00

12/20/991:065:241:020:321:050:410:380:250:240:000:00

12/21/991:105:520:590:301:030:370:360:200:280:000:00

12/22/991:095:481:020:311:060:360:380:210:270:000:00

12/23/990:495:250:500:280:430:340:310:190:230:000:00

12/24/991:035:171:060:301:020:340:390:280:260:000:00

12/25/991:075:511:040:321:040:330:350:200:280:000:00

12/26/991:065:191:040:341:030:350:370:270:250:000:00

12/27/991:045:151:030:341:010:350:370:290:240:000:00

12/28/991:055:141:060:331:010:330:390:270:230:000:00

12/29/991:216:130:550:321:040:400:340:200:230:000:00

12/30/991:085:241:080:291:030:370:400:280:270:000:00

12/31/991:045:271:010:301:020:360:380:270:270:000:00

1/1/001:105:471:050:351:030:330:380:220:280:000:00

1/2/001:065:241:050:341:040:360:370:260:230:000:00

1/4/000:005:051:150:330:580:320:380:290:280:000:00

1/5/000:011:300:000:000:000:000:000:000:010:000:00

1/6/001:368:041:170:351:090:420:390:310:290:000:00

1/7/001:208:171:090:351:100:400:390:220:280:000:00

1/8/001:042:561:120:320:520:310:410:300:290:000:00

1/9/001:135:291:040:321:020:380:390:210:270:000:00

1/10/001:098:001:130:351:070:380:450:340:270:000:00

1/11/001:188:341:080:321:020:400:390:300:280:000:00

1/12/001:197:321:100:301:040:340:400:310:280:000:00

1/13/001:187:101:050:321:020:330:390:200:282:482:12

1/14/000:007:220:580:260:520:280:290:210:221:302:09

1/15/001:433:548:000:431:230:450:540:440:391:502:02

1/16/001:278:001:310:441:300:430:530:330:371:261:57

1/17/001:118:001:080:331:030:380:440:250:261:271:57

1/18/001:354:128:000:381:220:440:500:370:371:512:06

1/19/001:033:291:070:311:040:350:380:300:251:240:48

1/20/001:103:121:070:341:080:340:380:300:321:080:24

1/21/001:123:191:110:330:530:370:380:250:271:070:25

1/22/001:073:451:070:351:080:340:380:220:311:400:33

1/23/001:053:091:040:351:090:330:380:210:301:180:27

1/24/001:073:111:020:331:070:340:360:200:301:120:27

1/25/001:083:181:070:331:070:340:400:230:291:140:27

1/26/001:073:161:040:351:060:340:370:220:291:100:26

1/27/001:053:161:060:361:060:320:390:210:311:190:27

1/28/001:133:171:030:321:060:320:400:230:291:170:27

1/29/001:153:301:070:331:060:360:380:290:291:100:35

1/30/001:083:191:100:361:060:320:380:240:281:180:29

1/31/001:043:141:030:341:060:320:380:210:271:180:30

2/1/000:523:170:510:300:490:310:320:210:010:510:28

2/2/000:003:200:010:280:500:270:000:010:000:490:23

2/3/001:013:121:000:311:020:320:380:210:291:170:22

2/4/001:003:191:040:311:000:330:400:220:291:170:28

2/5/001:053:281:030:321:050:350:350:230:291:190:37

2/6/001:053:171:020:311:070:340:390:220:281:090:30

2/7/001:043:141:020:321:040:340:390:190:271:200:29

2/8/001:073:221:000:361:020:320:430:230:281:200:27

2/9/001:023:190:590:311:060:350:400:200:301:110:27

2/10/001:013:220:590:301:060:340:380:200:311:110:27

2/11/001:033:221:030:321:100:340:400:230:331:090:28

2/12/001:103:351:000:351:080:370:410:260:301:100:38

2/13/001:073:161:000:341:070:350:400:210:291:080:31

2/14/001:003:131:030:341:030:340:390:210:281:170:31

2/15/001:033:231:010:341:040:350:400:240:321:240:29

2/16/001:023:201:030:331:070:340:400:220:311:190:35

2/17/001:053:231:020:321:090:350:420:230:331:270:35

2/18/000:583:231:030:311:040:350:410:210:311:290:36

2/19/001:003:371:040:351:040:330:400:220:321:300:44

2/20/001:033:201:000:331:030:320:400:230:311:280:36

2/21/000:563:131:010:321:100:320:400:270:281:300:35

2/22/001:003:211:040:311:020:320:390:220:311:280:36

2/23/000:593:211:070:311:050:320:390:230:291:290:35

2/24/001:003:251:010:311:030:320:420:220:291:290:36

2/25/001:023:281:000:331:070:330:400:230:291:300:35

2/26/001:053:431:010:301:070:330:420:230:311:310:47

2/27/001:033:231:000:331:050:340:390:220:291:290:35

2/28/001:043:231:020:301:060:320:400:210:301:240:35

2/29/001:053:331:040:321:040:330:380:240:321:220:35

3/1/001:093:481:010:311:040:360:420:260:281:300:33

3/2/001:013:290:590:331:040:340:410:250:321:330:36

3/3/000:583:271:020:311:030:320:390:220:321:310:36

3/4/001:074:041:010:321:050:360:410:250:291:320:40

3/5/001:023:281:030:361:030:340:410:240:291:280:36

3/6/000:570:001:020:331:020:330:400:260:251:290:00

3/7/001:196:401:010:371:050:390:420:270:261:450:39

3/8/001:033:291:000:331:060:370:410:230:291:310:38

3/9/001:033:281:010:351:040:330:420:260:301:320:38

3/10/001:043:311:080:351:120:330:400:230:311:300:39

3/11/001:083:590:580:351:090:360:390:250:321:300:49

3/12/001:043:291:040:341:100:360:370:220:301:290:37

3/13/001:013:211:040:381:070:330:400:220:291:240:37

3/14/001:083:311:020:331:080:330:410:260:291:200:38

3/15/001:103:451:060:341:050:380:400:280:291:210:48

3/16/000:583:451:020:331:050:320:380:270:281:270:50

3/17/001:053:501:050:351:090:360:420:270:291:180:54

3/18/000:014:150:000:000:000:000:010:000:011:020:47

3/19/000:013:280:010:240:000:000:000:000:010:510:37

3/20/000:003:230:000:250:000:000:000:000:010:580:37

3/21/001:014:021:030:311:110:350:420:240:271:290:34

3/22/001:004:400:520:321:080:300:390:190:321:200:45

3/23/001:0010:251:060:361:080:300:410:280:401:170:39

3/24/001:033:530:530:321:020:370:380:240:271:180:45

3/25/001:044:040:560:311:030:360:380:200:341:210:56

3/26/001:013:360:550:321:060:340:400:240:291:120:46

3/27/001:043:500:540:341:050:340:390:240:261:310:44

3/28/001:034:040:560:311:060:360:430:210:291:260:47

3/29/000:593:221:000:331:100:340:430:220:341:281:12

3/30/001:113:441:070:371:130:380:400:280:321:251:06

3/31/001:143:411:040:361:120:400:450:260:311:221:12

4/1/001:134:131:040:371:090:370:430:260:351:161:23

4/2/001:044:371:080:371:080:380:450:220:321:291:03

4/3/001:144:241:110:401:110:360:480:260:331:301:09

4/4/001:033:181:080:361:110:130:480:230:331:251:06

4/5/001:123:371:130:391:100:390:440:260:321:241:15

4/6/001:073:341:100:371:080:370:430:230:341:351:16

4/7/001:083:361:060:401:080:370:420:220:331:261:17

4/8/001:124:111:100:401:070:330:420:250:341:251:30

4/9/001:043:211:060:401:050:350:430:250:311:251:12

4/10/001:023:181:090:401:050:350:460:260:291:231:11

4/11/001:103:521:030:381:110:400:450:210:321:111:26

4/12/001:093:481:120:381:100:360:460:230:321:441:22

SAS ITSV Server Counts

Chart1

36383431318226953640136401

3223335716154763640336403

30453499232051043640436404

3036938920198753640536405

34405591622207963640636406

2642537920147963640736407

34382471219198963640836408

34374421317158963640936409

31384491322216953641036410

343574714181571063641136411

30388441417219843641236412

33402421615189863641336413

33458501021168763641436414

32403431619189863641536415

3140034820179963641636416

31344471517187743641736417

32420421221168843641836418

29392541422169953641936419

363964410181271063642036420

2941640717128853642136421

39400431219148953642236422

36412441220138963642336423

38391521218187743642436424

38436451021159863642636426

384555017202191063642736427

34425461219198863643036430

414015014181981063643136431

3640538819178963643236432

313904911201471053643336433

30420481318188953643436434

32472519231381043643536435

344334714201881043643636436

33451531621198963643736437

28419461218177863643836438

33415461025138833643936439

3843849131919141043644036440

404474815181313953644136441

4944658172422141063644536445

404485014201591063644636446

374515111201710763644736447

334204313221213753644836448

26441408201310863644936449

274164713201511663645036450

18434427141112653645136451

294035111201212663645236452

24376508211212763645336453

28409539191513863645436454

264245012191613653645536455

264104110221211763645636456

243934111181316733645736457

334086616212015753645836458

393855414201714733645936459

405006017192316833646036460

52364616319222017933646136461

416084914211218863646236462

4767355162517171053646336463

5550169192523181083646436464

5670785202622161063646536465

57364668220262216983646636466

4567080202322161085736467

6167481192123161074736468

6268181202323141084036469

6067280202323161083336470

6068184202321161083136471

6068210818222215982536472

66681104202224141081736473

6468096202121159122036474

696841022023211819121936475

716851052123231820121736476

696871042224271820113036477

716911032023261819122736478

66685941623241619112636479

616881011824261719102336480

72687992226391522113536481

686941011824371722124136482

1069743251324936483

68693962124371822105736484

726991052121361723105136485

736841062122391922103836486

73698992223361922113236487

736991021924381722122836488

757011042225391823122636489

746991032125371821122536490

697011012225341921122436491

737001072122371923122036492

7169699222237172391836493

716991042020331822121936494

737021032322362022122036495

766561002123382223122336496

737031022230342223112736497

71702982027342122112636498

697021022327342121102236499

70695981925322022122036500

697031021826322221101936501

727021002327332122121936502

707021062328342022112136503

757021012327362123102436504

68704942227312023102236505

747041002326382123102036506

717041092027352221121736507

17087224269143936508

677001001926341918101736509

67704992124382220122436510

72703962125372122104236511

65703901924352123103636512

66702962124382220114036513

78699982125402219103136514

777011002226282123133536515

797011012225272222114836516

670374142325236517

68700103202324212198036518

71704972225222122127536519

73701992323262121137036520

767031052523302123126836521

737031032422262122125536522

70703972321282122124636523

73700982126251919114436524

67702982023281719134436525

68705991927262021133836526

727051022427302023134136527

777031102428312123132836529

77703110242831212313836530

767061102328302122134336531

717061052026291921135536532

71706962327292223137336533

72706962127292122138336534

767061072426282221138336535

726531012124262222137736536

686541052125252121127936537

69653962327272221128551

66653921926272122117551

69653922424252120127353

72653982325221822126353

716531012425262019116253

686531012225241922136453

1411309201465352414524135104

69656962128242022136154

746561032328272121145655

756561072428302123146355

68654962128242021145655

726481032228292022144955

74632982028272123125052

64646962025271921134753

67656991826272120144356

746541042127271920124459

63654931627191921134060

706541032326282121134159

71654942328282123133959

16547212211131860

16537171221131856

70652991626272221144954

706521012124262121135361

716511052327302223145660

69654942228232321124460

726531012126282219114261

67654962228262321134461

63652892129262522135061

63653972130282521134361

64652842126222222124965

59651822029222423144366

62650932228232420114068

66649942126242418123966

68655932024252422144567

696541022326282423134769

68652972326232721125770

67653911725252720116169

53653881522202323126872

66652921723252521135370

65653961826242521125373

67655952023232321115472

64655851627262221115672

65655951927282420125471

64655831925252420105370

53653812026212519145670

64653902224272718135269

71648992524292522115170

646531012222252521145872

616521022124282622136671

61654922024292422166970

59653972024292220137072

62653872325232121127071

68652912021282321126473

686521042522292623146273

67653982227272523137973

59651891829252218116773

59653982428252123126173

63653942530282221136276

63652842727212020156279

65653882226271919155679

61652902222262121134978

60650912128231521165177

59653922324262019164976

63651842223241620155181

59652842223221819154987

59652842223221819152190

5965284123221819151486

5964984123221819151584

65649972027282523165085

60403932328252420155174

586579223242725231350101

606589622263126221652105

576538025242826231654107

63597942425292722155696

686519721243126231666108

606559521262627171562114

6555010027303028231667247

6961110025303127221666291

6965510028293228241469319

636479328292828231466310

646559226252928191472321

6955110527293129211572226

6464910028293128211569321

7265110430292926241567323

6564910928283025231576328

6665110127263026241574331

496477926271224241562331

626518727282924221563326

646479728283126201456326

656489727303027241545328

6564910529303128191576331

Application

Branch

Comm

Database

Exchange

File/Print

Internet

Notes

TermServer

CallCentre

Branch1

Sheet1

DateApplicationBranchCommDatabaseExchangeFile/PrintInternetNotesTermServerCallCentreBranch1

8/29/993638343131822695

8/31/99322333571615476

9/1/993045349923205104

9/2/99303693892019875

9/3/993440559162220796

9/4/99264253792014796

9/5/993438247121919896

9/6/993437442131715896

9/7/993138449132221695

9/8/9934357471418157106

9/9/993038844141721984

9/10/993340242161518986

9/11/993345850102116876

9/12/993240343161918986

9/13/99314003482017996

9/14/993134447151718774

9/15/993242042122116884

9/16/992939254142216995

9/17/9936396441018127106

9/18/99294164071712885

9/19/993940043121914895

9/20/993641244122013896

9/21/993839152121818774

9/23/993843645102115986

9/24/9938455501720219106

9/27/993442546121919886

9/28/9941401501418198106

9/29/99364053881917896

9/30/9931390491120147105

10/1/993042048131818895

10/2/993247251923138104

10/3/9934433471420188104

10/4/993345153162119896

10/5/992841946121817786

10/6/993341546102513883

10/7/99384384913191914104

10/8/9940447481518131395

10/12/99494465817242214106

10/13/9940448501420159106

10/14/9937451511120171076

10/15/9933420431322121375

10/16/992644140820131086

10/17/9927416471320151166

10/18/991843442714111265

10/19/9929403511120121266

10/20/992437650821121276

10/21/992840953919151386

10/22/9926424501219161365

10/23/9926410411022121176

10/24/9924393411118131673

10/25/9933408661621201575

10/26/9939385541420171473

10/27/9940500601719231683

10/28/9952631922201793

10/29/9941608491421121886

10/30/99476735516251717105

10/31/99555016919252318108

11/1/99567078520262216106

11/2/9957822026221698

11/3/9945670802023221610857

11/4/9961674811921231610747

11/5/9962681812023231410840

11/6/9960672802023231610833

11/7/9960681842023211610831

11/8/9960682108182222159825

11/9/99666811042022241410817

11/10/9964680962021211591220

11/11/996968410220232118191219

11/12/997168510521232318201217

11/13/996968710422242718201130

11/14/997169110320232618191227

11/15/99666859416232416191126

11/16/996168810118242617191023

11/17/99726879922263915221135

11/18/996869410118243717221241

11/19/9910697432513249

11/20/99686939621243718221057

11/21/997269910521213617231051

11/22/997368410621223919221038

11/23/99736989922233619221132

11/24/997369910219243817221228

11/25/997570110422253918231226

11/26/997469910321253718211225

11/27/996970110122253419211224

11/28/997370010721223719231220

11/29/9971696992222371723918

11/30/997169910420203318221219

12/1/997370210323223620221220

12/2/997665610021233822231223

12/3/997370310222303422231127

12/4/99717029820273421221126

12/5/996970210223273421211022

12/6/99706959819253220221220

12/7/996970310218263222211019

12/8/997270210023273321221219

12/9/997070210623283420221121

12/10/997570210123273621231024

12/11/99687049422273120231022

12/12/997470410023263821231020

12/13/997170410920273522211217

12/14/99170872242691439

12/15/996770010019263419181017

12/16/99677049921243822201224

12/17/99727039621253721221042

12/18/99657039019243521231036

12/19/99667029621243822201140

12/20/99786999821254022191031

12/21/997770110022262821231335

12/22/997970110122252722221148

12/23/996703741423252

12/24/99687001032023242121980

12/25/99717049722252221221275

12/26/99737019923232621211370

12/27/997670310525233021231268

12/28/997370310324222621221255

12/29/99707039723212821221246

12/30/99737009821262519191144

12/31/99677029820232817191344

1/1/00687059919272620211338

1/2/007270510224273020231341

1/4/007770311024283121231328

1/5/00777031102428312123138

1/6/007670611023283021221343

1/7/007170610520262919211355

1/8/00717069623272922231373

1/9/00727069621272921221383

1/10/007670610724262822211383

1/11/007265310121242622221377

1/12/006865410521252521211279

1/13/0069653962327272221128551

1/14/0066653921926272122117551

1/15/0069653922424252120127353

1/16/0072653982325221822126353

1/17/00716531012425262019116253

1/18/00686531012225241922136453

1/19/001411309201465352414524135104

1/20/0069656962128242022136154

1/21/00746561032328272121145655

1/22/00756561072428302123146355

1/23/0068654962128242021145655

1/24/00726481032228292022144955

1/25/0074632982028272123125052

1/26/0064646962025271921134753

1/27/0067656991826272120144356

1/28/00746541042127271920124459

1/29/0063654931627191921134060

1/30/00706541032326282121134159

1/31/0071654942328282123133959

2/1/0016547212211131860

2/2/0016537171221131856

2/3/0070652991626272221144954

2/4/00706521012124262121135361

2/5/00716511052327302223145660

2/6/0069654942228232321124460

2/7/00726531012126282219114261

2/8/0067654962228262321134461

2/9/0063652892129262522135061

2/10/0063653972130282521134361

2/11/0064652842126222222124965

2/12/0059651822029222423144366

2/13/0062650932228232420114068

2/14/0066649942126242418123966

2/15/0068655932024252422144567

2/16/00696541022326282423134769

2/17/0068652972326232721125770

2/18/0067653911725252720116169

2/19/0053653881522202323126872

2/20/0066652921723252521135370

2/21/0065653961826242521125373

2/22/0067655952023232321115472

2/23/0064655851627262221115672

2/24/0065655951927282420125471

2/25/0064655831925252420105370

2/26/0053653812026212519145670

2/27/0064653902224272718135269

2/28/0071648992524292522115170

2/29/00646531012222252521145872

3/1/00616521022124282622136671

3/2/0061654922024292422166970

3/3/0059653972024292220137072

3/4/0062653872325232121127071

3/5/0068652912021282321126473

3/6/00686521042522292623146273

3/7/0067653982227272523137973

3/8/0059651891829252218116773

3/9/0059653982428252123126173

3/10/0063653942530282221136276

3/11/0063652842727212020156279

3/12/0065653882226271919155679

3/13/0061652902222262121134978

3/14/0060650912128231521165177

3/15/0059653922324262019164976

3/16/0063651842223241620155181

3/17/0059652842223221819154987

3/18/0059652842223221819152190

3/19/005965284123221819151486

3/20/005964984123221819151584

3/21/0065649972027282523165085

3/22/0060403932328252420155174

3/23/00586579223242725231350101

3/24/00606589622263126221652105

3/25/00576538025242826231654107

3/26/0063597942425292722155696

3/27/00686519721243126231666108

3/28/00606559521262627171562114

3/29/006555010027303028231667247

3/30/006961110025303127221666291

3/31/006965510028293228241469319

4/1/00636479328292828231466310

4/2/00646559226252928191472321

4/3/006955110527293129211572226

4/4/006464910028293128211569321

4/5/007265110430292926241567323

4/6/006564910928283025231576328

4/7/006665110127263026241574331

4/8/00496477926271224241562331

4/9/00626518727282924221563326

4/10/00646479728283126201456326

4/11/00656489727303027241545328

4/12/006564910529303128191576331

Web Reporting System

Web Reporting System

Web Reporting using Excel

Web Reporting using Excel

FutureEncourage application developers to use performance toolsSAN solutionsClustering & Win2000 DatacentreNTSMF on selected desktopsMeasure transactions & response time

Summary & Lessons LearnedAvoid writing your own toolsServer naming conventions importantAdministration required and not trivialData loss from sites inevitableRandomize FTP transmissions

Largest bank by assetsAll bank and head office employeesIT related functionsRetail branch infrastructure

Processing centres mostly mainframe systemsCampus environment in major cities that is on fibre network to connect our building togetherDistricts manage the branches within their domain. Service Delivery do the back office functions for the branchesBusiness Banking centres provide services to corporationsBranches are for retail servicesMigrating to a Windows/NT platform from OS/2Old environments being phased out (converted to NT)Servers in branch and Head Office locationsOld OS/2 servers being migrated to NTDeployed NT throughout the bank

some applications still run in DOS moderapid deployment occurs using Visual Basic but at a cost of hardware resources (I.e. memory)provide client access across the country, when people travelbecause data is centrally stored, most databases replaced by DB2 mainframeMethodology modified as data and tools initially hard to get and costly across all systems (due to the large numbers involved)Wrote own tools for OS/2 data collections and repository; spent a lot of time in development and maintenanceTraditional escalation of problems was reversed as support teams came to branches to fix problems that branch people werent even aware of.Tools had to be researched and bought

From communications gateways to internet & exchangeDifficult to visit locations (geographical)Rapid deployment due to business needs and marketsMost client/server systems are departmental solutions, expect database, to be close to applications and usersBackup, Disaster Recovery, Software Distribution are in early stagesproducing reports is costly and overwhelmingminimize amount of data moved across network for central processingNeed to monitor growth closelyNeed to control growthNeed to architect solution using existing technologies (I.e. Client always wants the latest hardware, but that has major risks for us, as we certify all of our hardware & software)

Separate performance monitoring (realtime) from capacity monitoring (after-the-fact)Only a small subset of data is required for capacity planningNew version of NTSMF will allow sending of subset of data (filtering)Work with business units to ensure enough capacity for 3 years (high costs of upgrading in the field)Technical users are keen to get their information if a simple tool is providedDont produce reports that users might want all of the time (most users lose interest after a while)

The performance agent is part of the NT image (a standard)Different types of servers run at different thresholds (clustering analysis)Each manager is aware of capacity issues for their area of responsibilityControl how file/print growth is occurringMake reporting a self-server function (I.e. through a browser)To understand the capacity of the various components of a system and how they work together in our environment, we benchmark themReal time alerting to operations is done by Tivoli agentNTSMF (by Demand Technology) is our performance/capacity collectorSAS ITSV is our data repositoryWeb reporting is done using IIS/CGI/Perl/Excel solutionTools to provide feedback to desktop user on their memory usageVarious benchmark tools that we use

Simple diagram of Data FlowServer that runs SAS ITSV and IIS web serviceSever that runs IIS FTP serviceWe run two copies of SAS ITSV on the serverNotice how we split the data by database type (to speed up SAS ITSV processing and to manage similar type of servers)Note time of server: Depends on similarity of servers (ITSV pre-scans the SMF files)Branch spends: 2:30 hrs for processing and 1:15 for reductionBranch was split due to limit of 999 servers per database (our server naming convention allows us to separate files by wildcard selection)You have to monitor your elapsed time very carefully (so that all processing finished within a reasonable time)Notice the first large growth, we had to drop a lot of fields to regain some time backSecond improvement came from limiting the amount of time slots allocated to ITSV (for calculating missing time periods)More stats on where growth is occurringBrowser based reporting system which requires no training to useUsers see what standard reports they can run and select a report

Users choose what type of server, its name and how many days to report onThere is a separate report which shows which server is in which databaseUsers can control when their job is run (foreground or background)Users can also schedule their job (the report will only repeat a limited number of times, so their scheduled jobs will expire over time)Excel being used as a viewer within the browser (no extra software to buy or install)7 days are shown here for cpu utilisation and queue lengthThis view opens with an explain screen (Excel tab) that tells user what to doUser can switch to pivot table and select a particular day to drill-down on the dataNO MACROS are used!With pivot tables, you can select a specific date to display (drill-down) on the graphUser has access to the underlying data, so they can analyze it themselves

To control growth, developers need feedback on their usage sooner. NT has good developer tools, which can be intimidating at first, because its rich in the amount of objects. Biggest issue is memory usage in NT and our OS/2 ported programs dont use new WIN32 memory APIs.Due to centralization of servers in the processing centre (we currently have about 300 servers in one location), we need better disk space deployment and management. We use an Enterprise box for our Exchange/FilePrint in one processing centre and its performance is great, but its management is still an issue (as each server owns it disk space)NT still perceived as a gateway or passthru service, we need to provide better availability and scalabilityLab measurements are fine, but dont match the real-world. Need to deploy collectors into key locations to measure usageDifficult to get response time measurements today, but being addressed at CMG and vendors (I.e. ARM, Candle ETEwatch, etc)

Too much maintenance/coding and programming skills hard to findNaming conventions would make it easier to classify types of serversStill requires care-and-feeding of ITSV and NTSMF (maintenance releases)No guarantee that all data will arrive at FTP server, so must expect gaps in data. Issue here is for availability stats and summarizationMust ensure data transmission is spread to reduce bandwidth requirements