Enterprise Server Performance Analysis & Capacity Planning in
54
Transcript of Enterprise Server Performance Analysis & Capacity Planning in
Enterprise Server Performance Analysis & Capacity Planning in High
Volume Batch and On- Line Environments
UNITE Technology ConferenceReno Nevada 22-24 September 2003
Session MCP4064Guy Bonney
MGS, Inc.
! Software Engineering, Product Development & Professional Services firm founded in 1986
! We provide products and services to solve business problems:
" Software Engineering Services" Professional Services
# Management Support Services# Consulting and Technical Services# Application Development Services# Training Services
" Product Development
MGS Areas of Expertise! Principals average over 25 years industry
experience! Junior staff - minimum 5 years experience! 60% of experience is Unisys ClearPath NX/A
Series! Remainder is PC, Windows™, UNIX, C,
Delphi, VB, LAN/WAN...
MGS Software Products
! File Manager for ClearPath MCP/A Series (FMA)
! SightLine™ Performance Analyzer! SightLine Capacity Manager! SightLine Workload Analyzer! HVFAX High Volume Facsimile Delivery
System! Proof of Correctness System (PCS)
Complex Service Delivery Environment
IT Performance/Capacity Issues! Business Issues
! Information Technology Management Issues
! Technical Issues
Business Issues! Change
" Growth$ Evolutionary
• New clients, new products, higher volumes$ Revolutionary
• Buy a new division / product line• Merger• Business exchange / partner collaboration • On Line customer service• On Line payment processing or store front
" Decline$ Lose business (cost reduction)$ Change business focus / direction$ Get bought or merged
Business Growth! Growth in most organizations requires
Information Technology services to change
! Growth is not possible without reliable IT service delivery and good performance
! Even without business growth IT must enhance offerings and offer new services
IT Management Issues! Where is the business going?! What technologies are now available?! What new & improved functionality do we
need to support the business?! What service levels are needed for effective
business function?! What capacity is required to deliver the
services needed?! What budget is available to procure
necessary resources?
Technical Issues! What resources are available for service
delivery?! How do we define the business workloads?! What service levels are we delivering?! How do we measure services and service
levels?! How should we report on performance and
capacity issues?! How do we do performance analysis?! How do we do capacity planning?! What tools are available?
What is Performance?! Meeting Expectations.! Complete Processing a Unit of Work in the
Expected Time." On-line transactions - Response Time" Batch runs - Elapsed time/Deadlines" “On-Line Batch” - Elapsed (Response) time" Delivering the email – Response Time
Monitoring the Critical Path
Message Time Message Time “Service” “Service” MeasuresMeasures
Network Service & Network Service & Resource MeasuresResource Measures
Server & Server & Application Application MeasuresMeasures
What is a Workload?
! Aggregate set of units of work to be processed
! Everything that must be accomplished to deliver a set of results to a requester
Workload Example
4 Jul 1999 - 3 Sep 2000
COMS Tran/Sec for All
Jul4 1999 Sep12 1999 Nov21 1999 Jan30 2000 Apr9 2000 Jun18 2000 Aug27 20000
20
40
60
80
100
120
What is Capacity?
! Simply the capability to process defined amounts of work in a specified period of time
! Measured in terms of units of work “processed”: MIPS, RPM, TPC-C transactions, MFLOPS, etc.
Performance vs. Capacity
! Capacity is the ability to process work
! Performance is the measure of workload processing effectiveness
! Performance and Capacity must be considered together as they are related
Performance Analysis Objectives! Recognize performance anomalies (response
time or service time issues)
! Detect bottlenecks (measure queuing and busy time)
! Determine cause of inadequate performance
! AVAILABILITY IS ASSUMED!
On-Line Response TimeTime Series Analysis
22 Aug 2000 14:00 - 15:00 A19: COMS Response Avg
PGU0020 PGU0034B PGU0034 PGU0015 PGU0043B PGU0043 PGU0060 PGU0060B
14:00 14:05 14:10 14:15 14:20 14:25 14:30 14:35 14:40 14:45 14:50 14:55 15:000.00.10.20.30.40.50.60.70.80.91.0
Batch Cycle AnalysisBatch Cycle AnalysisTaskName Mix# CPUSecs IOSecs Memory IOCount PrintLns Elapsed StartTime StopTime
(SFB)SFB/REPTAL0002 2004 1.6 1.3 9949 60 8 7.9 4/30/97 3:44 4/30/97 3:44
(SFB)SFB/REPACL0024 2018 111.3 93.5 15531 162324 4683 244.5 4/30/97 3:44 4/30/97 3:48
(SFB)SFB/REPACA0704 2152 56.1 25.6 16515 92726 85 71.2 4/30/97 3:52 4/30/97 3:53
(SFB)SFB/REPACA0706 2190 32.6 24.6 18091 135537 4835 52.6 4/30/97 3:53 4/30/97 3:54
(SFB)SFB/REPACA0760 2213 739.7 1916.3 19021 177340 156 2298.2 4/30/97 3:54 4/30/97 4:33
(SFB)SFB/REPAHA0760 2898 5.9 17.1 17566 1546 90 30 4/30/97 4:33 4/30/97 4:34
(SFB)SFB/REPEEA0002 2935 630.5 1274.7 21858 217908 744 2295.8 4/30/97 4:34 4/30/97 5:12
(SFB)SFB/REPAHL0033 3632 4.2 11.5 16134 3535 1120 26.2 4/30/97 5:13 4/30/97 5:13
(SFB)SFB/REPAHL0010 3648 26.8 92.3 14795 12989 1211 188.8 4/30/97 5:14 4/30/97 5:17
(SFB)SFB/REPCHA0612 3752 250 477.5 21567 53228 2057 921.4 4/30/97 5:19 4/30/97 5:35
(SFB)SFB/REPCCA0612 4032 1245 1786.9 22968 206670 9313 2696.4 4/30/97 5:35 4/30/97 6:20
TOTALS 3103.7 Secs. 8833 Secs.
2.45 Hours
Deadline Graphic AnalysisDeadline Graphic AnalysisBATCH CYCLE START AND STOP TIMES
0:002:004:006:008:00
10:0012:0014:0016:0018:0020:0022:00
0:00
30-Jun 10-Jul 20-Jul 30-Jul
BOJ Time EOJ Time
Batch Cycle Trend AnalysisBatch Cycle Trend Analysis
E LA P S E D TIM E F O R B A TC H C YC LE S
0:001:002:003:004:005:006:007:008:009:00
10:0011:0012:0013:0014:0015:0016:0017:0018:0019:0020:0021:0022:0023:00
0:00
7 /1 /97 7 /6 /97 7 /11 /97 7 /16 /97 7 /21 /97 7 /26 /97 7 /31 /97
E lapsed T im e
Program Run Exception AnalysisProgram Run Exception Analysis
Report Date: 8/11/97 MC0181 Exception Report for July 1997 Processing (30% > Avg. Elapsed Time) Page 1
Name Task Number CPU Secs IO Secs ReadyQ Secs Elapsed Secs Disk IO Cnt Tape IO Cnt Print Lines BOJ Date BOJ Time EOJ Date EOJ Time
MC0181 9461 5,058 902 1,152 13,893 1,065,431 4,113,762 13 7/2/97 0:56:44 7/2/97 4:48:17MC0181 6502 5,287 854 832 16,002 987,808 4,092,475 13 7/3/97 0:36:29 7/3/97 5:03:11MC0181 2565 5,010 1,215 184 15,796 963,257 4,081,849 13 7/5/97 22:07:06 7/6/97 2:30:21MC0181 4232 5,367 115 278 13,571 997,660 8 13 7/29/97 22:54:21 7/30/97 2:40:33
July 97 AVG = 10,258
Analyze Components! Review utilization, queuing and throughput
rates for key components:" CPU" Memory" I/O Subsystems
CPU Analysis26 Jun 2000 00:00 - 3 Jul 00:00
CPU Total Pct
Jun26 Jun27 Jun28 Jun29 Jun30 Jul1 Jul2 Jul30
102030405060708090
100
Workload CPU Distribution
21 Aug 2000 20:00 - 22 Aug 20:00
MCP Critical Path Nightly Batch Online Processing Adhoc Batch Processing
Aug21 20:00 Aug22 00:00 Aug22 04:00 Aug22 08:00 Aug22 12:00 Aug22 16:00 Aug22 20:000
10
20
30
40
50
60
70
80
90
Memory Analysis14 Aug 2000 08:00 - 20 Aug 08:00 HR: Work Mem Pct
0-Compiler 0-Database 0-Library 0-MCS 0-Other 0-Remote 0-ViewPoint 0-WFL
Aug14 08:00 Aug15 08:00 Aug16 08:00 Aug17 08:00 Aug18 08:00 Aug19 08:00 Aug20 08:000
5
10
15
20
25
30
35
40
45
I/O Anomaly
21 Aug 2000 20:00 - 22 Aug 20:03
Fam Secs/IO for BCDAPPL 0.113 Fam Secs/IO for BCDCIS 0.113 Fam Secs/IO for CPCURRPK 0.113 Fam Secs/IO for CPMASTPK 0.113 Fam Secs/IO for MERCHPK 0.113 Fam Secs/IO for PCISPK 0.113 IOP KB/Sec 93700.000
Scale0.113
0.113
0.113
0.113
0.113
0.113
93700.000
Aug21 20:00 Aug22 00:00 Aug22 04:00 Aug22 08:00 Aug22 12:00 Aug22 16:00 Aug22 20:000.00
0.02
0.04
0.06
0.08
0.10
I/O Block Sizes
21 Aug 2000 20:00 - 22 Aug 20:03 A19: Fam KB/Block
BCDAPPL BCDCIS CPCURRPK CPMASTPK MERCHPK PCISPK
Aug21 20:00 Aug22 00:00 Aug22 04:00 Aug22 08:00 Aug22 12:00 Aug22 16:00 Aug22 20:000
50
100
150
200
250
300
350
Disk I/O Transfer Rate
21 Aug 2000 20:00 - 22 Aug 20:03 A19: Fam KB/Sec
BCDAPPL BCDCIS CPCURRPK CPMASTPK MERCHPK PCISPK
Aug21 20:00 Aug22 00:00 Aug22 04:00 Aug22 08:00 Aug22 12:00 Aug22 16:00 Aug22 20:000
50001000015000200002500030000350004000045000
Disk Service by Type
0.000
0.005
0.010
0.015
0.020
0.025
EMC 3700 EMC 3930 EMC 5430 EMC 5500 PermaCache SSD USR3000
Capacity Planning Objectives! Determine resources needed to support
projected workload
! Verify desired performance / service levels can be met
! Define explicit list of questions to answer to avoid wasted effort
Capacity Planning Levels! Simple Projection
" Make Rule Of Thumb (ROT) assumptions for CPU, I/O and memory loading to stay within “safe bounds”
" Use operational analysis techniques to estimate" Must understand component limits" Need monitor and analysis tools to validate
! Model Projection" Define processing types to be analyzed" Define time periods for modeling
On Line Workload GrowthTransactions/Second
0102030405060708090
4/6/
1997
7/6/
1997
10/6
/199
7
1/6/
1998
4/6/
1998
7/6/
1998
10/6
/199
8
1/6/
1999
4/6/
1999
7/6/
1999
10/6
/199
9
1/6/
2000
4/6/
2000
7/6/
2000
Tran/Sec
Expon.(Tran/Sec)
Batch Workload GrowthActive Accounts vs Transaction Volume
0
10,000,000
20,000,000
30,000,000
40,000,000
50,000,000
Nov-97
Feb-98
May-98
Aug-98
Nov-98
Feb-99
May-99
Aug-99
Nov-99
Feb-00
May-00
Aug-00
Nov-00
Activ
e Ac
coun
ts
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
Tran
sact
ions
Active Accts Daily Trans Avg. Trans - Peak Days
CPU Usage vs Transactions
Weekly Transaction Rate
10 May 92 - 27 Feb 94 Average CPU Busy Percent
CPU BUSY Percent - Function of Transaction Rate
WK1: COMS Tran/Sec for All
0
10
20
30
40
50
60
70
80
90
100
0 1 2 3 4 5 6 7
CPU Workload Capacity Projection
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
140.0%
160.0%
A18
-632
CP
U U
SA
GE
1998 1999 2000 2001 2002 2003
Month - Year
APPLICATION CPU REQUIREMENT PROJECTIONSIMPLE GROWTH
SALES OPERATIONS LAWSON PURCHASING CASH-N-CARRYSUPPORT WAREHOUSE MISC. PROD ORDER ENTRY ADMININVENTORY CTRL PROMOTION DEVELOPMENT MCP
Z`
CPU Capacity ProjectionCPU REQUIREMENT PROJECTION
25% LATENT DEMAND - Optimistic Growth
0%
50%
100%
150%
200%
250%
1998 1999 2000 2001 2002 2003Month - Year
CPU
USAG
E
CPU SERVICE LIMIT A18-632 USAGE NX5802-72 USAGENX4822-73 USAGE NX5802-73 USAGE NX5802-74 USAGE
Capacity Models! Characterize the workloads (CPU, memory,
I/O, Delays)! Define the hardware configuration! Execute the model using current measured
data! Validate against actual measurement! Change the model workload levels – rerun! Change the model parameters (HW/SW
spec) and rerun! Analyze and report results
A Workload ProfileMC0150 (Run 1) Transaction Profile8 Copies MC0150 Running Concurrently
0.00000
0.00200
0.00400
0.00600
0.00800
0.01000
0.01200
Resource Component
Res
ourc
e Ti
me
per
Tran
sact
ion
Other Time 0.00115
IO Wait Time 0.00248
I/O Time/ Transaction 0.00323
ReadyQ per Transaction 0.00067
CPU Per Transaction 0.00280
1
Linear Projection from Model
0:00
1:00
2:00
3:00
4:00
5:00
6:00
5,574,863 6,968,579 8,362,295 9,756,010 11,149,726 12,543,442 13,937,158Total Transaction Growth
MC0150 (Run 1) Transaction Growth with Linear Projections(Elapsed Time)
CPU CPU Wait I/O IO Wait Other Time
Batch Model Results
0:00:00
2:00:00
4:00:00
6:00:00
8:00:00
10:00:00
12:00:00
14:00:00
16:00:00
18:00:00
20:00:00
22:00:00
24:00:00
Hour
s to
Com
plet
ion
28,8
00,0
00
36,0
00,0
00
43,2
00,0
00
50,4
00,0
00
57,6
00,0
00
28,8
00,0
00
36,0
00,0
00
43,2
00,0
00
50,4
00,0
00
57,6
00,0
00
28,8
00,0
00
36,0
00,0
00
43,2
00,0
00
50,4
00,0
00
57,6
00,0
00
0% 25% 50% 75% 100% 0% 25% 50% 75% 100% 0% 25% 50% 75% 100%
NX5822-710 NX5822-78 NX6826-PL10
Combined Simulation Modeling Projection for Critical Path Programs 8:51pm September 25th - 5:55am September 26, 2000
(1 Hour and 21 Minutes of Delay between Critical Path Programs)
Elapsed Time - MC0150 Elapsed Time - MC0182 Elapsed Time - MC0150-2 Elapsed Time - MC0338
Configuration Growth Scenario Total Accounts
ComponentData
Batch Cycle Table
CPU Secs ReadyQ Secs Other Delay Elapse IO SecPre-Batch Prep 5100 1083 0 11757
Critical Nightly Batch 10563 2425 614 16993Post Nightly Batch 17920 1777 2928 26556
Totals 33583 5286 3541 55305
Batch Cycle ModelEnd of Month Batch Cycle Projections
(projections include EM C 8350 IO upgrade)
0
5
10
15
20
25
30
Curre
nt N
X482
2-22
NX-68
32-P
L5CS
S780
17-2
8CSS
7801
7-39
CSS7
8017
-52
NX48x
2-32
NX48x
2-42
NX48
x2-5
2
Hours
Elapse IO HoursOther Delay HoursReadyQ HoursCPU Hours
On-Line Workload ProjectionAverage Weekly Transaction Counts vs. Total Accounts
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
1999 2000 2001 2002 2003 2004 2005 2006
Weekly COMSTransaction CountTotal Accounts
Workload Projection
Consolidated Workloads Peak Composite (based on NX6832-PL5):
Dec-02 Dec-03 Dec-04 Dec-05 Dec-06 Dec-07Sys-1 User Growth Rate 20% 28% 28% 11% 11% 11%Sys-2 User Growth Rate 20% 10% 10% 10% 10% 10%Sys-3 User Growth Rate 20% 15% 15% 11% 11% 11%
Aug-02 Dec-02 Dec-03 Dec-04 Dec-05 Dec-06 Dec-07CPU MCP % 5.0 5.0 5.0 5.0 5.0 5.0 5.0Sys-1 CPU User % 41.0 49.2 63.0 80.6 89.5 99.3 110.2Sys-2 CPU User % 12.3 14.8 16.2 17.9 19.6 21.6 23.8Sys-3 CPU User % 11.5 13.8 15.9 18.3 20.3 22.5 25.0
Total 69.8 82.8 100.1 121.7 134.4 148.4 164.0
RPM Growth ProjectionRPM Growth Projections
0
1000
2000
3000
4000
5000
6000
Aug-02
Feb-03
Aug-03
Feb-04
Aug-04
Feb-05
Aug-05
Feb-06
Aug-06
Feb-07
Aug-07
RPM RPMRequirements
RPM vs Capacity
RPM Requirements vs. System Capacity
0
1000
2000
3000
4000
5000
6000
Aug-02
Feb-03Aug-0
3Feb-04Aug-0
4Feb-05Aug-0
5Feb-06Aug-0
6Feb-07Aug-0
7
RPM
RPMRequirements
CSS78017-39Capacity
NX6832-PL5Capacity
NX4822-22Capacity
RPM Capacity Plan
RPM Requirements vs. Proposed System Capacity
010002000300040005000600070008000
Aug-02
Feb-03Aug-0
3Feb-04Aug-0
4Feb-05Aug-0
5Feb-06Aug-0
6Feb-07Aug-0
7
RPM
RPMRequirementsCSS78017-70CapacityCSS78017-52CapacityCSS78017-39CapacityCSS78017-28Capacity
IO CapacityIO Bandpass Growth Considerations for an NX6836-PLA
Actual Capital One and Theoretical I/O LevelsTheoretical Capacity I/O Projections Based on Actual IO Blocksize and Estimated I/O Bandpass
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
Processing Window
I/Os
per s
econ
d
NX6836-PLA I/O BandPass Million Bytes/Sec (Est) 101,351,250 102,945,107 103,762,817 111,834,557 108,997,543
Actual Avg IO Block in Bytes 11,700 17,100 14,700 18,900 17,800
Theoretical I/Os per Second 8,663 6,020 7,059 5,917 6,123
Actual Capital One IOs per Second Observed 5,515 3,947 4,695 4,447 4,425
Estimated % IORate Grow th over Observed Rates 57% 53% 50% 33% 38%
GUI MC0150-1 MC0182 MC0102 MC0338
The Resulting Plan! Install Libra Model 180 at capacity level
CSS78017-39 offering 3800 RPM useable Now.
! Purchase Capacity-on-Demand increments as needed in 2004
! Upgrade to CSS78017-52 (5000 RPM useable) in 2005 if growth projections hold into mid 2004.
In Conclusion
! IT Service Delivery has become increasingly complex with the distribution of processing to multiple networked platforms.
! Fortunately, tools to monitor and manage performance and service delivery of the entire IT infrastructure in real time are now available.
! Integrated support for capacity projection and modeling is increasingly important to avoid service level degradation
Questions?
Thank you for your attention.
Are there any questions?
DownLoad Available! This presentation, complete with the
speakers notes and annotations will be available on the MGS, Inc web site (www.mgsinc.com) for anonymous download by September 30, 2003.
Contact Information! MGS, Inc. is the software engineering
organization that builds the ViewPoint and SightLine products for A Series, ClearPath and ClearPath Plus (MCP).
! MGS is a distributor of the SightLine products built by FORTEL Inc.
! Contact Guy BonneyMGS, Inc.10901 Trade RoadRichmond, VA 23236(804) [email protected]
