CPE 619: Modeling and Analysis of Computer and Communications Systems Aleksandar Milenković The...
-
Upload
giles-harris -
Category
Documents
-
view
212 -
download
0
Transcript of CPE 619: Modeling and Analysis of Computer and Communications Systems Aleksandar Milenković The...
CPE 619: Modeling and Analysis of Computer and Communications
Systems
Aleksandar Milenković
The LaCASA Laboratory
Electrical and Computer Engineering Department
The University of Alabama in Huntsville
http://www.ece.uah.edu/~milenka
http://www.ece.uah.edu/~lacasa
2
Topics
Background About the Course
3
Professor Background
Dr. Aleksandar Milenković Research interests
LaCASA laboratory: www.ece.uah.edu/~lacasa Computer architecture: hardware/software structures for secure,
cost-effective, and performance-effective computation and communication
Performance analysis and evaluation Low-power deeply embedded systems (sensor networks)
Teaching interests CPE 631: Advanced Computer Systems Architecture CPE 619: Modeling and Analysis of Computer and
Communication Systems CPE 527: VLSI Design CPE 323: Introduction to Embedded Computer Systems
4
Goals of This Course
Comprehensive course on performance analysis Includes measurement, statistical modeling,
experimental design, simulation, and queuing theory How to avoid common mistakes
in performance analysis Graduate course: (Advanced Topics)
Lot of independent reading and writing Project/Survey paper (Research techniques)
5
Syllabus
Course Web page: http://www.ece.uah.edu/~milenka/cpe619-08S
Office hours: MW 2:15 PM – 3:15 PM Email: milenka at ece.uah.edu
6
Text Books
Required Raj Jain.
The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, John Wiley and Sons, Inc., New York, NY, 1991. ISBN:0471503363
Other Edward D. Lazowska, John Zahorjan, G. Scott Graham, and
Kenneth C. Sevcik. Computer System Analysis Using Queueing Network Models. NOTE: Available for free on the WEB at http://www.cs.washington.edu/homes/lazowska/qsp/
David J. Lilja. Measuring Computer Performance: A Practitioner's Guide, Cambridge University Press, New York, NY, 2000.
7
Objectives: What You Will Learn
Specifying performance requirements Evaluating design alternatives Comparing two or more systems Determining the optimal value of a parameter (system tuning) Finding the performance bottleneck (bottleneck identification) Characterizing the load on the system
(workload characterization) Determining the number and sizes of components
(capacity planning) Predicting the performance at future loads (forecasting)
8
Basic Terms
System: Any collection of hardware, software, and firmware
Metrics: Criteria used to evaluate the performance of the system components
Workloads: The requests made by the users of the system
9
Main Parts of the Course
Part I: An Overview of Performance Evaluation Part II: Measurement Techniques and Tools Part III: Probability Theory and Statistics Part IV: Experimental Design and Analysis Part V: Simulation Part VI: Queuing Theory
10
Part I: An Overview of Performance Evaluation
Introduction Common Mistakes and How To Avoid Them Selection of Techniques and Metrics
11
Part II: Measurement Techniques and Tools
Types of Workloads Popular Benchmarks The Art of Workload Selection Workload Characterization Techniques Monitors Accounting Logs Monitoring Distributed Systems Load Drivers Capacity Planning The Art of Data Presentation Ratio Games
12
Part III: Probability Theory and Statistics
Probability and Statistics Concepts Four Important Distributions Summarizing Measured Data By a Single Number Summarizing The Variability Of Measured Data Graphical Methods to Determine Distributions of Measured
Data Sample Statistics Confidence Interval Comparing Two Alternatives Measures of Relationship Simple Linear Regression Models Multiple Linear Regression Models Other Regression Models
13
Part IV: Experimental Design and Analysis
Introduction to Experimental Design 2k Factorial Designs 2kr Factorial Designs with Replications 2k-p Fractional Factorial Designs One Factor Experiments Two Factors Full Factorial Design without
Replications Two Factors Full Factorial Design with Replications General Full Factorial Designs With k Factors
14
Part V: Simulation
Introduction to Simulation Types of Simulations Model Verification and Validation Analysis of Simulation Results Random-Number Generation Testing Random-Number Generators Random-Variate Generation Commonly Used Distributions
15
Part VI: Queuing Theory
Introduction to Queueing Theory Analysis of A Single Queue Queuing Networks Operational Laws Mean Value Analysis and Related Techniques Convolution Algorithm Advanced Techniques
16
Grading Policy
Prerequisites MA 585 or EE 500 (soft requirement) General knowledge about computer systems Interest in performance evaluation
Grading Homeworks 20% Midterm Exam 25% Final Exam 25% Project 25% Class participation 5%
17
More on Projects
Goal: Provide an insight (or information) not obvious before the project
Two project types A survey paper on a performance topic A real case study on performance of a system
you are already working on
18
Other Course Policies
See the course web site
Introduction
20
Outline
Objectives The Art Common Mistakes Systematic Approach Case Study
21
Objectives (1 of 6)
Select appropriate evaluation techniques, performance metrics and workloads for a system
Techniques: measurement, simulation, analytic modeling Metrics: criteria to study performance (ex: response time) Workloads: requests by users/applications to the system
Example: What performance metrics should you use for the following systems?
a) Two disk drives b) Two transactions processing systems c) Two packet retransmission algorithms
22
Objectives (2 of 6)
Conduct performance measurements correctly Need two tools
Load generator – a tool to load the system Monitor – a tool to measure the results
Example: Which type of monitor (software and hardware) would be more suitable for measuring each of the following quantities? a) Number of instructions executed by a processor b) Degree of multiprogramming on a timesharing
system c) Response time of packets on a network
23
Objectives (3 of 6)
Use proper statistical techniques to compare several alternatives Find the best among a number of alternatives One run of workload often not sufficient
Many non-deterministic computer events that effect performance
Comparing average of several runs may also not lead to correct results Especially if variance is high
Example: Packets lost on a link. Which link is better?File Size Link A Link B
1000 5 101200 7 31300 3 050 0 1
24
Objectives (4 of 6)
Design measurement and simulation experiments to provide the most information with the least effort Often many factors that affect performance.
Separate out the effects that individual factors. Example: The performance of a system depends
upon three factors: A) garbage collection technique: G1, G2, or none B) type of workload: editing, compiling, AI C) type of CPU: P2, P4, Sparc
How many experiments are needed? How can the performance of each factor be estimated?
25
Objectives (5 of 6)
Perform simulations correctly Select correct language, seeds for random numbers,
length of simulation run, and analysis Before all of that, may need to validate simulator
Example: To compare the performance of two cache replacement algorithms: A) What type of simulation model should be used? B) How long should the simulation be run? C) What can be done to get the same accuracy with a
shorter run? D) How can one decide if the random-number
generator in the simulation is a good generator?
26
Objectives (6 of 6)
Use simple queuing models to analyze the performance of systems
Queuing models are commonly used for analytical modeling of computer systems
Often can model computer systems by service rate and arrival rate of load
Multiple servers Multiple queues
Example: The average response time of a database system is 3 seconds. During a 1-minute observation interval, the idle time on the system was 10 seconds. Using a queuing model for the system, determine the following:
System utilization, average service time per query, the number of queries completed during observation, average number of jobs in the system, …
27
Outline
Objectives The Art Common Mistakes Systematic Approach Case Study
28
The Art of Performance Evaluation
Evaluation cannot be produced mechanically Requires intimate knowledge of system Careful selection of methodology, workload, tools
No one correct answer as two performance analysts may choose different metrics or workloads
Like art, there are techniques to learn how to use them when to apply them
29
Example: Comparing Two Systems
Two systems, two workloads, measure transactions per second
Which is better?
System Workload 1 Workload 2
A 20 10
B 10 20
30
Example: Comparing Two Systems
Two systems, two workloads, measure transactions per second
They are equally good! … but is A better than B?
System Workload 1 Workload 2 Average
A 20 10 15
B 10 20 15
31
The Ratio Game
Take system B as the base
A is better! … but is B better than A?
System Workload 1 Workload 2 Average
A 2 0.5 1.25
B 1 1 1
32
The Ratio Game
Take system A as the base
B is better!?
System Workload 1 Workload 2 Average
A 1 1 1
B 0.5 2 1.25
33
Outline
Objectives The Art Common Mistakes Systematic Approach Case Study
34
Common Mistakes (1-4)
1. Undefined Goals (Don’t shoot and then draw target) There is no such thing as a general model Describe goals and then design experiments
2. Biased Goals (Performance analysis is like a jury) Don’t show YOUR system better than HERS
3. Unsystematic Approach Arbitrary selection of system parameters, factors, metrics, …
will lead to inaccurate conclusions 4. Analysis without Understanding (“A problem well-stated is
half solved”) Don’t rush to modeling before defining a problem
35
Common Mistakes (5-8)
5. Incorrect Performance Metrics E.g., MIPS
6. Unrepresentative Workload Wrong workload will lead to inaccurate conclusions
7. Wrong Evaluation Technique (Don’t have a hammer and see everything as a nail) Use most appropriate: model, simulation,
measurement 8. Overlooking Important Parameters
Start from a complete list of system and workload parameters that affect the performance
36
Common Mistakes (9-12)
9. Ignoring Significant Factors Parameters that are varied are called factors; others are fixed Identify parameters that make significant impact on performance
when varied 10. Inappropriate Experimental Design
Relates to the number of measurement or simulation experiments to be conducted
11. Inappropriate Level of Detail Can have too much! Ex: modeling disk Can have too little! Ex: analytic model for congested router
12. No Analysis Having a measurement expert is desirable but not enough Expertise in analyzing results is crucial
37
Common Mistakes (13-16)
13. Erroneous Analysis E.g., take averages on too short simulations
14. No Sensitivity Analysis Analysis is evidence and not fact Need to determine how sensitive results are to settings
15. Ignoring Errors in Input Often parameters of interest cannot be measured;
Instead, they are estimated using other variables Adjust the level of confidence on the model output
16. Improper Treatment of Outliers Outliers are values that are too high or too low
compared to a majority of values If possible in real systems or workloads, do not ignore them
38
Common Mistakes (17-20)
17. Assuming No Change in the Future Workload may change in the future
18. Ignoring Variability If variability is high, the mean performance alone
may be misleading 19. Too Complex Analysis
A simpler and easier to explain analysis should be preferred
20. Improper Presentation of Results It is not the number of graphs,
but the number of graphs that help make decisions
39
Common Mistakes (21-22)
21. Ignoring Social Aspects Writing and speaking are social skills
22. Omitting Assumptions and Limitations E.g.: may assume most traffic TCP, whereas some
links may have significant UDP traffic May lead to applying results
where assumptions do not hold
40
Checklist for Avoiding Common Mistakes in Performance Evaluation
1. Is the system correctly defined and the goals are clearly stated?2. Are the goals stated in an unbiased manner?3. Have all the steps of the analysis followed systematically?4. Is the problem clearly understood before analyzing it?5. Are the performance metrics relevant for this problem?6. Is the workload correct for this problem?7. Is the evaluation technique appropriate?8. Is the list of parameters that affect performance complete?9. Have all parameters that affect performance been chosen as factors to be varied?10. Is the experimental design efficient in terms of time and results?11. Is the level of detail proper?12. Is the measured data presented with analysis and interpretation?13. Is the analysis statistically correct?14. Has the sensitivity analysis been done?15. Would errors in the input cause an insignificant change in the results?16. Have the outliers in the input or the output been treated properly?17. Have the future changes in the system and workload been modeled?18. Has the variance of input been taken into account?19. Has the variance of the results been analyzed?20. Is the analysis easy to explain?21. Is the presentation style suitable for its audience?22. Have the results been presented graphically as much as possible?23. Are the assumptions and limitations of the analysis clearly documented?
41
Outline
Objectives The Art Common Mistakes Systematic Approach Case Study
42
A Systematic Approach
1. State goals and define boundaries
2. List services and outcomes
3. Select performance metrics
4. List system and workload parameters
5. Select factors and values
6. Select evaluation techniques
7. Select workload
8. Design experiments
9. Analyze and interpret the data
10. Present the results. Repeat.
43
State Goals and Define Boundaries
Just “measuring performance” or “seeing how it works” is too broad E.g.: goal is to decide which ISP
provides better throughput Definition of system may depend upon goals
E.g.: if measuring CPU instruction speed, system may include CPU + cache
E.g.: if measuring response time, system may include CPU + memory + … + OS + user workload
44
List Services and Outcomes
List services provided by the system E.g., a computer network allows users
to send packets to specified destinations E.g., a database system responds to queries E.g., a processor performs a number of tasks
A user request for any of these services results in a number of possible outcomes (desirable or not) E.g., a database system may answer correctly,
incorrectly (due to inconsistent updates), or not at all (due to deadlocks)
45
Select Metrics
Criteria to compare performance In general, related to speed, accuracy and/or
availability of system services E.g.: network performance
Speed: throughput and delay Accuracy: error rate Availability: data packets sent do arrive
E.g.: processor performance Speed: time to execute instructions
46
List Parameters
List all parameters that affect performance System parameters (hardware and software)
E.g.: CPU type, OS type, … Workload parameters
E.g.: Number of users, type of requests List may not be initially complete,
so have a working list and let grow as progress
47
Select Factors to Study
Divide parameters into those that are to be studied and those that are not E.g.: may vary CPU type but fix OS type E.g.: may fix packet size but vary
number of connections Select appropriate levels for each factor
Want typical and ones with potentially high impact For workload often smaller (1/2 or 1/10th) and
larger (2x or 10x) range Start small or number can quickly overcome
available resources!
48
Select Evaluation Technique
Depends upon time, resources, and desired level of accuracy
Analytic modeling Quick, less accurate
Simulation Medium effort, medium accuracy
Measurement Typical most effort, most accurate
Note, above are all typical but can be reversed in some cases!
49
Select Workload
Set of service requests to system Depends upon measurement technique
Analytic model may have probability of various requests
Simulation may have trace of requests from real system
Measurement may have scripts impose transactions Should be representative of real life
50
Design Experiments
Want to maximize results with minimal effort Phase 1:
Many factors, few levels See which factors matter
Phase 2: Few factors, more levels See where the range of impact for the factors is
51
Analyze and Interpret Data
Compare alternatives Take into account variability of results
Statistical techniques Interpret results
The analysis does not provide a conclusion Different analysts may come to different conclusions
52
Present Results
Make it easily understood Graphs Disseminate (entire methodology!)
"The job of a scientist is not merely to see: it is to see, understand, and communicate. Leave out any of these phases, and you're not doing science. If
you don't see, but you do understand and communicate, you're a prophet, not a scientist. If
you don't understand, but you do see and communicate, you're a reporter, not a scientist. If
you don't communicate, but you do see and understand, you're a mystic, not a scientist."
53
Outline
Objectives The Art Common Mistakes Systematic Approach Case Study
54
Case Study
Consider remote pipes (rpipe) versus remote procedure calls (rpc) rpc is like procedure call but procedure is handled on
remote server Client caller blocks until return
rpipe is like pipe but server gets output on remote machine
Client process can continue, non-blocking Results are returned asynchronously
Goal: study the performance of applications using rpipes to similar applications using rpcs
55
System Definition
Client and Server and Network Key component is “channel”, either a rpipe or an rpc
Only the subset of the client and server that handle channel are part of the system
Client Network Server
- Try to minimize effect of components outside system
56
Services
There are a variety of services that can happen over a rpipe or rpc
Choose data transfer as a common one, with data being a typical result of most client-server interactions
Classify amount of data as either large or small Thus, two services:
Small data transfer Large data transfer
57
Metrics
Limit metrics to correct operation only (no failure or errors)
Study service rate and resources consumed Performance metrics
A) elapsed time per call B) maximum call rate per unit time C) Local CPU time per call D) Remote CPU time per call E) Number of bytes sent per call
58
Parameters
Speed of CPUs Local Remote
Network Speed Reliability (retrans)
Operating system overhead For interfacing with
channels For interfacing with network
Time between calls Number and sizes
of parameters of results
Type of channel rpc Rpipe
Other loads On CPUs On network
System Workload
59
Key Factors
A) Type of channel rpipe or rpc
B) Speed of network Choose short (LAN) and across country (WAN)
C) Size of parameters Small or larger
D) Number of calls 11 values: 8, 16, 32 …1024
E) All other parameters are fixed (Note, try to run during “light” network load)
60
Evaluation Technique
Since there are prototypes, use measurement Use analytic modeling based on measured data
for values outside the scope of the experiments conducted
61
Workload
Synthetic program generated specified channel requests
Will also monitor resources consumed and log results Use “null” channel requests to get baseline resources
consumed by logging Heisenberg uncertainty principle in physics:
“the measurement of position necessarily disturbs a particle's momentum, and vice versa—i.e., that the uncertainty principle is a manifestation of the observer effect”
62
Experimental Design
Full factorial (all possible combinations of factors) 2 channels, 2 network speeds, 2 sizes, 11 numbers
of calls 2 x 2 x 2 x 11 = 88 experiments
63
Data Analysis
Analysis of variance will be used to quantify the first three factors Are they different?
Regression will be used to quantify the effects of n consecutive calls Performance is linear? Exponential?
64
Data Presentation
The final results will be plotted as a functionof the block size n