Course Information

Andy WangCIS 5930

Computer Systems Performance Analysis

2

Contact Information

• Andy Wang (awang@cs.fsu.edu)• Office: 269 Love Building• Office hours: after class

(also by appointments)• Class website:

http://www.cs.fsu.edu/~awang/courses/cis5930_s2015

Objectives

• Become exposed to common issues involved in evaluating software systems

• Learn statistical tools and experimental design techniques

• Gain experience in performance evaluation

3

4

What Will This CourseTeach You?

• Proper methods for designing and performing experiments on systems

• Proper methods for analyzing and presenting data gathered from such experiments

• Proper methods for critiquing experiments and data gathered

5

What Won’t This Course Teach You?

• Basic systems software principles• Systems software modeling• Queueing theory• Simulation techniques for systems

software

6

Who Should TakeThis Course

Well, everyone, but especially -• Software developers• Software researchers• Software purchasers• Software evaluators

Course Materials

• Lecture notes and papers (posted on the class website)

• Textbook– The Art of Computer Systems

Performance Analysis, by Raj Jain

8

Grading

• Up to 5 assignments (take home exams) - 10% each

• Peer evaluation of projects by other groups - 10%

• Project – remaining grade

Project

• You need to develop a project in teams of two or three

• Goal:– Publishable results

10

Suitable Subjectsfor Projects

• OSes or their components (file systems, I/O subsystems, process handling, etc.)

• Compilers• Databases• Real-time applications• Application packages• Networks/networking systems• The Web

Possible Projects

• Measure/simulate/compare systems• Develop and validate a simulator• Characterize workloads of a system• Analyze the effects of various factors on

a system• Develop a software monitor to observe

the performance of a system

11

Project Proposal

• Due on the 2nd week– All team members are required to participate

• 2-page written proposal– System background and evaluation goals– Metrics– Parameters, factors, levels– Evaluation technique– Workloads– Division of labors

Experimental Design

• Due on the 4th week– All team members are required to participate

• 2-page written design– Brief background– Experimental design– Data analysis techniques– Anticipated graphs

Project Presentation

• During the last week of the course• 15-page (max) written paper due by the

last lecture (double column, single-space, 10-pt font)

15

Grading of Projects

Several criteria will be used:• Proper design of the experiment• Care and thoroughness of its execution• Completeness of analysis• Quality of data presentation• Insight gained from experiment

Insight is the most important

16

Evaluation of Projectsby Other Groups

• Submitted by each student individually• Two 1-page critiques on other group’s

proposed experiment – Due by 5th week

• Two 1-page critiques on other group’s results– Due by final’s week

• Graded on basis of insight into strengths and flaws of each project