1

Ashish Gupta

Bin Lin

Peter Dinda

Department of Computer Science

Northwestern University

3

Outline

• Motivation behind Resource Borrowing

• Measuring user comfort

• System Design for Measuring User Comfort

• Controlled User Study

• Its Results

• Conclusions

4

New Models of Distributed Computing

BreakScreenSaver time

Working

9 am 11 am 1 pm 3 pm 4 pm 5 pm 9 am

5

Observation and ideas

Idea: Why not borrow the unused resources ?

Unused Resources

Problem: Performance Slowdown

6

Effect of performance slowdown

7

Effect of performance slowdown

8

The question we want to answer

Depends on Other Factors as well…

What level of resource borrowing leads to user

discomfort for significant fraction of users ?

9

CPU RAM Disk

10

Context Person Hardware

Others ?

11

Power Users Typical Users Beginners

12

13

Outline

• Motivation behind Resource Borrowing

• Measuring user comfort

• System Design for Measuring User Comfort

• Controlled Study

• Its Results

• Conclusions

14

How can we find this out ?

Actually test it with user feedback

A system for emulating resource borrowing and measuring user discomfort using direct feedback

http://comfort.cs.northwestern.edu

15

How can we find this out ?

1Measure existing contention

16

How can we find this out ?

Emulate Resource Borrowing

Real Contention

Borrowed Contention

Final Contention

2 % CPU Usage vs. Time

17

How can we find this out ?

Allow User Feedback

3

F11

18

How can we find this out ?

Find MAX Level of tolerable contention

Borrowed Contention

Final Contention4

MAX

% CPU Usage vs. Time

19

How is it useful ?

Comfort = f (Contention)

To find a quantitative relationship between resource borrowing and user comfort

20

How is it useful ?

Enhances the degree of workstation sharing

21

Interactivity Aware Scheduling

Interactive

More intelligent scheduling of interactive tasks possible

via user feedback and our results

22

Outline

• Motivation behind Resource Borrowing

• Measuring user comfort

• System Design for Measuring User Comfort

• Controlled Study

• Its Results

• Conclusions

24

The client design

25

Testcases

• Each testcase stores details of resource borrowing• Testcase consists of exercise functions for each resource

• Exercise function is a vector of contention values• Example:

CPU contention: [0,0.5,1.0,1.5,2.0] at 1 Hz

27

Flowchart of Testcase Execution

No Testcase running

Resource Borrowing Phase Begins

(Testcase Runs)

No user feedback User expressed discomfort

Testcase Exhaustsafter some time

Testcase Ends immediately

User feedback

28

Resource Exercisers

• CPU Exerciser– Contention describes the expected extra number of

threads in ready queue– Fractional resource borrowing using stochastic scheduling

methods– Validated to contention level of 10

• Disk Exerciser– Random seek and read/write in a large file (twice the

memory)– Validated to contention level of 7

• Memory Exerciser– Borrows a fraction of physical memory: from 0 to 1

29

Outline

• Motivation behind Resource Borrowing

• Measuring user comfort

• System Design for Measuring User Comfort

• Controlled Study

• Its Results

• Conclusions

30

The control study description

• 33 undergrad and grad students

• 1.5 hrs each

• Each user was assigned 4 popular tasks to do– MS Word– MS Powerpoint– MS Explorer searching and saving information– Quake III

31

Machine Configuration

32

Testcases for the Control Study

Three types of testcases: step, ramp and blank

• Ramp allows us to know the point of discomfort for each resource• Step and Ramps: comparing the manner of borrowing resources• Blank: To test the background level of discomfort

34

Testcases for the Control Study

• Testcases different for each application–Some apps more intensive than

others

• Example ： Word and Quake

• Requires Calibration

35

Outline

• Motivation behind Resource Borrowing

• Measuring user comfort

• System Design for Measuring User Comfort

• Controlled Study

• Its Results

• Conclusions

36

Resource borrowing vs User Discomfort

• CDFs for discomfort contention level

• Not all contentions cause discomfort: exhausted region

37• C0.05 = 0.35 ( aggregated over all applications)

Exhausted Region

Discomfort Region

Aggregated Results

38

Resource borrowing vs User Discomfort

• CDFs for discomfort contention level

• Not all contentions cause discomfort: exhausted region

• CDFs allow us to read %age of people discomforted for a given contention

–Metric c0.05 : At what contention do we

discomfort only 5% of the people ?

39

Results overview

Aggregated Results

Dependence on Resource

Dependence on Context

Dependence on Users

Dependence on Step/Ramp

40

Aggregated Results

41• C0.05 = 0.35 ( aggregated over all applications)

Exhausted Region

Discomfort Region

Aggregated Results

42

• Can borrow a lot, discomforting only a small fraction\

• C0.05=0.33

Aggregated Results

Exhausted Region

Discomfort Region

43

• Can borrow a lot, discomforting only a small fraction• C0.05 = 1.11

Aggregated Results

Exhausted Region

Discomfort Region

44

Dependence on Resource

45

An overview of factors Dependenceon Resource

User sensitivity by task and resource (Low, Medium, High)

Users more tolerant for disk and memory borrowing, than CPU

Results Qualitative , but the testcase range is wide

46CPU

Dependenceon Resource

47Memory

Dependenceon Resource

48Disk

Dependenceon Resource

49

Dependence on Context

50

Comparison of CPU discomfort

WordIE

Powerpoint

Quake

Dependenceon Context

51

Dependence On Context - CPU

Word

Powerpoint

IE

Quake

Dependenceon Context

52

Dependence on Context - Memory

Word

Powerpoint

IE

Quake

Dependenceon Context

53

Dependence on Context - Disk

Word

Powerpoint

IE

Quake

Dependenceon Context

55

Internet Wide-Area Study

• No controls here, hope to get more general and complete data

• Users can download a client from http://comfort.cs.northwestern.edu

• Clean installation/un-installation and privacy options

Join us !

56

Outline

• Motivation behind Resource Borrowing

• Measuring user comfort

• System Design for Measuring User Comfort

• Controlled Study

• Its Results

• Conclusions

57

Conclusion

Comfort = f(contention)

Disk and memory can be borrowed more aggressively, CPU less so

A throttle for resource borrowing very useful

CDFs can be used to set the throttle

58

Conclusion

Comfort = f(contention)

Knowing the user context helps

Using Direct user feedback can be useful

59

Current and Future Work

User feedback for interactive VM Scheduling

Real time Scheduling to give exact resources to VMs

Minimal resources required to keep the user happy

60

Acknowledgements

http://comfort.cs.northwestern.edu

Special Thanks to:

Don Norman

Andrew Ortony

Ben Watson

Jack Tumblin

Leslie Lamport

and

All our beta testers

Study participants

61

Conclusion

Comfort = f(contention)

Disk and Memory surprisingly easy to borrow, CPU borrowing also liberal

Set of empirical CDFs provided

Depends of other factors: Resource, Context and Users

62

Dependence on User

63

Dependence on User

Power

Typical

Beginner

Dependenceon User

67

The placebo effect

• Users express discomfort even when no testcase is running• Can never guarantee that the user will never express

feedback

68

Memory Exerciser

• Contention indicates fraction of total memory allocated in addition to occupied memory

• Pages are touched frequently to keep them in physical memory

0MB 1024 MB

Occupied Mem. 0.5

Memory Contention

Clipping Point

69

70

Related Work

• User Frustration in HCI• Computer Response to User Frustration [K99]• Sensing and measurement of frustration with

computers [R01]

• Impact of latency of user utility• Behavioral aspects of text editors [EN81]• Psychological Upper and lower limits of

system response time and user’s preference on skill level [K97]

71

Virtual Machine Scheduling

• Virtuoso: Distributed Computing with Virtual Machines

• User pays for the resources he uses

• Need to know minimal resources needed to keep an interactive VM user happy

72

Questions and Answers• Other types of irritation• Bursty natures of Desktop Computing, bursts indicate sudden action-response tasks• Large number of factors for irritation• Since there is usually a power law for process run times, why do we need this model ? (CPU

is idle most of the time)• Why can’t we just run the bg app at a lower priority ? It may benefit much more than just

running at a fixed contention.• Memory exerciser: What does contention mean ? How do you find out free physical memory

?• How do you avoid thrashing ? Why do you avoid thrashing ?• How does the CPU exerciser work ? • Why do you disable buffering in the Disk exerciser ?• The aggregated graphs are not useful. Memory usage may be much more dynamic then in

your tests due to user multitasking etc. So this may not represent reality ?– Our study is a start in this direction and we realize that there are limitations to the data we have

collected. But we expect to collect and analyze more results from the Internet Wide area study and generalize our results to more real environments. There are no controls in the internet wide area study. The control study was meant to specifically answer the questions we have posed, like the affect of other factors in user discomfort and resource borrowing.

73

• Lower priority ?– Disk and memory ?

– There is a limit (max) possible amount of cycles stolen. Our goal is to s=estimate that. Seperately, there are many possible sched mech that would get closer to that max. Lower priority is an example. Linger longer is an example. Screen saver is an example.

– Quake example

– Don’t argue for/against a particular scheduling scheme

– It is a step in the right direction… (for results)

74

Testcase Transfer

Results Transfer

Server

Client

Resource borrowing using testcase profile

75

The client-server system

• Data Exchange• Hotsync

– Results transfer– Testcase update

• Testcase Tools

76

77

78

An irritating Interface

79

Interactive Computation

Bursty Nature of Interactive Computation

80

81

82

Understanding Testcases

CPU

ramp 1ramp 2ramp 3step 1step 2

…..

MEM

ramp 1ramp 2ramp 3step 1step 2

…..

Ramp Step

83

Combination of Resources

• CPU Usage• Memory Usage• Hard Disk activity• Network Traffic

Comfort Zone surface

85