OpenStack Summit :: Profiling the Nova Scheduler

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Profiling the Nova Scheduler

A Case Study

Joe Gordon

1

Wednesday, October 17, 12


About Me

• Engineer at Cloudscaling• Contributor• Deployer• Folsom Contributions

o Top 10 developer (by commits)

o Mostly in Nova

http://bitergia.com/public/reports/openstack/2012_09_folsom/

2





By the Numbers: Nova Folsom

• 190+ Contributors•

• Code churn:

Release Python Lines Other Lines Python Files Other Files

Folsom 186,738 242,721 666 788

Essex 150,894 221,109 593 302

Diablo 110,581 110,393 427 389

Code churn generated with git log --numstat --pretty="%H" $A..$B| grep \.py$ | awk 'NF==3 {plus+=$1; minus+=$2} END {printf("+%d, -%d\n", plus, minus)}'

Release Lines Insertions Lines Deletions Insertions/LoC %

Folsom 110,308 71,911 59.0%

Essex 182,298 138,346 120.8%

3



Who Wrote Nova Folsom?

4

1 138 Russell Bryant <[email protected]> 2 112 Johannes Erdfelt <[email protected]> 3 97 Dan Prince <[email protected]> 4 88 Vishvananda Ishaya <[email protected]> 5 81 Joe Gordon <[email protected]> 6 63 Michael Still <[email protected]> 7 59 Mark McLoughlin <[email protected]> 8 58 Rick Harris <[email protected]> 9 50 Yun Mao <[email protected]> 10 45 Daniel P. Berrange <[email protected]> 11 36 Chris Behrens <[email protected]> 12 31 Eoghan Glynn <[email protected]> 13 29 Brian Waldon <[email protected]> 14 26 Pádraig Brady <[email protected]> 15 25 Chuck Short <[email protected]> 16 23 Sean Dague <[email protected]> 17 21 Alex Meade <[email protected]> 18 18 Kevin L. Mitchell <[email protected]> 19 17 Brian Elliott <[email protected]> 20 17 Zhongyue Luo <[email protected]> 21 16 John Griffith <[email protected]> 22 13 Dan Smith <[email protected]> 23 13 Andrew Bogott <[email protected]> 24 12 Renuka Apte <[email protected]> 25 12 Thierry Carrez <[email protected]> 26 12 Monty Taylor <[email protected]> 27 10 MotoKen <[email protected]>

git shortlog -sne --since="Tue Mar 20 08:17:40 2012 +0100" --no-merges | cat -n


mailto:[email protected]























































Problems

5



Problemseuca-run-instances -n 1000

1. When trying to schedule 1000 VMs, more than 1000 VMs were scheduled. Why were too many VMs scheduled?

2. Scheduling 100 VMs took 24 seconds. Why is the scheduler so slow?

6

Environment: default nova essex scheduler



Outline

• How the nova essex scheduler works• P1: Why were too many VMs scheduled?• Nova performance • P2: Why is the essex scheduler is so slow?• How the folsom scheduler performs

7



How the nova essex scheduler works

8



Scheduling

9



Default SchedulerFilter Scheduler

10

http://docs.openstack.org/developer/nova/devref/filter_scheduler.html





Limitations

• Cannot run multiple schedulers• Race condition• Fixed in folsom

• O(nm)• n = number of nodes• m = number of VMs requested

11



Problem 1: Why were too many VMs scheduled?

12



Problem #1: Too Many VMs

13

• scheduling is slow• RPC call instead of cast• boto retries upon API timeout

$ time euca-run-instances -t m1.tiny -n 100 real 0m25.250s user 0m0.100s sys 0m0.020s



nova/compute/api.pyrpc.call instead of rpc.cast

14

def _create_instance(...): """Verify all the input parameters regardless of the provisioning strategy being performed and schedule the instance(s) for creation."""... # We need to wait for the scheduler to create the instance # DB entries, because the instance *could* be # created in # a child zone. rpc_method = rpc.call

# TODO(comstud): We should use rpc.multicall when we can # retrieve the full instance dictionary from the scheduler. # Otherwise, we could exceed the AMQP max message size limit. # This would require the schedulers' schedule_run_instances # methods to return an iterator vs a list. instances = self._schedule_run_instance( rpc_method, context, base_options, instance_type, availability_zone, injected_files, admin_password, image, num_instances, requested_networks, block_device_mapping, security_group, filter_properties)

# We can create the DB entry for the instance here if we're # only going to create 1 instance. # This speeds up API responses for builds # as we don't need to wait for the scheduler. create_instance_here = max_count == 1



nova/scheduler/filter_scheduler.pyeach VM is scheduled individually

15

def schedule_run_instance(self, context, request_spec, *args, **kwargs): """This method is called from nova.compute.api to provision an instance. We first create a build plan (a list of WeightedHosts) and then provision.

Returns a list of the instances created. """... for num in xrange(num_instances): if not weighted_hosts: break weighted_host = weighted_hosts.pop(0)

request_spec['instance_properties']['launch_index'] = num instance = self._provision_resource(elevated, weighted_host, request_spec, kwargs)

if instance: instances.append(instance)



Nova

16



Retries

17

EC2User



Retries

17

EC2User

Nova EC2API



Retries

17

EC2User

Pound

Timeout=15Nova EC2

API



Retries

17

EC2User

Pound

Timeout=15Nova EC2

API

euca-run-instances -n 100



Retries

17

EC2User

Pound

Timeout=15Nova EC2

API

euca-run-instances -n 100 Slow



Retries

17

EC2User

Pound

Timeout=15Nova EC2

API

euca-run-instances -n 100 SlowSendTimeout



Retries

17

EC2User

Pound

Timeout=15Nova EC2

API

euca-run-instances -n 100 SlowSendTimeout

Retry



Nova Performance

18



Nova Performance

19

• How eventlet changes everything• coroutines

• interplay between MySQL and eventlet• RPC• CPU frequency scaling



Nova Logslogging format:

%(asctime)s %(levelname)s %(name)s [%(request_id)s %(user_id)s %(project_id)s]

Example:

2012-10-11 21:48:08 DEBUG nova.api.ec2 [req-d54b7cfa-fd51-4a04-92f2-5509c8411ed4 d97c90fd1a4f49e7b7fb38f4d358f4dd a6f5079f10464a6f9aefd1b2e9d5aab3] action: RunInstances from (pid=30579) __call__ /usr/local/lib/python2.7/dist-packages/nova-2012.1.3-py2.7.egg/nova/api/ec2/__init__.py:435

20



Why is scheduling so slow?24 seconds in the life of nova

21


CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses. 22

nova-api.log2012-10-11 21:48:08 - POST RunInstances Request2012-10-11 21:48:08 - RPC RR CALL ‘scheduler’2012-10-11 21:48:32 - POST Response (24 seconds)

nova-scheduler.log2012-10-11 21:48:08 - Attempting to build 100 instance(s)2012-10-11 21:48:09 - Last VM filtered and weighted... - Write to DB, and Cast (19 seconds)2012-10-11 21:48:28 - Respond to api CALL

mysql.logfilter and weighting: 733 DB calls, 0.07151 secondsWrite to DB, and Cast: 3563 DB Calls, 1.77971 seconds

Problem #2: Slow24 seconds in the life of nova



Problem #2: Slow24 seconds in the life of nova

• DB is fast• filter and weight is fast• Python code around DB is slow

•SQLAlchemy?

23



Folsom

24



Folsom

• RPC.Call to scheduler is now RPC.Cast to scheduler• API server doesn’t wait for scheduling

• Improved DB access patterns• Essex: 29 DB calls in nova/scheduler/driver.py• Folsom: 12 DB calls in nova/scheduler/driver.py

25



Questions?

26



Raw Datagoal: show how much time spent where setup: essex filter scheduler, 2 compute nodes

27

$ time euca-run-instances -t m1.tiny -n 50real 0m12.183suser 0m0.104ssys 0m0.020s

$ time euca-run-instances -t m1.tiny -n 100real 0m25.250suser 0m0.100ssys 0m0.020s

req-d54b7cfa-fd51-4a04-92f2-5509c8411ed4

nova-api.log2012-10-11 21:48:08 - POST RunInstances Request2012-10-11 21:48:08 - RPC RR CALL ‘scheduler’2012-10-11 21:48:32 - POST response

Note: show euca -n * loop in nova-scheduler (1 RPC to scheduler 100 separate schedule commands)

nova-scheduler.log2012-10-11 21:48:08 - Attempting to build 100 instance(s)2012-10-11 21:48:09 - Last VM filtered wand weighted... - Cast to compute, block_device_mapping etc.2012-10-11 21:48:28 - respond to api CALL

mysql.logfilter and weighting: 733 DB calls, 0.071512 secondscast to compute, block_device_mapping, security groups: 3563 DB Calls, 1.77971


OpenStack Summit :: Profiling the Nova Scheduler

Documents

Transcript of OpenStack Summit :: Profiling the Nova Scheduler