Controlling volume and instance placement in OpenStack (take 1)

An OpenStack cluster is deployed using bare metal hardware provisionned from various hosting companies (eNovance, Hetzner etc.). The storage volume location is controlled by shutting down all nova-volume daemons, except the one running on the desired node. Each node (bare metal machine) is defined as an availability zone, which allows the user to target the desired node with the nova boot … –availability_zone=node … command line option.

OpenStack setup

The OpenStack cluster is setup using the Debian GNU/Linux puppet HOWTO. All hosts are named bm0001.the.re, bm0002.the.re, bm0003.the.re etc.

Availability zone

An availability zone is defined by adding the node_availability_zone configuration flag in the /etc/nova/nova.conf file for each node. For instance, the following puppet manifest snippet defines that each node is an availability zone.

  $availability_zone = regsubst($::fqdn, '^(bm\d+).*', '\1')
  nova_config { 'node_availability_zone': value => $availability_zone }

translates into the following line in /etc/nova/nova.conf for the node bm0001.the.re:

--node_availability_zone=bm0001

Locating the node for a volume

Create the volume

# nova volume-create --display_name volume01 1

Figure out the ID of the volume

# nova volume-list
+----+-----------+--------------+------+-------------+-------------+
| ID |   Status  | Display Name | Size | Volume Type | Attached to |
+----+-----------+--------------+------+-------------+-------------+
| 8  | available | volume01     | 1    | None        |             |
+----+-----------+--------------+------+-------------+-------------+

Get the volume provider_location and guess the node from it

# mysql -e "select provider_location from volumes where id = 8" nova
+------------------------------------------------------------------+
| provider_location                                                |
+------------------------------------------------------------------+
| 192.168.100.1:3260,2 iqn.2010-10.org.openstack:volume-00000008 0 |
+------------------------------------------------------------------+

Each node in the cluster has an IP address matching the node name. 192.168.100.1 is bm0001.the.re, 192.168.100.2 is bm0002.the.re etc. Hence the node on which the volume was created in the example above is bm0001.the.re.

SameHostFilter does not work in Essex

The scheduler configuration can be displayed with:

grep scheduler_ /var/log/nova/nova-scheduler.log

immediately after a

/etc/init.d/nova-scheduler restart

otherwise the logs rotatation will move them to another file. For instance:

root@bm0001:~# grep scheduler_ /var/log/nova/nova-scheduler.log
2012-09-13 11:08:14 DEBUG nova.service [-] scheduler_json_config_location :  from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] volume_scheduler_driver : nova.scheduler.chance.ChanceScheduler from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] scheduler_default_filters : ['AvailabilityZoneFilter', 'RamFilter', 'ComputeFilter'] from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] compute_scheduler_driver : nova.scheduler.filter_scheduler.FilterScheduler from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] scheduler_topic : scheduler from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] scheduler_host_manager : nova.scheduler.host_manager.HostManager from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] scheduler_manager : nova.scheduler.manager.SchedulerManager from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] scheduler_driver : nova.scheduler.multi.MultiScheduler from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411
2012-09-13 11:08:14 DEBUG nova.service [-] scheduler_available_filters : ['nova.scheduler.filters.standard_filters'] from (pid=534) wait /usr/lib/python2.7/dist-packages/nova/service.py:411

In the example above, the filters deciding on which node a new resource is located are listed in:

scheduler_default_filters : ['AvailabilityZoneFilter', 'RamFilter', 'ComputeFilter'

The SameHostFilter can be used by adding it to the list, in /etc/nova/nova.conf:

scheduler_default_filters=AvailabilityZoneFilter,RamFilter, SameHostFilter,ComputeFilter]

However, using

 nova boot ... --hint 'same_host=[f7f3b33c-e5e7-43b2-b1b6-687dc6d0d8a6]'  ...

as instructed in the documentation won’t work because the –debug shows that it translates into

{"os:scheduler_hints": {"same_host": "[f7f3b33c-e5e7-43b2-b1b6-687dc6d0d8a6]"}...

instead of

{"os:scheduler_hints": {"same_host": ["f7f3b33c-e5e7-43b2-b1b6-687dc6d0d8a6"]}...

It is a string and not an array and the corresponding code fails to select the appropriate host. There are no bugs filled against this problem. The same parsing problem happens with the force_hosts hint, which requires admin privileges.

Create a volume and a virtual machine on a designated node

Shutdown the nova-volume daemon on each node except the one that is targeted ( for instance bm0001.the.re ) with :

/etc/init.d/nova-volume stop

and check that only one remains:

# nova-manage service list | grep nova-volume
nova-volume      bm0001.the.re                        bm0001           enabled    :-)    2012-09-13 13:22:07
nova-volume      bm0002.the.re                        bm0002           enabled    XXX   2012-09-13 13:11:41

Create the volume:

nova volume-create --display_name volume01 1

Restart the nova-volume daemon on each node with

/etc/init.d/nova-volume start

and check that they are all back online:

# nova-manage service list | grep nova-volume
nova-volume      bm0001.the.re                        bm0001           enabled    :-)    2012-09-13 13:24:48
nova-volume      bm0002.the.re                        bm0002           enabled    :-)    2012-09-13 13:24:44

Create the virtual machine on the same node using the availability zone defined to include the designated node:

nova boot --image fe44546e-66db-4983-8a49-6bed4afc1301 --flavor 1 --key_name test_keypair  --availability_zone=bm0001 instance01 --poll

Check that it shows when listing the instances hosted on a designated node:

# nova list --host bm0001
+--------------------------------------+------------+--------+------------------------+
|                  ID                  |    Name    | Status |        Networks        |
+--------------------------------------+------------+--------+------------------------+
| 2194fc07-9443-4ce8-8ae0-4b9360757d36 | instance01 | ACTIVE | novanetwork=10.145.0.8 |
+--------------------------------------+------------+--------+------------------------+

Attach the volume to the server

# nova volume-attach 2194fc07-9443-4ce8-8ae0-4b9360757d36 8 /dev/vda

Check that it has been attached successfully:

# nova volume-list
+----+--------+--------------+------+-------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type |             Attached to              |
+----+--------+--------------+------+-------------+--------------------------------------+
| 8  | in-use | volume01     | 1    | None        | 2194fc07-9443-4ce8-8ae0-4b9360757d36 |
+----+--------+--------------+------+-------------+--------------------------------------+

Check that it is accessible from the instance with:

root@bm0001:~# ssh -i test_keypair.pem cirros@10.145.0.8
$ sudo fdisk /dev/vdb
Command (m for help): p
Disk /dev/vdb: 1073 MB, 1073741824 bytes
This entry was posted in Essex, openstack. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>