anatomy of an OpenStack based integration test for a backuppc puppet module

An integration test is run by jenkins within an OpenStack tenant. It checks that the backuppc puppet module is installed

ssh root@$instance test -f /etc/backuppc/hosts || return 3

A full backup is run

ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
   backup nagios.novalocal nagios.novalocal backuppc 1"' \
                              backuppc || return 4
ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* | \
    sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'

and a nagios plugin asserts its status is monitored

    while ! ( echo "GET services"
        echo "Filter: host_alias = $instance.novalocal"
        echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
        ssh root@nagios unixcat /var/lib/nagios3/rw/live |
        grep "BACKUPPC OK - (0/" ; do
        sleep 1
    done

The test environment

The test is run by the run-test-in-openstack.sh script which is called from jenkins in a dedicated OpenStack tenant.

function run_tests() {
    local directory="$1"

    for path in $(find * -name test-in-openstack.sh) ; do
        source "$path" || return 1
    done
}

A puppet master ( puppet.novalocal ) and a nagios server ( nagios.novalocal ) are run before calling the script. The puppet master is loaded with the master branch of the git repository and the nagios server is deployed from it using the april_nagios puppet module with:

node 'nagios-hetzner.vm.april-int',
     'nagios.vm.april-int',
     'nagios.novalocal' inherits openstack-instance {
  include april_nagios::server
}

The search domain within the OpenStack environment is novalocal and the nagios server can be referred to as nagios instead of nagios.novalocal.
The puppet master is configured to accept all new puppet clients without human interaction to allow for a fully automated run:

echo "'DAEMON_OPTS=\"--autosign true\"'" \>\> /etc/default/puppetmaster

The ssh public key of the user under which the tests are running is installed in the OpenStack tenant and will be used when creating instances so that the tests can run ssh commands using ssh root@instancename

function key_add() {
    local name="$1"
    local pubkey="$2"
    local command="nova keypair-add --pub_key \"$pubkey\" \"$name\""

    eval $command

    if ! key_exists "$name" ; then
        echo "The command '$command' was run but 'nova keypair-list' does not show $name"
        return 1
    else
        return 0
    fi
}

Running puppet unit tests

The puppet.novalocal puppet master is setup to run puppet rspec unit tests. A specific set of ruby gems is installed and known to work with ruby-1.8 and puppet-2.7 or puppet-2.6.

gem install --ignore-dependencies --version 1.1.3 diff-lcs
gem install --ignore-dependencies --version 1.6.14 facter
gem install --ignore-dependencies --version 0.0.1 metaclass
gem install --ignore-dependencies --version 0.13.0 mocha
gem install --ignore-dependencies --version 2.7.18 puppet
gem install --ignore-dependencies --version 0.1.13 puppet-lint
gem install --ignore-dependencies --version 0.2.0 puppetlabs_spec_helper
gem install --ignore-dependencies --version 10.0.2 rake
gem install --ignore-dependencies --version 2.12.0 rspec
gem install --ignore-dependencies --version 2.12.0 rspec-core
gem install --ignore-dependencies --version 2.12.0 rspec-expectations
gem install --ignore-dependencies --version 2.12.0 rspec-mocks
gem install --ignore-dependencies --version 0.1.4 rspec-puppet

So that the tests can be run with:

'GEM_HOME=$HOME/.gem-installed' 'PATH=$HOME/.gem-installed/bin:$PATH' rake spec

The puppet client

The integration tests for the puppet modules make use of a puppet client image. It is booted in the OpenStack tenant with:

nova boot --image puppet ... backuppc

and it will ask for a configuration to the puppet master under the name backuppc.novalocal. The puppet master configuration must contain such host name for test purposes. For instance:

node 'harmine.pavot.vm.april-int',
     'backuppc.novalocal'  inherits vserver-pavot {
  include backuppc::server
  include april_nagios::nrpe_server
  include april_nagios::check_backuppc
}

Helper functions

The instance_run function is a wrapper around nova boot that waits for the instance to show with nova list.

function instance_run() {
    local instance="$1"
    local image="$2"
    local flavor="$3"

    nova boot \
        --image "$image" \
        --flavor "$flavor" \
        --key_name ${KEYPAIR_NAME} \
        --availability_zone ${AVAILABILITY_ZONE} \
        --poll \
        "$instance"
    while ! nova list --name="$instance" | grep "$instance" ; do sleep 1 ; done
}

The instance_delete function is a wrapper around nova delete that waits until is disapears from the output of nova list and cleanup the puppet certificate.

function instance_delete() {
    local instance="$1"

    nova delete "$instance" || return 0
    while nova list --name="$instance" |
          grep "$instance" > /dev/null ; do sleep 1 ; done
    ssh root@${PUPPETMASTER[instance]} puppetca clean $instance.novalocal || true
}

Commented test script for backuppc

The test script starts by running the puppet rspec unit tests:

local instance=backuppc
ssh root@puppet cd /etc/puppet/modules/$instance \; ' \
GEM_HOME=$HOME/.gem-installed' \
'PATH=$HOME/.gem-installed/bin:$PATH' rake spec || return 1

If the test is interrupted, a previous instance named backuppc.novalocal remains and must be deleted:

instance_delete $instance

A puppet client with the hostname backuppc.novalocal is created

instance_run $instance puppet e.1-cpu.10GB-disk.512MB-ram || return 2

and the script pauses until an ssh server becomes avaiable:

while ! nmap $instance -PN -p ssh | grep open ; do sleep 1 ; done

The /var/log/daemon.log file is monitored to wait for the completion of the initial puppet agent run at boot time ( when Finished catalog run is found ).

ssh -o 'StrictHostKeyChecking=false' root@$instance \
 tail -f /var/log/daemon.log | sed --unbuffered -e "s/^/$instance: /" \
                  -e '/Finished catalog run/q'

A minimal side effect of the backuppc puppet module

  package { 'backuppc':
    ensure => installed,
  }

is checked : the presence of the /etc/backuppc/hosts file.

ssh root@$instance test -f /etc/backuppc/hosts || return 3

The puppet class configuring nagios makes use of exported resources, which requires an extra puppet pass:

  • nagios is configured by the puppet master
  • backuppc is configured by the puppet master and records ( in the puppet master ) the fact that it is a candidate to be monitored ( this is what exported resources basically do )
  • nagios runs the puppet client again and is notified by the puppet master that backuppc needs to be monitored and modifies its configuration accordingly
ssh root@nagios puppet agent -vt

The next test will be to verify that the backuppc.novalocal performs a backup of nagios.novalocal because the openstack-instance.pp manifests contains:

node 'nagios-hetzner.vm.april-int',
     'nagios.vm.april-int',
     'nagios.novalocal' inherits openstack-instance {
  include april_nagios::server
}

which inherits from openstack-instance

node openstack-instance inherits default {
  april_nagios::host { $fqdn: address => $ipaddress }
  include april_ssh
  include backuppc::client
}

declaring the instance to be a backuppc::client. This class makes use of exported resources and requires an extra run of puppet on backuppc.novalocal to configure backuppc so that it schedules the backup of the nagios.novalocal instance.

ssh root@$instance puppet agent -vt

Instead of waiting for the backup to start, it is forced to run immediately using the backuppc command line. It will start the backup of the nagios.novalocal instance and the logs ( /var/lib/backuppc/pc/nagios.novalocal/LOG.* ) will be displayed until the full backup 0 complete string shows, proving the backup succeeded.

    ssh root@$instance su -c '"/usr/share/backuppc/bin/BackupPC_serverMesg \
           backup nagios.novalocal nagios.novalocal backuppc 1"' backuppc || return 4
    ssh root@$instance tail -f /var/lib/backuppc/pc/nagios.novalocal/LOG.* |
           sed --unbuffered -e "s/^/$instance: /" -e '/full backup 0 complete/q'

Once the backup is complete, nagios.novalocal will notice because a test specific to backuppc was installed ( it comes with the nagios-plugins-contribs package ).

    while ! ( echo "GET services"
        echo "Filter: host_alias = $instance.novalocal"
        echo "Filter: check_command = check_nrpe_1arg"'!'"check_backuppc" ) |
        ssh root@nagios unixcat /var/lib/nagios3/rw/live |
        grep "BACKUPPC OK - (0/" ; do
        sleep 1
    done

The livestatus query language is used against the nagios server database : GET the services but Filter to get only those with the host_alias field matching backuppc.novalocal and the check_command matching check_nrpe_1arg!check_backuppc command. The line found will display the string BACKUPPC OK – (0/ on success.
Finally the backuppc.novalocal instance is destroyed.

instance_delete $instance
This entry was posted in Essex, debian, openstack, puppet. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>