unit testing ceph : the Throttle.cc example

The throttle implementation for ceph can be unit tested using threads when it needs to block. The gtest framework produces coverage information to lcov showing that 100% of the lines of code are covered.

add a new unit test

A new test program is declared in the src/Makefile.am

--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -655,6 +655,12 @@ unittest_log_LDADD = libcommon.la ${UNITTEST_LDADD}
 unittest_log_CXXFLAGS = ${AM_CXXFLAGS} ${UNITTEST_CXXFLAGS} -O2
 check_PROGRAMS += unittest_log

+unittest_throttle_SOURCES = test/common/Throttle.cc
+unittest_throttle_LDFLAGS = $(PTHREAD_CFLAGS) ${AM_LDFLAGS}
+unittest_throttle_LDADD = libcommon.la ${LIBGLOBAL_LDA} ${UNITTEST_LDADD}
+unittest_throttle_CXXFLAGS = ${AM_CXXFLAGS} ${UNITTEST_CXXFLAGS} -O2
+check_PROGRAMS += unittest_throttle
+
 unittest_base64_SOURCES = test/base64.cc
 unittest_base64_LDFLAGS = $(PTHREAD_CFLAGS) ${AM_LDFLAGS}
 unittest_base64_LDADD = libcephfs.la -lm ${UNITTEST_LDADD}

the source is located in a subdirectory of test matching the location of the source being tested in the src directory, i.e. common.

test skeleton

The headers of the sources being tested are included

#include "common/Throttle.h"

and the ceph helper is included to define the main function

#include "test/unit.h"

The verbosity is controlled with a macro:

#define TEST_DEBUG 20
//#define TEST_DEBUG 0

used with the set_log_level function

    g_ceph_context->_conf->subsys.set_log_level(dout_subsys, TEST_DEBUG);

The dout_subsys should be defined as a macro identical to the one used in the source being tested

#define dout_subsys ceph_subsys_throttle

This is achieved by defining a test class SetUp function:

class ThrottleTest : public ::testing::Test {
protected:
  virtual void SetUp() {
    g_ceph_context->_conf->subsys.set_log_level(dout_subsys, TEST_DEBUG);
 }
...
};

which requires to define each test using the TEST_F macro instead of the simpler TEST macro:

TEST_F(ThrottleTest, take) {
  int64_t throttle_max = 10;
  Throttle throttle(g_ceph_context, "throttle", throttle_max);
  ASSERT_THROW(throttle.take(-1), FailedAssertion);
  ASSERT_EQ(throttle.take(3), 3);
}

using threads

When a Throttle function blocks, it must be run in a thread otherwise the test would never complete. A class derived from Thread is defined

  class Thread_get : public Thread {
  public:
    bool waited;
    Throttle &throttle;
    int64_t count;

    Thread_get(Throttle& _throttle, int64_t _count) :
      throttle(_throttle),
      count(_count),
      waited(false)
    {
    }

    virtual void *entry() {
      waited = throttle.get(count);
      throttle.put(count);
      return NULL;
    }
  };

It will run a get ( which may block ) immediately followed by put and it keeps a record of wether it blocked ( waited ) or not. It is called from the test with:

    Thread_get t(throttle, 7);
    t.create();

which will call the entry function in the thread while the Thread_get constructor is called within the main thread.
There is no convenient way to check if a thread is asleep from the main thread. Instead, the main thread goes to sleep ( see usleep(delay); below) and expects the thread to be asleep when it wakes up.

  do {
    cout << "Trying with delay " << delay << "ms\n";

    ASSERT_FALSE(throttle.get(throttle_max));
    ASSERT_FALSE(throttle.get_or_fail(throttle_max));

    Thread_get t(throttle, 7);
    t.create();
    usleep(delay);
    ASSERT_EQ(throttle.put(throttle_max), 0);
    t.join();

    if(!(waited = t.waited))
      delay *= 2;
  } while(!waited);

However, there is no guarantee that it will actually happen because it may take longer than delay for the thread to go to sleep. The main thread will notice that the expected event did not happen ( waited is false ) and increase the delay to give more time to the thread. It may loop forever if the thread never goes to sleep because of a bug. Either way, it will only end the loop if the expected sequence of event occured.

licensing

Contributing to ceph does not require a CLA and the contributor is responsible for adding a license notice compatible with the project license at the begining of each file. For instance:

/*
 * Ceph - scalable distributed file system
 *
 * Copyright (C) 2013 Cloudwatt <libre.licensing@cloudwatt.com>
 *
 * Author: Loic Dachary <loic@dachary.org>
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU Library Public License as published by
 * the Free Software Foundation; either version 2, or (at your option)
 * any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU Library Public License for more details.
 *
 */

He is also responsible for checking with his employer that such a publication is authorized and must follow the ceph rules for submitting patches. In the above example, the copyright belongs to the company ( cloudwatt ) and the author of the work ( moral rights ) is also listed.
Since a new file is added, the inventory of the files and their respective license and copyright holder must be updated:

--- a/COPYING
+++ b/COPYING
@@ -98,3 +98,6 @@ License:

+Files: test/common/Throttle.cc
+Copyright: Copyright (C) 2013 Cloudwatt <libre.licensing@cloudwatt.com>
+License: LGPL2 or later

A new Contributors section to the AUTHORS list is proposed:

--- a/AUTHORS
+++ b/AUTHORS
@@ -13,3 +13,7 @@ Patience Warnick

 Yehuda Sadeh-Weinraub 
 Greg Farnum 

+Contributors
+------------
+
+Loïc Dachary <loic@dachary.org>

debugging a test

When a working on a given test ( for instance get_or_fail), it can be compiled and run with the following one-liner from the src directory:

make unittest_throttle ; ./unittest_throttle --gtest_filter=ThrottleTest.get_or_fail

GDB can be run with

gdb --annotate=3 unittest_throttle

from the src directory.

pull request and discussions

The tests are proposed for inclusion in ceph

This entry was posted in ceph. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>