You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
weston/tests/weston-test-runner.h

257 lines
8.0 KiB

/*
* Copyright © 2012 Intel Corporation
* Copyright © 2013 Sam Spilsbury <smspillaz@gmail.com>
*
* Permission is hereby granted, free of charge, to any person obtaining
* a copy of this software and associated documentation files (the
* "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish,
* distribute, sublicense, and/or sell copies of the Software, and to
* permit persons to whom the Software is furnished to do so, subject to
* the following conditions:
*
* The above copyright notice and this permission notice (including the
* next paragraph) shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#ifndef _WESTON_TEST_RUNNER_H_
#define _WESTON_TEST_RUNNER_H_
#include "config.h"
#include <stdlib.h>
#include <wayland-util.h>
#include "shared/helpers.h"
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
#include "weston-test-fixture-compositor.h"
#include "weston-testsuite-data.h"
#ifdef NDEBUG
#error "Tests must not be built with NDEBUG defined, they rely on assert()."
#endif
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
/** Test program entry
*
* Each invocation of TEST(), TEST_P(), or PLUGIN_TEST() will create one
* more weston_test_entry in a custom named section in the final binary.
* Iterating through the section then allows to iterate through all
* the defined tests.
*
* \ingroup testharness_private
*/
struct weston_test_entry {
const char *name;
void (*run)(void *);
const void *table_data;
size_t element_size;
int n_elements;
} __attribute__ ((aligned (32)));
#define TEST_BEGIN(name, arg) \
static void name(arg)
#define TEST_COMMON(func, name, data, size, n_elem) \
static void func(void *); \
\
const struct weston_test_entry test##name \
__attribute__ ((used, section ("test_section"))) = \
{ \
#name, func, data, size, n_elem \
};
#define NO_ARG_TEST(name) \
TEST_COMMON(wrap##name, name, NULL, 0, 1) \
static void name(void); \
static void wrap##name(void *data) \
{ \
(void) data; \
name(); \
} \
\
TEST_BEGIN(name, void)
#define ARG_TEST(name, test_data) \
TEST_COMMON(name, name, test_data, \
sizeof(test_data[0]), \
ARRAY_LENGTH(test_data)) \
TEST_BEGIN(name, void *data) \
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
/** Add a test with no parameters
*
* This defines one test as a new function. Use this macro in place of the
* function signature and put the function body after this.
*
* \param name Name for the test, must be a valid function name.
*
* \ingroup testharness
*/
#define TEST(name) NO_ARG_TEST(name)
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
/** Add an array driven test with a parameter
*
* This defines an array of tests as a new function. Use this macro in place
* of the function signature and put the function body after this. The function
* will be executed once for each element in \c data_array, passing the
* element as the argument <tt>void *data</tt> to the function.
*
* This macro is not usable if fixture setup is using
* weston_test_harness_execute_as_plugin().
*
* \param name Name for the test, must be a valid function name.
* \param data_array A static const array of any type. The length will be
* recorded automatically.
*
* \ingroup testharness
*/
#define TEST_P(name, data_array) ARG_TEST(name, data_array)
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
/** Add a test with weston_compositor argument
*
* This defines one test as a new function. Use this macro in place of the
* function signature and put the function body after this. The function
* will have one argument <tt>struct weston_compositor *compositor</tt>.
*
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
* This macro is only usable if fixture setup is using
* weston_test_harness_execute_as_plugin().
*
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
* \param name Name for the test, must be a valid function name.
*
* \ingroup testharness
*/
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
#define PLUGIN_TEST(name) \
TEST_COMMON(wrap##name, name, NULL, 0, 1) \
static void name(struct weston_compositor *); \
static void wrap##name(void *compositor) \
{ \
name(compositor); \
} \
TEST_BEGIN(name, struct weston_compositor *compositor)
void
testlog(const char *fmt, ...) WL_PRINTF(1, 2);
const char *
get_test_name(void);
tests: thread-based client harness This replaces the old test harness with a new one. The old harness relied on fork()'ing each test which makes tests independent, but makes debugging them harder. The new harness runs client code in a thread instead of a new process. A side-effect of not fork()'ing anymore is that any failure will stop running a test series short. Fortunately we do not have any tests that are expected to crash or fail. The old harness executed 'weston' from Meson, with lots of setup as both command line options and environment variables. The new harness executes wet_main() instead: the test program itself calls the compositor main function to execute the compositor in-process. Command line arguments are configured in the test program itself, not in meson.build. Environment variables aside, you are able to run a test by simply executing the test program, even if it is a plugin test. The new harness adds a new type of iteration: fixtures. For now, fixtures are used to set up the compositor for tests that need a compositor. If necessary, a fixture setup may include a data array of arbitrary type for executing the test series for each element in the array. This will be most useful for running screenshooting tests with both Pixman- and GL-renderers. The new harness outputs TAP formatted results into stdout. Meson is not switched to consume TAP yet though, because it would require a Meson version requirement bump and would not have any benefits at this time. OTOH outputting TAP is trivial and sets up a clear precedent of random test chatter belonging to stderr. This commit migrates only few tests to actually make use of the new features: roles is a basic client test, subsurface-shot is a client test that demonstrates the fixture array, and plugin-registry is a plugin test. The rest of the tests will be migrated later. Once all tests are migrated, we can remove the test-specific setup from meson.build, leaving only the actual build instructions in there. The not migrated tests and stand-alone tests suffer only a minor change: they no longer fork() for each TEST(), otherwise they keep running as before. Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
5 years ago
/** Fixture setup array record
*
* Helper to store the attributes of the data array passed in to
* DECLARE_FIXTURE_SETUP_WITH_ARG().
*
* \ingroup testharness_private
*/
struct fixture_setup_array {
const void *array;
size_t element_size;
int n_elements;
};
const struct fixture_setup_array *
fixture_setup_array_get_(void);
/** Test harness context
*
* \ingroup testharness
*/
struct weston_test_harness;
enum test_result_code
fixture_setup_run_(struct weston_test_harness *harness, const void *arg_);
/** Register a fixture setup function
*
* This registers the given (preferably static) function to be used for setting
* up any fixtures you might need. The function must have the signature:
*
* \code
* enum test_result_code func_(struct weston_test_harness *harness)
* \endcode
*
* The function must call one of weston_test_harness_execute_standalone(),
* weston_test_harness_execute_as_plugin() or
* weston_test_harness_execute_as_client() passing in the \c harness argument,
* and return the return value from that call. The function can also return a
* test_result_code on its own if it does not want to run the tests,
* e.g. RESULT_SKIP or RESULT_HARD_ERROR.
*
* The function will be called once to run all tests.
*
* \param func_ The function to be used as fixture setup.
*
* \ingroup testharness
*/
#define DECLARE_FIXTURE_SETUP(func_) \
enum test_result_code \
fixture_setup_run_(struct weston_test_harness *harness, \
const void *arg_) \
{ \
return func_(harness); \
}
/** Register a fixture setup function with a data array
*
* This registers the given (preferably static) function to be used for setting
* up any fixtures you might need. The function must have the signature:
*
* \code
* enum test_result_code func_(struct weston_test_harness *harness, typeof(array_[0]) *arg)
* \endcode
*
* The function must call one of weston_test_harness_execute_standalone(),
* weston_test_harness_execute_as_plugin() or
* weston_test_harness_execute_as_client() passing in the \c harness argument,
* and return the return value from that call. The function can also return a
* test_result_code on its own if it does not want to run the tests,
* e.g. RESULT_SKIP or RESULT_HARD_ERROR.
*
* The function will be called once with each element of the array pointed to
* by \c arg, so that all tests would be repeated for each element in turn.
*
* \param func_ The function to be used as fixture setup.
* \param array_ A static const array of arbitrary type.
*
* \ingroup testharness
*/
#define DECLARE_FIXTURE_SETUP_WITH_ARG(func_, array_) \
const struct fixture_setup_array * \
fixture_setup_array_get_(void) \
{ \
static const struct fixture_setup_array arr = { \
.array = array_, \
.element_size = sizeof(array_[0]), \
.n_elements = ARRAY_LENGTH(array_) \
}; \
return &arr; \
} \
\
enum test_result_code \
fixture_setup_run_(struct weston_test_harness *harness, \
const void *arg_) \
{ \
typeof(array_[0]) *arg = arg_; \
return func_(harness, arg); \
}
enum test_result_code
weston_test_harness_execute_as_client(struct weston_test_harness *harness,
const struct compositor_setup *setup);
enum test_result_code
weston_test_harness_execute_as_plugin(struct weston_test_harness *harness,
const struct compositor_setup *setup);
enum test_result_code
weston_test_harness_execute_standalone(struct weston_test_harness *harness);
#endif