The new field in struct scalar_stat allows recording all tested values
into a file. This is intended to replace ad hoc dumping code like in
alpha-blending-test.c.
To make it easy to set up, also offer a helper to open a writable file
whose name consists of a custom prefix and test name.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
These are the last places in weston-test-client-helper.c where using
image_header_from() will reduce the code line count and simplify the
code a little.
No change in behaviour.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
I will be needing it in a new test, so let's share it.
For convenience, this also adds client back-pointer in struct surface so
that I don't need to pass client explicitly.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
skip() is a left-over from an old test harness design, the comment even
refers to automake. Calling skip() cannot do anything good anymore,
because it wouldn't print the skips in the TAP report, so it would
probably be considered a failure.
Delete this unused and nowadays incorrect function, so it doesn't
confuse people.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Store the pointer serial for events that provide one, so that it can be
used by tests to send requests that require it (e.g., setting the cursor
surface).
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
expect_protocol_error() ensures that the connection has failed in the
expected way. To satisfy ASan leak detection, we still need to tear down
everything, including call client_destroy().
client_destroy() needs to check that tear-down does not cause protocol
errors, so it does one last roundtrip that checks that is succeeds. But
if the connection is already down in an expected way, this roundtrip
cannot succeed and must be skipped.
Also moves the roundtrip under 'if (wl_display)', assuming the 'if' is
there for a reason - but obviously that reason was never used as it
would have crashed.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Fixes a leak found by ASan in alpha-blending-test.
Direct leak of 160 byte(s) in 2 object(s) allocated from:
#0 0x7f511fe11518 in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
#1 0x7f511fc76373 in zalloc ../../git/wayland/src/wayland-private.h:232
#2 0x7f511fc76373 in proxy_create ../../git/wayland/src/wayland-client.c:422
#3 0x7f511fc79dcc in create_outgoing_proxy ../../git/wayland/src/wayland-client.c:651
#4 0x7f511fc79dcc in wl_proxy_marshal_array_constructor_versioned ../../git/wayland/src/wayland-client.c:736
#5 0x7f511fc7b17b in wl_proxy_marshal_constructor_versioned ../../git/wayland/src/wayland-client.c:873
#6 0x5583e5348f43 in wl_registry_bind /home/pq/local/include/wayland-client-protocol.h:1165
#7 0x5583e534cfbe in handle_global ../../git/weston/tests/weston-test-client-helper.c:800
#8 0x7f511f34b8ed in ffi_call_unix64 (/lib/x86_64-linux-gnu/libffi.so.6+0x68ed)
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
This refactors a new function verify_image() out of
verify_screen_content().
verify_image() will be useful with a test that verifies a screenshot
against a reference image but also wants to do additional testing on the
screenshot.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Const has documentary value saying the code will not modify the
parameter contents. Everything that can be const, should be const.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Do not write out PNG files for successful screenshot tests. It clutters
the build directory, but the biggest reason is to keep the CI artifacts
smaller.
internal-screenshot test still writes a PNG, it's good to keep one PNG
written so that we exercise the PNG writing code always.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Until now we had two different protocol extensions: one for the
test suite screenshooter and other for the screenshooter client.
As they are identical and this won't change, make the test suite
use the same protocol that the screenshooter client uses.
Besides the cleanup, this change will also allow us to use the
DRM writeback screenshooter in the test suite, as the test suite
implementation is hardcoded to use a renderer based screenshooter.
Signed-off-by: Leandro Ribeiro <leandro.ribeiro@collabora.com>
The 'struct test' has a field 'int buffer_copy_done', but it
is in a fact a boolean. Change it to 'bool buffer_copy_done'.
Signed-off-by: Leandro Ribeiro <leandro.ribeiro@collabora.com>
The wl_drm protocol is not being used by the test client. So
remove 'bool has_wl_drm' from 'struct client' and also the
branch that initializes this variable in handle_global().
Signed-off-by: Leandro Ribeiro <leandro.ribeiro@collabora.com>
There will be a new test program using viewports and would like to share this
bit of code.
There are two behavioral changes:
- Compositor wp_viewporter interface version is no longer checked.
- client_create_viewport() does not leak the viewporter object.
test_viewporter_double_create needs to call bind_to_singleton_global() itself
so that the viewporter object still exists when the error event arrives.
Otherwise error verification fails.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
When a test fails and it produces a difference image, also compute the min/max
per-channel signed difference statistics. These numbers can be used to adjust
the fuzz needed for fuzzy_match_pixels() to pass. Otherwise one would have to
manually inspect the reference and result images and figure out the values.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
This goes through all output transforms with two different buffer transforms
and verifies the visual output against reference images.
This commit introduces a new test input image 'basic-test-card.png'. It is a
small image with deliberately odd and indivisible dimensions to provoke bad
assumptions about image sizes. It contains red, green and blue areas which are
actually text that makes it very obvious if you have e.g. color channels
swapped. It has a white thick circle to highlight aspect ratio issues, and an
orange cross to show a mixed color. The white border is for contrast and a 1px
wide detail. The whole design makes it clear if the image happens to be rotated
or flipped in any way.
The image has one pixel wide transparent border so that bilinear sampling
filter near the edges of the image would produce the same colors with both
Pixman- and GL-renderers which handle the out-of-image samples fundamentally
differently: Pixman assumes (0, 0, 0, 0) samples outside of the image, while
GL-renderer clamps sample coordinates to the edge essentially repeating the
edge pixels.
It would have been "easy" to create a full matrix of
every output scale & transform x every buffer scale & transform, but that
would have resulted in 2 renderers * 8 output transforms * 3 output scales *
8 buffer transforms * 3 buffer scales = 1152 test cases that would have all
ran strictly serially because our test harness has no parallelism inside one
test program. That would have been slow to run, and need a lot more reference
images too.
Instead, I chose to iterate separately through all output scales & transforms
(this patch) and all buffer scales & transforms (next patch). This limits the
number of test cases in this patch to 56, and allows the two test programs to
run in parallel.
I did not even pick all possible scale & transform combinations here, but just
what I think is a representative sub-set to hopefully exercise all the code
paths.
https://gitlab.freedesktop.org/wayland/weston/issues/52
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Running with Mesa 20.1.0-devel (git-c7617d8908) GL renderer:
Radeon RX 550 Series (POLARIS11, DRM 3.27.0, 4.19.0-2-amd64, LLVM 8.0.1)
I found output-tranform test (a future patch) to produce exactly this much more
difference between Pixman and GL rendererers.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
It turns out that if the client is not explicitly destroyed, it will remain
connected until the compositor shuts down because there is no more a client
process that would terminate.
Usually this is not a problem, but if a test file has multiple screenshooting
tests, the windows from earlier tests in the file will remain on screen. That
is not wanted, hence implement client destruction.
To properly destroy a client, we also need a list of outputs. They used to be
simply leaked. This does not fix wl_registry.global_remove for wl_outputs, that
is left for a time when a test will actually need that.
This patch makes only ivi-shell-app test use the new client_destroy() to show
that it actually works. The added log scopes prove it: destroy requests get
sent. Sprinkling client_destroy() around in all other tests is left for a time
when it is actually necessary.
ivi-shell-app is a nicely simple test doing little else, hence I picked it.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Allow the reference image to be NULL or missing so that it does not even
attempt to load a reference image or compare it. You cannot just point the
reference image to an arbitrary image because the comparison functions can
abort due to size mismatch. This makes bootstrapping new tests easier when you
do not yet have a reference image.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
The old name felt too... short.
The return type is changed to bool; fits better for a success/failure.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
This will be useful in more tests.
No changes to the code, aside from dropping one 'static'.
Copyright 2017 is taken from git-blame of the moved code.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
The fuzzy range will be used with GL-renderer testing, as it may produce
slightly different images than Pixman-renderer yet still correct results.
Such allowed differences are due to different rounding.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Instead of relying on Meson setting up environment so that Weston and tests
find all their files, build those values into the tests. This way one can
execute a test program successfully wihtout Meson, simply by running it.
The old environment variables are still honoured if set. This might change in
the future.
Baking the source or build directory paths into the tests should not regress
reproducible builds, because the binaries where test-config.h values are used
will not be installed.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
When we move on to TAP, stdout will be reserved for TAP and stderr is for free
chatter. Set up an example that tests should use testlog() instead of fprintf
or printf to chat in the right place.
Most statements were already printing to stderr, so this just makes then a
little shorter. There are also some statements that printed to stdout and are
now corrected.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
This reverts 50b7b70835.
We didn't make Meson create a logs directory, so writing the images fails
because the directory does not exist. If you run a test without Meson, there is
even less expectation that it would write somewhere else than CWD by default.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
It is a public installed header used by libweston.h.
See "Rename compositor.h to libweston/libweston.h" for rationale.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
There is no automake anymore, I suppose it is ninja that handles it now.
There are still a couple references to automake left to point out where the
conventions originated, e.g. the exit code 77.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
Introduce helper test code to implement the client side of the
input_timestamps_unstable_v1 protocol. This helper will be used in
upcoming commits to test the server side implementation of the protocol
in libweston.
The input_timestamps_unstable_v1 protocol was introduced in version 1.13
of wayland-protocols, so this commit updates the version dependency in
configure.ac accordingly.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
The current test client code waits for all wl_seat globals to arrive
before checking them and deciding which one is the test seat global to
use for the input object. Test code that needs to add/remove test seats
would have to call the client_set_input() function for any seat changes
to take effect. Although we could allow this by making
client_set_input() public, we would be exposing unecessary
implementation details.
This commit applies any seat changes immediately upon arrival of the
seat name, freeing test code from needing to call extra functions like
client_set_input(). To achieve this the call to input_data_devices() is
moved from client_set_input() to the seat name event handler.
This commit also moves the check that all seats have names to an
explicit test. To support this test, inputs corresponding to non-test
seats are not destroyed (unless their seat global is removed), as
was previously the case.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
The current test client code completely ignores removal of globals.
This commit updates the code to properly handle removal of globals in
general, and of seat globals in particular. This ensures that the test
client objects are in sync with the server and any relevant resources
are released accordingly.
This update will be used by upcoming tests to check that seat removal
and re-addition is working properly.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Add test to verify the server correctly emits pointer axis events. This
requires updating the weston-test protocol with a new request for
pointer axis events.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Add test to verify that the server correctly sets the timestamps of
touch events. This requires updating the weston-test protocol with a new
request for touch events.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Add test to verify that the server correctly sets the timestamps of
keyboard key events. This requires updating the weston-test protocol to
support passing key event timestamps.
simple_keyboard_test now uses the create_client_with_keyboard_focus()
helper function which changes the initial state of the surface to be
focused. This leads to one additional iteration of the test loop when
starting, during which the surface is deactivated, i.e., loses focus.
After this initial iteration the test continues as before.
Furthermore, simple_keyboard_test now uses the send_key() helper
function which performs a roundtrip internally. To account for this, the
client_roundtrip() function is now directly called in the loop only when
it is still required, i.e., when deactivating the surface.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Enhance the existing pointer motion and button event tests to
additionally verify the event timestamps. This requires updating the
weston-test protocol to support passing motion and button event
timestamps.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
This doesn't attach a buffer to the surface. This is needed for the
next commit, where we have a test case with a surface that doesn't
have a buffer attached.
Signed-off-by: Emilio Pozuelo Monfort <emilio.pozuelo@collabora.co.uk>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Logs is where we write all our custom test logs, let's also put the
screenshots in the same place by default from cluttering the base
directory.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Emilio Pozuelo Monfort <emilio.pozuelo@collabora.co.uk>
Reviewed-by: Micah Fedke <micah.fedke@collabora.co.uk>
Reviewed-by: Quentin Glidic <sardemff7+git@sardemff7.net>
buffer-count was introduced in line with a Mesa change which forced
an earlier block on frame events to try to enforce double-buffering
where available.
The Mesa change has since been reverted (Mesa commit 9ca6711faa), as
this had unpleasant interactions with buffer_age in particular, so this
test is no longer valid.
Additionally, it only worked on backends which initialised EGL (not
headless-backend, where tests generally run), which can be flaky due to
initialisation races. Not only that, but on the DRM backend, we can
legitimately enter triple-buffering due to promoting the surface to a
hardware plane, skipping GPU composition.
In light of all this, just remove the test.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Derek Foreman <derekf@osg.samsung.com>
Useful for pointing out where the image comparisons fail.
Internal-screenshot-test is modified to save the visualization if the
test fails.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Screenshooting does not involve creating a wl_surface, so using struct
surface is superfluous.
Return a struct buffer instead. It could have been just a
pixman_image_t, but setting up proper destruction would be a bit more
work. Should not hurt to keep the wl_buffer around until the user is
ready to free the image.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
This rewrites write_surface_as_png() into write_image_as_png(), which
operates on a pixman_image_t instead of a struct surface.
This is part of the migration to use pixman_image_t everywhere without
superfluous parameters/members.
Now the image saving handles more than just ARGB32 format, presumably.
At least it does not assume everything is always ARGB32.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
This rewrites load_surface_from_png() to load_image_from_png(), to
return a pixman_image_t instead of a struct surface.
A loaded image has no need for wl_buffer or wl_surface or any of the
associated attributes. This is part of unifying to make everything use
pixman_image_t.
cairo_surface_flush() is added, because Cairo documentation for
cairo_image_surface_get_data() says you have to flush after drawing,
before using the data. It is unclear if loading a PNG counts as drawing,
so stay on the safe side.
load_image_from_png() now pays attention to the pixel format returned by
Cairo, which seems to come out as CAIRO_FORMAT_RGB24 in
internal-screenshot-test, not as CAIRO_FORMAT_ARGB32 as expected. I do
not know if Cairo actually guarantees the x8/a8 channel to be 0xff for
RGB24, but better to not trust it. Therefore the image is explicitly
converted to a8r8g8b8 as needed. This also adds support for loading A8
and RGB16_565 images, provided that Cairo delivers them.
The cairo surface is now wrapped directly into a pixman_image_t. If the
pixel format conversion is not needed, this eliminates a copy of the
image data. The Cairo surface will get automatically destroyed with the
Pixman image.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
check_surfaces_geometry() is removed as it was not used by anything, and
unlikely would be.
check_surfaces_equal() is merged into check_surfaces_match_in_clip(),
passing a NULL clip means to compare whole images.
check_surfaces_match_in_clip() is converted to work on pixman_image_t
instead of struct surface. The function is only concerned about
comparing images in memory, and does not care about a wl_buffer or a
wl_surface.
The verbosity of image comparisons is greatly reduced. An image mismatch
no longer prints a flood of raw pixel values. This will be replaced
later with a function writing out an error image instead.
Degenerate comparisons are no longer accepted, be that clip outside
images or zero area. Those are an indication of a programmer error.
The pixel format assumptions are made more visible in the code.
A new internal helper image_check_get_roi() computes and verifies the
area to be compared. Image iterator helper makes it simpler to write
manual pixel-poking loops.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Change create_shm_buffer() to handle any pixel format known to Pixman.
Presumably in the future we might want to test e.g. RGB565 content with
screenshot tests.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
No users remain outside the file. This will allow to fix the assumptions
in the function.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
We are growing more tests that need to handle buffers, both just images
and wl_buffers. Particularly the screenshooting facility needs these.
Currently everything is in struct surface, which contains more than we
need. It is a bit messy.
Create a new struct buffer to encapsulate the image representation, the
wl_buffer, and enough information to tear it all down (munmap) so we
don't have to leak everything. Some tests might start doing things in
loops, and leaking would accumulate.
Instead of inventing our own image representation, use pixman_image_t.
It is a well-tested library worth using, and we already rely on it in
other places.
This makes the tests depend on Pixman, which requires the fix for
building buffer-count, which would otherwise not find pixman.h.
The new create_shm_buffer_a8r8g8b8() creates an image with an explicit
format, and pixman_image_t keeps track of it. And stride and size and
data. This implementation is still a little hacky due to calling
create_shm_buffer().
A very new thing is buffer_destroy(). Previously we didn't really free
any buffers. It is not a problem when the process will exit soon anyway,
but it may become a problem if tests start iterating things.
Manual memset() on a image is converted to a pixman action, just to show
how to do it properly with pixman.
Stride and pixel format assumptions still linger all around, but those
are for another patch.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Pekka Paalanen
Alexandros Frantzis
Maxime Roussin-Bélanger
Leandro Ribeiro
Emilio Pozuelo Monfort
Pekka Paalanen
Daniel Stone
Jussi Kukkonen