pixman-renderer: half-fix bilinear sampling on edges

When weston-desktop-shell uses a solid color for the wallpaper, it creates a
1x1 buffer and uses wp_viewport to scale that up to fullscreen. It's a very
nice memory saving optimization.

If you also have output scale != buffer scale, it means pixman-renderer chooses
bilinear filter. Arguably pixman-renderer should choose bilinear filter also
when wp_viewport implies scaling, but it does not. As w-d-s always sets buffer
scale from output scale, triggering the bilinear filter needs some effort.

What happens when you sample with bilinear filter from a 1x1 buffer, stretching
it to cover a big area? Depends on the repeat mode. The default repeat mode is
NONE, which means that samples outside of the buffer come out as (0,0,0,0).
Bilinear filter makes it so that every sampling point on the 1x1 buffer except
the very center is actually a mixture of the pixel value and (0,0,0,0). The
resulting color is no longer opaque, but the renderer and damage tracking
assume it is. This leads to the issue 373.

Fix half of the issue by using repeat mode PAD which corresponds to OpenGL
CLAMP_TO_EDGE. GL-renderer already uses CLAMP_TO_EDGE always.

This is only a half-fix, because composite_clipped() cannot actually be fixed.
It relies on repeat mode NONE to work. It would need a whole different approach
to rendering potentially non-axis-aligned regions exactly like GL-renderer.

Fixes: https://gitlab.freedesktop.org/wayland/weston/issues/373

Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>

libweston/pixman-renderer.c

@@ -229,6 +229,12 @@ composite_whole(pixman_op_t op,
 	pixman_image_set_transform(src, transform);
 	pixman_image_set_filter(src, filter, NULL, 0);
 
+	/* bilinear filtering needs the equivalent of OpenGL CLAMP_TO_EDGE */
+	if (filter == PIXMAN_FILTER_NEAREST)
+		pixman_image_set_repeat(src, PIXMAN_REPEAT_NONE);
+	else
+		pixman_image_set_repeat(src, PIXMAN_REPEAT_PAD);
+
 	pixman_image_composite32(op, src, mask, dest,
 				 0, 0, /* src_x, src_y */
 				 0, 0, /* mask_x, mask_y */
@@ -254,9 +260,17 @@ composite_clipped(pixman_image_t *src,
 	void *src_data;
 	int i;
 
-	/* Hardcoded to use PIXMAN_OP_OVER, because sampling outside of
+	/*
+	 * Hardcoded to use PIXMAN_OP_OVER, because sampling outside of
 	 * a Pixman image produces (0,0,0,0) instead of discarding the
 	 * fragment.
+	 *
+	 * Also repeat mode must be PIXMAN_REPEAT_NONE (the default) to
+	 * actually sample (0,0,0,0). This may cause issues for clients that
+	 * expect OpenGL CLAMP_TO_EDGE sampling behavior on their buffer.
+	 * Using temporary 'boximg' it is not possible to apply CLAMP_TO_EDGE
+	 * correctly with bilinear filter. Maybe trapezoid rendering could be
+	 * the answer instead of source clip?
 	 */
 
 	dest_width = pixman_image_get_width(dest);