compositor: forbid sub-surface nesting loops

The only way to create a sub-surface loop by recursive nesting is to
make the main surface (which does not have a role) a sub-surface of any
of its sub-surfaces. All other cases should already be cought.

This change checks for that exact case, and sends a protocol error.

Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>

src/compositor.c

@@ -2295,6 +2295,14 @@ subcompositor_get_subsurface(struct wl_client *client,
 		return;
 	}
 
+	if (weston_surface_get_main_surface(parent) == surface) {
+		wl_resource_post_error(resource,
+			WL_SUBCOMPOSITOR_ERROR_BAD_SURFACE,
+			"%s%d: wl_surface@%d is an ancestor of parent",
+			where, id, surface_resource->object.id);
+		return;
+	}
+
 	/* make sure the parent is in its own list */
 	if (wl_list_empty(&parent->subsurface_list)) {
 		if (!weston_subsurface_create_for_parent(parent)) {