If there are regressions dump the number of changes and the head
of the diff file so one doesn't have to search in the artifacts
for a quick look at the problem.
Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
When running locally and with softpipe and the hardware driver then one
would sometimes pickup the nam of a temporary file. Make sure to set the
test file name always.
Signed-off-by: Gert Wollny <gert.wollny@collabora.com>-ne
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>.
There is no reason to handle the test separately, at the high-level at
least.
If needed we can make the results stand out better in the [near] future.
Note: this removes the (now dead) --compare-previous. I've considered
keeping it, yet since it doesn't attribute the flip-flopping tests it's
better to remove it for now.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
There is nothing test specific to it, move it outside the piglit/deqp
hunks. The latter of which made a bit clearer with a switch statement.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
Fold the duplicates and kill the unreachable default statement. While
we're here add a comment that we need similar must-pass test files for
piglit.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
We somewhat consistently use the latter anyway.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
There has been no wildcard or driver specifics ever since the code was
introduced.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
Instead of manually looping over the file, pass the file to the runner.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
Move the validation before the tests, since it's completely agnostic.
Any empty lines have already been stripped away.
This effectively removes fairly confusing output such as:
Running unreliable deqp-gl32 on vtest-softpipe: Fail - passed 0/0 tests
It used to happen since we run piglit/deqp with empty list, when we did
not need to run it at all.
Aside: this has uncovered a piglit bug, where it will run all tests,
when the list is empty.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
Update the piglit ignore_tests and sed pattern to use the real format
which is path/to/test, with @ instead of /.
This will allow us to pass the file directly into piglit, just like we
do with deqp ;-)
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Elie Tournier <elie.tournier@collabora.com>
Unused and fairly convoluted for little reason. In practise we want a
full run when changes are made - fixing one test can easily regress
another.
Furthermore, this only works with deqp and does not throw a error when
attempted with piglit.
If needed we can reinstate this, it should handle both deqp and piglit.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Elie Tournier <elie.tournier@collabora.com>
All the supported backends are explicitly handled, drop the default
statement.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Elie Tournier <elie.tournier@collabora.com>
Note this affectively adds GALLIVM_PERF to vtest-llvmpipe, just like the
normal llvmpipe backend.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Elie Tournier <elie.tournier@collabora.com>
Add test script flags to run desktop GL 3.0, 3.1 and 3.2 tests.
Signed-off-by: Alexandros Frantzis <alexandros.frantzis@collabora.com>
Reviewed-by: <Gurchetan Singh gurchetansingh@chromium.org>
Reviewed-By: Gert Wollny <gert.wollny@collabora.com>
Reduce the number of test processes run by the CI to 1/8 of the number of
available processors.
Normally only run the build and sanity checks, and only run the test suites
for merge requests.
Also fix some string marker that seems to make problems.
Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
Signed-off-by: Dave Airlie <airlied@redhat.com>
To try this out, start a run like using multiple `--test`
arguments.
ci/run_test_suite.sh \
--deqp \
--host-gles \
--backend vtest-gpu \
--backend gpu \
--test TEST1 \
--test TEST2
Signed-off-by: Robert Foss <robert.foss@collabora.com>
This patch cleans up the CI path handling, and
allows for external path variables to be used.
Environment variables are now respected for MESA_PATH,
VIRGL_PATH and CTS_PATH.
Additionally these variables can be picked up from a
config file `~/.virgl-ci.conf`
$ cat ~/.virgl-ci.conf
MESA_PATH=/opt/virgl/mesa
VIRGL_PATH=/opt/virgl/virglrenderer
Or given as a local environment variable like below.
CTS_PATH=/opt/virgl/ ci/run_test_suite.sh --piglit --gles3 --backend softpipe
Signed-off-by: Robert Foss <robert.foss@collabora.com>
This commit introduces a few new parts:
- A new deqp & piglit runner
- New lists of previous results
- New ignore lists for unreliable tests
- A switch to using CTS provided test-list for deqp
The new runner supports a few different modes:
- Running a single test using the '-t' flag
- Comparing tests using the same driver with and without vtest using '--compare-backends'
- Running piglit or deqp suites using '--piglit' or '--deqp'
- Running suites of piglit or deqp tests using, '--gles2'/'--gles3','--gles31'
- Selecting a specific backend to use for example, '-b vtest-softpipe' or '-b softpipe'
For example:
# Run piglit gles2 tests using gpu&softpipe drivers through vtest
$ ci/run_test_suite.sh --piglit --gles2 --backend vtest-gpu --backend vtest-softpipe
# Run ignored&unreliable CTS gles3 tests using the softpipe driver
$ ci/run_test_suite.sh --deqp --gles3 --backend softpipe --unreliable
# Run CTS GLES2 using softpipe driver, and then compare results with results
# stored in ci/previous_results/, while ignoring known unreliable tests.
$ ci/run_test_suite.sh --piglit --gles2 --backend softpipe --compare-previous
Reviewed-By: Gert Wollny <gert.wollny@collabora.com>
Signed-off-by: Robert Foss <robert.foss@collabora.com>
Signed-off-by: Jakob Bornecrantz <jakob@collabora.com>
Gert Wollny
Emil Velikov
Emil Velikov
Alexandros Frantzis
Robert Foss