scripts/mtest2make.py | 11 +++++++++-- tests/qemu-iotests/meson.build | 2 +- 2 files changed, 10 insertions(+), 3 deletions(-)
make check is currently by default running tests tagged
"slow". Running make check with no SPEED variable set should be
equivalent to SPEED=quick and run the meson 'quick' test setup, which
explicitly excludes slow tests:
add_test_setup('quick', exclude_suites: ['slow', 'thorough'] ...
The problem is that the mtest2make.py script is effectively overriding
the "meson test --setup" setting by using --suite which takes
precedence over --setup according to meson documentation.
Fix by excluding suites directly with the "--no-suite" option while
generating Makefile.mtest.
Although the 'exclude_suites' in meson.build are useless for make
check, they still help when running 'meson test' from the
command-line, so keep them around.
For the configure line I'm using:
make check goes from 1100 to 996 tests
make SPEED=slow check stays at 2158 tests
make SPEED=thorough check stays at 2279 tests
(the rest of the test targets look sane, but maintainers please
double-check)
Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
- With this change we obviously lose coverage in make check. I suggest
we use this opportunity to review what gets put in the quick
suite.
For migration, I want to keep only the smoke tests in make
check. I'll send a patch adding them back if this change is
accepted.
- Some of the suffixes from functional tests could probably be removed
and rely on having the tests in the proper suite instead. I.e. use
'slow' instead of 'foobar_slow'.
---
scripts/mtest2make.py | 11 +++++++++--
tests/qemu-iotests/meson.build | 2 +-
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 383ea68b16..a61a83f775 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -26,6 +26,11 @@ def names(self, base):
.speed.slow = $(sort $(filter-out %-thorough, $1))
.speed.thorough = $(sort $1)
+.speed_exclude_suites.quick = --no-suite slow --no-suite thorough \
+ --no-suite optional
+.speed_exclude_suites.slow = --no-suite thorough
+.speed_exclude_suites.thorough =
+
TIMEOUT_MULTIPLIER ?= 1
.mtestargs = --no-rebuild -t $(TIMEOUT_MULTIPLIER)
ifneq ($(SPEED), quick)
@@ -34,7 +39,8 @@ def names(self, base):
.mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword -j1 $(filter -j%, $(MAKEFLAGS)))))
.check.mtestargs = $(MTESTARGS) $(.mtestargs) $(if $(V),--verbose,--print-errorlogs) \
- $(foreach s, $(sort $(.check.mtest-suites)), --suite $s)
+ $(foreach s, $(sort $(.check.mtest-suites)), --suite $s) \
+ $(.speed_exclude_suites.$(SPEED))
.bench.mtestargs = $(MTESTARGS) $(.mtestargs) --benchmark --verbose \
$(foreach s, $(sort $(.bench.mtest-suites)), --suite $s)''')
@@ -102,7 +108,8 @@ def emit_suite(name, suite, prefix):
targets += f' {prefix} {prefix}-report.junit.xml'
print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
# for the "base" suite possibly add FOO-slow and FOO-thorough
- print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names})")
+ print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names}) "
+ ".speed_exclude_suites.$(SPEED)")
print(f'endif')
targets = {t['id']: [os.path.relpath(f) for f in t['filename']]
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 8857f25f29..34cfcdc53b 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -68,7 +68,7 @@ foreach driver, speed: qemu_iotests_drivers
endif
# Every driver gets put in the driver specific suite
- suites = ['block-' + driver + '-optional']
+ suites = ['block-' + driver + '-optional', 'optional']
# Any driver tagged quick or slow also gets added to slow
# otherwise its tagged thorough
if speed != 'thorough'
--
2.53.0On Fri, Sep 18, 2026 at 05:31:52PM -0300, Fabiano Rosas wrote:
> make check is currently by default running tests tagged
> "slow". Running make check with no SPEED variable set should be
> equivalent to SPEED=quick and run the meson 'quick' test setup, which
> explicitly excludes slow tests:
Can you give an example of some tests that you think should not
be running.
I just run 'make check' and looked at two examples
tests/unit/test-crypto-pbkdf.c skips registering some tests
unless "SPEED=slow" and that correctly skipped them.
tests/qemu-iotests/meson.build skips running 'raw' format tests
unless SPEED=slow, and that appears to be correctly working,
only running qcow2 format tests by default.
>
> add_test_setup('quick', exclude_suites: ['slow', 'thorough'] ...
>
> The problem is that the mtest2make.py script is effectively overriding
> the "meson test --setup" setting by using --suite which takes
> precedence over --setup according to meson documentation.
>
> Fix by excluding suites directly with the "--no-suite" option while
> generating Makefile.mtest.
>
> Although the 'exclude_suites' in meson.build are useless for make
> check, they still help when running 'meson test' from the
> command-line, so keep them around.
>
> For the configure line I'm using:
>
> make check goes from 1100 to 996 tests
> make SPEED=slow check stays at 2158 tests
> make SPEED=thorough check stays at 2279 tests
>
> (the rest of the test targets look sane, but maintainers please
> double-check)
>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> - With this change we obviously lose coverage in make check. I suggest
> we use this opportunity to review what gets put in the quick
> suite.
>
> For migration, I want to keep only the smoke tests in make
> check. I'll send a patch adding them back if this change is
> accepted.
>
> - Some of the suffixes from functional tests could probably be removed
> and rely on having the tests in the proper suite instead. I.e. use
> 'slow' instead of 'foobar_slow'.
> ---
> scripts/mtest2make.py | 11 +++++++++--
> tests/qemu-iotests/meson.build | 2 +-
> 2 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
> index 383ea68b16..a61a83f775 100644
> --- a/scripts/mtest2make.py
> +++ b/scripts/mtest2make.py
> @@ -26,6 +26,11 @@ def names(self, base):
> .speed.slow = $(sort $(filter-out %-thorough, $1))
> .speed.thorough = $(sort $1)
>
> +.speed_exclude_suites.quick = --no-suite slow --no-suite thorough \
> + --no-suite optional
> +.speed_exclude_suites.slow = --no-suite thorough
> +.speed_exclude_suites.thorough =
> +
> TIMEOUT_MULTIPLIER ?= 1
> .mtestargs = --no-rebuild -t $(TIMEOUT_MULTIPLIER)
> ifneq ($(SPEED), quick)
> @@ -34,7 +39,8 @@ def names(self, base):
> .mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword -j1 $(filter -j%, $(MAKEFLAGS)))))
>
> .check.mtestargs = $(MTESTARGS) $(.mtestargs) $(if $(V),--verbose,--print-errorlogs) \
> - $(foreach s, $(sort $(.check.mtest-suites)), --suite $s)
> + $(foreach s, $(sort $(.check.mtest-suites)), --suite $s) \
> + $(.speed_exclude_suites.$(SPEED))
> .bench.mtestargs = $(MTESTARGS) $(.mtestargs) --benchmark --verbose \
> $(foreach s, $(sort $(.bench.mtest-suites)), --suite $s)''')
>
> @@ -102,7 +108,8 @@ def emit_suite(name, suite, prefix):
> targets += f' {prefix} {prefix}-report.junit.xml'
> print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
> # for the "base" suite possibly add FOO-slow and FOO-thorough
> - print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names})")
> + print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names}) "
> + ".speed_exclude_suites.$(SPEED)")
> print(f'endif')
>
> targets = {t['id']: [os.path.relpath(f) for f in t['filename']]
> diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
> index 8857f25f29..34cfcdc53b 100644
> --- a/tests/qemu-iotests/meson.build
> +++ b/tests/qemu-iotests/meson.build
> @@ -68,7 +68,7 @@ foreach driver, speed: qemu_iotests_drivers
> endif
>
> # Every driver gets put in the driver specific suite
> - suites = ['block-' + driver + '-optional']
> + suites = ['block-' + driver + '-optional', 'optional']
> # Any driver tagged quick or slow also gets added to slow
> # otherwise its tagged thorough
> if speed != 'thorough'
> --
> 2.53.0
>
With regards,
Daniel
--
|: https://berrange.com ~~ https://hachyderm.io/@berrange :|
|: https://libvirt.org ~~ https://entangle-photo.org :|
|: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
Thomas Huth <thuth@redhat.com> writes:
> On 18/09/2026 22.31, Fabiano Rosas wrote:
>> make check is currently by default running tests tagged
>> "slow".
> That's a regression, it did not used to be that way. Thus could you please
> add a "Fixes:" tag to your patch?
>
So, in a sense, it's this commit:
f4b2607228 ("tests: tag slow tests with 'slow' suite for easy filtering").
It added some qtests to the 'slow' suite. But the problem actually has
always existed, it's just that we usually put tests not only the 'slow'
suite but also some 'foo-slow'. The mtest2make.py script then omits the
'slow' suite...
def process_tests(test, targets, suites):
...
if s == 'slow' or s == 'thorough':
continue
...while the filtering omits the foo-slow:
.speed.quick = $(foreach s,$(sort $(filter-out %-slow %-thorough,
$1)), --suite $s)
But using the 'exclude_suites' in the main meson.build never did
anything. So the 'slow' suite is actually useless aside from the fact
that it carries the environment variables along.
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Fri, Sep 18, 2026 at 05:31:52PM -0300, Fabiano Rosas wrote:
>> make check is currently by default running tests tagged
>> "slow". Running make check with no SPEED variable set should be
>> equivalent to SPEED=quick and run the meson 'quick' test setup, which
>> explicitly excludes slow tests:
>
> Can you give an example of some tests that you think should not
> be running.
>
Sorry, I could have worded this patch a bit better. I'm aiming
specifically at the (presumed) assumption that 'excluded_suites' in
meson.build is sufficient to stop a test from running if that test is in
one of the suites listed there.
For instance, all tests listed in the tests/qtest/meson.build
'slow_qtests' variable are affected:
$ ../configure
--target-list=x86_64-softmmu,i386-softmmu,x86_64-linux-user,aarch64-softmmu,arm-softmmu,ppc64-softmmu,ppc-softmmu,s390x-softmmu,riscv64-softmmu,aarch64-linux-user,mips64-softmmu,loongarch64-softmmu
--enable-plugins --enable-modules --enable-werror --enable-debug
--disable-docs --enable-slirp --disable-containers
$ make -j$(nproc) check | grep "+slow"
1/1100 qemu:qtest+qtest-s390x+slow / qtest-s390x/qom-test
2/1100 qemu:qtest+qtest-mips64+slow / qtest-mips64/qom-test
3/1100 qemu:qtest+qtest-loongarch64+slow / qtest-loongarch64/qom-test
4/1100 qemu:qtest+qtest-i386+slow / qtest-i386/qom-test
<plus a bunch>
> I just run 'make check' and looked at two examples
>
> tests/unit/test-crypto-pbkdf.c skips registering some tests
> unless "SPEED=slow" and that correctly skipped them.
>
> tests/qemu-iotests/meson.build skips running 'raw' format tests
> unless SPEED=slow, and that appears to be correctly working,
> only running qcow2 format tests by default.
>
Right, make SPEED=speed check works in the sense that:
1) it sets the SPEED properly
2) it excludes 'foo-slow' and 'slow' tests ("manually" via mtest2make.py)
however,
3) it does not exclude _the same test_ if it's part of another suite.
On 18/09/2026 22.31, Fabiano Rosas wrote: > make check is currently by default running tests tagged > "slow". That's a regression, it did not used to be that way. Thus could you please add a "Fixes:" tag to your patch? Thanks, Thomas
© 2016 - 2026 Red Hat, Inc.