From: Thomas Huth <thuth@redhat.com>
When running the tests in thorough mode, e.g. with:
make -j$(nproc) check SPEED=thorough
we currently always get a huge amount of total tests that the test
runner tries to execute (2457 in my case), but a big bunch of them are
only skipped (1099 in my case, meaning that only 1358 got executed).
This happens because we try to run the whole set of iotests for multiple
image formats while a lot of the tests can only run with one certain
format only and thus are marked as SKIP during execution. This is quite a
waste of time during each test run, and also unnecessarily blows up the
displayed list of executed tests in the console output.
Thus let's try to be a little bit smarter: If the "check" script is run
with "-n" and an image format switch (like "-qed") at the same time (which
is what we do already for discovering the tests for the meson test runner),
only report the tests that likely support the given format instead of
providing the whole list of all tests. We can determine whether a test
supports a format or not by looking at the lines in the file that contain
a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
of course, but it is good enough for running the iotests via "make
check-block" - I double-checked that the list of executed tests does not
get changed by this patch, it's only the tests that are skipped anyway that
are now not run anymore.
This way the amount of total tests drops from 2457 to 1432 for me, and
the amount of skipped tests drops from 1099 to just 74 (meaning that we
still properly run 1432 - 74 = 1358 tests as we did before).
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
1 file changed, 32 insertions(+), 3 deletions(-)
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 545f9ec7bdd..fb3d8f95b94 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -81,7 +81,7 @@ def make_argparser() -> argparse.ArgumentParser:
g_env.add_argument('-i', dest='aiomode', default='threads',
help='sets AIOMODE environment variable')
- p.set_defaults(imgfmt='raw', imgproto='file')
+ p.set_defaults(imgproto='file')
format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2',
'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg', 'vvfat']
@@ -136,12 +136,41 @@ def make_argparser() -> argparse.ArgumentParser:
return p
+def dry_run_list(test_dir, imgfmt, testlist):
+ for t in testlist:
+ if not imgfmt:
+ print('\n'.join([os.path.basename(t)]))
+ continue
+ # If a format has been given, we look for the "supported_fmt"
+ # and the "unsupported_fmt" lines in the test and try to find out
+ # whether the format is supported or not. This is only heuristics,
+ # but it should be good enough for "make check-block"
+ with open(os.path.join(test_dir, t), 'r', encoding='utf-8') as fh:
+ supported = True
+ check_next_line = False
+ for line in fh:
+ if 'unsupported_fmt' in line:
+ if imgfmt in line:
+ supported = False
+ break
+ elif 'supported_fmt' in line or check_next_line:
+ supported = imgfmt in line or 'generic' in line
+ check_next_line = not ']' in line and \
+ ('supported_fmts=[' in line or check_next_line)
+ if supported or not check_next_line:
+ break
+ if supported:
+ print('\n'.join([os.path.basename(t)]))
+
+
if __name__ == '__main__':
args = make_argparser().parse_args()
+ image_format = args.imgfmt or 'raw'
+
env = TestEnv(source_dir=args.source_dir,
build_dir=args.build_dir,
- imgfmt=args.imgfmt, imgproto=args.imgproto,
+ imgfmt=image_format, imgproto=args.imgproto,
aiomode=args.aiomode, cachemode=args.cachemode,
imgopts=args.imgopts, misalign=args.misalign,
debug=args.debug, valgrind=args.valgrind,
@@ -185,7 +214,7 @@ if __name__ == '__main__':
if args.dry_run:
with env:
- print('\n'.join([os.path.basename(t) for t in tests]))
+ dry_run_list(env.source_iotests, args.imgfmt, tests)
else:
with TestRunner(env, tap=args.tap,
color=args.color) as tr:
--
2.51.0Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> From: Thomas Huth <thuth@redhat.com>
>
> When running the tests in thorough mode, e.g. with:
>
> make -j$(nproc) check SPEED=thorough
>
> we currently always get a huge amount of total tests that the test
> runner tries to execute (2457 in my case), but a big bunch of them are
> only skipped (1099 in my case, meaning that only 1358 got executed).
> This happens because we try to run the whole set of iotests for multiple
> image formats while a lot of the tests can only run with one certain
> format only and thus are marked as SKIP during execution. This is quite a
> waste of time during each test run, and also unnecessarily blows up the
> displayed list of executed tests in the console output.
>
> Thus let's try to be a little bit smarter: If the "check" script is run
> with "-n" and an image format switch (like "-qed") at the same time (which
> is what we do already for discovering the tests for the meson test runner),
> only report the tests that likely support the given format instead of
> providing the whole list of all tests. We can determine whether a test
> supports a format or not by looking at the lines in the file that contain
> a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
> of course, but it is good enough for running the iotests via "make
> check-block" - I double-checked that the list of executed tests does not
> get changed by this patch, it's only the tests that are skipped anyway that
> are now not run anymore.
>
> This way the amount of total tests drops from 2457 to 1432 for me, and
> the amount of skipped tests drops from 1099 to just 74 (meaning that we
> still properly run 1432 - 74 = 1358 tests as we did before).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
> 1 file changed, 32 insertions(+), 3 deletions(-)
> +def dry_run_list(test_dir, imgfmt, testlist):
> + for t in testlist:
> + if not imgfmt:
> + print('\n'.join([os.path.basename(t)]))
> + continue
> + # If a format has been given, we look for the "supported_fmt"
> + # and the "unsupported_fmt" lines in the test and try to find out
> + # whether the format is supported or not. This is only heuristics,
> + # but it should be good enough for "make check-block"
I'm not completely sure if this is a good idea at all, but I think we
should at least mention the possible surprising cases where the
heuristics fails in this comment.
One thing is that '_unsupported_fmt qcow' will also disable the test
case for qcow2. 181 would almost run into this, but because it has
'_supported_fmt generic' first (which is an order not required by
anything else) and you stop after seeing this line, the test as written
currently is still returned (for both qcow and qcow2).
Actually, I suspect that the '_supported_fmt generic' tests that are
then followed by an '_unsupported_fmt' line are the majority of the
still skipped cases you see. (We don't have to do anything about it,
just an observation.)
There are other theoretical scenarios, like multiple options on a single
line in Python, but I think in practice you'll never combine block
driver names that this would skip something it shouldn't skip (apart
from the same 'qcow2' and 'qcow' problem).
Kevin
On 02/10/2025 16.04, Kevin Wolf wrote:
> Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
>> From: Thomas Huth <thuth@redhat.com>
>>
>> When running the tests in thorough mode, e.g. with:
>>
>> make -j$(nproc) check SPEED=thorough
>>
>> we currently always get a huge amount of total tests that the test
>> runner tries to execute (2457 in my case), but a big bunch of them are
>> only skipped (1099 in my case, meaning that only 1358 got executed).
>> This happens because we try to run the whole set of iotests for multiple
>> image formats while a lot of the tests can only run with one certain
>> format only and thus are marked as SKIP during execution. This is quite a
>> waste of time during each test run, and also unnecessarily blows up the
>> displayed list of executed tests in the console output.
>>
>> Thus let's try to be a little bit smarter: If the "check" script is run
>> with "-n" and an image format switch (like "-qed") at the same time (which
>> is what we do already for discovering the tests for the meson test runner),
>> only report the tests that likely support the given format instead of
>> providing the whole list of all tests. We can determine whether a test
>> supports a format or not by looking at the lines in the file that contain
>> a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
>> of course, but it is good enough for running the iotests via "make
>> check-block" - I double-checked that the list of executed tests does not
>> get changed by this patch, it's only the tests that are skipped anyway that
>> are now not run anymore.
>>
>> This way the amount of total tests drops from 2457 to 1432 for me, and
>> the amount of skipped tests drops from 1099 to just 74 (meaning that we
>> still properly run 1432 - 74 = 1358 tests as we did before).
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>> tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
>> 1 file changed, 32 insertions(+), 3 deletions(-)
>
>> +def dry_run_list(test_dir, imgfmt, testlist):
>> + for t in testlist:
>> + if not imgfmt:
>> + print('\n'.join([os.path.basename(t)]))
>> + continue
>> + # If a format has been given, we look for the "supported_fmt"
>> + # and the "unsupported_fmt" lines in the test and try to find out
>> + # whether the format is supported or not. This is only heuristics,
>> + # but it should be good enough for "make check-block"
>
> I'm not completely sure if this is a good idea at all, but I think we
> should at least mention the possible surprising cases where the
> heuristics fails in this comment.
Yeah, it's not perfect, but I also failed to come up with a better
solution... do you have any other ideas?
If not, would you be fine with the patch if I respin it with the comment
here extended with some words about those surprising cases?
Thomas
Am 02.10.2025 um 16:33 hat Thomas Huth geschrieben:
> On 02/10/2025 16.04, Kevin Wolf wrote:
> > Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> > > From: Thomas Huth <thuth@redhat.com>
> > >
> > > When running the tests in thorough mode, e.g. with:
> > >
> > > make -j$(nproc) check SPEED=thorough
> > >
> > > we currently always get a huge amount of total tests that the test
> > > runner tries to execute (2457 in my case), but a big bunch of them are
> > > only skipped (1099 in my case, meaning that only 1358 got executed).
> > > This happens because we try to run the whole set of iotests for multiple
> > > image formats while a lot of the tests can only run with one certain
> > > format only and thus are marked as SKIP during execution. This is quite a
> > > waste of time during each test run, and also unnecessarily blows up the
> > > displayed list of executed tests in the console output.
> > >
> > > Thus let's try to be a little bit smarter: If the "check" script is run
> > > with "-n" and an image format switch (like "-qed") at the same time (which
> > > is what we do already for discovering the tests for the meson test runner),
> > > only report the tests that likely support the given format instead of
> > > providing the whole list of all tests. We can determine whether a test
> > > supports a format or not by looking at the lines in the file that contain
> > > a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
> > > of course, but it is good enough for running the iotests via "make
> > > check-block" - I double-checked that the list of executed tests does not
> > > get changed by this patch, it's only the tests that are skipped anyway that
> > > are now not run anymore.
> > >
> > > This way the amount of total tests drops from 2457 to 1432 for me, and
> > > the amount of skipped tests drops from 1099 to just 74 (meaning that we
> > > still properly run 1432 - 74 = 1358 tests as we did before).
> > >
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > > tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
> > > 1 file changed, 32 insertions(+), 3 deletions(-)
> >
> > > +def dry_run_list(test_dir, imgfmt, testlist):
> > > + for t in testlist:
> > > + if not imgfmt:
> > > + print('\n'.join([os.path.basename(t)]))
> > > + continue
> > > + # If a format has been given, we look for the "supported_fmt"
> > > + # and the "unsupported_fmt" lines in the test and try to find out
> > > + # whether the format is supported or not. This is only heuristics,
> > > + # but it should be good enough for "make check-block"
> >
> > I'm not completely sure if this is a good idea at all, but I think we
> > should at least mention the possible surprising cases where the
> > heuristics fails in this comment.
>
> Yeah, it's not perfect, but I also failed to come up with a better
> solution... do you have any other ideas?
Maybe fixing the qcow vs. qcow2 case could be possible relatively easily
by operating only on full words, but I don't have any fundamentally
different solution without touching every script.
If touching everything is an option, maybe we could have a standardised
comment line like we have with "# group:" today. This would be easy to
parse and work in both bash and Python scripts. But I'm not sure if it's
worth it.
> If not, would you be fine with the patch if I respin it with the comment
> here extended with some words about those surprising cases?
Yes, I just want to have potentially surprising behaviour documented.
Kevin
On 13/10/2025 14.51, Kevin Wolf wrote:
> Am 02.10.2025 um 16:33 hat Thomas Huth geschrieben:
>> On 02/10/2025 16.04, Kevin Wolf wrote:
>>> Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
>>>> From: Thomas Huth <thuth@redhat.com>
>>>>
>>>> When running the tests in thorough mode, e.g. with:
>>>>
>>>> make -j$(nproc) check SPEED=thorough
>>>>
>>>> we currently always get a huge amount of total tests that the test
>>>> runner tries to execute (2457 in my case), but a big bunch of them are
>>>> only skipped (1099 in my case, meaning that only 1358 got executed).
>>>> This happens because we try to run the whole set of iotests for multiple
>>>> image formats while a lot of the tests can only run with one certain
>>>> format only and thus are marked as SKIP during execution. This is quite a
>>>> waste of time during each test run, and also unnecessarily blows up the
>>>> displayed list of executed tests in the console output.
>>>>
>>>> Thus let's try to be a little bit smarter: If the "check" script is run
>>>> with "-n" and an image format switch (like "-qed") at the same time (which
>>>> is what we do already for discovering the tests for the meson test runner),
>>>> only report the tests that likely support the given format instead of
>>>> providing the whole list of all tests. We can determine whether a test
>>>> supports a format or not by looking at the lines in the file that contain
>>>> a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
>>>> of course, but it is good enough for running the iotests via "make
>>>> check-block" - I double-checked that the list of executed tests does not
>>>> get changed by this patch, it's only the tests that are skipped anyway that
>>>> are now not run anymore.
>>>>
>>>> This way the amount of total tests drops from 2457 to 1432 for me, and
>>>> the amount of skipped tests drops from 1099 to just 74 (meaning that we
>>>> still properly run 1432 - 74 = 1358 tests as we did before).
>>>>
>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>> ---
>>>> tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
>>>> 1 file changed, 32 insertions(+), 3 deletions(-)
>>>
>>>> +def dry_run_list(test_dir, imgfmt, testlist):
>>>> + for t in testlist:
>>>> + if not imgfmt:
>>>> + print('\n'.join([os.path.basename(t)]))
>>>> + continue
>>>> + # If a format has been given, we look for the "supported_fmt"
>>>> + # and the "unsupported_fmt" lines in the test and try to find out
>>>> + # whether the format is supported or not. This is only heuristics,
>>>> + # but it should be good enough for "make check-block"
>>>
>>> I'm not completely sure if this is a good idea at all, but I think we
>>> should at least mention the possible surprising cases where the
>>> heuristics fails in this comment.
>>
>> Yeah, it's not perfect, but I also failed to come up with a better
>> solution... do you have any other ideas?
>
> Maybe fixing the qcow vs. qcow2 case could be possible relatively easily
> by operating only on full words, but I don't have any fundamentally
> different solution without touching every script.
Ack, using some proper regex seems to be better, indeed.
> If touching everything is an option, maybe we could have a standardised
> comment line like we have with "# group:" today. This would be easy to
> parse and work in both bash and Python scripts. But I'm not sure if it's
> worth it.
Actually, introducing special groups for every format was my initial thought
when I started working on this. But this is very, very cumbersome, you have
to make sure to get the groups of each test right in addition to the
"_supported_fmt" and "_unsupported_fmt" statements, so I quickly abandonded
that idea again.
So I'll send a v2 with regex statements, I hope that's good enough for the
average testing via "make check".
Thomas
© 2016 - 2026 Red Hat, Inc.