1. Patchew
  2. QEMU
  3. View series

[PATCH] meson: use thorough test setup as default

Pierrick Bouvier posted 1 patch 7 months, 2 weeks ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20250503201806.3045723-1-pierrick.bouvier@linaro.org
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, "Daniel P. Berrangé" <berrange@redhat.com>, "Philippe Mathieu-Daudé" <philmd@linaro.org>
meson.build | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
[PATCH] meson: use thorough test setup as default
Posted by Pierrick Bouvier 7 months, 2 weeks ago 
Allows all tests to be visible by default when using meson test
directly.

This has no impact on make check-* commands, which use SPEED=quick by
default (see scripts/mtest2make.py).

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 meson.build | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meson.build b/meson.build
index f8bf6e7bb66..57ff3f722d8 100644
--- a/meson.build
+++ b/meson.build
@@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0',
 
 meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() })
 
-add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true,
+add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
                env: ['RUST_BACKTRACE=1'])
 add_test_setup('slow', exclude_suites: ['thorough'],
                env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1'])
 add_test_setup('thorough',
-               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'])
+               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'],
+               is_default: true)
 
 meson.add_postconf_script(find_program('scripts/symlink-install-tree.py'))
 
-- 
2.47.2
Re: [PATCH] meson: use thorough test setup as default
Posted by Daniel P. Berrangé 7 months, 2 weeks ago 
On Sat, May 03, 2025 at 01:18:06PM -0700, Pierrick Bouvier wrote:
> Allows all tests to be visible by default when using meson test
> directly.
> 
> This has no impact on make check-* commands, which use SPEED=quick by
> default (see scripts/mtest2make.py).

IMHO it would be conceptually confusing if we cause 'make check' to
be running different stuff from 'meson test'. As long as we keep a
makefile wrapper around running tests, I think we should keep them
matching what they do by default.



> 
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  meson.build | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/meson.build b/meson.build
> index f8bf6e7bb66..57ff3f722d8 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0',
>  
>  meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() })
>  
> -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true,
> +add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
>                 env: ['RUST_BACKTRACE=1'])
>  add_test_setup('slow', exclude_suites: ['thorough'],
>                 env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1'])
>  add_test_setup('thorough',
> -               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'])
> +               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'],
> +               is_default: true)
>  
>  meson.add_postconf_script(find_program('scripts/symlink-install-tree.py'))
>  
> -- 
> 2.47.2
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [PATCH] meson: use thorough test setup as default
Posted by Thomas Huth 7 months, 2 weeks ago 
On 06/05/2025 11.36, Daniel P. Berrangé wrote:
> On Sat, May 03, 2025 at 01:18:06PM -0700, Pierrick Bouvier wrote:
>> Allows all tests to be visible by default when using meson test
>> directly.
>>
>> This has no impact on make check-* commands, which use SPEED=quick by
>> default (see scripts/mtest2make.py).
> 
> IMHO it would be conceptually confusing if we cause 'make check' to
> be running different stuff from 'meson test'. As long as we keep a
> makefile wrapper around running tests, I think we should keep them
> matching what they do by default.

But what we should maybe do instead: Properly document in 
docs/devel/testing/functional.rst how to get the list of all tests and how 
to run a single test with the "meson test" way... I could do it later when 
time permits (likely not this week), but if anybody else wants to have a 
try, that would be welcome!

  Thomas
Re: [PATCH] meson: use thorough test setup as default
Posted by Thomas Huth 7 months, 2 weeks ago 
On 03/05/2025 22.18, Pierrick Bouvier wrote:
> Allows all tests to be visible by default when using meson test
> directly.
> 
> This has no impact on make check-* commands, which use SPEED=quick by
> default (see scripts/mtest2make.py).
> 
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>   meson.build | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/meson.build b/meson.build
> index f8bf6e7bb66..57ff3f722d8 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0',
>   
>   meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() })
>   
> -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true,
> +add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
>                  env: ['RUST_BACKTRACE=1'])
>   add_test_setup('slow', exclude_suites: ['thorough'],
>                  env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1'])
>   add_test_setup('thorough',
> -               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'])
> +               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'],
> +               is_default: true)

I'd rather not make it the default: The thorough functional tests download a 
lot of assets from the internet, so if someone just runs "meson test" 
without further parameters, I think we should not trigger these downloads in 
that case.

  Thomas
Re: [PATCH] meson: use thorough test setup as default
Posted by Pierrick Bouvier 7 months, 2 weeks ago 
On 5/5/25 3:32 AM, Thomas Huth wrote:
> On 03/05/2025 22.18, Pierrick Bouvier wrote:
>> Allows all tests to be visible by default when using meson test
>> directly.
>>
>> This has no impact on make check-* commands, which use SPEED=quick by
>> default (see scripts/mtest2make.py).
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>    meson.build | 5 +++--
>>    1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/meson.build b/meson.build
>> index f8bf6e7bb66..57ff3f722d8 100644
>> --- a/meson.build
>> +++ b/meson.build
>> @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0',
>>    
>>    meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() })
>>    
>> -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true,
>> +add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
>>                   env: ['RUST_BACKTRACE=1'])
>>    add_test_setup('slow', exclude_suites: ['thorough'],
>>                   env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1'])
>>    add_test_setup('thorough',
>> -               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'])
>> +               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'],
>> +               is_default: true)
> 
> I'd rather not make it the default: The thorough functional tests download a
> lot of assets from the internet, so if someone just runs "meson test"
> without further parameters, I think we should not trigger these downloads in
> that case.
>

$ cat tests/Makefile.include
...
check-functional:
	@$(NINJA) precache-functional
	@QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func
...

What's the rationale to run check-func with SPEED=thorough and 
QEMU_TEST_NO_DOWNLOAD=1 with precache-functional having cached only 
quick tests by default?
Either we should remove SPEED=thorough, or download all tests by default.

>    Thomas
>
Re: [PATCH] meson: use thorough test setup as default
Posted by Daniel P. Berrangé 7 months, 2 weeks ago 
On Mon, May 05, 2025 at 10:46:52AM -0700, Pierrick Bouvier wrote:
> On 5/5/25 3:32 AM, Thomas Huth wrote:
> > On 03/05/2025 22.18, Pierrick Bouvier wrote:
> > > Allows all tests to be visible by default when using meson test
> > > directly.
> > > 
> > > This has no impact on make check-* commands, which use SPEED=quick by
> > > default (see scripts/mtest2make.py).
> > > 
> > > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> > > ---
> > >    meson.build | 5 +++--
> > >    1 file changed, 3 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/meson.build b/meson.build
> > > index f8bf6e7bb66..57ff3f722d8 100644
> > > --- a/meson.build
> > > +++ b/meson.build
> > > @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0',
> > >    meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() })
> > > -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true,
> > > +add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
> > >                   env: ['RUST_BACKTRACE=1'])
> > >    add_test_setup('slow', exclude_suites: ['thorough'],
> > >                   env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1'])
> > >    add_test_setup('thorough',
> > > -               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'])
> > > +               env: ['G_TEST_SLOW=1', 'SPEED=thorough', 'RUST_BACKTRACE=1'],
> > > +               is_default: true)
> > 
> > I'd rather not make it the default: The thorough functional tests download a
> > lot of assets from the internet, so if someone just runs "meson test"
> > without further parameters, I think we should not trigger these downloads in
> > that case.
> > 
> 
> $ cat tests/Makefile.include
> ...
> check-functional:
> 	@$(NINJA) precache-functional
> 	@QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func
> ...
> 
> What's the rationale to run check-func with SPEED=thorough and
> QEMU_TEST_NO_DOWNLOAD=1 with precache-functional having cached only quick
> tests by default?

Are you sure about that ? The precache logic uses custom_target() in
meson and so does not filters on "suites" used to define quick vs
thorough tests.

IOW, precache should always be downloading all assets.

> Either we should remove SPEED=thorough, or download all tests by default.
> 
> >    Thomas
> > 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [PATCH] meson: use thorough test setup as default
Posted by Paolo Bonzini 7 months, 2 weeks ago 
Il sab 3 mag 2025, 22:18 Pierrick Bouvier <pierrick.bouvier@linaro.org> ha
scritto:

> Allows all tests to be visible by default when using meson test
> directly.
>
> This has no impact on make check-* commands, which use SPEED=quick by
> default (see scripts/mtest2make.py).
>

What's the advantage of having different defaults depending on whether you
use "make check" or "meson test"?

I don't oppose this change per se, but if it's useful it should be matched
by a change in the default SPEED.

Paolo


> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  meson.build | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/meson.build b/meson.build
> index f8bf6e7bb66..57ff3f722d8 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0',
>
>  meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() })
>
> -add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default:
> true,
> +add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
>                 env: ['RUST_BACKTRACE=1'])
>  add_test_setup('slow', exclude_suites: ['thorough'],
>                 env: ['G_TEST_SLOW=1', 'SPEED=slow', 'RUST_BACKTRACE=1'])
>  add_test_setup('thorough',
> -               env: ['G_TEST_SLOW=1', 'SPEED=thorough',
> 'RUST_BACKTRACE=1'])
> +               env: ['G_TEST_SLOW=1', 'SPEED=thorough',
> 'RUST_BACKTRACE=1'],
> +               is_default: true)
>
>  meson.add_postconf_script(find_program('scripts/symlink-install-tree.py'))
>
> --
> 2.47.2
>
>
Re: [PATCH] meson: use thorough test setup as default
Posted by Pierrick Bouvier 7 months, 2 weeks ago 
On 5/4/25 11:00 PM, Paolo Bonzini wrote:
> 
> 
> Il sab 3 mag 2025, 22:18 Pierrick Bouvier <pierrick.bouvier@linaro.org 
> <mailto:pierrick.bouvier@linaro.org>> ha scritto:
> 
>     Allows all tests to be visible by default when using meson test
>     directly.
> 
>     This has no impact on make check-* commands, which use SPEED=quick by
>     default (see scripts/mtest2make.py).
> 
> 
> What's the advantage of having different defaults depending on whether 
> you use "make check" or "meson test"?
>

"make check" can be seen as a blackbox, where you have different 
targets, but don't really do things on a per test basis. At least, it's 
not the workflow that was implemented.
"meson test", on the opposite, expose the list of tests to you.

My rationale here is to expose all tests for meson test (instead of 
having to add --setup thorough everytime), and let the "per setup" 
workflow to make command.

> I don't oppose this change per se, but if it's useful it should be 
> matched by a change in the default SPEED.
> 

As Thomas pointed, running thorough tests by default would take more 
time/more disk space, and I'm not sure we want to change what is done in 
our CI or for people by default.

This change is just supposed to be convenience for people using meson 
test directly, without impacting "make check*" users.
So when you want to run a single test, it's visible from
"meson test -C build --list", by default.

> Paolo
> 
> 
>     Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org
>     <mailto:pierrick.bouvier@linaro.org>>
>     ---
>       meson.build | 5 +++--
>       1 file changed, 3 insertions(+), 2 deletions(-)
> 
>     diff --git a/meson.build b/meson.build
>     index f8bf6e7bb66..57ff3f722d8 100644
>     --- a/meson.build
>     +++ b/meson.build
>     @@ -5,12 +5,13 @@ project('qemu', ['c'], meson_version: '>=1.5.0',
> 
>       meson.add_devenv({ 'MESON_BUILD_ROOT' : meson.project_build_root() })
> 
>     -add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
>     is_default: true,
>     +add_test_setup('quick', exclude_suites: ['slow', 'thorough'],
>                      env: ['RUST_BACKTRACE=1'])
>       add_test_setup('slow', exclude_suites: ['thorough'],
>                      env: ['G_TEST_SLOW=1', 'SPEED=slow',
>     'RUST_BACKTRACE=1'])
>       add_test_setup('thorough',
>     -               env: ['G_TEST_SLOW=1', 'SPEED=thorough',
>     'RUST_BACKTRACE=1'])
>     +               env: ['G_TEST_SLOW=1', 'SPEED=thorough',
>     'RUST_BACKTRACE=1'],
>     +               is_default: true)
> 
>       meson.add_postconf_script(find_program('scripts/symlink-install-
>     tree.py'))
> 
>     -- 
>     2.47.2
>

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.