I was annoyed that

  meson build --auto-features=disabled -Dtests=enabled

will fail in unit tests. Rather than just fix that scenario, I went
down the rabbit hole and tested (almost) every single minimal build
option.

By that I mean I looked at meson_options.txt and got a list of every
option that has 'auto' as its default. I then attempted a minimal
build with only that enabled. I quickly found some options have
mandatory pre-requisite options, so I re-did the tests with such
deps present.

This uncovered many scenarios where we didn't use the right conditions
for tests. Surprisingly (at first, but not in retrospect) it also
found many places where the tests failed to depend on the earlier
build artifacts.

My test process was:

  opts=$(grep auto meson_options.txt | sed -e 's/option(//' -e 's/,.*//' -e "s/'//g")

  for opt in $opts
  do
    deps=`grep --before 1 "option('$opt'" meson_options.txt | grep dep: | head -1 | sed -e 's/# //' -e 's/dep:/-D/g' -e 's/$/=enabled/' -e 's/ -D/=enabled -D/g'`
    echo "Try test $opt with $deps"
    rm -rf build
    meson setup build --auto-features=disabled -Dtests=enabled $deps -D$opt=enabled 1> opt-$opt.log 2>&1
    meson test -C build --no-suite syntax-check --print-errorlogs 1>> opt-$opt.log 2>&1 ;  done
  done

Then to check results

  grep '^Fail:' opt-*.log | grep -v 0

This is slow, but not as slow as you might think, because each
build is very minimal, so we're not actually building more than
300-400 files each time, rather than 1000's, and we're not running
as many tests either.

Might be interesting to get extensive test into CI.

Strictly speaking we care about the combinatorial expansion of
meson_options.txt, but that would be insanely slow and is likely
to be overkill.

Daniel P. Berrangé (12):
  scripts/rpcgen: skip tests if tirpc is not present
  tests: fix tests when test driver is disabled
  meson: record which other options are a pre-requisite
  docs: ensure HTML/images are built before running reference tests
  src: ensure augeas test file is generated before running test
  tests: build 'virsh' before running virsh-auth test
  tests: build driver modules before virdrivermoduletest
  test: conditionalize 'virsh-auth' on test driver
  tests: always build securityselinuxhelper if libselinux is present
  test: drop bogus check for YAJL from libxl test/mock
  tests: don't run mdevctl test if lacking YAJL
  src/node_device: don't overwrite error messages

 docs/images/meson.build              |  5 ++-
 docs/logos/meson.build               |  5 ++-
 docs/meson.build                     |  1 +
 meson_options.txt                    | 20 +++++++++
 scripts/rpcgen/meson.build           |  2 +-
 src/meson.build                      |  6 ++-
 src/node_device/node_device_driver.c |  2 -
 tests/fchosttest.c                   |  9 ++--
 tests/libxlmock.c                    |  4 +-
 tests/libxlxml2domconfigtest.c       |  4 +-
 tests/meson.build                    | 67 ++++++++++++++++++----------
 11 files changed, 87 insertions(+), 38 deletions(-)

-- 
2.43.0
_______________________________________________
Devel mailing list -- devel@lists.libvirt.org
To unsubscribe send an email to devel-leave@lists.libvirt.org

On Tue, May 07, 2024 at 14:58:55 +0100, Daniel P. Berrangé wrote:
> I was annoyed that
> 
>   meson build --auto-features=disabled -Dtests=enabled
> 
> will fail in unit tests. Rather than just fix that scenario, I went
> down the rabbit hole and tested (almost) every single minimal build
> option.
> 
> By that I mean I looked at meson_options.txt and got a list of every
> option that has 'auto' as its default. I then attempted a minimal
> build with only that enabled. I quickly found some options have
> mandatory pre-requisite options, so I re-did the tests with such
> deps present.
> 
> This uncovered many scenarios where we didn't use the right conditions
> for tests. Surprisingly (at first, but not in retrospect) it also
> found many places where the tests failed to depend on the earlier
> build artifacts.
> 
> My test process was:
> 
>   opts=$(grep auto meson_options.txt | sed -e 's/option(//' -e 's/,.*//' -e "s/'//g")
> 
>   for opt in $opts
>   do
>     deps=`grep --before 1 "option('$opt'" meson_options.txt | grep dep: | head -1 | sed -e 's/# //' -e 's/dep:/-D/g' -e 's/$/=enabled/' -e 's/ -D/=enabled -D/g'`
>     echo "Try test $opt with $deps"
>     rm -rf build
>     meson setup build --auto-features=disabled -Dtests=enabled $deps -D$opt=enabled 1> opt-$opt.log 2>&1
>     meson test -C build --no-suite syntax-check --print-errorlogs 1>> opt-$opt.log 2>&1 ;  done
>   done
> 
> Then to check results
> 
>   grep '^Fail:' opt-*.log | grep -v 0
> 
> This is slow, but not as slow as you might think, because each
> build is very minimal, so we're not actually building more than
> 300-400 files each time, rather than 1000's, and we're not running
> as many tests either.
> 
> Might be interesting to get extensive test into CI.
> 
> Strictly speaking we care about the combinatorial expansion of
> meson_options.txt, but that would be insanely slow and is likely
> to be overkill.

Whoah, that's rather cool. But as you note I don't think this is
anything for CI and even if, this'd be something to run once a month
perhaps. Otherwise it's almost as useful as mining cryptocurrencies

With the few things I've pointed out inline:

Series:

Reviewed-by: Peter Krempa <pkrempa@redhat.com>
_______________________________________________
Devel mailing list -- devel@lists.libvirt.org
To unsubscribe send an email to devel-leave@lists.libvirt.org