The final bout of conversions removes the remaining compiler tests
for emulators, applies a few cleanups that are enabled by version 0.63
of Meson, and updates the documentation.

Committing this however requires first a libvirt-ci update, in order to
enable using Python 3.8/3.9 on CentOS 8 and SLES 15.  I am still working
on it but the review can start early for these patches.

A final comparison for the effort has a net negative of around 4000
lines of code, despite the growth of QEMU itself in the meanwhile:

      before                             after
--------------------------------------------------------------------------
 8647 configure                     4240 meson.build
 1296 Makefile                      1970 configure                            
  985 tests/Makefile.include         337 Makefile
  440 rules.mak                      224 scripts/meson-buildoptions.py
  379 scripts/tap-driver.pl          171 tests/Makefile.include
  287 Makefile.target                129 scripts/nsis.py
  263 tests/tcg/configure.sh         113 scripts/mtest2make.py
  129 scripts/create_config           48 scripts/undefsym.py
~5500 various Makefile fragments   ~6500 various meson.build files
--------------------------------------------------------------------------
17926 total                        13732 total

Completing the conversion took roughly two years and a lot of the complexity
in how QEMU is built is indeed essential, but I think it's agreed that:

- the disruption to the developers was minimal, and even Windows only took
  a month or so to solidify;

- there is overall feature parity for both configure/Makefile<->meson and for
  tap-driver.pl<->"meson test", but with fewer bugs and fewer (though not zero)
  workarounds required.  For Meson, the main recurring issue is probably the
  gnutls headers issue on Homebrew; see for example commit 76f67bac79f6,
  "meson: Propagate gnutls dependency to migration", 2021-04-01)

- the original target of the conversion ("make it trivial to do trivial
  things; easy to extend existing things; possible to do everything else")
  has been achieved.  New dependencies such as ninja, were introduced in
  such a way as to make them familiar to shell/Make users, and did
  provide subtle quality of life improvements (e.g. rebuild targets
  on command line changes)

New build system features since the introduction of Meson include switching
to config/targets files, custom device configuration (--with-devices-ARCH=),
the modinfo mechanism for module loading, entitlement support for
Hypervisor.framework, the preinstall tree, autogenerated parsing of the
configure command line, and cross-compilation of pc-bios/ (admittedly
not done by Meson, but not hindered either).

Big thanks to Akihiko, Alex, Daniel, Marc-Andr\ufffd\ufffd, Richard, Thomas and
everybody else that shared the work and the reviews!

Thanks,

Paolo

Paolo Bonzini (30):
  configure: remove useless write_c_skeleton
  configure: remove dead function
  configure: remove useless test
  configure: preserve qemu-ga variables
  configure: remove backwards-compatibility and obsolete options
  meson: tweak hardening options for Windows
  meson: support meson 0.64 -Doptimization=plain
  meson: require 0.63.0
  meson: use prefer_static option
  meson: remove static_kwargs
  meson: cleanup dummy-cpus.c rules
  modinfo: lookup compile_commands.json by object
  configure: remove backwards-compatibility code
  configure: test all warnings
  meson: cleanup compiler detection
  build: move glib detection and workarounds to meson
  configure: remove pkg-config functions
  configure, meson: move --enable-modules to Meson
  configure, meson: move --enable-debug-info to Meson
  meson: prepare move of QEMU_CFLAGS to meson
  build: move sanitizer tests to meson
  build: move SafeStack tests to meson
  build: move coroutine backend selection to meson
  build: move stack protector flag selection to meson
  build: move warning flag selection to meson
  build: move remaining compiler flag tests to meson
  build: move compiler version check to meson
  docs: update build system docs
  configure: do not rerun the tests with -Werror
  meson: always log qemu-iotests verbosely

 accel/meson.build                             |   9 +-
 block/meson.build                             |   1 +
 configure                                     | 760 +-----------------
 contrib/plugins/Makefile                      |   7 +-
 docs/devel/build-system.rst                   | 286 ++++---
 meson                                         |   2 +-
 meson.build                                   | 684 +++++++++++-----
 meson_options.txt                             |  13 +
 plugins/meson.build                           |   2 +-
 qga/meson.build                               |   2 +-
 scripts/meson-buildoptions.py                 |   3 +
 scripts/meson-buildoptions.sh                 |  22 +
 scripts/modinfo-collect.py                    |  23 +-
 tcg/meson.build                               |   2 +-
 tests/qemu-iotests/meson.build                |   3 +-
 tests/unit/meson.build                        |   2 +-
 ...{coroutine-win32.c => coroutine-windows.c} |   0
 util/meson.build                              |   2 +-
 18 files changed, 716 insertions(+), 1107 deletions(-)
 rename util/{coroutine-win32.c => coroutine-windows.c} (100%)

-- 
2.38.1

On Fri, Dec 09, 2022 at 12:23:39PM +0100, Paolo Bonzini wrote:
> The final bout of conversions removes the remaining compiler tests
> for emulators, applies a few cleanups that are enabled by version 0.63
> of Meson, and updates the documentation.
> 
> Committing this however requires first a libvirt-ci update, in order to
> enable using Python 3.8/3.9 on CentOS 8 and SLES 15.  I am still working
> on it but the review can start early for these patches.
> 
> A final comparison for the effort has a net negative of around 4000
> lines of code, despite the growth of QEMU itself in the meanwhile:
> 
>       before                             after
> --------------------------------------------------------------------------
>  8647 configure                     4240 meson.build
>  1296 Makefile                      1970 configure                            
>   985 tests/Makefile.include         337 Makefile
>   440 rules.mak                      224 scripts/meson-buildoptions.py
>   379 scripts/tap-driver.pl          171 tests/Makefile.include
>   287 Makefile.target                129 scripts/nsis.py
>   263 tests/tcg/configure.sh         113 scripts/mtest2make.py
>   129 scripts/create_config           48 scripts/undefsym.py
> ~5500 various Makefile fragments   ~6500 various meson.build files
> --------------------------------------------------------------------------
> 17926 total                        13732 total

So after applying this series, at least on Fedora, I barely needed
configure at all, in so muc as the following successfully built AFAICT:

$ mkdir build
$ cat > build/config-host.mak <<EOF
all:
GIT=git
GIT_SUBMODULES=ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
GIT_SUBMODULES_ACTION=update
CONFIG_POSIX=y
CONFIG_LINUX=y
SRC_PATH=/home/berrange/src/virt/qemu
TARGET_DIRS=x86_64-softmmu
CONFIG_PLUGIN=y
HAVE_GDB_BIN=/usr/bin/gdb
ENGINE=podman
ROMS=pc-bios/optionrom
MAKE=make
PYTHON=/usr/bin/python3 -B
GENISOIMAGE=/usr/bin/genisoimage
MESON=/usr/bin/meson
NINJA=/usr/bin/ninja
PKG_CONFIG=
CC=cc
EXESUF=
TCG_TESTS_TARGETS= x86_64-softmmu
EOF
$ meson build
$ ninja -C build

Out of the things in config-host.mak there, a few meson already knows,
like CC, SRC_PATH, 50% would be just a meson find_program() command,
a couple would need meson args (eg TARGET_DIRS), and some just look
redundant, eg CONFIG_LINUX largely duplicates __linux__, and
CONFIG_POSIX is effectively  !_WIN32.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 12/16/22 16:35, Daniel P. Berrangé wrote:
> So after applying this series, at least on Fedora, I barely needed
> configure at all, in so muc as the following successfully built AFAICT:
> 
> $ mkdir build
> $ cat > build/config-host.mak <<EOF
> all:
> GIT=git
> GIT_SUBMODULES=ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
> GIT_SUBMODULES_ACTION=update
> CONFIG_POSIX=y
> CONFIG_LINUX=y
> SRC_PATH=/home/berrange/src/virt/qemu
> TARGET_DIRS=x86_64-softmmu
> CONFIG_PLUGIN=y
> HAVE_GDB_BIN=/usr/bin/gdb
> ENGINE=podman
> ROMS=pc-bios/optionrom
> MAKE=make
> PYTHON=/usr/bin/python3 -B
> GENISOIMAGE=/usr/bin/genisoimage
> MESON=/usr/bin/meson
> NINJA=/usr/bin/ninja
> PKG_CONFIG=
> CC=cc
> EXESUF=
> TCG_TESTS_TARGETS= x86_64-softmmu
> EOF
> $ meson build
> $ ninja -C build

Yep.  It wouldn't build the firmwares though, or be able to run TCG or 
Avocado tests.  This kind of build system orchestration is the remaining 
task of configure (see new docs in patch 28).  It's not really possible 
for it to go away, it's beyond Meson's capabilities and intentions.

I would also like to move Python virtual environment configuration to 
configure.  This would give a lot more flexibility in whether to use 
network or not, for example, and is in line with submodule handling that 
is already in configure.

> Out of the things in config-host.mak there, a few meson already knows,
> like CC, SRC_PATH, 50% would be just a meson find_program() command,
> a couple would need meson args (eg TARGET_DIRS), and some just look
> redundant, eg CONFIG_LINUX largely duplicates __linux__, and
> CONFIG_POSIX is effectively  !_WIN32.

There are a couple like ENGINE or HAVE_GDB_BIN that are only needed by 
non-QEMU parts of the build system, and I don't think it makes much 
sense to move those detections out of configure.

But yeah, the OS symbols are pretty low-hanging fruit.  In fact 
meson.build recomputes them:

   config_host_data.set('CONFIG_POSIX', targetos != 'windows')

... and with Windriver's patches to get much closer to POSIX/Windows 
feature parity, the Makefiles don't use them as much as they did.

Paolo