[v3] i386, build system, KVM changes for 2023-05-18

[PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Paolo Bonzini 2 years, 8 months ago

The following changes since commit d27e7c359330ba7020bdbed7ed2316cb4cf6ffc1:

  qapi/parser: Drop two bad type hints for now (2023-05-17 10:18:33 -0700)

are available in the Git repository at:

  https://gitlab.com/bonzini/qemu.git tags/for-upstream

for you to fetch changes up to fe3ab4eb2de46076cbafcbc86b22e71ad24894c6:

  docs/devel: update build system docs (2023-05-18 13:35:28 +0200)

----------------------------------------------------------------
* kvm: enable dirty ring for arm64
* target/i386: new features
* target/i386: AVX fixes
* configure: create a python venv unconditionally
* meson: bump to 0.63.0 and move tests from configure
* meson: Pass -j option to sphinx
* drop support for Python 3.6
* fix check-python-tox
* fix "make clean" in the source directory

----------------------------------------------------------------
v2->v3:
        fix cut-and-paste typo in "build: move warning flag selection to meson"

Emanuele Giuseppe Esposito (2):
      target/i386: add support for FLUSH_L1D feature
      target/i386: add support for FB_CLEAR feature

Fabiano Rosas (1):
      meson: Pass -j option to sphinx

Gavin Shan (4):
      migration: Add last stage indicator to global dirty log
      kvm: Synchronize the backup bitmap in the last stage
      kvm: Add helper kvm_dirty_ring_init()
      kvm: Enable dirty ring for arm64

John Snow (23):
      python: update pylint configuration
      python: add mkvenv.py
      mkvenv: add better error message for broken or missing ensurepip
      mkvenv: add nested venv workaround
      mkvenv: add ensure subcommand
      mkvenv: add --diagnose option to explain "ensure" failures
      mkvenv: add console script entry point generation
      mkvenv: use pip's vendored distlib as a fallback
      mkvenv: avoid ensurepip if pip is installed
      mkvenv: work around broken pip installations on Debian 10
      tests/docker: add python3-venv dependency
      tests/vm: Configure netbsd to use Python 3.10
      tests/vm: add py310-expat to NetBSD
      python: add vendor.py utility
      configure: create a python venv unconditionally
      python/wheels: add vendored meson package
      configure: use 'mkvenv ensure meson' to bootstrap meson
      qemu.git: drop meson git submodule
      tests: Use configure-provided pyvenv for tests
      configure: move --enable-docs and --disable-docs back to configure
      configure: bootstrap sphinx with mkvenv
      configure: add --enable-pypi and --disable-pypi
      configure: Add courtesy hint to Python version failure message

Paolo Bonzini (33):
      target/i386: fix operand size for VCOMI/VUCOMI instructions
      tests/tcg/i386: correct mask for VPERM2F128/VPERM2I128
      tcg: round-robin: do not use mb_read for rr_current_cpu
      coverity: the definitive COMPONENTS.md update
      scsi-generic: fix buffer overflow on block limits inquiry
      python: shut up "pip install" during "make check-minreqs"
      Python: Drop support for Python 3.6
      mkvenv: mark command as required
      python: bump some of the dependencies
      meson: regenerate meson-buildoptions.sh
      meson: require 0.63.0
      meson: use prefer_static option
      meson: remove static_kwargs
      meson: add more version numbers to the summary
      meson: drop unnecessary declare_dependency()
      build: move glib detection and workarounds to meson
      configure: remove pkg-config functions
      configure, meson: move --enable-modules to Meson
      meson: prepare move of QEMU_CFLAGS to meson
      build: move sanitizer tests to meson
      build: move SafeStack tests to meson
      build: move coroutine backend selection to meson
      build: move stack protector flag selection to meson
      build: move warning flag selection to meson
      build: move remaining compiler flag tests to meson
      build: move compiler version check to meson
      build: move --disable-debug-info to meson
      configure: remove compiler sanity check
      configure: do not rerun the tests with -Werror
      configure: remove unnecessary mkdir
      configure: reorder option parsing code
      configure: remove unnecessary check
      docs/devel: update build system docs

Ricky Zhou (3):
      target/i386: Fix and add some comments next to SSE/AVX instructions.
      target/i386: Fix exception classes for SSE/AVX instructions.
      target/i386: Fix exception classes for MOVNTPS/MOVNTPD.

Steve Sistare (1):
      make: clean after distclean deletes source files

Xinyu Li (1):
      target/i386: fix avx2 instructions vzeroall and vpermdq

 .gitlab-ci.d/buildtest-template.yml                |   4 +-
 .gitlab-ci.d/buildtest.yml                         |   6 +-
 .gitmodules                                        |   3 -
 Makefile                                           |   6 +-
 accel/kvm/kvm-all.c                                | 108 ++-
 accel/tcg/tcg-accel-ops-rr.c                       |  11 +-
 configure                                          | 980 ++++-----------------
 contrib/plugins/Makefile                           |   7 +-
 docs/about/build-platforms.rst                     |   2 +-
 docs/conf.py                                       |   9 -
 docs/devel/acpi-bits.rst                           |   6 +-
 docs/devel/build-system.rst                        | 296 ++++---
 docs/devel/testing.rst                             |  14 +-
 docs/meson.build                                   |  14 +-
 docs/sphinx/dbusdomain.py                          |   4 +
 docs/sphinx/fakedbusdoc.py                         |   5 +
 docs/sphinx/qmp_lexer.py                           |   5 +
 hw/scsi/scsi-generic.c                             |  14 +-
 include/exec/memory.h                              |   9 +-
 include/sysemu/kvm_int.h                           |   1 +
 meson                                              |   1 -
 meson.build                                        | 661 +++++++++-----
 meson_options.txt                                  |  15 +-
 migration/dirtyrate.c                              |   4 +-
 migration/ram.c                                    |  20 +-
 python/Makefile                                    |  19 +-
 python/scripts/mkvenv.py                           | 897 +++++++++++++++++++
 python/scripts/vendor.py                           |  74 ++
 python/setup.cfg                                   |  46 +-
 python/tests/flake8.sh                             |   1 +
 python/tests/isort.sh                              |   1 +
 python/tests/minreqs.txt                           |  19 +-
 python/tests/mypy.sh                               |   1 +
 python/tests/pylint.sh                             |   1 +
 python/wheels/meson-0.63.3-py3-none-any.whl        | Bin 0 -> 926526 bytes
 qga/meson.build                                    |   2 +-
 scripts/ci/org.centos/stream/8/x86_64/test-avocado |   4 +-
 scripts/coverity-scan/COMPONENTS.md                |  45 +-
 scripts/device-crash-test                          |   2 +-
 scripts/meson-buildoptions.py                      |   3 +
 scripts/meson-buildoptions.sh                      |  31 +-
 scripts/qapi/mypy.ini                              |   2 +-
 softmmu/memory.c                                   |  10 +-
 target/i386/cpu.c                                  |   4 +-
 target/i386/cpu.h                                  |   3 +
 target/i386/ops_sse.h                              |   8 +
 target/i386/tcg/decode-new.c.inc                   |  90 +-
 target/i386/tcg/emit.c.inc                         |   2 +-
 tcg/meson.build                                    |   2 +-
 tests/Makefile.include                             |  10 +-
 .../dockerfiles/debian-all-test-cross.docker       |   3 +-
 .../docker/dockerfiles/debian-hexagon-cross.docker |   3 +-
 .../docker/dockerfiles/debian-riscv64-cross.docker |   3 +-
 .../docker/dockerfiles/debian-tricore-cross.docker |   3 +-
 tests/qemu-iotests/meson.build                     |   2 +-
 tests/qtest/meson.build                            |   2 +-
 tests/requirements.txt                             |   7 +-
 tests/tcg/i386/test-avx.py                         |   2 +-
 tests/unit/meson.build                             |   2 +-
 tests/vm/netbsd                                    |   2 +
 util/{coroutine-win32.c => coroutine-windows.c}    |   0
 util/meson.build                                   |   6 +-
 62 files changed, 2165 insertions(+), 1352 deletions(-)
 delete mode 160000 meson
 create mode 100644 python/scripts/mkvenv.py
 create mode 100755 python/scripts/vendor.py
 create mode 100644 python/wheels/meson-0.63.3-py3-none-any.whl
 rename util/{coroutine-win32.c => coroutine-windows.c} (100%)
-- 
2.40.1

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Markus Armbruster 2 years, 8 months ago

Paolo Bonzini <pbonzini@redhat.com> writes:

> The following changes since commit d27e7c359330ba7020bdbed7ed2316cb4cf6ffc1:
>
>   qapi/parser: Drop two bad type hints for now (2023-05-17 10:18:33 -0700)
>
> are available in the Git repository at:
>
>   https://gitlab.com/bonzini/qemu.git tags/for-upstream
>
> for you to fetch changes up to fe3ab4eb2de46076cbafcbc86b22e71ad24894c6:
>
>   docs/devel: update build system docs (2023-05-18 13:35:28 +0200)
>
> ----------------------------------------------------------------
> * kvm: enable dirty ring for arm64
> * target/i386: new features
> * target/i386: AVX fixes
> * configure: create a python venv unconditionally
> * meson: bump to 0.63.0 and move tests from configure
> * meson: Pass -j option to sphinx
> * drop support for Python 3.6

Halleluja!  Thank you both.

> * fix check-python-tox
> * fix "make clean" in the source directory

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Peter Maydell 2 years, 8 months ago

On Thu, 18 May 2023 at 12:41, Paolo Bonzini <pbonzini@redhat.com> wrote:
> ----------------------------------------------------------------
> * kvm: enable dirty ring for arm64
> * target/i386: new features
> * target/i386: AVX fixes
> * configure: create a python venv unconditionally
> * meson: bump to 0.63.0 and move tests from configure
> * meson: Pass -j option to sphinx
> * drop support for Python 3.6
> * fix check-python-tox
> * fix "make clean" in the source directory

Hi; this seems to have broken builds from not-clean on my system.
Blowing away the build tree and recreating it from scratch
works, as does manually re-running configure,
but the Makefile ought to re-run parts of configure
etc when it needs to.

It does seem to try to, but (unlike a manual configure re-run)
it doesn't prevent the failure.

You can see that it says it needs to run configure, and mkvenv
claims to be installing meson 0.6.3, but then the actual build
says it didn't work. (A second run of 'make' at this point doesn't
do the rerun of configure but meson fails in the same way.)

Other weird things:
(1) the error message is printed three times
(2) "warn: ignoring non-existent submodule meson"

$ make -C build/x86-tgts
make: Entering directory
'/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts'
config-host.mak is out-of-date, running configure
python determined to be '/usr/bin/python3'
python version: Python 3.10.6
mkvenv: Creating non-isolated virtual environment at 'pyvenv'
mkvenv: checking for meson>=0.63.0
mkvenv: installing meson>=0.63.0
mkvenv: checking for sphinx>=1.6.0, sphinx-rtd-theme>=0.5.0
/usr/bin/ninja  build.ninja && touch build.ninja.stamp
[0/1] Regenerating build files.
The Meson build system
Version: 0.61.5
Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
Build type: native build

../../meson.build:1:0: ERROR: Meson version is 0.61.5 but project
requires >=0.63.0

A full log can be found at
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
FAILED: build.ninja
/usr/bin/python3
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
ninja
ninja: error: rebuilding 'build.ninja': subcommand failed
  GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc
warn: ignoring non-existent submodule meson
/usr/bin/ninja  build.ninja && touch build.ninja.stamp
[0/1] Regenerating build files.
The Meson build system
Version: 0.61.5
Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
Build type: native build

../../meson.build:1:0: ERROR: Meson version is 0.61.5 but project
requires >=0.63.0

A full log can be found at
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
FAILED: build.ninja
/usr/bin/python3
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
ninja
ninja: error: rebuilding 'build.ninja': subcommand failed
  GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc
[0/1] Regenerating build files.
The Meson build system
Version: 0.61.5
Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
Build type: native build

../../meson.build:1:0: ERROR: Meson version is 0.61.5 but project
requires >=0.63.0

A full log can be found at
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
ninja: error: rebuilding 'build.ninja': subcommand failed
FAILED: build.ninja
/usr/bin/python3
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
ninja
make: *** [Makefile:165: run-ninja] Error 1
make: Leaving directory
'/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts'

thanks
-- PMM

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Paolo Bonzini 2 years, 8 months ago

Il ven 19 mag 2023, 16:39 Peter Maydell <peter.maydell@linaro.org> ha
scritto:

> On Thu, 18 May 2023 at 12:41, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > ----------------------------------------------------------------
> > * kvm: enable dirty ring for arm64
> > * target/i386: new features
> > * target/i386: AVX fixes
> > * configure: create a python venv unconditionally
> > * meson: bump to 0.63.0 and move tests from configure
> > * meson: Pass -j option to sphinx
> > * drop support for Python 3.6
> > * fix check-python-tox
> > * fix "make clean" in the source directory
>
> Hi; this seems to have broken builds from not-clean on my system.
> Blowing away the build tree and recreating it from scratch
> works, as does manually re-running configure,
> but the Makefile ought to re-run parts of configure
> etc when it needs to.
>

Can you send your config-host.mak after a failed rebuild? I think what's
happening is that the path to meson has changed but Makefile still tries
the old one (which could be the system meson in /usr/bin).

If so, you should be able to get out of the breakage by adding

MESON=pyvenv/bin/meson

after the "-include config-host.mak" line.

Paolo


> It does seem to try to, but (unlike a manual configure re-run)
> it doesn't prevent the failure.
>
> You can see that it says it needs to run configure, and mkvenv
> claims to be installing meson 0.6.3, but then the actual build
> says it didn't work. (A second run of 'make' at this point doesn't
> do the rerun of configure but meson fails in the same way.)
>
> Other weird things:
> (1) the error message is printed three times
> (2) "warn: ignoring non-existent submodule meson"
>
> $ make -C build/x86-tgts
> make: Entering directory
> '/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts'
> config-host.mak is out-of-date, running configure
> python determined to be '/usr/bin/python3'
> python version: Python 3.10.6
> mkvenv: Creating non-isolated virtual environment at 'pyvenv'
> mkvenv: checking for meson>=0.63.0
> mkvenv: installing meson>=0.63.0
> mkvenv: checking for sphinx>=1.6.0, sphinx-rtd-theme>=0.5.0
> /usr/bin/ninja  build.ninja && touch build.ninja.stamp
> [0/1] Regenerating build files.
> The Meson build system
> Version: 0.61.5
> Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
> Build type: native build
>
> ../../meson.build:1:0: ERROR: Meson version is 0.61.5 but project
> requires >=0.63.0
>
> A full log can be found at
>
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
> FAILED: build.ninja
> /usr/bin/python3
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
> regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
> ninja
> ninja: error: rebuilding 'build.ninja': subcommand failed
>   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 dtc
> warn: ignoring non-existent submodule meson
> /usr/bin/ninja  build.ninja && touch build.ninja.stamp
> [0/1] Regenerating build files.
> The Meson build system
> Version: 0.61.5
> Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
> Build type: native build
>
> ../../meson.build:1:0: ERROR: Meson version is 0.61.5 but project
> requires >=0.63.0
>
> A full log can be found at
>
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
> FAILED: build.ninja
> /usr/bin/python3
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
> regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
> ninja
> ninja: error: rebuilding 'build.ninja': subcommand failed
>   GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 dtc
> [0/1] Regenerating build files.
> The Meson build system
> Version: 0.61.5
> Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
> Build type: native build
>
> ../../meson.build:1:0: ERROR: Meson version is 0.61.5 but project
> requires >=0.63.0
>
> A full log can be found at
>
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
> ninja: error: rebuilding 'build.ninja': subcommand failed
> FAILED: build.ninja
> /usr/bin/python3
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
> regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
> ninja
> make: *** [Makefile:165: run-ninja] Error 1
> make: Leaving directory
> '/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts'
>
>
> thanks
> -- PMM
>
>

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Peter Maydell 2 years, 8 months ago

On Fri, 19 May 2023 at 17:51, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
>
> Il ven 19 mag 2023, 16:39 Peter Maydell <peter.maydell@linaro.org> ha scritto:
>>
>> On Thu, 18 May 2023 at 12:41, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> > ----------------------------------------------------------------
>> > * kvm: enable dirty ring for arm64
>> > * target/i386: new features
>> > * target/i386: AVX fixes
>> > * configure: create a python venv unconditionally
>> > * meson: bump to 0.63.0 and move tests from configure
>> > * meson: Pass -j option to sphinx
>> > * drop support for Python 3.6
>> > * fix check-python-tox
>> > * fix "make clean" in the source directory
>>
>> Hi; this seems to have broken builds from not-clean on my system.
>> Blowing away the build tree and recreating it from scratch
>> works, as does manually re-running configure,
>> but the Makefile ought to re-run parts of configure
>> etc when it needs to.
>
>
> Can you send your config-host.mak after a failed rebuild? I think what's happening is that the path to meson has changed but Makefile still tries the old one (which could be the system meson in /usr/bin).

Attached; it has
MESON=/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson

That meson says:
$ /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson
--version
1.1.0

1.1.0 is neither the system meson (which is 0.61.2) nor
the version the build system wants (0.63.0) nor the
version that it complains that it's found (0.61.5),
so I'm not sure what's going on there...

> If so, you should be able to get out of the breakage by adding
>
> MESON=pyvenv/bin/meson
>
> after the "-include config-host.mak" line.

This had no effect (unsurprising given that it's the same meson
as the config-host.mak specifies).

-- PMM

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Paolo Bonzini 2 years, 8 months ago

On 5/19/23 19:01, Peter Maydell wrote:
>> Can you send your config-host.mak after a failed rebuild? I think
>> what's happening is that the path to meson has changed but Makefile
>> still tries the old one (which could be the system meson in
>> /usr/bin).
> 
> Attached; it has 
> MESON=/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson
>
>  That meson says: $
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson --version> 
> 1.1.0 is neither the system meson (which is 0.61.2) nor the version
> the build system wants (0.63.0) nor the version that it complains
> that it's found (0.61.5), so I'm not sure what's going on there...

1.1.0 must be something you have cached somewhere, but 0.61.5 comes from
the old submodule.  You should be able to find it with

grep -wA2 'rule REGENERATE_BUILD' ../build.ninja

You should be able to unhose the directory with

pyvenv/bin/meson setup --reconfigure ../..

(where ../.. should be the path to the sources) or even with the patch:

diff --git a/Makefile b/Makefile
index 3c7d67142f13..08fb6a3b058a 100644
--- a/Makefile
+++ b/Makefile
@@ -115,15 +115,15 @@ Makefile.ninja: build.ninja
  	  $(NINJA) -t query build.ninja | sed -n '1,/^  input:/d; /^  outputs:/q; s/$$/ \\/p'; \
  	} > $@.tmp && mv $@.tmp $@
  -include Makefile.ninja
+endif
  
+ifneq ($(MESON),)
  # A separate rule is needed for Makefile dependencies to avoid -n
  build.ninja: build.ninja.stamp
  $(build-files):
  build.ninja.stamp: meson.stamp $(build-files)
-	$(NINJA) $(if $V,-v,) build.ninja && touch $@
-endif
+	$(MESON) setup --reconfigure $(SRC_PATH) && touch $@
  
-ifneq ($(MESON),)
  Makefile.mtest: build.ninja scripts/mtest2make.py
  	$(MESON) introspect --targets --tests --benchmarks | $(PYTHON) scripts/mtest2make.py > $@
  -include Makefile.mtest

Paolo

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Peter Maydell 2 years, 8 months ago

On Fri, 19 May 2023 at 18:31, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 5/19/23 19:01, Peter Maydell wrote:
> >> Can you send your config-host.mak after a failed rebuild? I think
> >> what's happening is that the path to meson has changed but Makefile
> >> still tries the old one (which could be the system meson in
> >> /usr/bin).
> >
> > Attached; it has
> > MESON=/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson
> >
> >  That meson says: $
> > /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson --version>
> > 1.1.0 is neither the system meson (which is 0.61.2) nor the version
> > the build system wants (0.63.0) nor the version that it complains
> > that it's found (0.61.5), so I'm not sure what's going on there...
>
> 1.1.0 must be something you have cached somewhere

Well, I've never done anything with meson except use whatever
the QEMU build system provides...

> but 0.61.5 comes from
> the old submodule.  You should be able to find it with
>
> grep -wA2 'rule REGENERATE_BUILD' ../build.ninja

Yep:

$ grep -wA2 'rule REGENERATE_BUILD' build/x86-tgts/build.ninja
rule REGENERATE_BUILD
 command = /usr/bin/python3
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
ninja
 description = Regenerating build files.

$ /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --version
0.61.5

> You should be able to unhose the directory with
>
> pyvenv/bin/meson setup --reconfigure ../..
>
> (where ../.. should be the path to the sources) or even with the patch:
>
> diff --git a/Makefile b/Makefile
> index 3c7d67142f13..08fb6a3b058a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -115,15 +115,15 @@ Makefile.ninja: build.ninja
>           $(NINJA) -t query build.ninja | sed -n '1,/^  input:/d; /^  outputs:/q; s/$$/ \\/p'; \
>         } > $@.tmp && mv $@.tmp $@
>   -include Makefile.ninja
> +endif
>
> +ifneq ($(MESON),)
>   # A separate rule is needed for Makefile dependencies to avoid -n
>   build.ninja: build.ninja.stamp
>   $(build-files):
>   build.ninja.stamp: meson.stamp $(build-files)
> -       $(NINJA) $(if $V,-v,) build.ninja && touch $@
> -endif
> +       $(MESON) setup --reconfigure $(SRC_PATH) && touch $@
>
> -ifneq ($(MESON),)
>   Makefile.mtest: build.ninja scripts/mtest2make.py
>         $(MESON) introspect --targets --tests --benchmarks | $(PYTHON) scripts/mtest2make.py > $@
>   -include Makefile.mtest

This patch doesn't work, I'm afraid:

$ make -C build/x86-tgts/
make: Entering directory
'/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts'
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson
setup --reconfigure /mnt/nvmedisk/linaro/qemu-from-laptop/qemu &&
touch build.ninja.stamp
WARNING: Regenerating configuration from scratch.
Reason: Coredata file
'/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-private/coredata.dat'
references functions or classes that don't exist. This probably means
that it was generated with an old version of meson. Try running from
the source directory meson setup
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --wipe
The Meson build system
Version: 1.1.0
Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
Build type: native build

../../meson.build:1:0: ERROR: Unknown options: "sphinx_build"

A full log can be found at
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
  GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc
[0/1] Regenerating build files.
The Meson build system
Version: 0.61.5
Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
Build type: native build

../../meson.build:1:0: ERROR: Meson version is 0.61.5 but project
requires >=0.63.0

A full log can be found at
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-logs/meson-log.txt
ninja: error: rebuilding 'build.ninja': subcommand failed
FAILED: build.ninja
/usr/bin/python3
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
ninja
make: *** [Makefile:165: run-ninja] Error 1
make: Leaving directory
'/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts'

-- PMM

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Paolo Bonzini 2 years, 8 months ago

On 5/19/23 19:45, Peter Maydell wrote:
> On Fri, 19 May 2023 at 18:31, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>> On 5/19/23 19:01, Peter Maydell wrote:
>>>> Can you send your config-host.mak after a failed rebuild? I think
>>>> what's happening is that the path to meson has changed but Makefile
>>>> still tries the old one (which could be the system meson in
>>>> /usr/bin).
>>>
>>> Attached; it has
>>> MESON=/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson
>>>
>>>   That meson says: $
>>> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson --version>
>>> 1.1.0 is neither the system meson (which is 0.61.2) nor the version
>>> the build system wants (0.63.0) nor the version that it complains
>>> that it's found (0.61.5), so I'm not sure what's going on there...
>>
>> 1.1.0 must be something you have cached somewhere
> 
> Well, I've never done anything with meson except use whatever
> the QEMU build system provides...
> 
>> but 0.61.5 comes from
>> the old submodule.  You should be able to find it with
>>
>> grep -wA2 'rule REGENERATE_BUILD' ../build.ninja
> 
> Yep:
> 
> $ grep -wA2 'rule REGENERATE_BUILD' build/x86-tgts/build.ninja
> rule REGENERATE_BUILD
>   command = /usr/bin/python3
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --internal
> regenerate /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --backend
> ninja
>   description = Regenerating build files.
> 
> $ /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/meson/meson.py --version
> 0.61.5
> 
>> You should be able to unhose the directory with
>>
>> pyvenv/bin/meson setup --reconfigure ../..
>>
>> (where ../.. should be the path to the sources) or even with the patch:
>>
>> diff --git a/Makefile b/Makefile
>> index 3c7d67142f13..08fb6a3b058a 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -115,15 +115,15 @@ Makefile.ninja: build.ninja
>>            $(NINJA) -t query build.ninja | sed -n '1,/^  input:/d; /^  outputs:/q; s/$$/ \\/p'; \
>>          } > $@.tmp && mv $@.tmp $@
>>    -include Makefile.ninja
>> +endif
>>
>> +ifneq ($(MESON),)
>>    # A separate rule is needed for Makefile dependencies to avoid -n
>>    build.ninja: build.ninja.stamp
>>    $(build-files):
>>    build.ninja.stamp: meson.stamp $(build-files)
>> -       $(NINJA) $(if $V,-v,) build.ninja && touch $@
>> -endif
>> +       $(MESON) setup --reconfigure $(SRC_PATH) && touch $@
>>
>> -ifneq ($(MESON),)
>>    Makefile.mtest: build.ninja scripts/mtest2make.py
>>          $(MESON) introspect --targets --tests --benchmarks | $(PYTHON) scripts/mtest2make.py > $@
>>    -include Makefile.mtest
> 
> This patch doesn't work, I'm afraid:
> 
> $ make -C build/x86-tgts/
> make: Entering directory
> '/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts'
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/pyvenv/bin/meson
> setup --reconfigure /mnt/nvmedisk/linaro/qemu-from-laptop/qemu &&
> touch build.ninja.stamp
> WARNING: Regenerating configuration from scratch.
> Reason: Coredata file
> '/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts/meson-private/coredata.dat'
> references functions or classes that don't exist. This probably means
> that it was generated with an old version of meson. Try running from
> the source directory meson setup
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts --wipe
> The Meson build system
> Version: 1.1.0
> Source dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu
> Build dir: /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86-tgts
> Build type: native build
> 
> ../../meson.build:1:0: ERROR: Unknown options: "sphinx_build"

Well, it almost works---notice that now you are running the 1.1.0 from 
pyvenv/bin/meson.  For now you can proceed with

    sed -i /sphinx_build/d meson-private/cmd_line.txt

and I'll try to send a proper patch over the weekend.

Paolo

Re: [PULL v3 00/68] i386, build system, KVM changes for 2023-05-18

Posted by Richard Henderson 2 years, 8 months ago

On 5/18/23 04:40, Paolo Bonzini wrote:
> The following changes since commit d27e7c359330ba7020bdbed7ed2316cb4cf6ffc1:
> 
>    qapi/parser: Drop two bad type hints for now (2023-05-17 10:18:33 -0700)
> 
> are available in the Git repository at:
> 
>    https://gitlab.com/bonzini/qemu.git  tags/for-upstream
> 
> for you to fetch changes up to fe3ab4eb2de46076cbafcbc86b22e71ad24894c6:
> 
>    docs/devel: update build system docs (2023-05-18 13:35:28 +0200)
> 
> ----------------------------------------------------------------
> * kvm: enable dirty ring for arm64
> * target/i386: new features
> * target/i386: AVX fixes
> * configure: create a python venv unconditionally
> * meson: bump to 0.63.0 and move tests from configure
> * meson: Pass -j option to sphinx
> * drop support for Python 3.6
> * fix check-python-tox
> * fix "make clean" in the source directory
> 
> ----------------------------------------------------------------
> v2->v3:
>          fix cut-and-paste typo in "build: move warning flag selection to meson"

Applied, thanks.  Please update https://wiki.qemu.org/ChangeLog/8.1 as appropriate.


r~

[PULL 59/68] build: move warning flag selection to meson