.cirrus.yml | 6 +- .travis.yml | 13 + Makefile | 42 +- configure | 29 +- docs/devel/build-system.rst | 6 +- meson.build | 34 +- scripts/mtest2make.py | 4 +- scripts/ninjatool.py | 1008 -------------------- tests/Makefile.include | 2 +- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/centos8.docker | 4 + tests/docker/dockerfiles/debian10.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/travis.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 + tests/docker/dockerfiles/ubuntu1804.docker | 1 + tests/docker/dockerfiles/ubuntu2004.docker | 1 + tests/include/meson.build | 8 +- tests/meson.build | 14 +- tests/vm/centos | 2 +- tests/vm/centos.aarch64 | 2 +- tests/vm/fedora | 2 +- tests/vm/freebsd | 1 + tests/vm/netbsd | 1 + tests/vm/openbsd | 1 + tests/vm/ubuntu.aarch64 | 2 +- tests/vm/ubuntu.i386 | 2 +- 27 files changed, 121 insertions(+), 1070 deletions(-) delete mode 100755 scripts/ninjatool.py
This pull request is the last build system change for 5.2 from me, and it is simple: similar to how we are invoking ROM or tests/tcg "make" from the main Makefile, we now invoke ninja to build QEMU. Unlike those cases, however, build.ninja targets are forwarded transparently. The advantages cover various areas: - maintainability: we drop scripts/ninjatool.py, which is a large and hairy piece of code, and generally remove one thing that can go wrong and one thing that is unique to QEMU; - platform support: we remove the requirement for GNU make 3.82, which was annoying on Mac. We also avoid bugs on Windows due to meson emitting Windows rather than POSIX escapes (as expected by Ninja) and ninjatool ignoring the difference; - speed: invoking "configure" does not have to generate 44k lines of rules, while invoking "Make" does not anymore have to parse 44k lines of rules. - ease of use: Ninja tracks command lines, hence the problem of static library changing the objects they hold goes away Paolo Paolo Bonzini (7): tests/Makefile.include: unbreak non-tcg builds make: run shell with pipefail tests: add missing generated sources to testqapi configure: move QEMU_INCLUDES to meson dockerfiles: enable Centos 8 PowerTools add ninja to dockerfiles, CI configurations and test VMs build: replace ninjatool with ninja .cirrus.yml | 6 +- .travis.yml | 13 + Makefile | 42 +- configure | 29 +- docs/devel/build-system.rst | 6 +- meson.build | 34 +- scripts/mtest2make.py | 4 +- scripts/ninjatool.py | 1008 -------------------- tests/Makefile.include | 2 +- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/centos8.docker | 4 + tests/docker/dockerfiles/debian10.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/travis.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 + tests/docker/dockerfiles/ubuntu1804.docker | 1 + tests/docker/dockerfiles/ubuntu2004.docker | 1 + tests/include/meson.build | 8 +- tests/meson.build | 14 +- tests/vm/centos | 2 +- tests/vm/centos.aarch64 | 2 +- tests/vm/fedora | 2 +- tests/vm/freebsd | 1 + tests/vm/netbsd | 1 + tests/vm/openbsd | 1 + tests/vm/ubuntu.aarch64 | 2 +- tests/vm/ubuntu.i386 | 2 +- 27 files changed, 121 insertions(+), 1070 deletions(-) delete mode 100755 scripts/ninjatool.py -- 2.26.2
On Wed, Oct 14, 2020 at 3:57 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
> This pull request is the last build system change for 5.2 from
> me, and it is simple: similar to how we are invoking
> ROM or tests/tcg "make" from the main Makefile, we now invoke ninja
> to build QEMU. Unlike those cases, however, build.ninja targets are
> forwarded transparently.
>
> The advantages cover various areas:
>
> - maintainability: we drop scripts/ninjatool.py, which is
> a large and hairy piece of code, and generally remove one
> thing that can go wrong and one thing that is unique to QEMU;
>
> - platform support: we remove the requirement for GNU make
> 3.82, which was annoying on Mac. We also avoid bugs on Windows
> due to meson emitting Windows rather than POSIX escapes (as
> expected by Ninja) and ninjatool ignoring the difference;
>
> - speed: invoking "configure" does not have to generate
> 44k lines of rules, while invoking "Make" does not anymore have
> to parse 44k lines of rules.
>
> - ease of use: Ninja tracks command lines, hence the problem
> of static library changing the objects they hold goes away
>
> Paolo
>
>
Hi,
I still have some issues when compiling on Windows with msys2/mingw64 after
this patch set. Command line:
mkdir build && cd build && ../configure --cross-prefix=x86_64-w64-mingw32-
--target-list=ppc-softmmu --enable-gtk --enable-sdl
Traceback (most recent call last):
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/mesonmain.py",
line 131, in run
return options.run_func(options)
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/msetup.py",
line 245, in run
app.generate()
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/msetup.py",
line 159, in generate
self._generate(env)
File "C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/msetup.py",
line 215, in _generate
intr.backend.generate()
File
"C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/backend/ninjabackend.py",
line 483, in generate
ninja = environment.detect_ninja_command_and_version(log=True)
File
"C:/msys64/home/hsp/src/qemu-master/meson/mesonbuild/environment.py", line
177, in detect_ninja_command_and_version
name = os.path.basename(n)
File "C:/msys64/mingw64/lib/python3.8/ntpath.py", line 231, in basename
return split(p)[1]
File "C:/msys64/mingw64/lib/python3.8/ntpath.py", line 200, in split
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: meson setup failed
When I set --ninja=ninja explicitly, this error does not occur:
Found ninja.EXE-1.10.1 at C:\msys64\mingw64\bin/ninja.EXE
However, compiling then runs into a problem:
$ make -j8
/mingw64/bin/python3 -B /home/hsp/src/qemu-master/meson/meson.py introspect
--targets --tests --benchmarks | /mingw64/bin/python3 -B
scripts/mtest2make.py > Makefile.mtest
{ echo 'ninja-targets = \'; ninja -t targets all | sed 's/:.*//; $!s/$/
\\/'; } > Makefile.ninja
ninja -j8 all | cat
make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop.
make: *** [Makefile:171: pc-bios/optionrom/all] Error 2
make: *** Waiting for unfinished jobs....
The problem with optionrom was already present earlier and could be fixed
by applying:
https://patchwork.ozlabs.org/project/qemu-devel/list/?series=202798
That patch set does by now not apply cleanly anymore.
Best,
Howard
On 15/10/20 08:15, Howard Spoelstra wrote: > ERROR: meson setup failed > > When I set --ninja=ninja explicitly, this error does not occur: > Found ninja.EXE-1.10.1 at C:\msys64\mingw64\bin/ninja.EXE That is fixed by Meson 0.55.2, I'll add a submodule update. Thanks for the reminder! Paolo
On Thu, Oct 15, 2020 at 8:34 AM Paolo Bonzini <pbonzini@redhat.com> wrote: > On 15/10/20 08:15, Howard Spoelstra wrote: > > ERROR: meson setup failed > > > > When I set --ninja=ninja explicitly, this error does not occur: > > Found ninja.EXE-1.10.1 at C:\msys64\mingw64\bin/ninja.EXE > > That is fixed by Meson 0.55.2, I'll add a submodule update. Thanks for > the reminder! > > Paolo > > Thanks Paolo, Then only the issue regarding the pcbios/optionrom stuff remains ;-) make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop. make: *** [Makefile:171: pc-bios/optionrom/all] Error 2 make: *** Waiting for unfinished jobs.... Best, Howard
> > Thanks Paolo, > > Then only the issue regarding the pcbios/optionrom stuff remains ;-) > > make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop. > make: *** [Makefile:171: pc-bios/optionrom/all] Error 2 > make: *** Waiting for unfinished jobs.... > > Best, > Howard > Hi Howard, one solution for this issue is to uncomment the following line in msys2_shell.cmd. rem To activate windows native symlinks uncomment next line set MSYS=winsymlinks:nativestrict Then tell Windows 10 it's okay to create symlinks without elevated rights. Here is a link with a description how to do this. https://www.joshkel.com/2018/01/18/symlinks-in-windows/ I think since commit bf708f3c4a "optionrom: simplify Makefile" pc-bios/optionrom/Makefile in your build directory has to be a symbolic link. Without 'set MSYS=winsymlinks:nativestrict' msys2 ln -s copies the Makefile instead of creating a symbolic link. With best regards, Volker
On 15/10/2020 18:39, Volker Rümelin wrote: >> Thanks Paolo, >> >> Then only the issue regarding the pcbios/optionrom stuff remains ;-) >> >> make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop. >> make: *** [Makefile:171: pc-bios/optionrom/all] Error 2 >> make: *** Waiting for unfinished jobs.... >> >> Best, >> Howard >> > > Hi Howard, > > one solution for this issue is to uncomment the following line in msys2_shell.cmd. > > rem To activate windows native symlinks uncomment next line > set MSYS=winsymlinks:nativestrict > > Then tell Windows 10 it's okay to create symlinks without elevated rights. Here is a link with a description how to do this. > https://www.joshkel.com/2018/01/18/symlinks-in-windows/ > > I think since commit bf708f3c4a "optionrom: simplify Makefile" pc-bios/optionrom/Makefile in your build directory has to be a symbolic link. Without 'set MSYS=winsymlinks:nativestrict' msys2 ln -s copies the Makefile instead of creating a symbolic link. Wow thanks for this. I had the same issue as Howard, and I can confirm that this fixes the problem for me. Having said that it was quite fiddly to get this working correctly - is there any reason why https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html still can't be merged? ATB, Mark.
Il gio 15 ott 2020, 20:49 Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> ha scritto: > Is there any reason why > https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html still > can't be > merged? > Because it's not the right approach. There is no reason why building firmware cannot be done with cross compilers, so moving those directories to Meson (not because Meson can't handle them; more specifically, the issue is tying the firmware build to the QEMU build system) is going in the wrong direction. The "Canadian cross" scenario, where you build on Linux a mingw GCC but the compiler is s390, is not even enough to describe the complexity in the case of QEMU, because there are multiple firmware for different machines. However we already have all the infrastructure to do such builds, we just don't use it for the firmware. So, instead of the patch you recalled above, the tests/tcg machinery should be extended into something that can be reused for firmware. As an aside, orchestrating this multi-compiler part of the build is what the Makefiles will keep on handling for the foreseeable future. As an aside to the aside, tests/tcg is more than underdocumented and I forget everything about it 5 minutes after looking at it. This is not something that I will be able to work on anytime soon. But still I don't think that going in the wrong direction is a good idea, even if temporarily. Paolo > > ATB, > > Mark. > >
On 15/10/2020 22:41, Paolo Bonzini wrote: > Il gio 15 ott 2020, 20:49 Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk > <mailto:mark.cave-ayland@ilande.co.uk>> ha scritto: > > Is there any reason why > https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html > <https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg06997.html> still can't be > merged? > > > Because it's not the right approach. There is no reason why building firmware cannot > be done with cross compilers, so moving those directories to Meson (not because Meson > can't handle them; more specifically, the issue is tying the firmware build to the > QEMU build system) is going in the wrong direction. > > The "Canadian cross" scenario, where you build on Linux a mingw GCC but the compiler > is s390, is not even enough to describe the complexity in the case of QEMU, because > there are multiple firmware for different machines. > > However we already have all the infrastructure to do such builds, we just don't use > it for the firmware. So, instead of the patch you recalled above, the tests/tcg > machinery should be extended into something that can be reused for firmware. As an > aside, orchestrating this multi-compiler part of the build is what the Makefiles will > keep on handling for the foreseeable future. As an aside to the aside, tests/tcg is > more than underdocumented and I forget everything about it 5 minutes after looking at it. > > This is not something that I will be able to work on anytime soon. But still I don't > think that going in the wrong direction is a good idea, even if temporarily. That's a shame, although I do appreciate the huge amount of time and effort that you've put into this release in order to get the Meson build up and running, and so why taking on another large task is going to be lower down the list :) At the moment OpenBIOS doesn't have a docker image capable of building the required binaries: I did experiment with trying to use the QEMU docker images for openbios-ppc but whilst the binary built successfully, it did not run compared to my hand-rolled compilers. So there's still some debugging to be done there... ATB, Mark.
On Thu, Oct 15, 2020 at 7:39 PM Volker Rümelin <vr_qemu@t-online.de> wrote: > > > > > Thanks Paolo, > > > > Then only the issue regarding the pcbios/optionrom stuff remains ;-) > > > > make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. > Stop. > > make: *** [Makefile:171: pc-bios/optionrom/all] Error 2 > > make: *** Waiting for unfinished jobs.... > > > > Best, > > Howard > > > > Hi Howard, > > one solution for this issue is to uncomment the following line in > msys2_shell.cmd. > > rem To activate windows native symlinks uncomment next line > set MSYS=winsymlinks:nativestrict > > Then tell Windows 10 it's okay to create symlinks without elevated rights. > Here is a link with a description how to do this. > https://www.joshkel.com/2018/01/18/symlinks-in-windows/ > > I think since commit bf708f3c4a "optionrom: simplify Makefile" > pc-bios/optionrom/Makefile in your build directory has to be a symbolic > link. Without 'set MSYS=winsymlinks:nativestrict' msys2 ln -s copies the > Makefile instead of creating a symbolic link. > > Thanks Volker! I changed the Windows policy and setting in msys2_shell.cmd. However, I had to edit ming64.ini to uncomment the MSYS=winsymlinks:nativestrict there as well to get things going. While it is great to have this fix, I can't say I'm really happy with the need to change the Windows policy and to have to fix msys2. The patches Mark referred to and to which I also pointed earlier did fix this problem without changing Windows/Msys2/Mingw64 settings. Best regards, Howard
On 15/10/20 09:39, Howard Spoelstra wrote: > > > Thanks Paolo, > > Then only the issue regarding the pcbios/optionrom stuff remains ;-) > > make[1]: *** No rule to make target 'multiboot.bin', needed by 'all'. Stop. > make: *** [Makefile:171: pc-bios/optionrom/all] Error 2 > make: *** Waiting for unfinished jobs.... It seems to work on cirrus CI: https://cirrus-ci.com/task/5383432727429120 Paolo
On Wed, 14 Oct 2020 at 14:54, Paolo Bonzini <pbonzini@redhat.com> wrote: > > This pull request is the last build system change for 5.2 from > me, and it is simple: similar to how we are invoking > ROM or tests/tcg "make" from the main Makefile, we now invoke ninja > to build QEMU. Unlike those cases, however, build.ninja targets are > forwarded transparently. > 27 files changed, 121 insertions(+), 1070 deletions(-) The diffstat is certainly persuasive :-) -- PMM
On Wed, Oct 14, 2020 at 9:58 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> This pull request is the last build system change for 5.2 from
> me, and it is simple: similar to how we are invoking
> ROM or tests/tcg "make" from the main Makefile, we now invoke ninja
> to build QEMU. Unlike those cases, however, build.ninja targets are
> forwarded transparently.
>
> The advantages cover various areas:
>
> - maintainability: we drop scripts/ninjatool.py, which is
> a large and hairy piece of code, and generally remove one
> thing that can go wrong and one thing that is unique to QEMU;
>
> - platform support: we remove the requirement for GNU make
> 3.82, which was annoying on Mac. We also avoid bugs on Windows
> due to meson emitting Windows rather than POSIX escapes (as
> expected by Ninja) and ninjatool ignoring the difference;
>
> - speed: invoking "configure" does not have to generate
> 44k lines of rules, while invoking "Make" does not anymore have
> to parse 44k lines of rules.
>
> - ease of use: Ninja tracks command lines, hence the problem
> of static library changing the objects they hold goes away
>
> Paolo
>
> Paolo Bonzini (7):
> tests/Makefile.include: unbreak non-tcg builds
> make: run shell with pipefail
> tests: add missing generated sources to testqapi
> configure: move QEMU_INCLUDES to meson
> dockerfiles: enable Centos 8 PowerTools
> add ninja to dockerfiles, CI configurations and test VMs
> build: replace ninjatool with ninja
>
> .cirrus.yml | 6 +-
> .travis.yml | 13 +
> Makefile | 42 +-
> configure | 29 +-
> docs/devel/build-system.rst | 6 +-
> meson.build | 34 +-
> scripts/mtest2make.py | 4 +-
> scripts/ninjatool.py | 1008 --------------------
> tests/Makefile.include | 2 +-
> tests/docker/dockerfiles/centos7.docker | 1 +
> tests/docker/dockerfiles/centos8.docker | 4 +
> tests/docker/dockerfiles/debian10.docker | 1 +
> tests/docker/dockerfiles/fedora.docker | 1 +
> tests/docker/dockerfiles/travis.docker | 2 +-
> tests/docker/dockerfiles/ubuntu.docker | 1 +
> tests/docker/dockerfiles/ubuntu1804.docker | 1 +
> tests/docker/dockerfiles/ubuntu2004.docker | 1 +
> tests/include/meson.build | 8 +-
> tests/meson.build | 14 +-
> tests/vm/centos | 2 +-
> tests/vm/centos.aarch64 | 2 +-
> tests/vm/fedora | 2 +-
> tests/vm/freebsd | 1 +
> tests/vm/netbsd | 1 +
> tests/vm/openbsd | 1 +
> tests/vm/ubuntu.aarch64 | 2 +-
> tests/vm/ubuntu.i386 | 2 +-
> 27 files changed, 121 insertions(+), 1070 deletions(-)
> delete mode 100755 scripts/ninjatool.py
>
> --
> 2.26.2
>
>
That's good, it's broken my locally rST doc build
```
[226/1564] Generating input-keymap-qnum-to-qcode.c.inc with a custom
command (wrapped by meson to capture output)
[227/1564] Generating input-keymap-qcode-to-qnum.c.inc with a custom
command (wrapped by meson to capture output)
[228/1564] Generating shared QAPI source files with a custom command
[229/1564] Generating input-keymap-qcode-to-linux.c.inc with a custom
command (wrapped by meson to capture output)
[230/1564] Generating input-keymap-win32-to-qcode.c.inc with a custom
command (wrapped by meson to capture output)
[231/1564] Generating QAPI rST doc with a custom command
FAILED: tests/qapi-schema/doc-good.txt
"/mingw64/bin/sphinx-build" "-Dversion=5.1.50" "-Drelease=" "-W" "-b"
"text" "-E" "-c" "C:/work/xemu/qemu/docs" "-D" "master_doc=doc-good"
"C:/work/xemu/qemu/tests/qapi-schema"
"C:/work/xemu/qemu/build/tests/qapi-schema"
CreateProcess failed: The system cannot find the file specified.
[232/1564] Generating input-keymap-qcode-to-atset1.c.inc with a custom
command (wrapped by meson to capture output)
[233/1564] Generating generated-helpers-wrappers.h with a custom command
(wrapped by meson to capture output)
ninja: fatal: GetOverlappedResult: �������
make: *** [Makefile:87：run-ninja] 错误 1
/mingw64/bin/ninja -v -j1 tests/check-block-qdict.exe
tests/check-qdict.exe tests/check-qjson.exe tests/check-qlist.exe
tests/check-qlit.exe tests/check-qnull.exe tests/check-qnum.exe
tests/check-qobject.exe tests/check-qom-interface.exe
tests/check-qom-proplist.exe tests/check-qstring.exe tests/ptimer-test.exe
tests/qapi-schema/doc-good.ref.nocr tests/qapi-schema/doc-good.txt.nocr
tests/rcutorture.exe tests/test-aio-multithread.exe tests/test-aio.exe
tests/test-authz-list.exe tests/test-authz-listfile.exe
tests/test-authz-simple.exe tests/test-base64.exe tests/test-bdrv-drain.exe
tests/test-bdrv-graph-mod.exe tests/test-bitcnt.exe tests/test-bitmap.exe
tests/test-bitops.exe tests/test-block-backend.exe
tests/test-block-iothread.exe tests/test-blockjob-txn.exe
tests/test-blockjob.exe tests/test-bufferiszero.exe tests/test-char.exe
tests/test-clone-visitor.exe tests/test-coroutine.exe
tests/test-crypto-afsplit.exe tests/test-crypto-block.exe
tests/test-crypto-cipher.exe tests/test-crypto-hash.exe
tests/test-crypto-hmac.exe tests/test-crypto-ivgen.exe
tests/test-crypto-pbkdf.exe tests/test-crypto-secret.exe
tests/test-cutils.exe tests/test-hbitmap.exe tests/test-int128.exe
tests/test-io-channel-buffer.exe tests/test-io-channel-command.exe
tests/test-io-channel-file.exe tests/test-io-channel-socket.exe
tests/test-io-task.exe tests/test-iov.exe tests/test-keyval.exe
tests/test-logging.exe tests/test-mul64.exe tests/test-opts-visitor.exe
tests/test-qapi-util.exe tests/test-qdev-global-props.exe
tests/test-qdist.exe tests/test-qemu-opts.exe tests/test-qgraph.exe
tests/test-qht.exe tests/test-qmp-cmds.exe tests/test-qmp-event.exe
tests/test-qobject-input-visitor.exe tests/test-qobject-output-visitor.exe
tests/test-rcu-list.exe tests/test-rcu-simpleq.exe tests/test-rcu-slist.exe
tests/test-rcu-tailq.exe tests/test-replication.exe tests/test-shift128.exe
tests/test-string-input-visitor.exe tests/test-string-output-visitor.exe
tests/test-thread-pool.exe tests/test-throttle.exe
tests/test-timed-average.exe tests/test-util-sockets.exe
tests/test-uuid.exe tests/test-visitor-serialization.exe
tests/test-vmstate.exe tests/test-write-threshold.exe
tests/test-x86-cpuid.exe tests/test-xbzrle.exe | cat
```
That's because currently sphinx-build are found in configure not in
meson, maybe it's worth to move the sphinx related thing into meson totally.
--
此致
礼
罗勇刚
Yours
sincerely,
Yonggang Luo
© 2016 - 2025 Red Hat, Inc.