[PULL 00/40] Rust changes for QEMU 9.2 soft freeze

Paolo Bonzini posted 40 patches 2 weeks, 5 days ago
There is a newer version of this series
docs/about/build-platforms.rst                     |  12 +
meson.build                                        | 135 +++++---
include/hw/qdev-core.h                             |   4 +-
include/hw/qdev-properties.h                       |   4 +-
rust/wrapper.h                                     |  17 +
hw/core/qdev-properties.c                          |  26 +-
system/qdev-monitor.c                              |   2 +-
.gitattributes                                     |   2 +
.gitlab-ci.d/buildtest.yml                         |   6 +-
meson_options.txt                                  |   2 +
rust/{hw/char/pl011 => }/Cargo.lock                |   4 +
rust/Cargo.toml                                    |   7 +
rust/hw/char/Kconfig                               |   1 -
rust/hw/char/pl011/Cargo.toml                      |   3 -
rust/hw/char/pl011/src/device.rs                   | 162 +++++++---
rust/hw/char/pl011/src/device_class.rs             |  80 ++++-
rust/hw/char/pl011/src/lib.rs                      |   6 +-
rust/hw/char/pl011/src/memory_ops.rs               |  24 +-
rust/qemu-api-macros/Cargo.lock                    |  47 ---
rust/qemu-api-macros/Cargo.toml                    |   5 +-
rust/qemu-api-macros/meson.build                   |   2 +-
rust/qemu-api-macros/src/lib.rs                    | 103 ++++--
rust/qemu-api/Cargo.lock                           |   7 -
rust/qemu-api/Cargo.toml                           |  10 +-
rust/qemu-api/build.rs                             |   9 +
rust/qemu-api/meson.build                          |  44 ++-
rust/qemu-api/src/c_str.rs                         |  53 +++
rust/qemu-api/src/definitions.rs                   |  68 ++--
rust/qemu-api/src/device_class.rs                  | 114 ++-----
rust/qemu-api/src/lib.rs                           |  23 +-
rust/qemu-api/src/offset_of.rs                     | 161 +++++++++
rust/qemu-api/src/tests.rs                         |  49 ---
rust/qemu-api/src/vmstate.rs                       | 360 +++++++++++++++++++++
rust/qemu-api/src/zeroable.rs                      |  86 +++++
rust/qemu-api/tests/tests.rs                       |  79 +++++
scripts/ci/setup/ubuntu/ubuntu-2204-aarch64.yaml   |   1 -
scripts/ci/setup/ubuntu/ubuntu-2204-s390x.yaml     |   1 -
scripts/meson-buildoptions.sh                      |   4 +
subprojects/bilge-impl-0.2-rs.wrap                 |   1 +
subprojects/packagefiles/bilge-impl-1.63.0.patch   |  45 +++
.../packagefiles/proc-macro2-1-rs/meson.build      |   4 +-
subprojects/packagefiles/syn-2-rs/meson.build      |   1 +
tests/docker/dockerfiles/ubuntu2204.docker         |   6 +-
tests/lcitool/mappings.yml                         |   4 +
tests/lcitool/refresh                              |  11 +-
45 files changed, 1379 insertions(+), 416 deletions(-)
[PULL 00/40] Rust changes for QEMU 9.2 soft freeze
Posted by Paolo Bonzini 2 weeks, 5 days ago
The following changes since commit 15195de6a93438be99fdf9a90992c4228527130d:

  ci: enable rust in the Fedora system build job (2024-10-30 16:30:56 +0100)

are available in the Git repository at:

  https://gitlab.com/bonzini/qemu.git tags/for-upstream-rust

for you to fetch changes up to d20feaa9a5af597bd20630d041e5dc7808612be1:

  ci: enable rust in the Debian and Ubuntu system build job (2024-10-31 18:39:52 +0100)

----------------------------------------------------------------
* rust: cleanups
* rust: integration tests
* rust/pl011: add support for migration
* rust/pl011: add TYPE_PL011_LUMINARY device
* rust: add support for older compilers and bindgen
* rust: enable rust in the Debian, Fedora and Ubuntu system build job

----------------------------------------------------------------

This pull request enables Rust in QEMU's CI infrastructure, as a
first step towards collaborative development of Rust features. It matches
the plan that I mentioned last Thursday at
https://lore.kernel.org/qemu-devel/CABgObfb7=ZxgiasgB=dE8yV+bhd5-pd51n4qGpP8OFNBS3iMXQ@mail.gmail.com/.

There is a lot of new code in here that is specifically from me.  Because of
the worry that new Rust code may introduce hidden technical debt, others have
reviewed all the Rust code in here, with the exception of the mostly trivial
https://lore.kernel.org/qemu-devel/20241025160209.194307-17-pbonzini@redhat.com/.

The changes focus on CI integration and compilation infrastructure:

* support for older Rust versions as found in QEMU's Debian bookworm and
  Ubuntu CI targets.  Workarounds for older Rust compiler versions are grouped
  together for future cleanup.

* passing qtests with --enable-rust testing support: the pl011 code is closer
  to parity with the C version, though still experimental.

Regarding toolchain compatibility: Michael Tokarev expressed some
doubts about supporting the old toolchain in Debian bookworm.  However,
even trixie (currently in "testing") would require workarounds, mostly
due to lack of stable "offset_of!"  support, so maintaining bookworm
compatibility seemed reasonable.

I am also working on a documentation patch that explains the more recent
features that would be nice to have.

Note that the Rust code still contains what is technically undefined
behavior (also known as "unsound code", i.e. unsafe code that does not
respect aliasing rules). Fixing this and other aspects will be easier
with the CI infrastructure in place. For example bindings, static checking
and improved developer ergonomics (e.g., clippy and rustfmt integration),
documentation generation would all be valid starting points.

For developers testing locally with --enable-rust, run "meson subprojects
update --reset" after pulling and before building.  Meson does not do this
step automatically due to it being potentially destructive.  This should
only affect this initial stabilization period, but a fix is in progress
(I first need to check with the Meson folks whether my script is using
stable interfaces).

Paolo

Junjie Mao (1):
      rust: introduce alternative implementation of offset_of!

Manos Pitsidianakis (9):
      rust/wrapper.h: define memory_order enum
      Revert "rust: add PL011 device model"
      rust: add PL011 device model
      rust: add definitions for vmstate
      rust/pl011: add support for migration
      rust/pl011: move CLK_NAME static to function scope
      rust/pl011: add TYPE_PL011_LUMINARY device
      rust/pl011: remove commented out C code
      rust/pl011: Use correct masks for IBRD and FBRD

Paolo Bonzini (30):
      qdev: make properties array "const"
      meson: import rust module into a global variable
      meson: remove repeated search for rust_root_crate.sh
      meson: pass rustc_args when building all crates
      rust: do not always select X_PL011_RUST
      rust: do not use --no-size_t-is-usize
      rust: remove uses of #[no_mangle]
      rust: modernize link_section usage for ELF platforms
      rust: build integration test for the qemu_api crate
      rust: cleanup module_init!, use it from #[derive(Object)]
      rust: clean up define_property macro
      rust: make properties array immutable
      rust: provide safe wrapper for MaybeUninit::zeroed()
      rust: do not use TYPE_CHARDEV unnecessarily
      rust/pl011: fix default value for migrate-clock
      rust: patch bilge-impl to allow compilation with 1.63.0
      rust: fix cfgs of proc-macro2 for 1.63.0
      rust: use std::os::raw instead of core::ffi
      rust: introduce a c_str macro
      rust: silence unknown warnings for the sake of old compilers
      rust: synchronize dependencies between subprojects and Cargo.lock
      rust: create a cargo workspace
      rust: do not use MaybeUninit::zeroed()
      rust: clean up detection of the language
      rust: allow version 1.63.0 of rustc
      rust: do not use --generate-cstr
      rust: allow older version of bindgen
      rust: make rustfmt optional
      dockerfiles: install bindgen from cargo on Ubuntu 22.04
      ci: enable rust in the Debian and Ubuntu system build job


 docs/about/build-platforms.rst                     |  12 +
 meson.build                                        | 135 +++++---
 include/hw/qdev-core.h                             |   4 +-
 include/hw/qdev-properties.h                       |   4 +-
 rust/wrapper.h                                     |  17 +
 hw/core/qdev-properties.c                          |  26 +-
 system/qdev-monitor.c                              |   2 +-
 .gitattributes                                     |   2 +
 .gitlab-ci.d/buildtest.yml                         |   6 +-
 meson_options.txt                                  |   2 +
 rust/{hw/char/pl011 => }/Cargo.lock                |   4 +
 rust/Cargo.toml                                    |   7 +
 rust/hw/char/Kconfig                               |   1 -
 rust/hw/char/pl011/Cargo.toml                      |   3 -
 rust/hw/char/pl011/src/device.rs                   | 162 +++++++---
 rust/hw/char/pl011/src/device_class.rs             |  80 ++++-
 rust/hw/char/pl011/src/lib.rs                      |   6 +-
 rust/hw/char/pl011/src/memory_ops.rs               |  24 +-
 rust/qemu-api-macros/Cargo.lock                    |  47 ---
 rust/qemu-api-macros/Cargo.toml                    |   5 +-
 rust/qemu-api-macros/meson.build                   |   2 +-
 rust/qemu-api-macros/src/lib.rs                    | 103 ++++--
 rust/qemu-api/Cargo.lock                           |   7 -
 rust/qemu-api/Cargo.toml                           |  10 +-
 rust/qemu-api/build.rs                             |   9 +
 rust/qemu-api/meson.build                          |  44 ++-
 rust/qemu-api/src/c_str.rs                         |  53 +++
 rust/qemu-api/src/definitions.rs                   |  68 ++--
 rust/qemu-api/src/device_class.rs                  | 114 ++-----
 rust/qemu-api/src/lib.rs                           |  23 +-
 rust/qemu-api/src/offset_of.rs                     | 161 +++++++++
 rust/qemu-api/src/tests.rs                         |  49 ---
 rust/qemu-api/src/vmstate.rs                       | 360 +++++++++++++++++++++
 rust/qemu-api/src/zeroable.rs                      |  86 +++++
 rust/qemu-api/tests/tests.rs                       |  79 +++++
 scripts/ci/setup/ubuntu/ubuntu-2204-aarch64.yaml   |   1 -
 scripts/ci/setup/ubuntu/ubuntu-2204-s390x.yaml     |   1 -
 scripts/meson-buildoptions.sh                      |   4 +
 subprojects/bilge-impl-0.2-rs.wrap                 |   1 +
 subprojects/packagefiles/bilge-impl-1.63.0.patch   |  45 +++
 .../packagefiles/proc-macro2-1-rs/meson.build      |   4 +-
 subprojects/packagefiles/syn-2-rs/meson.build      |   1 +
 tests/docker/dockerfiles/ubuntu2204.docker         |   6 +-
 tests/lcitool/mappings.yml                         |   4 +
 tests/lcitool/refresh                              |  11 +-
 45 files changed, 1379 insertions(+), 416 deletions(-)
-- 
2.47.0
Re: [PULL 00/40] Rust changes for QEMU 9.2 soft freeze
Posted by Peter Maydell 2 weeks, 3 days ago
On Mon, 4 Nov 2024 at 17:35, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> The following changes since commit 15195de6a93438be99fdf9a90992c4228527130d:
>
>   ci: enable rust in the Fedora system build job (2024-10-30 16:30:56 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/bonzini/qemu.git tags/for-upstream-rust
>
> for you to fetch changes up to d20feaa9a5af597bd20630d041e5dc7808612be1:
>
>   ci: enable rust in the Debian and Ubuntu system build job (2024-10-31 18:39:52 +0100)
>
> ----------------------------------------------------------------
> * rust: cleanups
> * rust: integration tests
> * rust/pl011: add support for migration
> * rust/pl011: add TYPE_PL011_LUMINARY device
> * rust: add support for older compilers and bindgen
> * rust: enable rust in the Debian, Fedora and Ubuntu system build job
>
> ----------------------------------------------------------------

This probably isn't something worth not merging this for, but I
noticed while testing (via vm-build-openbsd) that Meson complains:

Compiler for language rust for the host machine not found.
Program bindgen skipped: feature rust disabled
../meson.build:111: WARNING: bindgen not found, disabling Rust compilation.
Message: To use Rust you can install a new version with "cargo install
bindgen-cli"

Rust is still disabled-by-default, so why is meson probing for bindgen?
It would be nice if we could avoid printing WARNING messages for
the normal case.

(I'm continuing with the CI test run.)

thanks
-- PMM
Re: [PULL 00/40] Rust changes for QEMU 9.2 soft freeze
Posted by Paolo Bonzini 2 weeks, 3 days ago
On Wed, Nov 6, 2024 at 2:10 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Mon, 4 Nov 2024 at 17:35, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > The following changes since commit 15195de6a93438be99fdf9a90992c4228527130d:
> >
> >   ci: enable rust in the Fedora system build job (2024-10-30 16:30:56 +0100)
> >
> > are available in the Git repository at:
> >
> >   https://gitlab.com/bonzini/qemu.git tags/for-upstream-rust
> >
> > for you to fetch changes up to d20feaa9a5af597bd20630d041e5dc7808612be1:
> >
> >   ci: enable rust in the Debian and Ubuntu system build job (2024-10-31 18:39:52 +0100)
> >
> > ----------------------------------------------------------------
> > * rust: cleanups
> > * rust: integration tests
> > * rust/pl011: add support for migration
> > * rust/pl011: add TYPE_PL011_LUMINARY device
> > * rust: add support for older compilers and bindgen
> > * rust: enable rust in the Debian, Fedora and Ubuntu system build job
> >
> > ----------------------------------------------------------------
>
> This probably isn't something worth not merging this for, but I
> noticed while testing (via vm-build-openbsd) that Meson complains:
>
> Compiler for language rust for the host machine not found.
> Program bindgen skipped: feature rust disabled
> ../meson.build:111: WARNING: bindgen not found, disabling Rust compilation.
> Message: To use Rust you can install a new version with "cargo install
> bindgen-cli"
>
> Rust is still disabled-by-default, so why is meson probing for bindgen?

It's not probing it ("Program bindgen skipped"), but I was a bit too
happy about printing warnings. This line:

   if not bindgen.found() or bindgen.version().version_compare('<0.60.0')

should simply have had an "if not have_rust", or something like that.

If you want I can resend. I know that Linaro people are in Dublin, so
whatever is easiest for you.

Paolo
Re: [PULL 00/40] Rust changes for QEMU 9.2 soft freeze
Posted by Peter Maydell 2 weeks, 3 days ago
On Wed, 6 Nov 2024 at 13:14, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On Wed, Nov 6, 2024 at 2:10 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Mon, 4 Nov 2024 at 17:35, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > >
> > > The following changes since commit 15195de6a93438be99fdf9a90992c4228527130d:
> > >
> > >   ci: enable rust in the Fedora system build job (2024-10-30 16:30:56 +0100)
> > >
> > > are available in the Git repository at:
> > >
> > >   https://gitlab.com/bonzini/qemu.git tags/for-upstream-rust
> > >
> > > for you to fetch changes up to d20feaa9a5af597bd20630d041e5dc7808612be1:
> > >
> > >   ci: enable rust in the Debian and Ubuntu system build job (2024-10-31 18:39:52 +0100)
> > >
> > > ----------------------------------------------------------------
> > > * rust: cleanups
> > > * rust: integration tests
> > > * rust/pl011: add support for migration
> > > * rust/pl011: add TYPE_PL011_LUMINARY device
> > > * rust: add support for older compilers and bindgen
> > > * rust: enable rust in the Debian, Fedora and Ubuntu system build job
> > >
> > > ----------------------------------------------------------------
> >
> > This probably isn't something worth not merging this for, but I
> > noticed while testing (via vm-build-openbsd) that Meson complains:
> >
> > Compiler for language rust for the host machine not found.
> > Program bindgen skipped: feature rust disabled
> > ../meson.build:111: WARNING: bindgen not found, disabling Rust compilation.
> > Message: To use Rust you can install a new version with "cargo install
> > bindgen-cli"
> >
> > Rust is still disabled-by-default, so why is meson probing for bindgen?
>
> It's not probing it ("Program bindgen skipped"), but I was a bit too
> happy about printing warnings. This line:
>
>    if not bindgen.found() or bindgen.version().version_compare('<0.60.0')
>
> should simply have had an "if not have_rust", or something like that.
>
> If you want I can resend. I know that Linaro people are in Dublin, so
> whatever is easiest for you.

(I'm not in Dublin, as it happens.)  I don't think this needs
to be fixed in this pullreq; it's fine to send a patch to
fix this cosmetic issue and we'll apply it sometime during freeze.

However, this does seem to be causing the functional-tests to timeout
on the CI job that now enables Rust:

https://gitlab.com/qemu-project/qemu/-/jobs/8284623145
https://gitlab.com/qemu-project/qemu/-/jobs/8283637798

both fail with

01-tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_virt:
INTERRUPTED
13-tests/avocado/replay_kernel.py:ReplayKernelNormal.test_arm_virt: INTERRUPTED
15-tests/avocado/replay_kernel.py:ReplayKernelNormal.test_arm_vexpressa9:
INTERRUPTED

(which I suspect will turn out to be "all the functional tests that
use a pl011").

Could you have a look, please?

thanks
-- PMM
Re: [PULL 00/40] Rust changes for QEMU 9.2 soft freeze
Posted by Paolo Bonzini 2 weeks, 3 days ago
On Wed, Nov 6, 2024 at 4:00 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> However, this does seem to be causing the functional-tests to timeout
> on the CI job that now enables Rust:
>
> https://gitlab.com/qemu-project/qemu/-/jobs/8284623145
> https://gitlab.com/qemu-project/qemu/-/jobs/8283637798
>
> both fail with
>
> 01-tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_virt:
> INTERRUPTED
> 13-tests/avocado/replay_kernel.py:ReplayKernelNormal.test_arm_virt: INTERRUPTED
> 15-tests/avocado/replay_kernel.py:ReplayKernelNormal.test_arm_vexpressa9:
> INTERRUPTED
>
> (which I suspect will turn out to be "all the functional tests that
> use a pl011").
>
> Could you have a look, please?

Drat, I'll just drop the patch that enables Rust in CI. Not really in
the mood of debugging device code.

Paolo