This is the preview of moving the minimum supported Rust version
forward to 1.83.0, which is the target for QEMU due to its support
for the const_refs_to_static feature.

Being able to autogenerate all the reflection-like structs in qdev and
VMState improves the type safety, but also requires annotating the
types with the information needed to generate the structs.  The
const_refs_to_static feature is needed because this information resides
in constants that refer to global variables (of types such as PropertyInfo,
VMStateField or VMStateDescription).

This series does not cover enabling the newer compiler in CI because,
while both Debian and Ubuntu have a new-enough Rust compiler to support
1.77, they pose problems for this further bump.  For Debian, the bookworm
release probably will not have new compilers and is supported by QEMU
for roughly two more years.  For Ubuntu, the situation is a bit weird
because while Ubuntu 22.04 had new Rust compilers added until the summer
of 2024, Ubuntu 24.04 is not adding packages for new versions.

A possible plan here is to split the configuration between "enable Rust"
and "enable all devices written in Rust" as soon as new devices are
contributed that are written in Rust.  This way, the C versions of
the pl011 and HPET devices can be used but the new boards/devices would
only be available on Debian or Ubuntu by using rustup.

This series does not use *all* features enabled between 1.77 and 1.83;
in particular it does not replace addr_of!/addr_of_mut! with "&raw"
expressions.

Paolo

Paolo Bonzini (5):
  meson, cargo: require Rust 1.83.0
  rust: use inline const expressions
  rust: vmstate: convert to use builder pattern
  rust: vmstate: use const_refs_to_static
  rust: qdev: const_refs_to_static

 docs/devel/rust.rst                    |  30 +-
 meson.build                            |   6 +-
 rust/Cargo.toml                        |   2 +-
 rust/clippy.toml                       |   2 +-
 rust/hw/char/pl011/src/device.rs       |  20 +-
 rust/hw/char/pl011/src/device_class.rs | 123 +++----
 rust/hw/timer/hpet/src/hpet.rs         | 173 ++++------
 rust/qemu-api/src/assertions.rs        |   4 -
 rust/qemu-api/src/callbacks.rs         |  27 +-
 rust/qemu-api/src/chardev.rs           |   2 +-
 rust/qemu-api/src/qdev.rs              |  16 +-
 rust/qemu-api/src/timer.rs             |   2 +-
 rust/qemu-api/src/vmstate.rs           | 432 +++++++++++++++----------
 rust/qemu-api/tests/tests.rs           |  20 +-
 rust/qemu-api/tests/vmstate_tests.rs   | 155 +++++----
 15 files changed, 517 insertions(+), 497 deletions(-)

-- 
2.49.0

> This series does not cover enabling the newer compiler in CI because,
> while both Debian and Ubuntu have a new-enough Rust compiler to support
> 1.77, they pose problems for this further bump.  For Debian, the bookworm
> release probably will not have new compilers and is supported by QEMU
> for roughly two more years.  For Ubuntu, the situation is a bit weird
> because while Ubuntu 22.04 had new Rust compilers added until the summer
> of 2024, Ubuntu 24.04 is not adding packages for new versions.
> 
> A possible plan here is to split the configuration between "enable Rust"
> and "enable all devices written in Rust" as soon as new devices are
> contributed that are written in Rust.  This way, the C versions of
> the pl011 and HPET devices can be used but the new boards/devices would
> only be available on Debian or Ubuntu by using rustup.

"enable Rust" supports v1.77 and "enable all devices written in Rust"
supports v1.83, correct?

If so, do we need two versions of vmstate? one is for v1.77 (that HPET &
pl011 can use), and another one is for v1.83 (newer devices based on
v1.83 can use).

The current vmstate builder is excellent, but I'm concerned it might not
land soon. Can we find a compromise?

Thanks,
Zhao

On Tue, May 6, 2025 at 10:35 AM Zhao Liu <zhao1.liu@intel.com> wrote:
>
> > This series does not cover enabling the newer compiler in CI because,
> > while both Debian and Ubuntu have a new-enough Rust compiler to support
> > 1.77, they pose problems for this further bump.  For Debian, the bookworm
> > release probably will not have new compilers and is supported by QEMU
> > for roughly two more years.  For Ubuntu, the situation is a bit weird
> > because while Ubuntu 22.04 had new Rust compilers added until the summer
> > of 2024, Ubuntu 24.04 is not adding packages for new versions.
> >
> > A possible plan here is to split the configuration between "enable Rust"
> > and "enable all devices written in Rust" as soon as new devices are
> > contributed that are written in Rust.  This way, the C versions of
> > the pl011 and HPET devices can be used but the new boards/devices would
> > only be available on Debian or Ubuntu by using rustup.
>
> "enable Rust" supports v1.77 and "enable all devices written in Rust"
> supports v1.83, correct?

Both support v1.83 only.  However, if Rust is missing or old, "enable
all devices written in Rust" will fail compilation (e.g. Kconfig would
fail for ARM/x86 targets due to unsatisfiable CONFIG_PL011); "enable
Rust" will simply pick the C version of the PL011 and HPET devices.

> The current vmstate builder is excellent, but I'm concerned it might not
> land soon. Can we find a compromise?

Do you think the above would be a good compromise?

Paolo

> > "enable Rust" supports v1.77 and "enable all devices written in Rust"
> > supports v1.83, correct?
> 
> Both support v1.83 only.  However, if Rust is missing or old, "enable
> all devices written in Rust" will fail compilation (e.g. Kconfig would
> fail for ARM/x86 targets due to unsatisfiable CONFIG_PL011);

In this case, a brand new Rust device (without a corresponding C
version) would be unable to compile on the above platforms which don't
support v1.83. I'm not sure if this is an acceptable limitation or
policy. (Has there been a similar case in history?)

> "enable Rust" will simply pick the C version of the PL011 and HPET devices.

I support this, at least the compatibility with the old QEMU won't be
broken! Then all C devices rewritten in Rust can be covered by this
category.

> > The current vmstate builder is excellent, but I'm concerned it might not
> > land soon. Can we find a compromise?
> 
> Do you think the above would be a good compromise?
 
Overall, I think it's OK (it's not even a compromise).

Thanks,
Zhao

On Tue, May 06, 2025 at 05:26:12PM +0800, Zhao Liu wrote:
> > > "enable Rust" supports v1.77 and "enable all devices written in Rust"
> > > supports v1.83, correct?
> > 
> > Both support v1.83 only.  However, if Rust is missing or old, "enable
> > all devices written in Rust" will fail compilation (e.g. Kconfig would
> > fail for ARM/x86 targets due to unsatisfiable CONFIG_PL011);
> 
> In this case, a brand new Rust device (without a corresponding C
> version) would be unable to compile on the above platforms which don't
> support v1.83. I'm not sure if this is an acceptable limitation or
> policy. (Has there been a similar case in history?)
> 
> > "enable Rust" will simply pick the C version of the PL011 and HPET devices.
> 
> I support this, at least the compatibility with the old QEMU won't be
> broken! Then all C devices rewritten in Rust can be covered by this
> category.

I don't really like this because it perpetuates a state where we have
parallel implementations of devices that have to be kept in sync.

If we're re-writing C devices in Rust, we need to be able to promptly
drop the C impl once the Rust impl is feature complete. Keeping 2 impls
is a general maint burden, as well as an ongoing vmstate compatibility
danger if a change in one impl is not matched by an identical change
in the other impl.

IMHO having Rust declared supported in QEMU should be aligned with being
able to drop C impls of any ported devices.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, May 6, 2025 at 11:49 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > [...] If Rust is missing or old, "enable
> > > all devices written in Rust" will fail compilation (e.g. Kconfig would
> > > fail for ARM/x86 targets due to unsatisfiable CONFIG_PL011);
> > > "enable Rust" will simply pick the C version of the PL011 and HPET devices.
>
> I don't really like this because it perpetuates a state where we have
> parallel implementations of devices that have to be kept in sync.

Me neither (see for example the Meson transition which avoided
parallel implementations at all costs).

On the other hand, this series shows that it's hard to have a baseline
version earlier than 1.83.  The bindings got pretty far while
supporting older versions, and the few hacks needed were nice testbeds
for the build system and the procedural macro infrastructure, but the
improvements that const_refs_to_static provides for reflection are
just too big. And for Debian that means waiting until July 2027 before
making Rust mandatory, and for Ubuntu that's April 2028 based on the
current situation. I hope that the effort proves itself either valid
or unviable in less than 2-3 years. :)

Now, it's certainly not the only possibility:

1) If someone contributes devices that are written in Rust then we
could just drop the PL011 and/or HPET sample device. That's a pity but
they would survive in git history and could be resurrected later.

2) Using RUSTC_BOOTSTRAP[1] allows enabling unstable features even in
versions older than 1.83. Disadvantage: build system changes that will
be obsolete soon(ish), plus the relevant compiler code obviously
wasn't as tested as after stabilization. I'd prefer to avoid that, but
hey---Linux does it.

3) Affected distros could use RUSTC_BOOTSTRAP themselves if they want,
while upstream QEMU would only support rustup toolchains for Debian
bookworm and Ubuntu up to 24.10. This only requires
tests/lcitool/refresh changes, the disadvantage is that the project
would renege on the general promise that we make on platform support.

[1] https://rustc-dev-guide.rust-lang.org/building/bootstrapping/what-bootstrapping-does.html#complications-of-bootstrapping

> If we're re-writing C devices in Rust, we need to be able to promptly
> drop the C impl once the Rust impl is feature complete. Keeping 2 impls
> is a general maint burden, as well as an ongoing vmstate compatibility
> danger if a change in one impl is not matched by an identical change
> in the other impl.

I agree. One more reason why "Let's Rewrite It In Rust" is more of a
necessary evil to bootstrap the creation of bindings, and not a good
idea in general.

> IMHO having Rust declared supported in QEMU should be aligned with being
> able to drop C impls of any ported devices.

I agree in principle, though theory and practice may diverge.

Paolo

On Tue, May 06, 2025 at 12:54:38PM +0200, Paolo Bonzini wrote:
> On Tue, May 6, 2025 at 11:49 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > [...] If Rust is missing or old, "enable
> > > > all devices written in Rust" will fail compilation (e.g. Kconfig would
> > > > fail for ARM/x86 targets due to unsatisfiable CONFIG_PL011);
> > > > "enable Rust" will simply pick the C version of the PL011 and HPET devices.
> >
> > I don't really like this because it perpetuates a state where we have
> > parallel implementations of devices that have to be kept in sync.
> 
> Me neither (see for example the Meson transition which avoided
> parallel implementations at all costs).
> 
> On the other hand, this series shows that it's hard to have a baseline
> version earlier than 1.83.  The bindings got pretty far while
> supporting older versions, and the few hacks needed were nice testbeds
> for the build system and the procedural macro infrastructure, but the
> improvements that const_refs_to_static provides for reflection are
> just too big.

Admittedly I'm not actively working on the QEMU Rust code, but to
me to feels the opposite - we've shown it is possible to write useful
Rust code with the older version baseline. It may not be the ideal
way we want the code to look, but that's a tradeoff we can make.

I very much worry that at any point in time there is *always* going
to be something in a newer Rust that is very attractive to use, so
we end up on a slippery slope where we're always going to be chasing
the latest version to get a better way.

We've had the same situation with Meson where we initially set a
temporary newer baseline to get some critical features we could
not do without, and now have ended up in a situation where we
are continually pushing newer & newer versions, because there is
always something attractive in the new release.

> Now, it's certainly not the only possibility:
> 
> 1) If someone contributes devices that are written in Rust then we
> could just drop the PL011 and/or HPET sample device. That's a pity but
> they would survive in git history and could be resurrected later.

IMHO for Rust in QEMU we should be targetting both new features
and existing feature ports - excluding existing feature ports
would be tieing one of our hands behind our back limiting the
potential benefits we can see.

> 2) Using RUSTC_BOOTSTRAP[1] allows enabling unstable features even in
> versions older than 1.83. Disadvantage: build system changes that will
> be obsolete soon(ish), plus the relevant compiler code obviously
> wasn't as tested as after stabilization. I'd prefer to avoid that, but
> hey---Linux does it.
> 
> 3) Affected distros could use RUSTC_BOOTSTRAP themselves if they want,
> while upstream QEMU would only support rustup toolchains for Debian
> bookworm and Ubuntu up to 24.10. This only requires
> tests/lcitool/refresh changes, the disadvantage is that the project
> would renege on the general promise that we make on platform support.

Yes, this increasing defeats the benefit of defining our distro
target. We wanted to set a clear baseline that we could unambiguously
target, to give clarity to both users & contributors on when we could/
would impose new version requirements. 

We've made exceptions for python, and then meson, and now Rust. We can
rationalize it is as "users only need to do x, y & z to get newer stuff",
but as we make more & more exceptions, this is a game of death by a 1000
cuts.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, May 06, 2025 at 05:26:12PM +0800, Zhao Liu wrote:
> > > "enable Rust" supports v1.77 and "enable all devices written in Rust"
> > > supports v1.83, correct?
> > 
> > Both support v1.83 only.  However, if Rust is missing or old, "enable
> > all devices written in Rust" will fail compilation (e.g. Kconfig would
> > fail for ARM/x86 targets due to unsatisfiable CONFIG_PL011);
> 
> In this case, a brand new Rust device (without a corresponding C
> version) would be unable to compile on the above platforms which don't
> support v1.83. I'm not sure if this is an acceptable limitation or
> policy. (Has there been a similar case in history?)

Brand new features are not required to support all existing QEMU build
targets, they can set whatever baseline is appropriate given the external
dependencies they have.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|