Hello everyone,

This is an early draft of my work on implementing a very simple device, 
in this case the ARM PL011 (which in C code resides in hw/char/pl011.c 
and is used in hw/arm/virt.c).

The device is functional, with copied logic from the C code but with 
effort not to make a direct C to Rust translation. In other words, do 
not write Rust as a C developer would.

That goal is not complete but a best-effort case. To give a specific 
example, register values are typed but interrupt bit flags are not (but 
could be). I will leave such minutiae for later iterations.

By the way, the wiki page for Rust was revived to keep track of all 
current series on the mailing list https://wiki.qemu.org/RustInQemu

a #qemu-rust IRC channel was also created for rust-specific discussion 
that might flood #qemu

------------------------------------------------------------------------
A request: keep comments to Rust in relation to the QEMU project and no 
debates on the merits of the language itself. These are valid concerns, 
but it'd be better if they were on separate mailing list threads.
------------------------------------------------------------------------

Table of contents: [TOC]

- How can I try it? [howcanItryit]
- What are the most important points to focus on, at this point? 
  [whatarethemostimportant]
  - What are the issues with not using the compiler, rustc, directly? 
    [whataretheissueswith]
    1. Tooling
    2. Rust dependencies

  - Should QEMU use third-party dependencies? [shouldqemuusethirdparty]
  - Should QEMU provide wrapping Rust APIs over QEMU internals? 
    [qemuprovidewrappingrustapis]
  - Will QEMU now depend on Rust and thus not build on my XYZ platform? 
    [qemudependonrustnotbuildonxyz]
- How is the compilation structured? [howisthecompilationstructured]
- The generated.rs rust file includes a bunch of junk definitions? 
  [generatedrsincludesjunk]
- The staticlib artifact contains a bunch of mangled .o objects? 
  [staticlibmangledobjects]

How can I try it?
=================
[howcanItryit] Back to [TOC]

Hopefully applying this patches (or checking out `master` branch from 
https://gitlab.com/epilys/rust-for-qemu/ current commit 
de81929e0e9d470deac2c6b449b7a5183325e7ee )

Tag for this RFC is rust-pl011-rfc-v1 

Rustdoc documentation is hosted on

https://rust-for-qemu-epilys-aebb06ca9f9adfe6584811c14ae44156501d935ba4.gitlab.io/pl011/index.html

If `cargo` and `bindgen` is installed in your system, you should be able 
to build qemu-system-aarch64 with configure flag --enable-rust and 
launch an arm virt VM. One of the patches hardcodes the default UART of 
the machine to the Rust one, so if something goes wrong you will see it 
upon launching qemu-system-aarch64.

To confirm it is there for sure, run e.g. info qom-tree on the monitor 
and look for x-pl011-rust.


What are the most important points to focus on, at this point?
==============================================================
[whatarethemostimportant] Back to [TOC]

In my opinion, integration of the go-to Rust build system (Cargo and 
crates.io) with the build system we use in QEMU. This is "easily" done 
in some definition of the word with a python wrapper script.

What are the issues with not using the compiler, rustc, directly?
-----------------------------------------------------------------
[whataretheissueswith] Back to [TOC]

1. Tooling
   Mostly writing up the build-sys tooling to do so. Ideally we'd 
   compile everything without cargo but rustc directly.

   If we decide we need Rust's `std` library support, we could 
   investigate whether building it from scratch is a good solution. This 
   will only build the bits we need in our devices.

2. Rust dependencies
   We could go without them completely. I chose deliberately to include 
   one dependency in my UART implementation, `bilge`[0], because it has 
   an elegant way of representing typed bitfields for the UART's 
   registers.

[0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
     Crates.io page: https://crates.io/crates/bilge
     Repository: https://github.com/hecatia-elegua/bilge

Should QEMU use third-party dependencies?
-----------------------------------------
[shouldqemuusethirdparty] Back to [TOC]

In my personal opinion, if we need a dependency we need a strong 
argument for it. A dependency needs a trusted upstream source, a QEMU 
maintainer to make sure it us up-to-date in QEMU etc.

We already fetch some projects with meson subprojects, so this is not a 
new reality. Cargo allows you to define "locked" dependencies which is 
the same as only fetching specific commits by SHA. No suspicious 
tarballs, and no disappearing dependencies a la left-pad in npm.

However, I believe it's worth considering vendoring every dependency by 
default, if they prove to be few, for the sake of having a local QEMU 
git clone buildable without network access.

Should QEMU provide wrapping Rust APIs over QEMU internals?
-----------------------------------------------------------
[qemuprovidewrappingrustapis] Back to [TOC]

My personal opinion is no, with the reasoning being that QEMU internals 
are not documented or stable. However I do not see why creating stable 
opt-in interfaces is bad. It just needs someone to volunteer to maintain 
it and ensure there are no breakages through versions.

Will QEMU now depend on Rust and thus not build on my XYZ platform?
-------------------------------------------------------------------
[qemudependonrustnotbuildonxyz] Back to [TOC]

No, worry about this in some years if this experiment takes off. Rust 
has broad platform support and is present in most distro package 
managers. In the future we might have gcc support for it as well.

For now, Rust will have an experimental status, and will be aimed to 
those who wish to try it. I leave it to the project leaders to make 
proper decisions and statements on this if necessary.


How is the compilation structured?
==================================
[howisthecompilationstructured] Back to [TOC]

First, a meson target that runs `bindgen` on a bunch of header files 
(defined in `rust/wrapper.h`) is created as a target and as a dependency 
for any rust hardware device that needs it. You can see the generated 
bindings by running

  ninja generated.rs

inside your build directory.

The devices are defined as dictionaries in rust/meson.build because they 
depend on the bindgen dependency, which is available much later in the 
meson process (when the static qemu lib and target emulator executables 
are defined).

A cargo wrapper python script under scripts/ exists to build the crate 
library, by providing the path to the generated.rs bindings via the 
environment. Then, the qemu-system-aarch64 binary links against the 
staticlib archive (i.e. libpl011.a)

The generated.rs rust file includes a bunch of junk definitions?
================================================================
[generatedrsincludesjunk] Back to [TOC]

Yes, bindgen allows you to block certain types and identifiers from 
being generated but they are simply too many. I have trimmed some of the 
fat but vast improvements can be made.

The staticlib artifact contains a bunch of mangled .o objects?
==============================================================
[staticlibmangledobjects] Back to [TOC]

Yes, until we compile without the `std` module library or we compile it 
manually instead of linking it, we will have some junk in it.

--

Manos Pitsidianakis (6):
  build-sys: Add rust feature option
  rust: add PL011 device model
  DO NOT MERGE: add rustdoc build for gitlab pages
  DO NOT MERGE: replace TYPE_PL011 with x-pl011-rust in arm virt machine
  rust: add bindgen step as a meson dependency
  DO NOT MERGE: update rustdoc gitlab pages gen

 .gitignore                     |   2 +
 .gitlab-ci.d/buildtest.yml     |  64 ++--
 configure                      |  12 +
 hw/arm/virt.c                  |   2 +-
 meson.build                    |  99 ++++++
 meson_options.txt              |   4 +
 rust/meson.build               |  93 ++++++
 rust/pl011/.cargo/config.toml  |   2 +
 rust/pl011/.gitignore          |   2 +
 rust/pl011/Cargo.lock          | 120 +++++++
 rust/pl011/Cargo.toml          |  26 ++
 rust/pl011/README.md           |  42 +++
 rust/pl011/build.rs            |  44 +++
 rust/pl011/meson.build         |   7 +
 rust/pl011/rustfmt.toml        |  10 +
 rust/pl011/src/definitions.rs  |  95 ++++++
 rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
 rust/pl011/src/device_class.rs |  95 ++++++
 rust/pl011/src/generated.rs    |   5 +
 rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
 rust/pl011/src/memory_ops.rs   |  38 +++
 rust/wrapper.h                 |  39 +++
 scripts/cargo_wrapper.py       | 221 +++++++++++++
 scripts/meson-buildoptions.sh  |   5 +
 24 files changed, 2113 insertions(+), 20 deletions(-)
 create mode 100644 rust/meson.build
 create mode 100644 rust/pl011/.cargo/config.toml
 create mode 100644 rust/pl011/.gitignore
 create mode 100644 rust/pl011/Cargo.lock
 create mode 100644 rust/pl011/Cargo.toml
 create mode 100644 rust/pl011/README.md
 create mode 100644 rust/pl011/build.rs
 create mode 100644 rust/pl011/meson.build
 create mode 100644 rust/pl011/rustfmt.toml
 create mode 100644 rust/pl011/src/definitions.rs
 create mode 100644 rust/pl011/src/device.rs
 create mode 100644 rust/pl011/src/device_class.rs
 create mode 100644 rust/pl011/src/generated.rs
 create mode 100644 rust/pl011/src/lib.rs
 create mode 100644 rust/pl011/src/memory_ops.rs
 create mode 100644 rust/wrapper.h
 create mode 100644 scripts/cargo_wrapper.py


base-commit: 01782d6b294f95bcde334386f0aaac593cd28c0d
-- 
γαῖα πυρί μιχθήτω

On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
> Hello everyone,
> 
> This is an early draft of my work on implementing a very simple device, 
> in this case the ARM PL011 (which in C code resides in hw/char/pl011.c 
> and is used in hw/arm/virt.c).

looking at the diffstat:

>  .gitignore                     |   2 +
>  .gitlab-ci.d/buildtest.yml     |  64 ++--
>  configure                      |  12 +
>  hw/arm/virt.c                  |   2 +-
>  meson.build                    |  99 ++++++
>  meson_options.txt              |   4 +
>  rust/meson.build               |  93 ++++++
>  rust/pl011/.cargo/config.toml  |   2 +
>  rust/pl011/.gitignore          |   2 +
>  rust/pl011/Cargo.lock          | 120 +++++++
>  rust/pl011/Cargo.toml          |  26 ++
>  rust/pl011/README.md           |  42 +++
>  rust/pl011/build.rs            |  44 +++
>  rust/pl011/meson.build         |   7 +
>  rust/pl011/rustfmt.toml        |  10 +
>  rust/pl011/src/definitions.rs  |  95 ++++++
>  rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
>  rust/pl011/src/device_class.rs |  95 ++++++
>  rust/pl011/src/generated.rs    |   5 +
>  rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
>  rust/pl011/src/memory_ops.rs   |  38 +++

My thought is that if we're going to start implementing devices
or other parts of QEMU, in Rust, then I do not want to see it
placed in a completely separate directory sub-tree.

In this example, I would expect to have hw/arm/pl011.rs, or hw/arm/pl011/*.rs
so that the device is part of the normal Arm hardware directory structure 
and maintainer assignments.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Jun 11, 2024 at 10:22 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
> > Hello everyone,
> >
> > This is an early draft of my work on implementing a very simple device,
> > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
> > and is used in hw/arm/virt.c).
>
> looking at the diffstat:
>
> >  .gitignore                     |   2 +
> >  .gitlab-ci.d/buildtest.yml     |  64 ++--
> >  configure                      |  12 +
> >  hw/arm/virt.c                  |   2 +-
> >  meson.build                    |  99 ++++++
> >  meson_options.txt              |   4 +
> >  rust/meson.build               |  93 ++++++
> >  rust/pl011/.cargo/config.toml  |   2 +
> >  rust/pl011/.gitignore          |   2 +
> >  rust/pl011/Cargo.lock          | 120 +++++++
> >  rust/pl011/Cargo.toml          |  26 ++
> >  rust/pl011/README.md           |  42 +++
> >  rust/pl011/build.rs            |  44 +++
> >  rust/pl011/meson.build         |   7 +
> >  rust/pl011/rustfmt.toml        |  10 +
> >  rust/pl011/src/definitions.rs  |  95 ++++++
> >  rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
> >  rust/pl011/src/device_class.rs |  95 ++++++
> >  rust/pl011/src/generated.rs    |   5 +
> >  rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
> >  rust/pl011/src/memory_ops.rs   |  38 +++
>
> My thought is that if we're going to start implementing devices
> or other parts of QEMU, in Rust, then I do not want to see it
> placed in a completely separate directory sub-tree.
>
> In this example, I would expect to have hw/arm/pl011.rs, or hw/arm/pl011/*.rs
> so that the device is part of the normal Arm hardware directory structure
> and maintainer assignments.

I think that's incompatible with the layout that Cargo expects.
rust/hw/arm/pl011/ could be another possibility.

Paolo

On Tue, Jun 11, 2024 at 03:16:19PM +0200, Paolo Bonzini wrote:
> On Tue, Jun 11, 2024 at 10:22 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
> > > Hello everyone,
> > >
> > > This is an early draft of my work on implementing a very simple device,
> > > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
> > > and is used in hw/arm/virt.c).
> >
> > looking at the diffstat:
> >
> > >  .gitignore                     |   2 +
> > >  .gitlab-ci.d/buildtest.yml     |  64 ++--
> > >  configure                      |  12 +
> > >  hw/arm/virt.c                  |   2 +-
> > >  meson.build                    |  99 ++++++
> > >  meson_options.txt              |   4 +
> > >  rust/meson.build               |  93 ++++++
> > >  rust/pl011/.cargo/config.toml  |   2 +
> > >  rust/pl011/.gitignore          |   2 +
> > >  rust/pl011/Cargo.lock          | 120 +++++++
> > >  rust/pl011/Cargo.toml          |  26 ++
> > >  rust/pl011/README.md           |  42 +++
> > >  rust/pl011/build.rs            |  44 +++
> > >  rust/pl011/meson.build         |   7 +
> > >  rust/pl011/rustfmt.toml        |  10 +
> > >  rust/pl011/src/definitions.rs  |  95 ++++++
> > >  rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
> > >  rust/pl011/src/device_class.rs |  95 ++++++
> > >  rust/pl011/src/generated.rs    |   5 +
> > >  rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
> > >  rust/pl011/src/memory_ops.rs   |  38 +++
> >
> > My thought is that if we're going to start implementing devices
> > or other parts of QEMU, in Rust, then I do not want to see it
> > placed in a completely separate directory sub-tree.
> >
> > In this example, I would expect to have hw/arm/pl011.rs, or hw/arm/pl011/*.rs
> > so that the device is part of the normal Arm hardware directory structure
> > and maintainer assignments.
> 
> I think that's incompatible with the layout that Cargo expects.
> rust/hw/arm/pl011/ could be another possibility.

It doesn't look like its a problem in this patch series. It is just
introducing a "rust/pl011/Cargo.toml", and I don't see anything
that has an fundamental assumption that it is below a 'rust/' top
level dir, as opposed to under our existing dir structure.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, 11 Jun 2024 11:22, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
>> Hello everyone,
>> 
>> This is an early draft of my work on implementing a very simple device, 
>> in this case the ARM PL011 (which in C code resides in hw/char/pl011.c 
>> and is used in hw/arm/virt.c).
>
>looking at the diffstat:
>
>>  .gitignore                     |   2 +
>>  .gitlab-ci.d/buildtest.yml     |  64 ++--
>>  configure                      |  12 +
>>  hw/arm/virt.c                  |   2 +-
>>  meson.build                    |  99 ++++++
>>  meson_options.txt              |   4 +
>>  rust/meson.build               |  93 ++++++
>>  rust/pl011/.cargo/config.toml  |   2 +
>>  rust/pl011/.gitignore          |   2 +
>>  rust/pl011/Cargo.lock          | 120 +++++++
>>  rust/pl011/Cargo.toml          |  26 ++
>>  rust/pl011/README.md           |  42 +++
>>  rust/pl011/build.rs            |  44 +++
>>  rust/pl011/meson.build         |   7 +
>>  rust/pl011/rustfmt.toml        |  10 +
>>  rust/pl011/src/definitions.rs  |  95 ++++++
>>  rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
>>  rust/pl011/src/device_class.rs |  95 ++++++
>>  rust/pl011/src/generated.rs    |   5 +
>>  rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
>>  rust/pl011/src/memory_ops.rs   |  38 +++
>
>My thought is that if we're going to start implementing devices
>or other parts of QEMU, in Rust, then I do not want to see it
>placed in a completely separate directory sub-tree.
>
>In this example, I would expect to have hw/arm/pl011.rs, or hw/arm/pl011/*.rs
>so that the device is part of the normal Arm hardware directory structure 
>and maintainer assignments.

I agree 100%, but I thought it was not my place to decide that, it's 
part of the "request for comments" side of this series.

On Tue, Jun 11, 2024 at 09:22:44AM +0100, Daniel P. Berrangé wrote:
> Date: Tue, 11 Jun 2024 09:22:44 +0100
> From: "Daniel P. Berrangé" <berrange@redhat.com>
> Subject: Re: [RFC PATCH v1 0/6] Implement ARM PL011 in Rust
> 
> On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
> > Hello everyone,
> > 
> > This is an early draft of my work on implementing a very simple device, 
> > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c 
> > and is used in hw/arm/virt.c).
> 
> looking at the diffstat:
> 
> >  .gitignore                     |   2 +
> >  .gitlab-ci.d/buildtest.yml     |  64 ++--
> >  configure                      |  12 +
> >  hw/arm/virt.c                  |   2 +-
> >  meson.build                    |  99 ++++++
> >  meson_options.txt              |   4 +
> >  rust/meson.build               |  93 ++++++
> >  rust/pl011/.cargo/config.toml  |   2 +
> >  rust/pl011/.gitignore          |   2 +
> >  rust/pl011/Cargo.lock          | 120 +++++++
> >  rust/pl011/Cargo.toml          |  26 ++
> >  rust/pl011/README.md           |  42 +++
> >  rust/pl011/build.rs            |  44 +++
> >  rust/pl011/meson.build         |   7 +
> >  rust/pl011/rustfmt.toml        |  10 +
> >  rust/pl011/src/definitions.rs  |  95 ++++++
> >  rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
> >  rust/pl011/src/device_class.rs |  95 ++++++
> >  rust/pl011/src/generated.rs    |   5 +
> >  rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
> >  rust/pl011/src/memory_ops.rs   |  38 +++
> 
> My thought is that if we're going to start implementing devices
> or other parts of QEMU, in Rust, then I do not want to see it
> placed in a completely separate directory sub-tree.
> 
> In this example, I would expect to have hw/arm/pl011.rs, or hw/arm/pl011/*.rs
> so that the device is part of the normal Arm hardware directory structure 
> and maintainer assignments.

It has its advantages. Otherwise, as the number of Rust implementations
grows, the same mirror directory as QEMU will have to be rebuilt again
in the Rust directory.

Further, putting C implementations in the same directory, there is again
the question of why it needs to be duplicated :-) . This topic is
probably also beyond the scope of this RFC, but it's nice to have a Rust
example to start with.

Currently, pl011 exclusively occupies a cargo as a package. In the
future, will other Rust implementations utilize the workspace mechanism
to act as a second package in the same cargo? Or will new cargo be created
again?

Under a unified Rust directory, using a workspace to manage multiple
packages looks as if it would be easier to maintain. Decentralized to an
existing directory, they're all separate cargos, and external dependencies
tend to become fragmented?

On Tue, 11 Jun 2024 12:45, Zhao Liu <zhao1.liu@intel.com> wrote:
>On Tue, Jun 11, 2024 at 09:22:44AM +0100, Daniel P. Berrangé wrote:
>> On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
>> > Hello everyone,
>> > 
>> > This is an early draft of my work on implementing a very simple device, 
>> > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c 
>> > and is used in hw/arm/virt.c).
>> 
>> looking at the diffstat:
>> 
>> >  .gitignore                     |   2 +
>> >  .gitlab-ci.d/buildtest.yml     |  64 ++--
>> >  configure                      |  12 +
>> >  hw/arm/virt.c                  |   2 +-
>> >  meson.build                    |  99 ++++++
>> >  meson_options.txt              |   4 +
>> >  rust/meson.build               |  93 ++++++
>> >  rust/pl011/.cargo/config.toml  |   2 +
>> >  rust/pl011/.gitignore          |   2 +
>> >  rust/pl011/Cargo.lock          | 120 +++++++
>> >  rust/pl011/Cargo.toml          |  26 ++
>> >  rust/pl011/README.md           |  42 +++
>> >  rust/pl011/build.rs            |  44 +++
>> >  rust/pl011/meson.build         |   7 +
>> >  rust/pl011/rustfmt.toml        |  10 +
>> >  rust/pl011/src/definitions.rs  |  95 ++++++
>> >  rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
>> >  rust/pl011/src/device_class.rs |  95 ++++++
>> >  rust/pl011/src/generated.rs    |   5 +
>> >  rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
>> >  rust/pl011/src/memory_ops.rs   |  38 +++
>> 
>> My thought is that if we're going to start implementing devices
>> or other parts of QEMU, in Rust, then I do not want to see it
>> placed in a completely separate directory sub-tree.
>> 
>> In this example, I would expect to have hw/arm/pl011.rs, or hw/arm/pl011/*.rs
>> so that the device is part of the normal Arm hardware directory structure 
>> and maintainer assignments.
>
>It has its advantages. Otherwise, as the number of Rust implementations
>grows, the same mirror directory as QEMU will have to be rebuilt again
>in the Rust directory.

Offtopic for this RFC but:

It'd also mean that each crate would have its own subdir in the tree. In 
the future free-standing .rs files like in the kernel would be nice to 
have.

For those who are not familiar with Cargo, a cargo library is a single 
compilation unit. You cannot have interspersed .rs files and compile 
with cargo as one does generally. You'd have to generate the rustc 
commands for building.

>
>Further, putting C implementations in the same directory, there is again
>the question of why it needs to be duplicated :-) . This topic is
>probably also beyond the scope of this RFC, but it's nice to have a Rust
>example to start with.

pl011 was suggested by Peter as a very simple device to model. The 
duplication is not meant to replace the C version for now or at the 
foreseeable future. It's more of a reference implementation.

>
>Currently, pl011 exclusively occupies a cargo as a package. In the
>future, will other Rust implementations utilize the workspace mechanism
>to act as a second package in the same cargo? Or will new cargo be created
>again?

What do you mean by "new cargo"? I didn't catch that :(

A workspace would make sense if we have "general" crate libraries that 
hardware crates depend on.

>
>Under a unified Rust directory, using a workspace to manage multiple
>packages looks as if it would be easier to maintain. Decentralized to an
>existing directory, they're all separate cargos, and external dependencies
>tend to become fragmented?

Hmm potentially yes, but that's a "what if" scenario. Let's worry about 
that bridge when we cross it!

On Tue, Jun 11, 2024 at 01:41:57PM +0300, Manos Pitsidianakis wrote:
> Date: Tue, 11 Jun 2024 13:41:57 +0300
> From: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
> Subject: Re: [RFC PATCH v1 0/6] Implement ARM PL011 in Rust
> 
> > Currently, pl011 exclusively occupies a cargo as a package. In the
> > future, will other Rust implementations utilize the workspace mechanism
> > to act as a second package in the same cargo? Or will new cargo be created
> > again?
> 
> What do you mean by "new cargo"? I didn't catch that :(
> 
> A workspace would make sense if we have "general" crate libraries that
> hardware crates depend on.

Thanks Manos!

I mean if we spread the rust device across the QEMU submodules, wouldn't
we have to create their own cargo directories (aka single-package cargo)
for each rust device?

However, if the Rust code is all centralized under the /Rust directory,
then it can be managed by multiple-packages in cargo workspace. 

About the "general" crate, I'm not sure whether a base lib to manage
external crates is a good idea, like I replied in [1].

[1]: https://lore.kernel.org/qemu-devel/CAJSP0QWLe6yPDE3rPztx=oS0g+vKT9W3GykrNU0EQZcaW06sog@mail.gmail.com/T/#mfaf9abf06ed82dd7f8ce5e7520bbb4447083b550

> > 
> > Under a unified Rust directory, using a workspace to manage multiple
> > packages looks as if it would be easier to maintain. Decentralized to an
> > existing directory, they're all separate cargos, and external dependencies
> > tend to become fragmented?
> 
> Hmm potentially yes, but that's a "what if" scenario. Let's worry about that
> bridge when we cross it!
>

Yes!

On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
> What are the issues with not using the compiler, rustc, directly?
> -----------------------------------------------------------------
> [whataretheissueswith] Back to [TOC]
> 
> 1. Tooling
>    Mostly writing up the build-sys tooling to do so. Ideally we'd 
>    compile everything without cargo but rustc directly.
> 
>    If we decide we need Rust's `std` library support, we could 
>    investigate whether building it from scratch is a good solution. This 
>    will only build the bits we need in our devices.

Re-building 'std' for QEMU would be a no-go for many distros who
will expect QEMU to use the distro provided 'std' package. So at
most that would have to be an optional feature.

> 2. Rust dependencies
>    We could go without them completely. I chose deliberately to include 
>    one dependency in my UART implementation, `bilge`[0], because it has 
>    an elegant way of representing typed bitfields for the UART's 
>    registers.
> 
> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
>      Crates.io page: https://crates.io/crates/bilge
>      Repository: https://github.com/hecatia-elegua/bilge
> 
> Should QEMU use third-party dependencies?
> -----------------------------------------
> [shouldqemuusethirdparty] Back to [TOC]
> 
> In my personal opinion, if we need a dependency we need a strong 
> argument for it. A dependency needs a trusted upstream source, a QEMU 
> maintainer to make sure it us up-to-date in QEMU etc.

"strong" is a rather fuzzy term. In C we already have a huge number
of build dependencies

 $ wc -l tests/lcitool/projects/qemu.yml 
 127 tests/lcitool/projects/qemu.yml

we would have many more than that except that we're conservative
about adding deps on things because getting new libraries into
distros is quite painful, or we lag behind where we would want
to be to stick with compat for old distro versions.

In terms of Rust dependancies, I'd expect us to have fairly arbitrary
dependancies used. If the dep avoids QEMU maintainers having to
re-invent the wheel for something there is already a common crate
for, then it is a good thing to use it. I'd almost go as far as
encouraging use of external crates. Our maintainers should focus tmie
on writing code that's delivers compelling features to QEMU, rather
than re-creating common APIs that already have good crates.

> We already fetch some projects with meson subprojects, so this is not a 
> new reality. Cargo allows you to define "locked" dependencies which is 
> the same as only fetching specific commits by SHA. No suspicious 
> tarballs, and no disappearing dependencies a la left-pad in npm.
> 
> However, I believe it's worth considering vendoring every dependency by 
> default, if they prove to be few, for the sake of having a local QEMU 
> git clone buildable without network access.

A local git clone is already not buildable without network access,
given that you have to install countless extra distro packages
ahead of time. I think its reasonable to expect people working from
git to have to download rust deps. We should consider whether we
want vendoring in the release tarballs though.

> Should QEMU provide wrapping Rust APIs over QEMU internals?
> -----------------------------------------------------------
> [qemuprovidewrappingrustapis] Back to [TOC]
> 
> My personal opinion is no, with the reasoning being that QEMU internals 
> are not documented or stable. However I do not see why creating stable 
> opt-in interfaces is bad. It just needs someone to volunteer to maintain 
> it and ensure there are no breakages through versions.

I expect this will evolve organically with people providing wrappers
where appropriate to suit their development neds.

> Will QEMU now depend on Rust and thus not build on my XYZ platform?
> -------------------------------------------------------------------
> [qemudependonrustnotbuildonxyz] Back to [TOC]
> 
> No, worry about this in some years if this experiment takes off. Rust 
> has broad platform support and is present in most distro package 
> managers. In the future we might have gcc support for it as well.

Rust isn't going away, so if a platform wants to remain relevant
to the modern software world, then people who care about that
platform need to ensure Rust works on it. I wouldn't say that
QEMU needs to massively worry about this, since all the common
platforms are now covered precisely because Rust is becoming
so wildly used that a platform cannot ignore it.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, 11 Jun 2024 11:18, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
>> What are the issues with not using the compiler, rustc, directly?
>> -----------------------------------------------------------------
>> [whataretheissueswith] Back to [TOC]
>> 
>> 1. Tooling
>>    Mostly writing up the build-sys tooling to do so. Ideally we'd 
>>    compile everything without cargo but rustc directly.
>> 
>>    If we decide we need Rust's `std` library support, we could 
>>    investigate whether building it from scratch is a good solution. This 
>>    will only build the bits we need in our devices.
>
>Re-building 'std' for QEMU would be a no-go for many distros who
>will expect QEMU to use the distro provided 'std' package. So at
>most that would have to be an optional feature.

Yes this wasn't meant for the distro case, you're correct.

>
>> 2. Rust dependencies
>>    We could go without them completely. I chose deliberately to include 
>>    one dependency in my UART implementation, `bilge`[0], because it has 
>>    an elegant way of representing typed bitfields for the UART's 
>>    registers.
>> 
>> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
>>      Crates.io page: https://crates.io/crates/bilge
>>      Repository: https://github.com/hecatia-elegua/bilge
>> 
>> Should QEMU use third-party dependencies?
>> -----------------------------------------
>> [shouldqemuusethirdparty] Back to [TOC]
>> 
>> In my personal opinion, if we need a dependency we need a strong 
>> argument for it. A dependency needs a trusted upstream source, a QEMU 
>> maintainer to make sure it us up-to-date in QEMU etc.
>
>"strong" is a rather fuzzy term. In C we already have a huge number
>of build dependencies

Rust crates.io dependencies tend to "explode" due to the large number of 
transitive dependencies and even different versions of the same crates.

Here's an example:

https://landaire.net/on-dependency-usage-in-rust/#what-about-dependency-explosion

This is something to be aware of in general when pulling dependencies.


>
> $ wc -l tests/lcitool/projects/qemu.yml 
> 127 tests/lcitool/projects/qemu.yml
>
>we would have many more than that except that we're conservative
>about adding deps on things because getting new libraries into
>distros is quite painful, or we lag behind where we would want
>to be to stick with compat for old distro versions.
>
>In terms of Rust dependancies, I'd expect us to have fairly arbitrary
>dependancies used. If the dep avoids QEMU maintainers having to
>re-invent the wheel for something there is already a common crate
>for, then it is a good thing to use it. I'd almost go as far as
>encouraging use of external crates. Our maintainers should focus tmie
>on writing code that's delivers compelling features to QEMU, rather
>than re-creating common APIs that already have good crates.

That was my reasoning for using the bitfield crate to represent UART 
registers.

>
>> We already fetch some projects with meson subprojects, so this is not a 
>> new reality. Cargo allows you to define "locked" dependencies which is 
>> the same as only fetching specific commits by SHA. No suspicious 
>> tarballs, and no disappearing dependencies a la left-pad in npm.
>> 
>> However, I believe it's worth considering vendoring every dependency by 
>> default, if they prove to be few, for the sake of having a local QEMU 
>> git clone buildable without network access.
>
>A local git clone is already not buildable without network access,
>given that you have to install countless extra distro packages
>ahead of time. I think its reasonable to expect people working from
>git to have to download rust deps. We should consider whether we
>want vendoring in the release tarballs though.


Sorry, I meant using cargo without network access. This requires setting 
up the registry index and caches on your $CARGO_HOME

>
>> Should QEMU provide wrapping Rust APIs over QEMU internals?
>> -----------------------------------------------------------
>> [qemuprovidewrappingrustapis] Back to [TOC]
>> 
>> My personal opinion is no, with the reasoning being that QEMU internals 
>> are not documented or stable. However I do not see why creating stable 
>> opt-in interfaces is bad. It just needs someone to volunteer to maintain 
>> it and ensure there are no breakages through versions.
>
>I expect this will evolve organically with people providing wrappers
>where appropriate to suit their development neds.
>
>> Will QEMU now depend on Rust and thus not build on my XYZ platform?
>> -------------------------------------------------------------------
>> [qemudependonrustnotbuildonxyz] Back to [TOC]
>> 
>> No, worry about this in some years if this experiment takes off. Rust 
>> has broad platform support and is present in most distro package 
>> managers. In the future we might have gcc support for it as well.
>
>Rust isn't going away, so if a platform wants to remain relevant
>to the modern software world, then people who care about that
>platform need to ensure Rust works on it. I wouldn't say that
>QEMU needs to massively worry about this, since all the common
>platforms are now covered precisely because Rust is becoming
>so wildly used that a platform cannot ignore it.

Agreed.

On Tue, Jun 11, 2024 at 09:18:25AM +0100, Daniel P. Berrangé wrote:
> Date: Tue, 11 Jun 2024 09:18:25 +0100
> From: "Daniel P. Berrangé" <berrange@redhat.com>
> Subject: Re: [RFC PATCH v1 0/6] Implement ARM PL011 in Rust
> 
> On Mon, Jun 10, 2024 at 09:22:35PM +0300, Manos Pitsidianakis wrote:
> > What are the issues with not using the compiler, rustc, directly?
> > -----------------------------------------------------------------
> > [whataretheissueswith] Back to [TOC]
> > 
> > 1. Tooling
> >    Mostly writing up the build-sys tooling to do so. Ideally we'd 
> >    compile everything without cargo but rustc directly.
> > 
> >    If we decide we need Rust's `std` library support, we could 
> >    investigate whether building it from scratch is a good solution. This 
> >    will only build the bits we need in our devices.
> 
> Re-building 'std' for QEMU would be a no-go for many distros who
> will expect QEMU to use the distro provided 'std' package. So at
> most that would have to be an optional feature.
> 
> > 2. Rust dependencies
> >    We could go without them completely. I chose deliberately to include 
> >    one dependency in my UART implementation, `bilge`[0], because it has 
> >    an elegant way of representing typed bitfields for the UART's 
> >    registers.
> > 
> > [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
> >      Crates.io page: https://crates.io/crates/bilge
> >      Repository: https://github.com/hecatia-elegua/bilge
> > 
> > Should QEMU use third-party dependencies?
> > -----------------------------------------
> > [shouldqemuusethirdparty] Back to [TOC]
> > 
> > In my personal opinion, if we need a dependency we need a strong 
> > argument for it. A dependency needs a trusted upstream source, a QEMU 
> > maintainer to make sure it us up-to-date in QEMU etc.
> 
> "strong" is a rather fuzzy term. In C we already have a huge number
> of build dependencies
> 
>  $ wc -l tests/lcitool/projects/qemu.yml 
>  127 tests/lcitool/projects/qemu.yml
> 
> we would have many more than that except that we're conservative
> about adding deps on things because getting new libraries into
> distros is quite painful, or we lag behind where we would want
> to be to stick with compat for old distro versions.
> 
> In terms of Rust dependancies, I'd expect us to have fairly arbitrary
> dependancies used. If the dep avoids QEMU maintainers having to
> re-invent the wheel for something there is already a common crate
> for, then it is a good thing to use it. I'd almost go as far as
> encouraging use of external crates. Our maintainers should focus tmie
> on writing code that's delivers compelling features to QEMU, rather
> than re-creating common APIs that already have good crates.

So should a base lib be introduced to import and wrap all external
dependencies?

Sort of like osdep.h, so that specific Rust implementations can't import
external third-party libraries directly, but only through the base lib.

The advantage of this is that we can unify the management of external
dependencies and avoid “potentially/overly arbitrary” importing of
specific Rust implementations.

On Mon, 10 Jun 2024 at 14:23, Manos Pitsidianakis
<manos.pitsidianakis@linaro.org> wrote:
>
> Hello everyone,
>
> This is an early draft of my work on implementing a very simple device,
> in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
> and is used in hw/arm/virt.c).
>
> The device is functional, with copied logic from the C code but with
> effort not to make a direct C to Rust translation. In other words, do
> not write Rust as a C developer would.
>
> That goal is not complete but a best-effort case. To give a specific
> example, register values are typed but interrupt bit flags are not (but
> could be). I will leave such minutiae for later iterations.
>
> By the way, the wiki page for Rust was revived to keep track of all
> current series on the mailing list https://wiki.qemu.org/RustInQemu
>
> a #qemu-rust IRC channel was also created for rust-specific discussion
> that might flood #qemu
>
> ------------------------------------------------------------------------
> A request: keep comments to Rust in relation to the QEMU project and no
> debates on the merits of the language itself. These are valid concerns,
> but it'd be better if they were on separate mailing list threads.
> ------------------------------------------------------------------------
>
> Table of contents: [TOC]
>
> - How can I try it? [howcanItryit]
> - What are the most important points to focus on, at this point?
>   [whatarethemostimportant]
>   - What are the issues with not using the compiler, rustc, directly?
>     [whataretheissueswith]
>     1. Tooling
>     2. Rust dependencies
>
>   - Should QEMU use third-party dependencies? [shouldqemuusethirdparty]
>   - Should QEMU provide wrapping Rust APIs over QEMU internals?
>     [qemuprovidewrappingrustapis]
>   - Will QEMU now depend on Rust and thus not build on my XYZ platform?
>     [qemudependonrustnotbuildonxyz]
> - How is the compilation structured? [howisthecompilationstructured]
> - The generated.rs rust file includes a bunch of junk definitions?
>   [generatedrsincludesjunk]
> - The staticlib artifact contains a bunch of mangled .o objects?
>   [staticlibmangledobjects]
>
> How can I try it?
> =================
> [howcanItryit] Back to [TOC]
>
> Hopefully applying this patches (or checking out `master` branch from
> https://gitlab.com/epilys/rust-for-qemu/ current commit
> de81929e0e9d470deac2c6b449b7a5183325e7ee )
>
> Tag for this RFC is rust-pl011-rfc-v1
>
> Rustdoc documentation is hosted on
>
> https://rust-for-qemu-epilys-aebb06ca9f9adfe6584811c14ae44156501d935ba4.gitlab.io/pl011/index.html
>
> If `cargo` and `bindgen` is installed in your system, you should be able
> to build qemu-system-aarch64 with configure flag --enable-rust and
> launch an arm virt VM. One of the patches hardcodes the default UART of
> the machine to the Rust one, so if something goes wrong you will see it
> upon launching qemu-system-aarch64.
>
> To confirm it is there for sure, run e.g. info qom-tree on the monitor
> and look for x-pl011-rust.
>
>
> What are the most important points to focus on, at this point?
> ==============================================================
> [whatarethemostimportant] Back to [TOC]
>
> In my opinion, integration of the go-to Rust build system (Cargo and
> crates.io) with the build system we use in QEMU. This is "easily" done
> in some definition of the word with a python wrapper script.
>
> What are the issues with not using the compiler, rustc, directly?
> -----------------------------------------------------------------
> [whataretheissueswith] Back to [TOC]
>
> 1. Tooling
>    Mostly writing up the build-sys tooling to do so. Ideally we'd
>    compile everything without cargo but rustc directly.

Why would that be ideal?

>
>    If we decide we need Rust's `std` library support, we could
>    investigate whether building it from scratch is a good solution. This
>    will only build the bits we need in our devices.

Whether or not to use std is a fundamental decision. It might be
difficult to back from std later on. This is something that should be
discussed in more detail.

Do you want to avoid std for maximum flexibility in the future, or are
there QEMU use cases today where std is unavailable?

>
> 2. Rust dependencies
>    We could go without them completely. I chose deliberately to include
>    one dependency in my UART implementation, `bilge`[0], because it has
>    an elegant way of representing typed bitfields for the UART's
>    registers.
>
> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
>      Crates.io page: https://crates.io/crates/bilge
>      Repository: https://github.com/hecatia-elegua/bilge

I guess there will be interest in using rust-vmm crates in some way.

Bindings to platform features that are not available in core or std
will also be desirable. We probably don't want to reinvent them.

>
> Should QEMU use third-party dependencies?
> -----------------------------------------
> [shouldqemuusethirdparty] Back to [TOC]
>
> In my personal opinion, if we need a dependency we need a strong
> argument for it. A dependency needs a trusted upstream source, a QEMU
> maintainer to make sure it us up-to-date in QEMU etc.
>
> We already fetch some projects with meson subprojects, so this is not a
> new reality. Cargo allows you to define "locked" dependencies which is
> the same as only fetching specific commits by SHA. No suspicious
> tarballs, and no disappearing dependencies a la left-pad in npm.
>
> However, I believe it's worth considering vendoring every dependency by
> default, if they prove to be few, for the sake of having a local QEMU
> git clone buildable without network access.

Do you mean vendoring by committing them to qemu.git or just the
practice of running `cargo vendor` locally for users who decide they
want to keep a copy of the dependencies?

>
> Should QEMU provide wrapping Rust APIs over QEMU internals?
> -----------------------------------------------------------
> [qemuprovidewrappingrustapis] Back to [TOC]
>
> My personal opinion is no, with the reasoning being that QEMU internals
> are not documented or stable. However I do not see why creating stable
> opt-in interfaces is bad. It just needs someone to volunteer to maintain
> it and ensure there are no breakages through versions.

Rust code will need to interface with QEMU's C APIs, so Rust wrappers
seem unavoidable. Using a protocol like vhost-user might be possible
in some cases. It separates the two codebases so they can both be
native and without bindings, but that won't work for all parts of the
QEMU source tree.

Stable APIs aren't necessary if most developers in the QEMU community
are willing to work in both languages. They can adjust both C and Rust
code when making changes to APIs. I find this preferable to having
Rust maintainers whose job is to keep wrappers up-to-date. Those Rust
maintainers would probably burn out. This seems like a question of
which approach the developer community is comfortable with.

>
> Will QEMU now depend on Rust and thus not build on my XYZ platform?
> -------------------------------------------------------------------
> [qemudependonrustnotbuildonxyz] Back to [TOC]
>
> No, worry about this in some years if this experiment takes off. Rust
> has broad platform support and is present in most distro package
> managers. In the future we might have gcc support for it as well.
>
> For now, Rust will have an experimental status, and will be aimed to
> those who wish to try it. I leave it to the project leaders to make
> proper decisions and statements on this if necessary.

This can be discussed in a separate email thread if you prefer, but I
do think it needs agreement soon so that people have the confidence to
invest their time in writing Rust. They need to know that the code
they develop will be available on most platforms where QEMU is
available and that others in the community won't object or insist on a
C implementation for platform support reasons.

>
>
> How is the compilation structured?
> ==================================
> [howisthecompilationstructured] Back to [TOC]
>
> First, a meson target that runs `bindgen` on a bunch of header files
> (defined in `rust/wrapper.h`) is created as a target and as a dependency
> for any rust hardware device that needs it. You can see the generated
> bindings by running
>
>   ninja generated.rs
>
> inside your build directory.
>
> The devices are defined as dictionaries in rust/meson.build because they
> depend on the bindgen dependency, which is available much later in the
> meson process (when the static qemu lib and target emulator executables
> are defined).
>
> A cargo wrapper python script under scripts/ exists to build the crate
> library, by providing the path to the generated.rs bindings via the
> environment. Then, the qemu-system-aarch64 binary links against the
> staticlib archive (i.e. libpl011.a)
>
> The generated.rs rust file includes a bunch of junk definitions?
> ================================================================
> [generatedrsincludesjunk] Back to [TOC]
>
> Yes, bindgen allows you to block certain types and identifiers from
> being generated but they are simply too many. I have trimmed some of the
> fat but vast improvements can be made.
>
> The staticlib artifact contains a bunch of mangled .o objects?
> ==============================================================
> [staticlibmangledobjects] Back to [TOC]
>
> Yes, until we compile without the `std` module library or we compile it
> manually instead of linking it, we will have some junk in it.

What is the consequence of this? As long as the linker is bringing in
.o files from the .a only through symbol dependencies, then unused .o
files in the .a won't be linked into the final QEMU binary.

>
> --
>
> Manos Pitsidianakis (6):
>   build-sys: Add rust feature option
>   rust: add PL011 device model
>   DO NOT MERGE: add rustdoc build for gitlab pages
>   DO NOT MERGE: replace TYPE_PL011 with x-pl011-rust in arm virt machine
>   rust: add bindgen step as a meson dependency
>   DO NOT MERGE: update rustdoc gitlab pages gen
>
>  .gitignore                     |   2 +
>  .gitlab-ci.d/buildtest.yml     |  64 ++--
>  configure                      |  12 +
>  hw/arm/virt.c                  |   2 +-
>  meson.build                    |  99 ++++++
>  meson_options.txt              |   4 +
>  rust/meson.build               |  93 ++++++
>  rust/pl011/.cargo/config.toml  |   2 +
>  rust/pl011/.gitignore          |   2 +
>  rust/pl011/Cargo.lock          | 120 +++++++
>  rust/pl011/Cargo.toml          |  26 ++
>  rust/pl011/README.md           |  42 +++
>  rust/pl011/build.rs            |  44 +++
>  rust/pl011/meson.build         |   7 +
>  rust/pl011/rustfmt.toml        |  10 +
>  rust/pl011/src/definitions.rs  |  95 ++++++
>  rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
>  rust/pl011/src/device_class.rs |  95 ++++++
>  rust/pl011/src/generated.rs    |   5 +
>  rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
>  rust/pl011/src/memory_ops.rs   |  38 +++
>  rust/wrapper.h                 |  39 +++
>  scripts/cargo_wrapper.py       | 221 +++++++++++++
>  scripts/meson-buildoptions.sh  |   5 +
>  24 files changed, 2113 insertions(+), 20 deletions(-)
>  create mode 100644 rust/meson.build
>  create mode 100644 rust/pl011/.cargo/config.toml
>  create mode 100644 rust/pl011/.gitignore
>  create mode 100644 rust/pl011/Cargo.lock
>  create mode 100644 rust/pl011/Cargo.toml
>  create mode 100644 rust/pl011/README.md
>  create mode 100644 rust/pl011/build.rs
>  create mode 100644 rust/pl011/meson.build
>  create mode 100644 rust/pl011/rustfmt.toml
>  create mode 100644 rust/pl011/src/definitions.rs
>  create mode 100644 rust/pl011/src/device.rs
>  create mode 100644 rust/pl011/src/device_class.rs
>  create mode 100644 rust/pl011/src/generated.rs
>  create mode 100644 rust/pl011/src/lib.rs
>  create mode 100644 rust/pl011/src/memory_ops.rs
>  create mode 100644 rust/wrapper.h
>  create mode 100644 scripts/cargo_wrapper.py
>
>
> base-commit: 01782d6b294f95bcde334386f0aaac593cd28c0d
> --
> γαῖα πυρί μιχθήτω
>
>

On 10/6/24 21:59, Stefan Hajnoczi wrote:
> On Mon, 10 Jun 2024 at 14:23, Manos Pitsidianakis
> <manos.pitsidianakis@linaro.org> wrote:


>> Should QEMU provide wrapping Rust APIs over QEMU internals?
>> -----------------------------------------------------------
>> [qemuprovidewrappingrustapis] Back to [TOC]
>>
>> My personal opinion is no, with the reasoning being that QEMU internals
>> are not documented or stable. However I do not see why creating stable
>> opt-in interfaces is bad. It just needs someone to volunteer to maintain
>> it and ensure there are no breakages through versions.
> 
> Rust code will need to interface with QEMU's C APIs, so Rust wrappers
> seem unavoidable. Using a protocol like vhost-user might be possible
> in some cases. It separates the two codebases so they can both be
> native and without bindings, but that won't work for all parts of the
> QEMU source tree.
> 
> Stable APIs aren't necessary if most developers in the QEMU community
> are willing to work in both languages. They can adjust both C and Rust
> code when making changes to APIs. I find this preferable to having
> Rust maintainers whose job is to keep wrappers up-to-date. Those Rust
> maintainers would probably burn out. This seems like a question of
> which approach the developer community is comfortable with.

Both APIs must be updated in sync in order to pass CI, so having
the same developer updating both languages seems a requisite.

On Mon, 10 Jun 2024 22:59, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> What are the issues with not using the compiler, rustc, directly?
>> -----------------------------------------------------------------
>> [whataretheissueswith] Back to [TOC]
>>
>> 1. Tooling
>>    Mostly writing up the build-sys tooling to do so. Ideally we'd
>>    compile everything without cargo but rustc directly.
>
>Why would that be ideal?

It remove the indirection level of meson<->cargo<->rustc. I don't have a 
concrete idea on how to tackle this, but if cargo ends up not strictly 
necessary, I don't see why we cannot use one build system.

>
>>
>>    If we decide we need Rust's `std` library support, we could
>>    investigate whether building it from scratch is a good solution. This
>>    will only build the bits we need in our devices.
>
>Whether or not to use std is a fundamental decision. It might be
>difficult to back from std later on. This is something that should be
>discussed in more detail.
>
>Do you want to avoid std for maximum flexibility in the future, or are
>there QEMU use cases today where std is unavailable?

For flexibility, and for being compatible with more versions.

But I do not want to avoid it, what I am saying is we can do a custom 
build of it instead of linking to the rust toolchain's prebuilt version.

>
>>
>> 2. Rust dependencies
>>    We could go without them completely. I chose deliberately to include
>>    one dependency in my UART implementation, `bilge`[0], because it has
>>    an elegant way of representing typed bitfields for the UART's
>>    registers.
>>
>> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
>>      Crates.io page: https://crates.io/crates/bilge
>>      Repository: https://github.com/hecatia-elegua/bilge
>
>I guess there will be interest in using rust-vmm crates in some way.
>
>Bindings to platform features that are not available in core or std
>will also be desirable. We probably don't want to reinvent them.


Agreed.

>
>>
>> Should QEMU use third-party dependencies?
>> -----------------------------------------
>> [shouldqemuusethirdparty] Back to [TOC]
>>
>> In my personal opinion, if we need a dependency we need a strong
>> argument for it. A dependency needs a trusted upstream source, a QEMU
>> maintainer to make sure it us up-to-date in QEMU etc.
>>
>> We already fetch some projects with meson subprojects, so this is not a
>> new reality. Cargo allows you to define "locked" dependencies which is
>> the same as only fetching specific commits by SHA. No suspicious
>> tarballs, and no disappearing dependencies a la left-pad in npm.
>>
>> However, I believe it's worth considering vendoring every dependency by
>> default, if they prove to be few, for the sake of having a local QEMU
>> git clone buildable without network access.
>
>Do you mean vendoring by committing them to qemu.git or just the
>practice of running `cargo vendor` locally for users who decide they
>want to keep a copy of the dependencies?


Committing, with an option to opt-out. They are generally not big in 
size. I am not of strong opinion on this one, I'm very open to 
alternatives.


>>
>> Should QEMU provide wrapping Rust APIs over QEMU internals?
>> -----------------------------------------------------------
>> [qemuprovidewrappingrustapis] Back to [TOC]
>>
>> My personal opinion is no, with the reasoning being that QEMU internals
>> are not documented or stable. However I do not see why creating stable
>> opt-in interfaces is bad. It just needs someone to volunteer to maintain
>> it and ensure there are no breakages through versions.
>
>Rust code will need to interface with QEMU's C APIs, so Rust wrappers
>seem unavoidable. Using a protocol like vhost-user might be possible
>in some cases. It separates the two codebases so they can both be
>native and without bindings, but that won't work for all parts of the
>QEMU source tree.
>
>Stable APIs aren't necessary if most developers in the QEMU community
>are willing to work in both languages. They can adjust both C and Rust
>code when making changes to APIs. I find this preferable to having
>Rust maintainers whose job is to keep wrappers up-to-date. Those Rust
>maintainers would probably burn out. This seems like a question of
>which approach the developer community is comfortable with.


Me too.

>
>>
>> Will QEMU now depend on Rust and thus not build on my XYZ platform?
>> -------------------------------------------------------------------
>> [qemudependonrustnotbuildonxyz] Back to [TOC]
>>
>> No, worry about this in some years if this experiment takes off. Rust
>> has broad platform support and is present in most distro package
>> managers. In the future we might have gcc support for it as well.
>>
>> For now, Rust will have an experimental status, and will be aimed to
>> those who wish to try it. I leave it to the project leaders to make
>> proper decisions and statements on this if necessary.
>
>This can be discussed in a separate email thread if you prefer, but I
>do think it needs agreement soon so that people have the confidence to
>invest their time in writing Rust. They need to know that the code
>they develop will be available on most platforms where QEMU is
>available and that others in the community won't object or insist on a
>C implementation for platform support reasons.

Definitely, also it's out of scope for this RFC since we're not writing 
and rules/guidelines yet.

>
>>
>>
>> How is the compilation structured?
>> ==================================
>> [howisthecompilationstructured] Back to [TOC]
>>
>> First, a meson target that runs `bindgen` on a bunch of header files
>> (defined in `rust/wrapper.h`) is created as a target and as a dependency
>> for any rust hardware device that needs it. You can see the generated
>> bindings by running
>>
>>   ninja generated.rs
>>
>> inside your build directory.
>>
>> The devices are defined as dictionaries in rust/meson.build because they
>> depend on the bindgen dependency, which is available much later in the
>> meson process (when the static qemu lib and target emulator executables
>> are defined).
>>
>> A cargo wrapper python script under scripts/ exists to build the crate
>> library, by providing the path to the generated.rs bindings via the
>> environment. Then, the qemu-system-aarch64 binary links against the
>> staticlib archive (i.e. libpl011.a)
>>
>> The generated.rs rust file includes a bunch of junk definitions?
>> ================================================================
>> [generatedrsincludesjunk] Back to [TOC]
>>
>> Yes, bindgen allows you to block certain types and identifiers from
>> being generated but they are simply too many. I have trimmed some of the
>> fat but vast improvements can be made.
>>
>> The staticlib artifact contains a bunch of mangled .o objects?
>> ==============================================================
>> [staticlibmangledobjects] Back to [TOC]
>>
>> Yes, until we compile without the `std` module library or we compile it
>> manually instead of linking it, we will have some junk in it.
>
>What is the consequence of this? As long as the linker is bringing in
>.o files from the .a only through symbol dependencies, then unused .o
>files in the .a won't be linked into the final QEMU binary.

No consequence, I just want to warn anyone peeking into the rust output 
(not the final qemu binary) to expect junk.

On Mon, 10 Jun 2024 at 16:27, Manos Pitsidianakis
<manos.pitsidianakis@linaro.org> wrote:
>
> On Mon, 10 Jun 2024 22:59, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >> What are the issues with not using the compiler, rustc, directly?
> >> -----------------------------------------------------------------
> >> [whataretheissueswith] Back to [TOC]
> >>
> >> 1. Tooling
> >>    Mostly writing up the build-sys tooling to do so. Ideally we'd
> >>    compile everything without cargo but rustc directly.
> >
> >Why would that be ideal?
>
> It remove the indirection level of meson<->cargo<->rustc. I don't have a
> concrete idea on how to tackle this, but if cargo ends up not strictly
> necessary, I don't see why we cannot use one build system.

The convenience of being able to use cargo dependencies without
special QEMU meson build system effort seems worth the overhead of
meson<->cargo<->rustc to me. There is a blog post that explores using
cargo crates using meson's wrap dependencies here, and it seems like
extra work:
https://coaxion.net/blog/2023/04/building-a-gstreamer-plugin-in-rust-with-meson-instead-of-cargo/

It's possible to use just meson today, but I don't think it's
practical when using cargo dependencies.

>
> >
> >>
> >>    If we decide we need Rust's `std` library support, we could
> >>    investigate whether building it from scratch is a good solution. This
> >>    will only build the bits we need in our devices.
> >
> >Whether or not to use std is a fundamental decision. It might be
> >difficult to back from std later on. This is something that should be
> >discussed in more detail.
> >
> >Do you want to avoid std for maximum flexibility in the future, or are
> >there QEMU use cases today where std is unavailable?
>
> For flexibility, and for being compatible with more versions.
>
> But I do not want to avoid it, what I am saying is we can do a custom
> build of it instead of linking to the rust toolchain's prebuilt version.

What advantages does a custom build of std bring?

>
> >
> >>
> >> 2. Rust dependencies
> >>    We could go without them completely. I chose deliberately to include
> >>    one dependency in my UART implementation, `bilge`[0], because it has
> >>    an elegant way of representing typed bitfields for the UART's
> >>    registers.
> >>
> >> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
> >>      Crates.io page: https://crates.io/crates/bilge
> >>      Repository: https://github.com/hecatia-elegua/bilge
> >
> >I guess there will be interest in using rust-vmm crates in some way.
> >
> >Bindings to platform features that are not available in core or std
> >will also be desirable. We probably don't want to reinvent them.
>
>
> Agreed.
>
> >
> >>
> >> Should QEMU use third-party dependencies?
> >> -----------------------------------------
> >> [shouldqemuusethirdparty] Back to [TOC]
> >>
> >> In my personal opinion, if we need a dependency we need a strong
> >> argument for it. A dependency needs a trusted upstream source, a QEMU
> >> maintainer to make sure it us up-to-date in QEMU etc.
> >>
> >> We already fetch some projects with meson subprojects, so this is not a
> >> new reality. Cargo allows you to define "locked" dependencies which is
> >> the same as only fetching specific commits by SHA. No suspicious
> >> tarballs, and no disappearing dependencies a la left-pad in npm.
> >>
> >> However, I believe it's worth considering vendoring every dependency by
> >> default, if they prove to be few, for the sake of having a local QEMU
> >> git clone buildable without network access.
> >
> >Do you mean vendoring by committing them to qemu.git or just the
> >practice of running `cargo vendor` locally for users who decide they
> >want to keep a copy of the dependencies?
>
>
> Committing, with an option to opt-out. They are generally not big in
> size. I am not of strong opinion on this one, I'm very open to
> alternatives.

Fedora and Debian want Rust applications to use distro-packaged
crates. No vendoring and no crates.io online access. It's a bit of a
pain because Rust developers need to make sure their code works with
whatever version of crates Fedora and Debian provide.

The `cargo vendor` command makes it easy for anyone wishing to collect
the required dependencies for offline builds (something I've used for
CentOS builds where vendoring is allowed).

I suggest not vendoring packages in qemu.git. Users can still run
`cargo vendor` for easy offline builds.

>
>
> >>
> >> Should QEMU provide wrapping Rust APIs over QEMU internals?
> >> -----------------------------------------------------------
> >> [qemuprovidewrappingrustapis] Back to [TOC]
> >>
> >> My personal opinion is no, with the reasoning being that QEMU internals
> >> are not documented or stable. However I do not see why creating stable
> >> opt-in interfaces is bad. It just needs someone to volunteer to maintain
> >> it and ensure there are no breakages through versions.
> >
> >Rust code will need to interface with QEMU's C APIs, so Rust wrappers
> >seem unavoidable. Using a protocol like vhost-user might be possible
> >in some cases. It separates the two codebases so they can both be
> >native and without bindings, but that won't work for all parts of the
> >QEMU source tree.
> >
> >Stable APIs aren't necessary if most developers in the QEMU community
> >are willing to work in both languages. They can adjust both C and Rust
> >code when making changes to APIs. I find this preferable to having
> >Rust maintainers whose job is to keep wrappers up-to-date. Those Rust
> >maintainers would probably burn out. This seems like a question of
> >which approach the developer community is comfortable with.
>
>
> Me too.
>
> >
> >>
> >> Will QEMU now depend on Rust and thus not build on my XYZ platform?
> >> -------------------------------------------------------------------
> >> [qemudependonrustnotbuildonxyz] Back to [TOC]
> >>
> >> No, worry about this in some years if this experiment takes off. Rust
> >> has broad platform support and is present in most distro package
> >> managers. In the future we might have gcc support for it as well.
> >>
> >> For now, Rust will have an experimental status, and will be aimed to
> >> those who wish to try it. I leave it to the project leaders to make
> >> proper decisions and statements on this if necessary.
> >
> >This can be discussed in a separate email thread if you prefer, but I
> >do think it needs agreement soon so that people have the confidence to
> >invest their time in writing Rust. They need to know that the code
> >they develop will be available on most platforms where QEMU is
> >available and that others in the community won't object or insist on a
> >C implementation for platform support reasons.
>
> Definitely, also it's out of scope for this RFC since we're not writing
> and rules/guidelines yet.
>
> >
> >>
> >>
> >> How is the compilation structured?
> >> ==================================
> >> [howisthecompilationstructured] Back to [TOC]
> >>
> >> First, a meson target that runs `bindgen` on a bunch of header files
> >> (defined in `rust/wrapper.h`) is created as a target and as a dependency
> >> for any rust hardware device that needs it. You can see the generated
> >> bindings by running
> >>
> >>   ninja generated.rs
> >>
> >> inside your build directory.
> >>
> >> The devices are defined as dictionaries in rust/meson.build because they
> >> depend on the bindgen dependency, which is available much later in the
> >> meson process (when the static qemu lib and target emulator executables
> >> are defined).
> >>
> >> A cargo wrapper python script under scripts/ exists to build the crate
> >> library, by providing the path to the generated.rs bindings via the
> >> environment. Then, the qemu-system-aarch64 binary links against the
> >> staticlib archive (i.e. libpl011.a)
> >>
> >> The generated.rs rust file includes a bunch of junk definitions?
> >> ================================================================
> >> [generatedrsincludesjunk] Back to [TOC]
> >>
> >> Yes, bindgen allows you to block certain types and identifiers from
> >> being generated but they are simply too many. I have trimmed some of the
> >> fat but vast improvements can be made.
> >>
> >> The staticlib artifact contains a bunch of mangled .o objects?
> >> ==============================================================
> >> [staticlibmangledobjects] Back to [TOC]
> >>
> >> Yes, until we compile without the `std` module library or we compile it
> >> manually instead of linking it, we will have some junk in it.
> >
> >What is the consequence of this? As long as the linker is bringing in
> >.o files from the .a only through symbol dependencies, then unused .o
> >files in the .a won't be linked into the final QEMU binary.
>
> No consequence, I just want to warn anyone peeking into the rust output
> (not the final qemu binary) to expect junk.

Okay, cool!

On Mon, Jun 10, 2024 at 10:47 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Mon, 10 Jun 2024 at 16:27, Manos Pitsidianakis
> <manos.pitsidianakis@linaro.org> wrote:
> >
> > On Mon, 10 Jun 2024 22:59, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >> What are the issues with not using the compiler, rustc, directly?
> > >> -----------------------------------------------------------------
> > >> [whataretheissueswith] Back to [TOC]
> > >>
> > >> 1. Tooling
> > >>    Mostly writing up the build-sys tooling to do so. Ideally we'd
> > >>    compile everything without cargo but rustc directly.
> > >
> > >Why would that be ideal?
> >
> > It remove the indirection level of meson<->cargo<->rustc. I don't have a
> > concrete idea on how to tackle this, but if cargo ends up not strictly
> > necessary, I don't see why we cannot use one build system.
>
> The convenience of being able to use cargo dependencies without
> special QEMU meson build system effort seems worth the overhead of
> meson<->cargo<->rustc to me. There is a blog post that explores using
> cargo crates using meson's wrap dependencies here, and it seems like
> extra work:
> https://coaxion.net/blog/2023/04/building-a-gstreamer-plugin-in-rust-with-meson-instead-of-cargo/

The worst part of using cargo from meson (like in libblkio) is the
lack of integration with Rust tests, but otherwise it's a much better
experience. IIUC Meson's cargo subprojects do not support build.rs,
which is a problem if one of your dependencies (for example libc)
needs it.

https://mesonbuild.com/Wrap-dependency-system-manual.html#cargo-wraps

On the other hand, I think it's possible, possibly even clearer, to
invoke bindgen from meson. I would prefer to have many small .rs files
produced by bindgen, to be imported via "use", and I would prefer an
allowlist approach that excludes symbols from system headers.

> > >I guess there will be interest in using rust-vmm crates in some way.

Yes, especially the ByteValued, VolatileMemory and Bytes traits.

One complication in that respect is that anything that does DMA
depends on either RCU or the BQL. We could, at least at the beginning,
introduce a dummy guard that simply enforces at run-time that the BQL
is taken.

Paolo

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Mon, 10 Jun 2024 at 16:27, Manos Pitsidianakis
> <manos.pitsidianakis@linaro.org> wrote:
>>
>> On Mon, 10 Jun 2024 22:59, Stefan Hajnoczi <stefanha@gmail.com> wrote:
<snip>
>> >>
>> >> 1. Tooling
>> >>    Mostly writing up the build-sys tooling to do so. Ideally we'd
>> >>    compile everything without cargo but rustc directly.
>> >
>> >Why would that be ideal?
>>
>> It remove the indirection level of meson<->cargo<->rustc. I don't have a
>> concrete idea on how to tackle this, but if cargo ends up not strictly
>> necessary, I don't see why we cannot use one build system.
>
> The convenience of being able to use cargo dependencies without
> special QEMU meson build system effort seems worth the overhead of
> meson<->cargo<->rustc to me. There is a blog post that explores using
> cargo crates using meson's wrap dependencies here, and it seems like
> extra work:
> https://coaxion.net/blog/2023/04/building-a-gstreamer-plugin-in-rust-with-meson-instead-of-cargo/
>
> It's possible to use just meson today, but I don't think it's
> practical when using cargo dependencies.

I did find the wrap approach very useful in giving a useful checkout
with --download that I can edit and tweak but is still integrated into
the build. This is helpful when working with very new libraries that
haven't been widely packaged yet. Distro's can choose or not choose to
use --download as they wish.

<snip>
>> >
>> >Do you mean vendoring by committing them to qemu.git or just the
>> >practice of running `cargo vendor` locally for users who decide they
>> >want to keep a copy of the dependencies?
>>
>>
>> Committing, with an option to opt-out. They are generally not big in
>> size. I am not of strong opinion on this one, I'm very open to
>> alternatives.

I think we generally host stuff we explicitly need in separate mirrors
or even a little light forking (testfloat does this).

> Fedora and Debian want Rust applications to use distro-packaged
> crates. No vendoring and no crates.io online access. It's a bit of a
> pain because Rust developers need to make sure their code works with
> whatever version of crates Fedora and Debian provide.
>
> The `cargo vendor` command makes it easy for anyone wishing to collect
> the required dependencies for offline builds (something I've used for
> CentOS builds where vendoring is allowed).
>
> I suggest not vendoring packages in qemu.git. Users can still run
> `cargo vendor` for easy offline builds.
>
<snip>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

On Mon, Jun 10, 2024 at 04:47:33PM -0400, Stefan Hajnoczi wrote:
> On Mon, 10 Jun 2024 at 16:27, Manos Pitsidianakis
> <manos.pitsidianakis@linaro.org> wrote:
> >
> > On Mon, 10 Jun 2024 22:59, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >> Should QEMU use third-party dependencies?
> > >> -----------------------------------------
> > >> [shouldqemuusethirdparty] Back to [TOC]
> > >>
> > >> In my personal opinion, if we need a dependency we need a strong
> > >> argument for it. A dependency needs a trusted upstream source, a QEMU
> > >> maintainer to make sure it us up-to-date in QEMU etc.
> > >>
> > >> We already fetch some projects with meson subprojects, so this is not a
> > >> new reality. Cargo allows you to define "locked" dependencies which is
> > >> the same as only fetching specific commits by SHA. No suspicious
> > >> tarballs, and no disappearing dependencies a la left-pad in npm.
> > >>
> > >> However, I believe it's worth considering vendoring every dependency by
> > >> default, if they prove to be few, for the sake of having a local QEMU
> > >> git clone buildable without network access.
> > >
> > >Do you mean vendoring by committing them to qemu.git or just the
> > >practice of running `cargo vendor` locally for users who decide they
> > >want to keep a copy of the dependencies?
> >
> >
> > Committing, with an option to opt-out. They are generally not big in
> > size. I am not of strong opinion on this one, I'm very open to
> > alternatives.
> 
> Fedora and Debian want Rust applications to use distro-packaged
> crates. No vendoring and no crates.io online access. It's a bit of a
> pain because Rust developers need to make sure their code works with
> whatever version of crates Fedora and Debian provide.

NB Fedora isn't actually that strict for Rust.  The "no vendoring"
policy is merely a "SHOULD", rather than a "MUST" requirement:

  https://docs.fedoraproject.org/en-US/packaging-guidelines/Rust/#_vendored_dependencies

which is a more pragmmatic approach to the real world packaging where
there's potentially 100's of deps in an application chain.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Hello Manos,

On 6/10/24 11:22, Manos Pitsidianakis wrote:
> Hello everyone,
> 
> This is an early draft of my work on implementing a very simple device,
> in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
> and is used in hw/arm/virt.c).
> 
> The device is functional, with copied logic from the C code but with
> effort not to make a direct C to Rust translation. In other words, do
> not write Rust as a C developer would.
> 
> That goal is not complete but a best-effort case. To give a specific
> example, register values are typed but interrupt bit flags are not (but
> could be). I will leave such minutiae for later iterations.
> 
> By the way, the wiki page for Rust was revived to keep track of all
> current series on the mailing list https://wiki.qemu.org/RustInQemu
> 
> a #qemu-rust IRC channel was also created for rust-specific discussion
> that might flood #qemu
> 

Excellent work, and thanks for posting this RFC!

IMHO, having patches 2 and 5 splitted is a bit confusing, and exposing 
(temporarily) the generated.rs file in patches is not a good move.
Any reason you kept it this way?

Maybe it could be better if build.rs file was *not* needed for new 
devices/folders, and could be abstracted as a detail of the python 
wrapper script instead of something that should be committed.

Having a simple rust/pl011/meson.build is nice and good taste!

> ------------------------------------------------------------------------
> A request: keep comments to Rust in relation to the QEMU project and no
> debates on the merits of the language itself. These are valid concerns,
> but it'd be better if they were on separate mailing list threads.
> ------------------------------------------------------------------------
> 
> Table of contents: [TOC]
> 
> - How can I try it? [howcanItryit]
> - What are the most important points to focus on, at this point?
>    [whatarethemostimportant]
>    - What are the issues with not using the compiler, rustc, directly?
>      [whataretheissueswith]
>      1. Tooling
>      2. Rust dependencies
> 
>    - Should QEMU use third-party dependencies? [shouldqemuusethirdparty]
>    - Should QEMU provide wrapping Rust APIs over QEMU internals?
>      [qemuprovidewrappingrustapis]
>    - Will QEMU now depend on Rust and thus not build on my XYZ platform?
>      [qemudependonrustnotbuildonxyz]
> - How is the compilation structured? [howisthecompilationstructured]
> - The generated.rs rust file includes a bunch of junk definitions?
>    [generatedrsincludesjunk]
> - The staticlib artifact contains a bunch of mangled .o objects?
>    [staticlibmangledobjects]
> 
> How can I try it?
> =================
> [howcanItryit] Back to [TOC]
> 
> Hopefully applying this patches (or checking out `master` branch from
> https://gitlab.com/epilys/rust-for-qemu/ current commit
> de81929e0e9d470deac2c6b449b7a5183325e7ee )
> 
> Tag for this RFC is rust-pl011-rfc-v1
> 
> Rustdoc documentation is hosted on
> 
> https://rust-for-qemu-epilys-aebb06ca9f9adfe6584811c14ae44156501d935ba4.gitlab.io/pl011/index.html
> 
> If `cargo` and `bindgen` is installed in your system, you should be able
> to build qemu-system-aarch64 with configure flag --enable-rust and
> launch an arm virt VM. One of the patches hardcodes the default UART of
> the machine to the Rust one, so if something goes wrong you will see it
> upon launching qemu-system-aarch64.
> 
> To confirm it is there for sure, run e.g. info qom-tree on the monitor
> and look for x-pl011-rust.
> 
> 
> What are the most important points to focus on, at this point?
> ==============================================================
> [whatarethemostimportant] Back to [TOC]
> 
> In my opinion, integration of the go-to Rust build system (Cargo and
> crates.io) with the build system we use in QEMU. This is "easily" done
> in some definition of the word with a python wrapper script.
> 
> What are the issues with not using the compiler, rustc, directly?
> -----------------------------------------------------------------
> [whataretheissueswith] Back to [TOC]
> 
> 1. Tooling
>     Mostly writing up the build-sys tooling to do so. Ideally we'd
>     compile everything without cargo but rustc directly.
> 
>     If we decide we need Rust's `std` library support, we could
>     investigate whether building it from scratch is a good solution. This
>     will only build the bits we need in our devices.
>  > 2. Rust dependencies
>     We could go without them completely. I chose deliberately to include
>     one dependency in my UART implementation, `bilge`[0], because it has
>     an elegant way of representing typed bitfields for the UART's
>     registers.
> 
> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
>       Crates.io page: https://crates.io/crates/bilge
>       Repository: https://github.com/hecatia-elegua/bilge
> 
> Should QEMU use third-party dependencies?
> -----------------------------------------
> [shouldqemuusethirdparty] Back to [TOC]
> 
> In my personal opinion, if we need a dependency we need a strong
> argument for it. A dependency needs a trusted upstream source, a QEMU
> maintainer to make sure it us up-to-date in QEMU etc.
> 
> We already fetch some projects with meson subprojects, so this is not a
> new reality. Cargo allows you to define "locked" dependencies which is
> the same as only fetching specific commits by SHA. No suspicious
> tarballs, and no disappearing dependencies a la left-pad in npm.
>

As a complement to this, and for other readers, in more than having a 
lock file (fixing version you use), cargo crates system is designed to 
be immutable (see: https://crates.io/policies), and it means there is a 
strong guarantee that a published package will stay there, to the 
opposite of npm, pypi, or most of other similar systems.

"Crate deletion by their owners is not possible to keep the registry as 
immutable as possible."

I believe this is a *key* feature of Rust ecosystem and should be 
emphasized regarding the policy for Rust dependencies to come.

> However, I believe it's worth considering vendoring every dependency by
> default, if they prove to be few, for the sake of having a local QEMU
> git clone buildable without network access.
>

I would not be in favor to vendor all dependencies. Beyond the "offline 
build" scenario, it has only downsides.

Sure, we should really debate before introducing a new dependency, but 
the technical difficulty to mirror its sources and dependencies should 
not be an argument for or against it.

What will happen the day we want to introduce something bigger than a 
simple dependency? (let's say "serde" for instance)

> Should QEMU provide wrapping Rust APIs over QEMU internals?
> -----------------------------------------------------------
> [qemuprovidewrappingrustapis] Back to [TOC]
> 
> My personal opinion is no, with the reasoning being that QEMU internals
> are not documented or stable. However I do not see why creating stable
> opt-in interfaces is bad. It just needs someone to volunteer to maintain
> it and ensure there are no breakages through versions.
> 
> Will QEMU now depend on Rust and thus not build on my XYZ platform?
> -------------------------------------------------------------------
> [qemudependonrustnotbuildonxyz] Back to [TOC]
> 
> No, worry about this in some years if this experiment takes off. Rust
> has broad platform support and is present in most distro package
> managers. In the future we might have gcc support for it as well.
> 
> For now, Rust will have an experimental status, and will be aimed to
> those who wish to try it. I leave it to the project leaders to make
> proper decisions and statements on this if necessary.
> 
> 
> How is the compilation structured?
> ==================================
> [howisthecompilationstructured] Back to [TOC]
> 
> First, a meson target that runs `bindgen` on a bunch of header files
> (defined in `rust/wrapper.h`) is created as a target and as a dependency
> for any rust hardware device that needs it. You can see the generated
> bindings by running
> 
>    ninja generated.rs
> 
> inside your build directory.
> 
> The devices are defined as dictionaries in rust/meson.build because they
> depend on the bindgen dependency, which is available much later in the
> meson process (when the static qemu lib and target emulator executables
> are defined).
> 
> A cargo wrapper python script under scripts/ exists to build the crate
> library, by providing the path to the generated.rs bindings via the
> environment. Then, the qemu-system-aarch64 binary links against the
> staticlib archive (i.e. libpl011.a)
> 
> The generated.rs rust file includes a bunch of junk definitions?
> ================================================================
> [generatedrsincludesjunk] Back to [TOC]
> 
> Yes, bindgen allows you to block certain types and identifiers from
> being generated but they are simply too many. I have trimmed some of the
> fat but vast improvements can be made.
> 
> The staticlib artifact contains a bunch of mangled .o objects?
> ==============================================================
> [staticlibmangledobjects] Back to [TOC]
> 
> Yes, until we compile without the `std` module library or we compile it
> manually instead of linking it, we will have some junk in it.
> 

Besides the size aspect, which potential advantage would there be to 
switch to no_std?
We don't build a bare metal or kernel binary here, so why introduce this 
restriction willingly?

> --
> 
> Manos Pitsidianakis (6):
>    build-sys: Add rust feature option
>    rust: add PL011 device model
>    DO NOT MERGE: add rustdoc build for gitlab pages
>    DO NOT MERGE: replace TYPE_PL011 with x-pl011-rust in arm virt machine
>    rust: add bindgen step as a meson dependency
>    DO NOT MERGE: update rustdoc gitlab pages gen
> 
>   .gitignore                     |   2 +
>   .gitlab-ci.d/buildtest.yml     |  64 ++--
>   configure                      |  12 +
>   hw/arm/virt.c                  |   2 +-
>   meson.build                    |  99 ++++++
>   meson_options.txt              |   4 +
>   rust/meson.build               |  93 ++++++
>   rust/pl011/.cargo/config.toml  |   2 +
>   rust/pl011/.gitignore          |   2 +
>   rust/pl011/Cargo.lock          | 120 +++++++
>   rust/pl011/Cargo.toml          |  26 ++
>   rust/pl011/README.md           |  42 +++
>   rust/pl011/build.rs            |  44 +++
>   rust/pl011/meson.build         |   7 +
>   rust/pl011/rustfmt.toml        |  10 +
>   rust/pl011/src/definitions.rs  |  95 ++++++
>   rust/pl011/src/device.rs       | 531 ++++++++++++++++++++++++++++++
>   rust/pl011/src/device_class.rs |  95 ++++++
>   rust/pl011/src/generated.rs    |   5 +
>   rust/pl011/src/lib.rs          | 575 +++++++++++++++++++++++++++++++++
>   rust/pl011/src/memory_ops.rs   |  38 +++
>   rust/wrapper.h                 |  39 +++
>   scripts/cargo_wrapper.py       | 221 +++++++++++++
>   scripts/meson-buildoptions.sh  |   5 +
>   24 files changed, 2113 insertions(+), 20 deletions(-)
>   create mode 100644 rust/meson.build
>   create mode 100644 rust/pl011/.cargo/config.toml
>   create mode 100644 rust/pl011/.gitignore
>   create mode 100644 rust/pl011/Cargo.lock
>   create mode 100644 rust/pl011/Cargo.toml
>   create mode 100644 rust/pl011/README.md
>   create mode 100644 rust/pl011/build.rs
>   create mode 100644 rust/pl011/meson.build
>   create mode 100644 rust/pl011/rustfmt.toml
>   create mode 100644 rust/pl011/src/definitions.rs
>   create mode 100644 rust/pl011/src/device.rs
>   create mode 100644 rust/pl011/src/device_class.rs
>   create mode 100644 rust/pl011/src/generated.rs
>   create mode 100644 rust/pl011/src/lib.rs
>   create mode 100644 rust/pl011/src/memory_ops.rs
>   create mode 100644 rust/wrapper.h
>   create mode 100644 scripts/cargo_wrapper.py
> 
> 
> base-commit: 01782d6b294f95bcde334386f0aaac593cd28c0d

Thanks,
Pierrick

On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>Hello Manos,
>
>On 6/10/24 11:22, Manos Pitsidianakis wrote:
>> Hello everyone,
>> 
>> This is an early draft of my work on implementing a very simple device,
>> in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
>> and is used in hw/arm/virt.c).
>> 
>> The device is functional, with copied logic from the C code but with
>> effort not to make a direct C to Rust translation. In other words, do
>> not write Rust as a C developer would.
>> 
>> That goal is not complete but a best-effort case. To give a specific
>> example, register values are typed but interrupt bit flags are not (but
>> could be). I will leave such minutiae for later iterations.
>> 
>> By the way, the wiki page for Rust was revived to keep track of all
>> current series on the mailing list https://wiki.qemu.org/RustInQemu
>> 
>> a #qemu-rust IRC channel was also created for rust-specific discussion
>> that might flood #qemu
>> 
>
>Excellent work, and thanks for posting this RFC!
>
>IMHO, having patches 2 and 5 splitted is a bit confusing, and exposing 
>(temporarily) the generated.rs file in patches is not a good move.
>Any reason you kept it this way?

That was my first approach, I will rework it on the second version. The 
generated code should not exist in committed code at all.

It was initally tricky setting up the dependency orders correctly, so I 
first committed it and then made it a dependency.

>
>Maybe it could be better if build.rs file was *not* needed for new 
>devices/folders, and could be abstracted as a detail of the python 
>wrapper script instead of something that should be committed.


That'd mean you cannot work on the rust files with a LanguageServer, you 
cannot run cargo build or cargo check or cargo clippy, etc. That's why I 
left the alternative choice of including a manually generated bindings 
file (generated.rs.inc)

>
>Having a simple rust/pl011/meson.build is nice and good taste!
>
>> ------------------------------------------------------------------------
>> A request: keep comments to Rust in relation to the QEMU project and no
>> debates on the merits of the language itself. These are valid concerns,
>> but it'd be better if they were on separate mailing list threads.
>> ------------------------------------------------------------------------
>> 
>> Table of contents: [TOC]
>> 
>> - How can I try it? [howcanItryit]
>> - What are the most important points to focus on, at this point?
>>    [whatarethemostimportant]
>>    - What are the issues with not using the compiler, rustc, directly?
>>      [whataretheissueswith]
>>      1. Tooling
>>      2. Rust dependencies
>> 
>>    - Should QEMU use third-party dependencies? [shouldqemuusethirdparty]
>>    - Should QEMU provide wrapping Rust APIs over QEMU internals?
>>      [qemuprovidewrappingrustapis]
>>    - Will QEMU now depend on Rust and thus not build on my XYZ platform?
>>      [qemudependonrustnotbuildonxyz]
>> - How is the compilation structured? [howisthecompilationstructured]
>> - The generated.rs rust file includes a bunch of junk definitions?
>>    [generatedrsincludesjunk]
>> - The staticlib artifact contains a bunch of mangled .o objects?
>>    [staticlibmangledobjects]
>> 
>> How can I try it?
>> =================
>> [howcanItryit] Back to [TOC]
>> 
>> Hopefully applying this patches (or checking out `master` branch from
>> https://gitlab.com/epilys/rust-for-qemu/ current commit
>> de81929e0e9d470deac2c6b449b7a5183325e7ee )
>> 
>> Tag for this RFC is rust-pl011-rfc-v1
>> 
>> Rustdoc documentation is hosted on
>> 
>> https://rust-for-qemu-epilys-aebb06ca9f9adfe6584811c14ae44156501d935ba4.gitlab.io/pl011/index.html
>> 
>> If `cargo` and `bindgen` is installed in your system, you should be able
>> to build qemu-system-aarch64 with configure flag --enable-rust and
>> launch an arm virt VM. One of the patches hardcodes the default UART of
>> the machine to the Rust one, so if something goes wrong you will see it
>> upon launching qemu-system-aarch64.
>> 
>> To confirm it is there for sure, run e.g. info qom-tree on the monitor
>> and look for x-pl011-rust.
>> 
>> 
>> What are the most important points to focus on, at this point?
>> ==============================================================
>> [whatarethemostimportant] Back to [TOC]
>> 
>> In my opinion, integration of the go-to Rust build system (Cargo and
>> crates.io) with the build system we use in QEMU. This is "easily" done
>> in some definition of the word with a python wrapper script.
>> 
>> What are the issues with not using the compiler, rustc, directly?
>> -----------------------------------------------------------------
>> [whataretheissueswith] Back to [TOC]
>> 
>> 1. Tooling
>>     Mostly writing up the build-sys tooling to do so. Ideally we'd
>>     compile everything without cargo but rustc directly.
>> 
>>     If we decide we need Rust's `std` library support, we could
>>     investigate whether building it from scratch is a good solution. This
>>     will only build the bits we need in our devices.
>>  > 2. Rust dependencies
>>     We could go without them completely. I chose deliberately to include
>>     one dependency in my UART implementation, `bilge`[0], because it has
>>     an elegant way of representing typed bitfields for the UART's
>>     registers.
>> 
>> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
>>       Crates.io page: https://crates.io/crates/bilge
>>       Repository: https://github.com/hecatia-elegua/bilge
>> 
>> Should QEMU use third-party dependencies?
>> -----------------------------------------
>> [shouldqemuusethirdparty] Back to [TOC]
>> 
>> In my personal opinion, if we need a dependency we need a strong
>> argument for it. A dependency needs a trusted upstream source, a QEMU
>> maintainer to make sure it us up-to-date in QEMU etc.
>> 
>> We already fetch some projects with meson subprojects, so this is not a
>> new reality. Cargo allows you to define "locked" dependencies which is
>> the same as only fetching specific commits by SHA. No suspicious
>> tarballs, and no disappearing dependencies a la left-pad in npm.
>>
>
>As a complement to this, and for other readers, in more than having a 
>lock file (fixing version you use), cargo crates system is designed to 
>be immutable (see: https://crates.io/policies), and it means there is a 
>strong guarantee that a published package will stay there, to the 
>opposite of npm, pypi, or most of other similar systems.
>
>"Crate deletion by their owners is not possible to keep the registry as 
>immutable as possible."
>
>I believe this is a *key* feature of Rust ecosystem and should be 
>emphasized regarding the policy for Rust dependencies to come.
>
>> However, I believe it's worth considering vendoring every dependency by
>> default, if they prove to be few, for the sake of having a local QEMU
>> git clone buildable without network access.
>>
>
>I would not be in favor to vendor all dependencies. Beyond the "offline 
>build" scenario, it has only downsides.
>
>Sure, we should really debate before introducing a new dependency, but 
>the technical difficulty to mirror its sources and dependencies should 
>not be an argument for or against it.
>
>What will happen the day we want to introduce something bigger than a 
>simple dependency? (let's say "serde" for instance)


Yes, vendor-the-world is a different topic than vendor e.g. two crates 
such as the dependencies I'm using here.

>
>> Should QEMU provide wrapping Rust APIs over QEMU internals?
>> -----------------------------------------------------------
>> [qemuprovidewrappingrustapis] Back to [TOC]
>> 
>> My personal opinion is no, with the reasoning being that QEMU internals
>> are not documented or stable. However I do not see why creating stable
>> opt-in interfaces is bad. It just needs someone to volunteer to maintain
>> it and ensure there are no breakages through versions.
>> 
>> Will QEMU now depend on Rust and thus not build on my XYZ platform?
>> -------------------------------------------------------------------
>> [qemudependonrustnotbuildonxyz] Back to [TOC]
>> 
>> No, worry about this in some years if this experiment takes off. Rust
>> has broad platform support and is present in most distro package
>> managers. In the future we might have gcc support for it as well.
>> 
>> For now, Rust will have an experimental status, and will be aimed to
>> those who wish to try it. I leave it to the project leaders to make
>> proper decisions and statements on this if necessary.
>> 
>> 
>> How is the compilation structured?
>> ==================================
>> [howisthecompilationstructured] Back to [TOC]
>> 
>> First, a meson target that runs `bindgen` on a bunch of header files
>> (defined in `rust/wrapper.h`) is created as a target and as a dependency
>> for any rust hardware device that needs it. You can see the generated
>> bindings by running
>> 
>>    ninja generated.rs
>> 
>> inside your build directory.
>> 
>> The devices are defined as dictionaries in rust/meson.build because they
>> depend on the bindgen dependency, which is available much later in the
>> meson process (when the static qemu lib and target emulator executables
>> are defined).
>> 
>> A cargo wrapper python script under scripts/ exists to build the crate
>> library, by providing the path to the generated.rs bindings via the
>> environment. Then, the qemu-system-aarch64 binary links against the
>> staticlib archive (i.e. libpl011.a)
>> 
>> The generated.rs rust file includes a bunch of junk definitions?
>> ================================================================
>> [generatedrsincludesjunk] Back to [TOC]
>> 
>> Yes, bindgen allows you to block certain types and identifiers from
>> being generated but they are simply too many. I have trimmed some of the
>> fat but vast improvements can be made.
>> 
>> The staticlib artifact contains a bunch of mangled .o objects?
>> ==============================================================
>> [staticlibmangledobjects] Back to [TOC]
>> 
>> Yes, until we compile without the `std` module library or we compile it
>> manually instead of linking it, we will have some junk in it.
>> 
>
>Besides the size aspect, which potential advantage would there be to 
>switch to no_std?
>We don't build a bare metal or kernel binary here, so why introduce this 
>restriction willingly?

We'll see that as we progress. Might enable more platform support, for 
example. I have no definite answers here. Also, I know binary bloat is a 
big complaint from people with dislike of Rust, so I pre-emptively 
addressed it.

On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
> > Hello Manos,
> > 
> > On 6/10/24 11:22, Manos Pitsidianakis wrote:
> > > Hello everyone,
> > > 
> > > This is an early draft of my work on implementing a very simple device,
> > > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
> > > and is used in hw/arm/virt.c).
> > > 
> > > The device is functional, with copied logic from the C code but with
> > > effort not to make a direct C to Rust translation. In other words, do
> > > not write Rust as a C developer would.
> > > 
> > > That goal is not complete but a best-effort case. To give a specific
> > > example, register values are typed but interrupt bit flags are not (but
> > > could be). I will leave such minutiae for later iterations.

snip

> > Maybe it could be better if build.rs file was *not* needed for new
> > devices/folders, and could be abstracted as a detail of the python
> > wrapper script instead of something that should be committed.
> 
> 
> That'd mean you cannot work on the rust files with a LanguageServer, you
> cannot run cargo build or cargo check or cargo clippy, etc. That's why I
> left the alternative choice of including a manually generated bindings file
> (generated.rs.inc)

I would not expect QEMU developers to be running 'cargo <anything>'
directly at all.

QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
convenience facade.

Any use of 'cargo' would be an internal impl detail of meson rules
for building rust code, and developers should still exclusively work
with 'make' or 'ninja' to run builds & tests. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, 11 Jun 2024 13:57, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
>> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>> > Hello Manos,
>> > 
>> > On 6/10/24 11:22, Manos Pitsidianakis wrote:
>> > > Hello everyone,
>> > > 
>> > > This is an early draft of my work on implementing a very simple device,
>> > > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
>> > > and is used in hw/arm/virt.c).
>> > > 
>> > > The device is functional, with copied logic from the C code but with
>> > > effort not to make a direct C to Rust translation. In other words, do
>> > > not write Rust as a C developer would.
>> > > 
>> > > That goal is not complete but a best-effort case. To give a specific
>> > > example, register values are typed but interrupt bit flags are not (but
>> > > could be). I will leave such minutiae for later iterations.
>
>snip
>
>> > Maybe it could be better if build.rs file was *not* needed for new
>> > devices/folders, and could be abstracted as a detail of the python
>> > wrapper script instead of something that should be committed.
>> 
>> 
>> That'd mean you cannot work on the rust files with a LanguageServer, you
>> cannot run cargo build or cargo check or cargo clippy, etc. That's why I
>> left the alternative choice of including a manually generated bindings file
>> (generated.rs.inc)
>
>I would not expect QEMU developers to be running 'cargo <anything>'
>directly at all.
>
>QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
>convenience facade.
>
>Any use of 'cargo' would be an internal impl detail of meson rules
>for building rust code, and developers should still exclusively work
>with 'make' or 'ninja' to run builds & tests.

No, that's not true. If I wrote the pl011 device with this workflow I'd 
just waste time using meson. Part of the development is making sure the 
library type checks, compiles, using cargo to run style formatting, to 
check for lints, perhaps run tests. Doing this only through meson is an 
unnecessary complication.

To compile and run QEMU with a rust component, sure, you'd use meson.

Hi there :)

On 11/06/2024 12:58, Manos Pitsidianakis wrote:
> On Tue, 11 Jun 2024 13:57, "Daniel P. Berrangé" <berrange@redhat.com> 
> wrote:
>> On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
>>> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier 
>>> <pierrick.bouvier@linaro.org> wrote:
>>> > Hello Manos,
>>> > > On 6/10/24 11:22, Manos Pitsidianakis wrote:
>>> > > Hello everyone,
>>> > > > > This is an early draft of my work on implementing a very 
>>> simple device,
>>> > > in this case the ARM PL011 (which in C code resides in 
>>> hw/char/pl011.c
>>> > > and is used in hw/arm/virt.c).
>>> > > > > The device is functional, with copied logic from the C code 
>>> but with
>>> > > effort not to make a direct C to Rust translation. In other 
>>> words, do
>>> > > not write Rust as a C developer would.
>>> > > > > That goal is not complete but a best-effort case. To give a 
>>> specific
>>> > > example, register values are typed but interrupt bit flags are 
>>> not (but
>>> > > could be). I will leave such minutiae for later iterations.
>>
>> snip
>>
>>> > Maybe it could be better if build.rs file was *not* needed for new
>>> > devices/folders, and could be abstracted as a detail of the python
>>> > wrapper script instead of something that should be committed.
>>>
>>>
>>> That'd mean you cannot work on the rust files with a LanguageServer, you
>>> cannot run cargo build or cargo check or cargo clippy, etc. That's why I
>>> left the alternative choice of including a manually generated 
>>> bindings file
>>> (generated.rs.inc)
>>
>> I would not expect QEMU developers to be running 'cargo <anything>'
>> directly at all.
>>
>> QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
>> convenience facade.
>>
>> Any use of 'cargo' would be an internal impl detail of meson rules
>> for building rust code, and developers should still exclusively work
>> with 'make' or 'ninja' to run builds & tests.
> 
> No, that's not true. If I wrote the pl011 device with this workflow I'd 
> just waste time using meson. Part of the development is making sure the 
> library type checks, compiles, using cargo to run style formatting, to 
> check for lints, perhaps run tests. Doing this only through meson is an 
> unnecessary complication.
> 

My favorite tool for Rust development is rust-analyzer, which works very 
well with cargo-based projects. Making it work with meson is just a 
matter of pointing rust-analyzer to the rust-project.json file generated 
by meson at configuration time (just like compile_commands.json).

Unfortunately, rust-analyzer also relies on cargo for doing its check. I 
was able to override that with ninja, but it requires `meson setup` with 
`RUSTFLAGS="--emit=metadata --error-format=json"`. That makes 
rust-analyzer happy, but compilation output is not readable anymore 
being json-like.

I ended up working with 2 build folders, one for me, one for 
rust-analyzer. So, yeah, it complicates a bit.

> To compile and run QEMU with a rust component, sure, you'd use meson.
> 

Cheers,
Antonio

Hello Antonio!

On Tue, 11 Jun 2024 15:45, Antonio Caggiano <quic_acaggian@quicinc.com> wrote:
>Hi there :)
>
>On 11/06/2024 12:58, Manos Pitsidianakis wrote:
>> On Tue, 11 Jun 2024 13:57, "Daniel P. Berrangé" <berrange@redhat.com> 
>> wrote:
>>> On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
>>>> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier 
>>>> <pierrick.bouvier@linaro.org> wrote:
>>>> > Hello Manos,
>>>> > > On 6/10/24 11:22, Manos Pitsidianakis wrote:
>>>> > > Hello everyone,
>>>> > > > > This is an early draft of my work on implementing a very 
>>>> simple device,
>>>> > > in this case the ARM PL011 (which in C code resides in 
>>>> hw/char/pl011.c
>>>> > > and is used in hw/arm/virt.c).
>>>> > > > > The device is functional, with copied logic from the C code 
>>>> but with
>>>> > > effort not to make a direct C to Rust translation. In other 
>>>> words, do
>>>> > > not write Rust as a C developer would.
>>>> > > > > That goal is not complete but a best-effort case. To give a 
>>>> specific
>>>> > > example, register values are typed but interrupt bit flags are 
>>>> not (but
>>>> > > could be). I will leave such minutiae for later iterations.
>>>
>>> snip
>>>
>>>> > Maybe it could be better if build.rs file was *not* needed for new
>>>> > devices/folders, and could be abstracted as a detail of the python
>>>> > wrapper script instead of something that should be committed.
>>>>
>>>>
>>>> That'd mean you cannot work on the rust files with a LanguageServer, you
>>>> cannot run cargo build or cargo check or cargo clippy, etc. That's why I
>>>> left the alternative choice of including a manually generated 
>>>> bindings file
>>>> (generated.rs.inc)
>>>
>>> I would not expect QEMU developers to be running 'cargo <anything>'
>>> directly at all.
>>>
>>> QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
>>> convenience facade.
>>>
>>> Any use of 'cargo' would be an internal impl detail of meson rules
>>> for building rust code, and developers should still exclusively work
>>> with 'make' or 'ninja' to run builds & tests.
>> 
>> No, that's not true. If I wrote the pl011 device with this workflow I'd 
>> just waste time using meson. Part of the development is making sure the 
>> library type checks, compiles, using cargo to run style formatting, to 
>> check for lints, perhaps run tests. Doing this only through meson is an 
>> unnecessary complication.
>> 
>
>My favorite tool for Rust development is rust-analyzer, which works very 
>well with cargo-based projects. Making it work with meson is just a 
>matter of pointing rust-analyzer to the rust-project.json file generated 
>by meson at configuration time (just like compile_commands.json).

That's only generated for meson rust targets, whereas we are currently 
compiling with a cargo wrapper script.

>
>Unfortunately, rust-analyzer also relies on cargo for doing its check. I 
>was able to override that with ninja, but it requires `meson setup` with 
>`RUSTFLAGS="--emit=metadata --error-format=json"`. That makes 
>rust-analyzer happy, but compilation output is not readable anymore 
>being json-like.
>
>I ended up working with 2 build folders, one for me, one for 
>rust-analyzer. So, yeah, it complicates a bit.
>
>> To compile and run QEMU with a rust component, sure, you'd use meson.
>> 
>
>Cheers,
>Antonio

On Tue, Jun 11, 2024 at 01:58:10PM +0300, Manos Pitsidianakis wrote:
> On Tue, 11 Jun 2024 13:57, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
> > On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
> > > On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
> > > > Hello Manos,
> > > > > On 6/10/24 11:22, Manos Pitsidianakis wrote:
> > > > > Hello everyone,
> > > > > > > This is an early draft of my work on implementing a very
> > > simple device,
> > > > > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
> > > > > and is used in hw/arm/virt.c).
> > > > > > > The device is functional, with copied logic from the C code
> > > but with
> > > > > effort not to make a direct C to Rust translation. In other words, do
> > > > > not write Rust as a C developer would.
> > > > > > > That goal is not complete but a best-effort case. To give a
> > > specific
> > > > > example, register values are typed but interrupt bit flags are not (but
> > > > > could be). I will leave such minutiae for later iterations.
> > 
> > snip
> > 
> > > > Maybe it could be better if build.rs file was *not* needed for new
> > > > devices/folders, and could be abstracted as a detail of the python
> > > > wrapper script instead of something that should be committed.
> > > 
> > > 
> > > That'd mean you cannot work on the rust files with a LanguageServer, you
> > > cannot run cargo build or cargo check or cargo clippy, etc. That's why I
> > > left the alternative choice of including a manually generated bindings file
> > > (generated.rs.inc)
> > 
> > I would not expect QEMU developers to be running 'cargo <anything>'
> > directly at all.
> > 
> > QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
> > convenience facade.
> > 
> > Any use of 'cargo' would be an internal impl detail of meson rules
> > for building rust code, and developers should still exclusively work
> > with 'make' or 'ninja' to run builds & tests.
> 
> No, that's not true. If I wrote the pl011 device with this workflow I'd just
> waste time using meson. Part of the development is making sure the library
> type checks, compiles, using cargo to run style formatting, to check for
> lints, perhaps run tests. Doing this only through meson is an unnecessary
> complication.

I don't see why it should waste time, when we ultimately end up calling
the same underlying tools. We need to have a consistent experiance for
developers working on QEMU, not have to use different tools for different
parts of QEMU depending on whether a piece of code happens to be rust
or C.

> To compile and run QEMU with a rust component, sure, you'd use meson.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Jun 11, 2024 at 01:58:10PM +0300, Manos Pitsidianakis wrote:
>> On Tue, 11 Jun 2024 13:57, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>> > On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
>> > > On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>> > > > Hello Manos,
>> > > > > On 6/10/24 11:22, Manos Pitsidianakis wrote:
>> > > > > Hello everyone,
>> > > > > > > This is an early draft of my work on implementing a very
>> > > simple device,
>> > > > > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
>> > > > > and is used in hw/arm/virt.c).
>> > > > > > > The device is functional, with copied logic from the C code
>> > > but with
>> > > > > effort not to make a direct C to Rust translation. In other words, do
>> > > > > not write Rust as a C developer would.
>> > > > > > > That goal is not complete but a best-effort case. To give a
>> > > specific
>> > > > > example, register values are typed but interrupt bit flags are not (but
>> > > > > could be). I will leave such minutiae for later iterations.
>> > 
>> > snip
>> > 
>> > > > Maybe it could be better if build.rs file was *not* needed for new
>> > > > devices/folders, and could be abstracted as a detail of the python
>> > > > wrapper script instead of something that should be committed.
>> > > 
>> > > 
>> > > That'd mean you cannot work on the rust files with a LanguageServer, you
>> > > cannot run cargo build or cargo check or cargo clippy, etc. That's why I
>> > > left the alternative choice of including a manually generated bindings file
>> > > (generated.rs.inc)
>> > 
>> > I would not expect QEMU developers to be running 'cargo <anything>'
>> > directly at all.
>> > 
>> > QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
>> > convenience facade.
>> > 
>> > Any use of 'cargo' would be an internal impl detail of meson rules
>> > for building rust code, and developers should still exclusively work
>> > with 'make' or 'ninja' to run builds & tests.
>> 
>> No, that's not true. If I wrote the pl011 device with this workflow I'd just
>> waste time using meson. Part of the development is making sure the library
>> type checks, compiles, using cargo to run style formatting, to check for
>> lints, perhaps run tests. Doing this only through meson is an unnecessary
>> complication.
>
> I don't see why it should waste time, when we ultimately end up calling
> the same underlying tools. We need to have a consistent experiance for
> developers working on QEMU, not have to use different tools for different
> parts of QEMU depending on whether a piece of code happens to be rust
> or C.

For example if I wanted to run rust-based unit tests (which I think
potentially offer an easier solution than qtest) I would expect that to
be done from the normal make/ninja targets.

>
>> To compile and run QEMU with a rust component, sure, you'd use meson.
>
> With regards,
> Daniel

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

On Tue, Jun 11, 2024 at 01:51:13PM +0100, Alex Bennée wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Tue, Jun 11, 2024 at 01:58:10PM +0300, Manos Pitsidianakis wrote:
> >> On Tue, 11 Jun 2024 13:57, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
> >> > On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
> >> > > On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
> >> > > > Hello Manos,
> >> > > > > On 6/10/24 11:22, Manos Pitsidianakis wrote:
> >> > > > > Hello everyone,
> >> > > > > > > This is an early draft of my work on implementing a very
> >> > > simple device,
> >> > > > > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
> >> > > > > and is used in hw/arm/virt.c).
> >> > > > > > > The device is functional, with copied logic from the C code
> >> > > but with
> >> > > > > effort not to make a direct C to Rust translation. In other words, do
> >> > > > > not write Rust as a C developer would.
> >> > > > > > > That goal is not complete but a best-effort case. To give a
> >> > > specific
> >> > > > > example, register values are typed but interrupt bit flags are not (but
> >> > > > > could be). I will leave such minutiae for later iterations.
> >> > 
> >> > snip
> >> > 
> >> > > > Maybe it could be better if build.rs file was *not* needed for new
> >> > > > devices/folders, and could be abstracted as a detail of the python
> >> > > > wrapper script instead of something that should be committed.
> >> > > 
> >> > > 
> >> > > That'd mean you cannot work on the rust files with a LanguageServer, you
> >> > > cannot run cargo build or cargo check or cargo clippy, etc. That's why I
> >> > > left the alternative choice of including a manually generated bindings file
> >> > > (generated.rs.inc)
> >> > 
> >> > I would not expect QEMU developers to be running 'cargo <anything>'
> >> > directly at all.
> >> > 
> >> > QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
> >> > convenience facade.
> >> > 
> >> > Any use of 'cargo' would be an internal impl detail of meson rules
> >> > for building rust code, and developers should still exclusively work
> >> > with 'make' or 'ninja' to run builds & tests.
> >> 
> >> No, that's not true. If I wrote the pl011 device with this workflow I'd just
> >> waste time using meson. Part of the development is making sure the library
> >> type checks, compiles, using cargo to run style formatting, to check for
> >> lints, perhaps run tests. Doing this only through meson is an unnecessary
> >> complication.
> >
> > I don't see why it should waste time, when we ultimately end up calling
> > the same underlying tools. We need to have a consistent experiance for
> > developers working on QEMU, not have to use different tools for different
> > parts of QEMU depending on whether a piece of code happens to be rust
> > or C.
> 
> For example if I wanted to run rust-based unit tests (which I think
> potentially offer an easier solution than qtest) I would expect that to
> be done from the normal make/ninja targets.

Meson provides a nice "suite" concept to facilitate selection of a
subset of tests.

eg, to limit to running just 'rust' unit tests, I might expect we
should have

  meson test --suite rustunit

and have this invoked by 'make check-rustunit' 

Similar can be done for clippy, or other types of rust tests

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, 11 Jun 2024 14:09, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>On Tue, Jun 11, 2024 at 01:58:10PM +0300, Manos Pitsidianakis wrote:
>> On Tue, 11 Jun 2024 13:57, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>> > On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
>> > > On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>> > > > Hello Manos,
>> > > > > On 6/10/24 11:22, Manos Pitsidianakis wrote:
>> > > > > Hello everyone,
>> > > > > > > This is an early draft of my work on implementing a very
>> > > simple device,
>> > > > > in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
>> > > > > and is used in hw/arm/virt.c).
>> > > > > > > The device is functional, with copied logic from the C code
>> > > but with
>> > > > > effort not to make a direct C to Rust translation. In other words, do
>> > > > > not write Rust as a C developer would.
>> > > > > > > That goal is not complete but a best-effort case. To give a
>> > > specific
>> > > > > example, register values are typed but interrupt bit flags are not (but
>> > > > > could be). I will leave such minutiae for later iterations.
>> > 
>> > snip
>> > 
>> > > > Maybe it could be better if build.rs file was *not* needed for new
>> > > > devices/folders, and could be abstracted as a detail of the python
>> > > > wrapper script instead of something that should be committed.
>> > > 
>> > > 
>> > > That'd mean you cannot work on the rust files with a LanguageServer, you
>> > > cannot run cargo build or cargo check or cargo clippy, etc. That's why I
>> > > left the alternative choice of including a manually generated bindings file
>> > > (generated.rs.inc)
>> > 
>> > I would not expect QEMU developers to be running 'cargo <anything>'
>> > directly at all.
>> > 
>> > QEMU's build system is 'meson' + 'ninja' with a 'configure' + 'make'
>> > convenience facade.
>> > 
>> > Any use of 'cargo' would be an internal impl detail of meson rules
>> > for building rust code, and developers should still exclusively work
>> > with 'make' or 'ninja' to run builds & tests.
>> 
>> No, that's not true. If I wrote the pl011 device with this workflow I'd just
>> waste time using meson. Part of the development is making sure the library
>> type checks, compiles, using cargo to run style formatting, to check for
>> lints, perhaps run tests. Doing this only through meson is an unnecessary
>> complication.
>
>I don't see why it should waste time, when we ultimately end up calling
>the same underlying tools. We need to have a consistent experiance for
>developers working on QEMU, not have to use different tools for different
>parts of QEMU depending on whether a piece of code happens to be rust
>or C.

This is a technicality but for the spirit of conversation and curiosity 
I will reply with my view: we end up calling meson and ninja, yes, but 
they wrap cargo under the hood. Cargo has its own tooling and workflows. 
At the end of compilation it spits out a dynamic or static library. That 
happens independently of QEMU.

If you start writing a new project, you don't go linking it to QEMU 
right away (you can, but there's no need to yet). You make an idiomatic 
Rust crate with rust tooling. FFI calls can exist and you are still able 
to compile/check your crate. This is the typical rust workflow, and it's 
ubiquitous, so I think we should not deprive anyone of it.

The only thing we cannot check with cargo alone is if the crate uses the 
QEMU apis correctly (logic errors, not compilation errors)

Until/if we can build with just meson and rustc (a hypothetical scenario 
at the moment) cargo becomes part of our development and build tools.

I understand your argument and it's not irrational at all. I offer this 
perspective as a Rust developer and not a QEMU developer. As a QEMU 
developer I shouldn't need to care about cargo unless I'm touching part 
of that code; the build system details (the cargo wrapper) can be 
abstracted away for everyone else.

On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
> > > The staticlib artifact contains a bunch of mangled .o objects?
> > > ==============================================================
> > > [staticlibmangledobjects] Back to [TOC]
> > > 
> > > Yes, until we compile without the `std` module library or we compile it
> > > manually instead of linking it, we will have some junk in it.
> > > 
> > 
> > Besides the size aspect, which potential advantage would there be to
> > switch to no_std?
> > We don't build a bare metal or kernel binary here, so why introduce this
> > restriction willingly?
> 
> We'll see that as we progress. Might enable more platform support, for
> example. I have no definite answers here. Also, I know binary bloat is a big
> complaint from people with dislike of Rust, so I pre-emptively addressed it.

Requiring 'no_std' would significantly limit what 3rd party crates QEMU
can make use of, and thus would put more burden on QEMU maintainers.
I don't find "binary bloat" a credible technical argument on its own
either, so certainly not sufficient justification to take on the pain
of 'no_std'.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jun 10, 2024 at 11:29:36PM +0300, Manos Pitsidianakis wrote:
>> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>> > > The staticlib artifact contains a bunch of mangled .o objects?
>> > > ==============================================================
>> > > [staticlibmangledobjects] Back to [TOC]
>> > > 
>> > > Yes, until we compile without the `std` module library or we compile it
>> > > manually instead of linking it, we will have some junk in it.
>> > > 
>> > 
>> > Besides the size aspect, which potential advantage would there be to
>> > switch to no_std?
>> > We don't build a bare metal or kernel binary here, so why introduce this
>> > restriction willingly?
>> 
>> We'll see that as we progress. Might enable more platform support, for
>> example. I have no definite answers here. Also, I know binary bloat is a big
>> complaint from people with dislike of Rust, so I pre-emptively addressed it.
>
> Requiring 'no_std' would significantly limit what 3rd party crates QEMU
> can make use of, and thus would put more burden on QEMU maintainers.
> I don't find "binary bloat" a credible technical argument on its own
> either, so certainly not sufficient justification to take on the pain
> of 'no_std'.

no_std is great for OS's and micro controllers but I don't think its
something we have to worry about for QEMU. One potential area of
co-operation would be the rust-vmm libraries and they definitely take
advantage of the stdlibs.

>
>
> With regards,
> Daniel

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

On 6/10/24 13:29, Manos Pitsidianakis wrote:
> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>> Hello Manos,
>>
>> On 6/10/24 11:22, Manos Pitsidianakis wrote:
>>> Hello everyone,
>>>
>>> This is an early draft of my work on implementing a very simple device,
>>> in this case the ARM PL011 (which in C code resides in hw/char/pl011.c
>>> and is used in hw/arm/virt.c).
>>>
>>> The device is functional, with copied logic from the C code but with
>>> effort not to make a direct C to Rust translation. In other words, do
>>> not write Rust as a C developer would.
>>>
>>> That goal is not complete but a best-effort case. To give a specific
>>> example, register values are typed but interrupt bit flags are not (but
>>> could be). I will leave such minutiae for later iterations.
>>>
>>> By the way, the wiki page for Rust was revived to keep track of all
>>> current series on the mailing list https://wiki.qemu.org/RustInQemu
>>>
>>> a #qemu-rust IRC channel was also created for rust-specific discussion
>>> that might flood #qemu
>>>
>>
>> Excellent work, and thanks for posting this RFC!
>>
>> IMHO, having patches 2 and 5 splitted is a bit confusing, and exposing
>> (temporarily) the generated.rs file in patches is not a good move.
>> Any reason you kept it this way?
> 
> That was my first approach, I will rework it on the second version. The
> generated code should not exist in committed code at all.
> 
> It was initally tricky setting up the dependency orders correctly, so I
> first committed it and then made it a dependency.
> 
>>
>> Maybe it could be better if build.rs file was *not* needed for new
>> devices/folders, and could be abstracted as a detail of the python
>> wrapper script instead of something that should be committed.
> 
> 
> That'd mean you cannot work on the rust files with a LanguageServer, you
> cannot run cargo build or cargo check or cargo clippy, etc. That's why I
> left the alternative choice of including a manually generated bindings
> file (generated.rs.inc)
> 

Maybe I missed something, but it seems like it just checks/copies the 
generated.rs file where it's expected. Definitely something that could 
be done as part of the rust build.

Having to run the build before getting completion does not seem to be a 
huge compromise.

>>
>> Having a simple rust/pl011/meson.build is nice and good taste!
>>
>>> ------------------------------------------------------------------------
>>> A request: keep comments to Rust in relation to the QEMU project and no
>>> debates on the merits of the language itself. These are valid concerns,
>>> but it'd be better if they were on separate mailing list threads.
>>> ------------------------------------------------------------------------
>>>
>>> Table of contents: [TOC]
>>>
>>> - How can I try it? [howcanItryit]
>>> - What are the most important points to focus on, at this point?
>>>     [whatarethemostimportant]
>>>     - What are the issues with not using the compiler, rustc, directly?
>>>       [whataretheissueswith]
>>>       1. Tooling
>>>       2. Rust dependencies
>>>
>>>     - Should QEMU use third-party dependencies? [shouldqemuusethirdparty]
>>>     - Should QEMU provide wrapping Rust APIs over QEMU internals?
>>>       [qemuprovidewrappingrustapis]
>>>     - Will QEMU now depend on Rust and thus not build on my XYZ platform?
>>>       [qemudependonrustnotbuildonxyz]
>>> - How is the compilation structured? [howisthecompilationstructured]
>>> - The generated.rs rust file includes a bunch of junk definitions?
>>>     [generatedrsincludesjunk]
>>> - The staticlib artifact contains a bunch of mangled .o objects?
>>>     [staticlibmangledobjects]
>>>
>>> How can I try it?
>>> =================
>>> [howcanItryit] Back to [TOC]
>>>
>>> Hopefully applying this patches (or checking out `master` branch from
>>> https://gitlab.com/epilys/rust-for-qemu/ current commit
>>> de81929e0e9d470deac2c6b449b7a5183325e7ee )
>>>
>>> Tag for this RFC is rust-pl011-rfc-v1
>>>
>>> Rustdoc documentation is hosted on
>>>
>>> https://rust-for-qemu-epilys-aebb06ca9f9adfe6584811c14ae44156501d935ba4.gitlab.io/pl011/index.html
>>>
>>> If `cargo` and `bindgen` is installed in your system, you should be able
>>> to build qemu-system-aarch64 with configure flag --enable-rust and
>>> launch an arm virt VM. One of the patches hardcodes the default UART of
>>> the machine to the Rust one, so if something goes wrong you will see it
>>> upon launching qemu-system-aarch64.
>>>
>>> To confirm it is there for sure, run e.g. info qom-tree on the monitor
>>> and look for x-pl011-rust.
>>>
>>>
>>> What are the most important points to focus on, at this point?
>>> ==============================================================
>>> [whatarethemostimportant] Back to [TOC]
>>>
>>> In my opinion, integration of the go-to Rust build system (Cargo and
>>> crates.io) with the build system we use in QEMU. This is "easily" done
>>> in some definition of the word with a python wrapper script.
>>>
>>> What are the issues with not using the compiler, rustc, directly?
>>> -----------------------------------------------------------------
>>> [whataretheissueswith] Back to [TOC]
>>>
>>> 1. Tooling
>>>      Mostly writing up the build-sys tooling to do so. Ideally we'd
>>>      compile everything without cargo but rustc directly.
>>>
>>>      If we decide we need Rust's `std` library support, we could
>>>      investigate whether building it from scratch is a good solution. This
>>>      will only build the bits we need in our devices.
>>>   > 2. Rust dependencies
>>>      We could go without them completely. I chose deliberately to include
>>>      one dependency in my UART implementation, `bilge`[0], because it has
>>>      an elegant way of representing typed bitfields for the UART's
>>>      registers.
>>>
>>> [0]: Article: https://hecatia-elegua.github.io/blog/no-more-bit-fiddling/
>>>        Crates.io page: https://crates.io/crates/bilge
>>>        Repository: https://github.com/hecatia-elegua/bilge
>>>
>>> Should QEMU use third-party dependencies?
>>> -----------------------------------------
>>> [shouldqemuusethirdparty] Back to [TOC]
>>>
>>> In my personal opinion, if we need a dependency we need a strong
>>> argument for it. A dependency needs a trusted upstream source, a QEMU
>>> maintainer to make sure it us up-to-date in QEMU etc.
>>>
>>> We already fetch some projects with meson subprojects, so this is not a
>>> new reality. Cargo allows you to define "locked" dependencies which is
>>> the same as only fetching specific commits by SHA. No suspicious
>>> tarballs, and no disappearing dependencies a la left-pad in npm.
>>>
>>
>> As a complement to this, and for other readers, in more than having a
>> lock file (fixing version you use), cargo crates system is designed to
>> be immutable (see: https://crates.io/policies), and it means there is a
>> strong guarantee that a published package will stay there, to the
>> opposite of npm, pypi, or most of other similar systems.
>>
>> "Crate deletion by their owners is not possible to keep the registry as
>> immutable as possible."
>>
>> I believe this is a *key* feature of Rust ecosystem and should be
>> emphasized regarding the policy for Rust dependencies to come.
>>
>>> However, I believe it's worth considering vendoring every dependency by
>>> default, if they prove to be few, for the sake of having a local QEMU
>>> git clone buildable without network access.
>>>
>>
>> I would not be in favor to vendor all dependencies. Beyond the "offline
>> build" scenario, it has only downsides.
>>
>> Sure, we should really debate before introducing a new dependency, but
>> the technical difficulty to mirror its sources and dependencies should
>> not be an argument for or against it.
>>
>> What will happen the day we want to introduce something bigger than a
>> simple dependency? (let's say "serde" for instance)
> 
> 
> Yes, vendor-the-world is a different topic than vendor e.g. two crates
> such as the dependencies I'm using here.
>

If there must be a discussion about dependencies, it's probably better 
to consider the "worse" case to take a decison about vendoring this or not.

>>
>>> Should QEMU provide wrapping Rust APIs over QEMU internals?
>>> -----------------------------------------------------------
>>> [qemuprovidewrappingrustapis] Back to [TOC]
>>>
>>> My personal opinion is no, with the reasoning being that QEMU internals
>>> are not documented or stable. However I do not see why creating stable
>>> opt-in interfaces is bad. It just needs someone to volunteer to maintain
>>> it and ensure there are no breakages through versions.
>>>
>>> Will QEMU now depend on Rust and thus not build on my XYZ platform?
>>> -------------------------------------------------------------------
>>> [qemudependonrustnotbuildonxyz] Back to [TOC]
>>>
>>> No, worry about this in some years if this experiment takes off. Rust
>>> has broad platform support and is present in most distro package
>>> managers. In the future we might have gcc support for it as well.
>>>
>>> For now, Rust will have an experimental status, and will be aimed to
>>> those who wish to try it. I leave it to the project leaders to make
>>> proper decisions and statements on this if necessary.
>>>
>>>
>>> How is the compilation structured?
>>> ==================================
>>> [howisthecompilationstructured] Back to [TOC]
>>>
>>> First, a meson target that runs `bindgen` on a bunch of header files
>>> (defined in `rust/wrapper.h`) is created as a target and as a dependency
>>> for any rust hardware device that needs it. You can see the generated
>>> bindings by running
>>>
>>>     ninja generated.rs
>>>
>>> inside your build directory.
>>>
>>> The devices are defined as dictionaries in rust/meson.build because they
>>> depend on the bindgen dependency, which is available much later in the
>>> meson process (when the static qemu lib and target emulator executables
>>> are defined).
>>>
>>> A cargo wrapper python script under scripts/ exists to build the crate
>>> library, by providing the path to the generated.rs bindings via the
>>> environment. Then, the qemu-system-aarch64 binary links against the
>>> staticlib archive (i.e. libpl011.a)
>>>
>>> The generated.rs rust file includes a bunch of junk definitions?
>>> ================================================================
>>> [generatedrsincludesjunk] Back to [TOC]
>>>
>>> Yes, bindgen allows you to block certain types and identifiers from
>>> being generated but they are simply too many. I have trimmed some of the
>>> fat but vast improvements can be made.
>>>
>>> The staticlib artifact contains a bunch of mangled .o objects?
>>> ==============================================================
>>> [staticlibmangledobjects] Back to [TOC]
>>>
>>> Yes, until we compile without the `std` module library or we compile it
>>> manually instead of linking it, we will have some junk in it.
>>>
>>
>> Besides the size aspect, which potential advantage would there be to
>> switch to no_std?
>> We don't build a bare metal or kernel binary here, so why introduce this
>> restriction willingly?
> 
> We'll see that as we progress. Might enable more platform support, for
> example. I have no definite answers here. Also, I know binary bloat is a
> big complaint from people with dislike of Rust, so I pre-emptively
> addressed it.

I understand.
Stripping rust binary does a big difference in size, including for 
release builds.
I hope we'll explore other options than using no_std if binary size 
should become a decisive criteria.

Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:

> On 6/10/24 13:29, Manos Pitsidianakis wrote:
>> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>>> Hello Manos,
>>>
<snip>
>>> Excellent work, and thanks for posting this RFC!
>>>
>>> IMHO, having patches 2 and 5 splitted is a bit confusing, and exposing
>>> (temporarily) the generated.rs file in patches is not a good move.
>>> Any reason you kept it this way?
>> That was my first approach, I will rework it on the second version.
>> The
>> generated code should not exist in committed code at all.
>> It was initally tricky setting up the dependency orders correctly,
>> so I
>> first committed it and then made it a dependency.
>> 
>>>
>>> Maybe it could be better if build.rs file was *not* needed for new
>>> devices/folders, and could be abstracted as a detail of the python
>>> wrapper script instead of something that should be committed.
>> That'd mean you cannot work on the rust files with a LanguageServer,
>> you
>> cannot run cargo build or cargo check or cargo clippy, etc. That's why I
>> left the alternative choice of including a manually generated bindings
>> file (generated.rs.inc)
>> 
>
> Maybe I missed something, but it seems like it just checks/copies the
> generated.rs file where it's expected. Definitely something that could
> be done as part of the rust build.
>
> Having to run the build before getting completion does not seem to be
> a huge compromise.
>
<snip>

As long as the Language Server can kick in after a first build. Rust
definitely leans in to the concept of the tooling helping you out while
coding.

I think for the C LSPs compile_commands.json is generated during the
configure step but I could be wrong.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

On 6/11/24 02:21, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:
> 
>> On 6/10/24 13:29, Manos Pitsidianakis wrote:
>>> On Mon, 10 Jun 2024 22:37, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>>>> Hello Manos,
>>>>
> <snip>
>>>> Excellent work, and thanks for posting this RFC!
>>>>
>>>> IMHO, having patches 2 and 5 splitted is a bit confusing, and exposing
>>>> (temporarily) the generated.rs file in patches is not a good move.
>>>> Any reason you kept it this way?
>>> That was my first approach, I will rework it on the second version.
>>> The
>>> generated code should not exist in committed code at all.
>>> It was initally tricky setting up the dependency orders correctly,
>>> so I
>>> first committed it and then made it a dependency.
>>>
>>>>
>>>> Maybe it could be better if build.rs file was *not* needed for new
>>>> devices/folders, and could be abstracted as a detail of the python
>>>> wrapper script instead of something that should be committed.
>>> That'd mean you cannot work on the rust files with a LanguageServer,
>>> you
>>> cannot run cargo build or cargo check or cargo clippy, etc. That's why I
>>> left the alternative choice of including a manually generated bindings
>>> file (generated.rs.inc)
>>>
>>
>> Maybe I missed something, but it seems like it just checks/copies the
>> generated.rs file where it's expected. Definitely something that could
>> be done as part of the rust build.
>>
>> Having to run the build before getting completion does not seem to be
>> a huge compromise.
>>
> <snip>
> 
> As long as the Language Server can kick in after a first build. Rust
> definitely leans in to the concept of the tooling helping you out while
> coding.
> 
> I think for the C LSPs compile_commands.json is generated during the
> configure step but I could be wrong.
> 

Yes, meson generates it.
I agree having support for completion tooling is important nowadays, 
whether in C or in Rust.

On Tue, 11 Jun 2024 00:38, Pierrick Bouvier <pierrick.bouvier@linaro.org> wrote:
>>> Maybe it could be better if build.rs file was *not* needed for new 
>>> devices/folders, and could be abstracted as a detail of the python 
>>> wrapper script instead of something that should be committed.
>>
>> That'd mean you cannot work on the rust files with a LanguageServer, 
>> you cannot run cargo build or cargo check or cargo clippy, etc. 
>> That's why I left the alternative choice of including a manually 
>> generated bindings file (generated.rs.inc)
>
>Maybe I missed something, but it seems like it just checks/copies the 
>generated.rs file where it's expected. Definitely something that could 
>be done as part of the rust build.
>
>Having to run the build before getting completion does not seem to be a 
>huge compromise.

It only checks if it's called from meson, hence it should update the 
should choose meson's generated.rs file. Otherwise it falls back to any 
manual bindings you have put there that are not checked into git. So 
essentially it does what you suggest, I think :)


>>
>> Yes, vendor-the-world is a different topic than vendor e.g. two 
>> crates such as the dependencies I'm using here.
>>
>
>If there must be a discussion about dependencies, it's probably better 
>to consider the "worse" case to take a decison about vendoring this or not.
>

Agreed. To re-cap, my opinion is that vendoring 1-2 small crates is 
fine, but any more than that needs rethinking.