QEMU bundles a copy of dtc and keycodemapdb.  They both support meson,
so we can run their build system via subproject() instead of hardcoding
their contents.

In fact, now that QEMU's configure script knows how to install missing
Python dependencies, we can replace submodules altogether with .wrap
files, which have several advantages, either immediate or potential:

* option parsing and downloading is delegated to meson

* the commit is stored in a text file instead of a magic entry in the
  git tree object, and can be a branch name or a version number as well

* we could stop shipping external dependencies that are only used as a
  fallback, but not break compilation on platforms that lack them.
  For example it may make sense to download dtc at build time,
  controlled by --enable-download, even when building from a tarball.
  This is _not_ done in this patch series, but Marc-André has tried
  it before[1].

* we could also add .wrap files for other dependencies that are missing
  on the GCC compile farm machines.  People who build on Windows might also
  enjoy getting the mandatory dependencies (pixman, zlib, glib, possibly
  SDL?) via wraps.

It is possible to use subprojects also for berkeley-softfloat-3
and berkeley-testfloat-3.  This would require moving the corresponding
parts of tests/fp/meson.build to an overlay file.

Paolo

[1] https://patchew.org/QEMU/20230302131848.1527460-1-marcandre.lureau@redhat.com/20230302131848.1527460-5-marcandre.lureau@redhat.com/


Paolo Bonzini (6):
  remove remaining traces of meson submodule
  meson: simplify logic for -Dfdt
  meson: use subproject for internal libfdt
  meson: use subproject for keycodemapdb
  configure: rename --enable-pypi to --enable-download, control
    subprojects too
  meson: subprojects: replace submodules with wrap files

 .gitignore                          |  2 -
 .gitlab-ci.d/buildtest-template.yml |  5 +-
 .gitmodules                         |  9 ----
 configure                           | 71 ++++++-----------------------
 dtc                                 |  1 -
 meson.build                         | 56 ++++++-----------------
 meson_options.txt                   |  1 +
 scripts/archive-source.sh           | 11 ++++-
 scripts/make-release                |  5 ++
 subprojects/.gitignore              |  6 +++
 subprojects/dtc.wrap                |  4 ++
 subprojects/keycodemapdb.wrap       |  4 ++
 subprojects/libvfio-user            |  1 -
 subprojects/libvfio-user.wrap       |  4 ++
 ui/keycodemapdb                     |  1 -
 ui/meson.build                      |  8 ++--
 16 files changed, 68 insertions(+), 121 deletions(-)
 delete mode 160000 dtc
 create mode 100644 subprojects/.gitignore
 create mode 100644 subprojects/dtc.wrap
 create mode 100644 subprojects/keycodemapdb.wrap
 delete mode 160000 subprojects/libvfio-user
 create mode 100644 subprojects/libvfio-user.wrap
 delete mode 160000 ui/keycodemapdb

-- 
2.40.1

On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> QEMU bundles a copy of dtc and keycodemapdb.  They both support meson,
> so we can run their build system via subproject() instead of hardcoding
> their contents.
>
> In fact, now that QEMU's configure script knows how to install missing
> Python dependencies, we can replace submodules altogether with .wrap
> files, which have several advantages, either immediate or potential:

So do we have a path for getting rid of submodules entirely?
I'd be all in favour of that, but I'm somewhat less in favour
of "some things are submodules and some things are this new
.wrap file setup"...

thanks
-- PMM

On Fri, May 19, 2023 at 10:21:03AM +0100, Peter Maydell wrote:
> On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > QEMU bundles a copy of dtc and keycodemapdb.  They both support meson,
> > so we can run their build system via subproject() instead of hardcoding
> > their contents.
> >
> > In fact, now that QEMU's configure script knows how to install missing
> > Python dependencies, we can replace submodules altogether with .wrap
> > files, which have several advantages, either immediate or potential:
> 
> So do we have a path for getting rid of submodules entirely?
> I'd be all in favour of that, but I'm somewhat less in favour
> of "some things are submodules and some things are this new
> .wrap file setup"...

I think we can ignore 'roms' submodules on the basis they are never used
during a normal build, they're mostly just a maintainer convenience and
for release archive source bundling. That would leave us with

$ grep submodule .gitmodules  | sort | grep -v roms
[submodule "dtc"]
[submodule "subprojects/libvfio-user"]
[submodule "tests/fp/berkeley-softfloat-3"]
[submodule "tests/fp/berkeley-testfloat-3"]
[submodule "tests/lcitool/libvirt-ci"]
[submodule "ui/keycodemapdb"]


as submodules which are commonly used. This series handles dtc, libvfio-user
and keycodemapdb leaving the float tests and the libvirt-ci module. The
libvirt-ci module isn't used by meson, so I guess that doesn't directly
fit into its wrap handling.

Maybe we can none the less wire up libvirt-ci into wrap, or alternatively
since its python, could we make our new venv setup tools put libvirt-ci
into the venv ?

Paolo mentioned the float stuff as future work, but not sure what his
plans are specifically?

Feels like we should be able to figure out some way to get rid of all
the submodules though, except for the roms, which are special and ok to
leave IMHO

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 5/19/23 11:32, Daniel P. Berrangé wrote:
> Feels like we should be able to figure out some way to get rid of all
> the submodules though, except for the roms, which are special and ok to
> leave IMHO

Hmm, almost.  roms/SLOF is used to build 
pc-bios/s390-ccw/s390-netboot.img, so right now GIT_SUBMODULE and 
GIT_SUBMODULE_ACTION could not be removed even without counting the 
tests/fp submodules.  That's about 10,000 lines of code, or about 20% of 
all of SLOF; it might be a bit too much for vendoring.

Adding Thomas for ideas...

Paolo

On 19/05/2023 13.48, Paolo Bonzini wrote:
> On 5/19/23 11:32, Daniel P. Berrangé wrote:
>> Feels like we should be able to figure out some way to get rid of all
>> the submodules though, except for the roms, which are special and ok to
>> leave IMHO
> 
> Hmm, almost.  roms/SLOF is used to build pc-bios/s390-ccw/s390-netboot.img, 
> so right now GIT_SUBMODULE and GIT_SUBMODULE_ACTION could not be removed 
> even without counting the tests/fp submodules.  That's about 10,000 lines of 
> code, or about 20% of all of SLOF; it might be a bit too much for vendoring.
> 
> Adding Thomas for ideas...

Sorry, I also don't have any good ideas here. We could maybe copy the libs 
from the SLOF repository over into the s390-ccw folder, but that's kind of 
ugly, too (there aren't many updates in SLOF anymore, so it wouldn't at 
least hurt too much from a synchronization point of view) ... CC:-ing 
qemu-s390x mailing list, maybe somebody else there has a better idea.

  Thomas

On Fri, May 19, 2023 at 11:32 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> Maybe we can none the less wire up libvirt-ci into wrap, or alternatively
> since its python, could we make our new venv setup tools put libvirt-ci
> into the venv ?

libvirt-ci is used from the source tree so I think it doesn't even fit
into the venv. It's not unlike the roms.

> Paolo mentioned the float stuff as future work, but not sure what his
> plans are specifically?

No plans in particular---after all the whole meson conversion was like
"let's try to find a day every month to proceed", and these two
submodules are not particularly big. I know it's doable but I heard
there were plans of embedding it too, so I left it aside and focused
on the more interesting ones.

Paolo

> Feels like we should be able to figure out some way to get rid of all
> the submodules though, except for the roms, which are special and ok to
> leave IMHO
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

On Fri, 19 May 2023 at 10:32, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, May 19, 2023 at 10:21:03AM +0100, Peter Maydell wrote:
> > On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > >
> > > QEMU bundles a copy of dtc and keycodemapdb.  They both support meson,
> > > so we can run their build system via subproject() instead of hardcoding
> > > their contents.
> > >
> > > In fact, now that QEMU's configure script knows how to install missing
> > > Python dependencies, we can replace submodules altogether with .wrap
> > > files, which have several advantages, either immediate or potential:
> >
> > So do we have a path for getting rid of submodules entirely?
> > I'd be all in favour of that, but I'm somewhat less in favour
> > of "some things are submodules and some things are this new
> > .wrap file setup"...
>
> I think we can ignore 'roms' submodules on the basis they are never used
> during a normal build, they're mostly just a maintainer convenience and
> for release archive source bundling.

Yeah, I agree -- or at least I agree they're a distinct use of
submodules. We might at some point want to look at whether there's
a better way to deal with ROM blobs than what we do today, especially
as the size of the source archives for things like UEFI is much
bigger than I expect anybody anticipated when they set the rom
blob machinery up. I don't think the way we do ROM blobs is
very convenient for our distro downstreams either.

-- PMM

On Fri, May 19, 2023 at 11:21 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > QEMU bundles a copy of dtc and keycodemapdb.  They both support meson,
> > so we can run their build system via subproject() instead of hardcoding
> > their contents.
> >
> > In fact, now that QEMU's configure script knows how to install missing
> > Python dependencies, we can replace submodules altogether with .wrap
> > files, which have several advantages, either immediate or potential:
>
> So do we have a path for getting rid of submodules entirely?

For building QEMU, yes.

The firmware submodules would not be affected, but those are really
just a way to ensure we respect the GPL when distributing firmware
blobs. We could keep them, as they mostly stay out of the way, or we
could move the information to files that are then consulted by
scripts/make-release. And those files could very well reuse the same
".ini" format that is used by .wrap files.

Paolo

> I'd be all in favour of that, but I'm somewhat less in favour
> of "some things are submodules and some things are this new
> .wrap file setup"...

On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> It is possible to use subprojects also for berkeley-softfloat-3
> and berkeley-testfloat-3.  This would require moving the corresponding
> parts of tests/fp/meson.build to an overlay file.

I think we also talked about simply putting those directly in
the main qemu repo, since we've forked the tests anyway.

-- PMM