Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116
MiB. If you look at the contents, approx. 80% of the size is used for the
firmware sources that we ship along to provide the sources for the ROM
binaries. This feels very wrong, why do we urge users to download such
huge tarballs while 99.9% of them never will rebuilt the firmware sources?
We were also struggeling a bit in the past already with server load and
costs, so we should really try to decrease the size of our release tarballs
to a saner level.

So let's split the firmware sources into a separate tarball to decrease
the size of the main QEMU sources tarball a lot (which should help us
to safe a lot of traffic on the server).

Additional improvements for the make-release script add a little help text
and speed it up by downloading less data from the various git repositories.

v2:
 - Move the firmware sources into a separate tarball instead of dropping
   the edk2 and skiboot sources.

Thomas Huth (5):
  scripts/make-release: Add a simple help text for the script
  scripts/make-release: Only clone single branches to speed up the
    script
  scripts/make-release: Remove CI yaml and more git files from the
    tarball
  roms: Add a README file with some basic information
  scripts/make-release: Move roms into separate tarball

 roms/README.rst      | 24 ++++++++++++++++++++++++
 scripts/make-release | 34 +++++++++++++++++++++++++++++-----
 2 files changed, 53 insertions(+), 5 deletions(-)
 create mode 100644 roms/README.rst

-- 
2.31.1

This is great. It will reduce qemu.org's network bandwidth consumption
and make QEMU release tarballs nicer to use due to reduced size.

I left a comment because I don't like patching the roms/ directory,
but if I'm the only one who doesn't like it then feel free to keep
that approach.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

On Mon, Nov 28, 2022 at 10:25:50AM +0100, Thomas Huth wrote:
> Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116
> MiB. If you look at the contents, approx. 80% of the size is used for the
> firmware sources that we ship along to provide the sources for the ROM
> binaries. This feels very wrong, why do we urge users to download such
> huge tarballs while 99.9% of them never will rebuilt the firmware sources?
> We were also struggeling a bit in the past already with server load and
> costs, so we should really try to decrease the size of our release tarballs
> to a saner level.

The main reason for shipping the source in the tarball was to
guarantee license compliance for anyone who is distributing
qemu release tarballs, including ourselves.

Splitting off the firmware source, but not the firmware binaries,
means people are now at risk of not complying with the license
but failing to provide complete and corresponding source.

Technically the license requirement is only critical for GPL
licenses ROMs, but as good practice we do it for all our ROMs.

> So let's split the firmware sources into a separate tarball to decrease
> the size of the main QEMU sources tarball a lot (which should help us
> to safe a lot of traffic on the server).

With my distro maintainer hat I would rather QEMU ship neither the
ROM source, nor the ROM binaries.

Still the binaries are convenient for people doing their own QEMU
builds from source.

How about shipping two distinct options:

  qemu-x.y.z.tar.xz          (QEMU source only)
  qemu-bundled-x.y.z.tar.xz  (QEMU source + bundled ROM binaries + ROM sources)

though I'm not sure how much of an impact that will have on the download
traffic - depends what is causing the traffic.

Another option is

  qemu-x.y.z.tar.xz        (QEMU source only)
  qemu-roms-x.y.z.tar.xz   (bundled ROM binaries + ROM sources)

though this is slightly more inconvenient for users, and there's the
risk they'll use new QEMU with old ROMs.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 28/11/2022 18.04, Daniel P. Berrangé wrote:
> On Mon, Nov 28, 2022 at 10:25:50AM +0100, Thomas Huth wrote:
>> Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116
>> MiB. If you look at the contents, approx. 80% of the size is used for the
>> firmware sources that we ship along to provide the sources for the ROM
>> binaries. This feels very wrong, why do we urge users to download such
>> huge tarballs while 99.9% of them never will rebuilt the firmware sources?
>> We were also struggeling a bit in the past already with server load and
>> costs, so we should really try to decrease the size of our release tarballs
>> to a saner level.
> 
> The main reason for shipping the source in the tarball was to
> guarantee license compliance for anyone who is distributing
> qemu release tarballs, including ourselves.
> 
> Splitting off the firmware source, but not the firmware binaries,
> means people are now at risk of not complying with the license
> but failing to provide complete and corresponding source.
> 
> Technically the license requirement is only critical for GPL
> licenses ROMs, but as good practice we do it for all our ROMs.
> 
>> So let's split the firmware sources into a separate tarball to decrease
>> the size of the main QEMU sources tarball a lot (which should help us
>> to safe a lot of traffic on the server).
> 
> With my distro maintainer hat I would rather QEMU ship neither the
> ROM source, nor the ROM binaries.
> 
> Still the binaries are convenient for people doing their own QEMU
> builds from source.
> 
> How about shipping two distinct options:
> 
>    qemu-x.y.z.tar.xz          (QEMU source only)
>    qemu-bundled-x.y.z.tar.xz  (QEMU source + bundled ROM binaries + ROM sources)
> 
> though I'm not sure how much of an impact that will have on the download
> traffic - depends what is causing the traffic.
> 
> Another option is
> 
>    qemu-x.y.z.tar.xz        (QEMU source only)
>    qemu-roms-x.y.z.tar.xz   (bundled ROM binaries + ROM sources)
> 
> though this is slightly more inconvenient for users, and there's the
> risk they'll use new QEMU with old ROMs.

Maybe that would work for distros, but I don't think that these are good 
options for the average users who just want to download and recompile the 
latest version of QEMU on their own.
I assume that most users don't have an environment with cross-compilers or 
for running things in a container, so I think they still want to use the 
pre-built binaries. Thus, if you bundle the binaries along with their 
sources, people will still continue to download the big tarball and we 
haven't gained anything.

So do you really really think shipping the binaries in the main tarball is a 
problem? Honestly, it's not a problem for us as long as we publish both 
tarballs together - and if someone wants to mirror the main tarball to their 
webserver and fails to mirror the rom-sources tarball, too, it's their 
fault, not ours.

Anyway, what about splitting the binaries into a separate tarball, so we 
would have three tarballs:

     qemu-x.y.z.tar.xz               (QEMU source only)
     qemu-roms-x.y.z.tar.xz          (ROM binaries)
     qemu-roms-sources-x.y.z.tar.xz  (ROM sources)

That should make it hopefully obvious that the two qemu-roms* tarballs 
belong together. Would that be OK for you?

  Thomas

On Wed, Nov 30, 2022 at 11:49:50AM +0100, Thomas Huth wrote:
> On 28/11/2022 18.04, Daniel P. Berrangé wrote:
> > On Mon, Nov 28, 2022 at 10:25:50AM +0100, Thomas Huth wrote:
> > > Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116
> > > MiB. If you look at the contents, approx. 80% of the size is used for the
> > > firmware sources that we ship along to provide the sources for the ROM
> > > binaries. This feels very wrong, why do we urge users to download such
> > > huge tarballs while 99.9% of them never will rebuilt the firmware sources?
> > > We were also struggeling a bit in the past already with server load and
> > > costs, so we should really try to decrease the size of our release tarballs
> > > to a saner level.
> > 
> > The main reason for shipping the source in the tarball was to
> > guarantee license compliance for anyone who is distributing
> > qemu release tarballs, including ourselves.
> > 
> > Splitting off the firmware source, but not the firmware binaries,
> > means people are now at risk of not complying with the license
> > but failing to provide complete and corresponding source.
> > 
> > Technically the license requirement is only critical for GPL
> > licenses ROMs, but as good practice we do it for all our ROMs.
> > 
> > > So let's split the firmware sources into a separate tarball to decrease
> > > the size of the main QEMU sources tarball a lot (which should help us
> > > to safe a lot of traffic on the server).
> > 
> > With my distro maintainer hat I would rather QEMU ship neither the
> > ROM source, nor the ROM binaries.
> > 
> > Still the binaries are convenient for people doing their own QEMU
> > builds from source.
> > 
> > How about shipping two distinct options:
> > 
> >    qemu-x.y.z.tar.xz          (QEMU source only)
> >    qemu-bundled-x.y.z.tar.xz  (QEMU source + bundled ROM binaries + ROM sources)
> > 
> > though I'm not sure how much of an impact that will have on the download
> > traffic - depends what is causing the traffic.
> > 
> > Another option is
> > 
> >    qemu-x.y.z.tar.xz        (QEMU source only)
> >    qemu-roms-x.y.z.tar.xz   (bundled ROM binaries + ROM sources)
> > 
> > though this is slightly more inconvenient for users, and there's the
> > risk they'll use new QEMU with old ROMs.
> 
> Maybe that would work for distros, but I don't think that these are good
> options for the average users who just want to download and recompile the
> latest version of QEMU on their own.
> I assume that most users don't have an environment with cross-compilers or
> for running things in a container, so I think they still want to use the
> pre-built binaries. Thus, if you bundle the binaries along with their
> sources, people will still continue to download the big tarball and we
> haven't gained anything.
> 
> So do you really really think shipping the binaries in the main tarball is a
> problem? Honestly, it's not a problem for us as long as we publish both
> tarballs together - and if someone wants to mirror the main tarball to their
> webserver and fails to mirror the rom-sources tarball, too, it's their
> fault, not ours.

I think we would be contributing to mistakes by providing a tarball
that contains a mixture of sources and binaries, but not all the
sources for all the binaries. 

> Anyway, what about splitting the binaries into a separate tarball, so we
> would have three tarballs:
> 
>     qemu-x.y.z.tar.xz               (QEMU source only)
>     qemu-roms-x.y.z.tar.xz          (ROM binaries)
>     qemu-roms-sources-x.y.z.tar.xz  (ROM sources)
> 
> That should make it hopefully obvious that the two qemu-roms* tarballs
> belong together. Would that be OK for you?

Yes, I think that's better, as it is cleanly separating the
binaries and sources.

Even bikeshedding a bit more I woud have probably suggested
'qemu-roms-prebuilt-x.y.z.tar.xz' for ROM binaries and
'qemu-roms-x.y.z.tar.xz for the ROM sources, since sources
is the normal tarball content.

The downside is that there's the risk of ROMS not matching
the QEMU version, if people updates to latest qemu tarball
but forgot the corresponding ROMs tarball. Most of the time
a mismatch would not matter, but we should think about if
there is a way to make it easier to diagnose such a
mismatch if only for easier bug triage.

Perhaps the ROMs should install into a versioned subdir
of /usr/share/qemu, instead of the root of it, and the
QEMU binary preferentially look at the versioned subdir
Maybe that's overthinking things though, and we would
suffic to have a /usr/share/qemu/ROM-VERSION.txt file

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 30/11/2022 13.09, Daniel P. Berrangé wrote:
> On Wed, Nov 30, 2022 at 11:49:50AM +0100, Thomas Huth wrote:
>> On 28/11/2022 18.04, Daniel P. Berrangé wrote:
>>> On Mon, Nov 28, 2022 at 10:25:50AM +0100, Thomas Huth wrote:
>>>> Our release tarballs are huge - qemu-7.2.0-rc2.tar.xz has a size of 116
>>>> MiB. If you look at the contents, approx. 80% of the size is used for the
>>>> firmware sources that we ship along to provide the sources for the ROM
>>>> binaries. This feels very wrong, why do we urge users to download such
>>>> huge tarballs while 99.9% of them never will rebuilt the firmware sources?
>>>> We were also struggeling a bit in the past already with server load and
>>>> costs, so we should really try to decrease the size of our release tarballs
>>>> to a saner level.
>>>
>>> The main reason for shipping the source in the tarball was to
>>> guarantee license compliance for anyone who is distributing
>>> qemu release tarballs, including ourselves.
>>>
>>> Splitting off the firmware source, but not the firmware binaries,
>>> means people are now at risk of not complying with the license
>>> but failing to provide complete and corresponding source.
>>>
>>> Technically the license requirement is only critical for GPL
>>> licenses ROMs, but as good practice we do it for all our ROMs.
>>>
>>>> So let's split the firmware sources into a separate tarball to decrease
>>>> the size of the main QEMU sources tarball a lot (which should help us
>>>> to safe a lot of traffic on the server).
>>>
>>> With my distro maintainer hat I would rather QEMU ship neither the
>>> ROM source, nor the ROM binaries.
>>>
>>> Still the binaries are convenient for people doing their own QEMU
>>> builds from source.
>>>
>>> How about shipping two distinct options:
>>>
>>>     qemu-x.y.z.tar.xz          (QEMU source only)
>>>     qemu-bundled-x.y.z.tar.xz  (QEMU source + bundled ROM binaries + ROM sources)
>>>
>>> though I'm not sure how much of an impact that will have on the download
>>> traffic - depends what is causing the traffic.
>>>
>>> Another option is
>>>
>>>     qemu-x.y.z.tar.xz        (QEMU source only)
>>>     qemu-roms-x.y.z.tar.xz   (bundled ROM binaries + ROM sources)
>>>
>>> though this is slightly more inconvenient for users, and there's the
>>> risk they'll use new QEMU with old ROMs.
>>
>> Maybe that would work for distros, but I don't think that these are good
>> options for the average users who just want to download and recompile the
>> latest version of QEMU on their own.
>> I assume that most users don't have an environment with cross-compilers or
>> for running things in a container, so I think they still want to use the
>> pre-built binaries. Thus, if you bundle the binaries along with their
>> sources, people will still continue to download the big tarball and we
>> haven't gained anything.
>>
>> So do you really really think shipping the binaries in the main tarball is a
>> problem? Honestly, it's not a problem for us as long as we publish both
>> tarballs together - and if someone wants to mirror the main tarball to their
>> webserver and fails to mirror the rom-sources tarball, too, it's their
>> fault, not ours.
> 
> I think we would be contributing to mistakes by providing a tarball
> that contains a mixture of sources and binaries, but not all the
> sources for all the binaries.
> 
>> Anyway, what about splitting the binaries into a separate tarball, so we
>> would have three tarballs:
>>
>>      qemu-x.y.z.tar.xz               (QEMU source only)
>>      qemu-roms-x.y.z.tar.xz          (ROM binaries)
>>      qemu-roms-sources-x.y.z.tar.xz  (ROM sources)
>>
>> That should make it hopefully obvious that the two qemu-roms* tarballs
>> belong together. Would that be OK for you?
> 
> Yes, I think that's better, as it is cleanly separating the
> binaries and sources.

I pondered about this idea quite a while now, but I think that will 
definitely be a decrease in the user's experience, e.g. "make install" won't 
work by default anymore if the user did not download the binaries before. 
Sure, we can instruct the users to download and uncompress the roms tarball 
here or there ... but still, it's getting more cumbersome for the users this 
way. Thus I think I'd rather would like to avoid packaging the binaries in 
an extra tarball.

> The downside is that there's the risk of ROMS not matching
> the QEMU version, if people updates to latest qemu tarball
> but forgot the corresponding ROMs tarball. Most of the time
> a mismatch would not matter, but we should think about if
> there is a way to make it easier to diagnose such a
> mismatch if only for easier bug triage.

Yes, that's another disadvantage which will lead to very subtle bug reports 
if we don't prevent it from happening somehow.

> Perhaps the ROMs should install into a versioned subdir
> of /usr/share/qemu, instead of the root of it, and the
> QEMU binary preferentially look at the versioned subdir
> Maybe that's overthinking things though, and we would
> suffic to have a /usr/share/qemu/ROM-VERSION.txt file

I don't think that a text file will be sufficient to prevent people running 
into the problem. Having versioned subdirs might help, but it has the 
disadvantage that it might cause lots of old files lying around in the file 
system if the users forget to delete the old versions after an upgrade.


... Ok, let's start again from scratch. So what we are trying to solve is 
that our tarballs are too huge and cause a non-neglectable traffic on the 
server. A huge part of the tarball size is caused by the edk2 sources. And 
we don't want to separate the GPL firmware sources from the binaries. ... so 
what if we just move the edk2 sources (and maybe skiboot sources) into a 
separate tarball? These sources are not licensed under a strong copyleft 
license, so it should be OK if we don't ship the sources for these binaries 
in the main tarball (similar to my v1 series, but I was rather completely 
removing the sources there instead). We could call the tarball with the 
separate source like qemu-extra-rom-sources.tar.xz or so?

  Thomas

Il lun 28 nov 2022, 18:04 Daniel P. Berrangé <berrange@redhat.com> ha
scritto:

> With my distro maintainer hat I would rather QEMU ship neither the
> ROM source, nor the ROM binaries.
>

Annd since QEMU can finally cross compile its embedded firmware modules,
too, it is now easier for distros not to use any prebuilt binary.

However some firmware sources are only available from QEMU's submodules. So
either we distribute those submodules as separate tarballs, or distros
would need to use the bundled tarball as well.

Separately, I am not even sure what compiler is needed for the old
Macintosh ROMs...

Paolo


> Still the binaries are convenient for people doing their own QEMU
> builds from source.
>
> How about shipping two distinct options:
>
>   qemu-x.y.z.tar.xz          (QEMU source only)
>   qemu-bundled-x.y.z.tar.xz  (QEMU source + bundled ROM binaries + ROM
> sources)
>
> though I'm not sure how much of an impact that will have on the download
> traffic - depends what is causing the traffic.
>
> Another option is
>
>   qemu-x.y.z.tar.xz        (QEMU source only)
>   qemu-roms-x.y.z.tar.xz   (bundled ROM binaries + ROM sources)
>
> though this is slightly more inconvenient for users, and there's the
> risk they'll use new QEMU with old ROMs.
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

On 28/11/2022 19:25, Paolo Bonzini wrote:

> Il lun 28 nov 2022, 18:04 Daniel P. Berrangé <berrange@redhat.com 
> <mailto:berrange@redhat.com>> ha scritto:
> 
>     With my distro maintainer hat I would rather QEMU ship neither the
>     ROM source, nor the ROM binaries.
> 
> 
> Annd since QEMU can finally cross compile its embedded firmware modules, too, it is 
> now easier for distros not to use any prebuilt binary.
> 
> However some firmware sources are only available from QEMU's submodules. So either we 
> distribute those submodules as separate tarballs, or distros would need to use the 
> bundled tarball as well.
> 
> Separately, I am not even sure what compiler is needed for the old Macintosh ROMs...
> 
> Paolo

I can quickly summarise what is needed for the ROMs that I'm involved with:

- OpenBIOS is currently built using the builder docker image hosted at 
https://github.com/openbios/openbios/pkgs/container/openbios-builder which is based 
upon the kernel crosstool archives.

I initially tried using the QEMU docker cross-compilers, however whilst the builds 
work the binaries generated would crash early for PPC/SPARC64. I suspect it's likely 
because of specific patches applied by Debian or the options used for the build but 
didn't really dig into it further.

- qemu_vga.ndrv is a tricky one: it's actually built by Metrowerks CodeWarrior 
running in an OS 9 VM in QEMU!

There is a project called Retro68 (https://github.com/autc04/Retro68) which is a port 
of gcc for 68k/ppc Macs which could potentially be used, however it would need 
conversion of the driver from Metrowerks to gcc, and also likely require the use of 
Apple's MPW headers which were free to distribute but whose copyright status is 
currently undetermined.


ATB,

Mark.

On Mon, Nov 28, 2022 at 08:25:24PM +0100, Paolo Bonzini wrote:
> Il lun 28 nov 2022, 18:04 Daniel P. Berrangé <berrange@redhat.com> ha
> scritto:
> 
> > With my distro maintainer hat I would rather QEMU ship neither the
> > ROM source, nor the ROM binaries.
> >
> 
> Annd since QEMU can finally cross compile its embedded firmware modules,
> too, it is now easier for distros not to use any prebuilt binary.
> 
> However some firmware sources are only available from QEMU's submodules. So
> either we distribute those submodules as separate tarballs, or distros
> would need to use the bundled tarball as well.

If the firmware doesn't exist as a standalone project, IMHO, it is
fine to bundle their sources with QEMU. If they move off into a
separate project some time in arbitrary future, they can be unbundled
at that point.

> Separately, I am not even sure what compiler is needed for the old
> Macintosh ROMs...

That we're not sure how to build some ROMS is exactly why
distros have their build everything from source policy !

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Il mar 29 nov 2022, 09:12 Daniel P. Berrangé <berrange@redhat.com> ha
scritto:

> > However some firmware sources are only available from QEMU's submodules.
> So
> > either we distribute those submodules as separate tarballs, or distros
> > would need to use the bundled tarball as well.
>
> If the firmware doesn't exist as a standalone project, IMHO, it is
> fine to bundle their sources with QEMU


> They're not small though, for example we have a fork of uboot.

> Separately, I am not even sure what compiler is needed for the old
> > Macintosh ROMs...
>
> That we're not sure how to build some ROMS is exactly why
> distros have their build everything from source policy !
>

What I mean is that it might be that an ancient proprietary compiler is
needed. I am not sure though.

Paolo


> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>