1. Patchew
  2. Xen
  3. View series

[PATCH] releases: use newer compression methods for tarballs

Jan Beulich posted 1 patch 3 months, 2 weeks ago
Failed in applying to current master (apply log)
There is a newer version of this series
[PATCH] releases: use newer compression methods for tarballs
Posted by Jan Beulich 3 months, 2 weeks ago 
Other projects have long switched to xz and/or lzip.

In the mktarball script, don't rely on the tar used supporting the -J
(xz) or --lzip (lzip) options.

Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
Question (to the release technicians) is whether this would be adding
undue overhead, primarily as a result of anything done outside of the
build system.

It's unclear to me why we have git create a tarball, extract that, just
to then make a tarball again (without any special options, like to
override owner or timestamps; in this context I notice that tarballs
created by Andrew have file ownership of andrew/andrew, while ones made
by Julien use root/root).

Without passing -9, I observe lzip to compress worse than xz; the win
of passing -9 to xz isn't overly big anyway (about 100k, compared to
about 250k with lzip).

lzip, unlike the other two tools, doesn't really show a progress
indicator with -v. Merely having final statistics may make the use of
the option here questionable.

--- a/docs/process/release-technician-checklist.txt
+++ b/docs/process/release-technician-checklist.txt
@@ -119,7 +119,7 @@ RELEASE TARBALL
        make src-tarball           # uses git-describe (best for RCs)
         # ^find some way to add git-cache-proxy to this (done in ~iwj/.gitconfig)
        mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v
-       mv dist/xen-$v.tar.gz /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
+       mv dist/xen-$v.tar.[glx]z /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
 
        # website-thing/xen.org is cvs -d mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org
 	cd /volatile/iwj/website-thing/xen.org
@@ -139,9 +139,12 @@ RELEASE TARBALL
 	cvs add -kb oss-xen/release/$v/
 
         cd oss-xen/release/$v
-        gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz
-	cvs add -kb xen-$v.tar.gz
-        cvs add -kb xen-$v.tar.gz.sig
+        for t in xen-$v.tar.[glx]z
+        do
+            gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t
+            cvs add -kb $t
+            cvs add -kb $t.sig
+        done
         cd ../../..
 
 	cvs ci -m $v
@@ -152,6 +155,10 @@ RELEASE TARBALL
 	# should show something like
 	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz
 	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig
 
 After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above).
 
--- a/docs/process/xen-release-management.pandoc
+++ b/docs/process/xen-release-management.pandoc
@@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
 https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
 
 For your convenience there is also a tarball at:
-https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
+https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
 
 And the signature is at:
-https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
+https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
 
 Please send bug reports and test reports to xen-devel@lists.xenproject.org.
 When sending bug reports, please CC relevant maintainers and me
--- a/tools/misc/mktarball
+++ b/tools/misc/mktarball
@@ -31,4 +31,14 @@ git_archive_into $xen_root $tdir/xen-$de
 
 GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
 
-echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
+if [ -n "$(command -v xz)" ]
+then
+  tar c -C $tdir xen-$desc | $(command -v xz) -v -9 >$xen_root/dist/xen-$desc.tar.xz
+fi
+
+if [ -n "$(command -v lzip)" ]
+then
+  tar c -C $tdir xen-$desc | $(command -v lzip) -v -9 >$xen_root/dist/xen-$desc.tar.lz
+fi
+
+echo "Source tarball in $xen_root/dist/xen-$desc".tar.[glx]z
Re: [PATCH] releases: use newer compression methods for tarballs
Posted by Andrew Cooper 2 months, 1 week ago 
On 15/07/2025 7:33 am, Jan Beulich wrote:
> Other projects have long switched to xz and/or lzip.
>
> In the mktarball script, don't rely on the tar used supporting the -J
> (xz) or --lzip (lzip) options.
>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> ---
> Question (to the release technicians) is whether this would be adding
> undue overhead, primarily as a result of anything done outside of the
> build system.

I'm happy with this.

>
> It's unclear to me why we have git create a tarball, extract that, just
> to then make a tarball again (without any special options, like to
> override owner or timestamps;

That's because in c5be91eb8140 you deleted the intermediate step of
merging qemus


> in this context I notice that tarballs
> created by Andrew have file ownership of andrew/andrew, while ones made
> by Julien use root/root).

Ownership where exactly?  the tarball itself (which will be down to
accounts on downloads.xenproject.org) or the tarball contents itself?

>
> Without passing -9, I observe lzip to compress worse than xz; the win
> of passing -9 to xz isn't overly big anyway (about 100k, compared to
> about 250k with lzip).

As these are created once and downloaded many times, we should always do
max compression.  Even if it takes minutes extra to create, that's still
a win overall.

>
> lzip, unlike the other two tools, doesn't really show a progress
> indicator with -v. Merely having final statistics may make the use of
> the option here questionable.

I can't say I find any of the stats relevant.

> --- a/docs/process/xen-release-management.pandoc
> +++ b/docs/process/xen-release-management.pandoc
> @@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
>  https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
>  
>  For your convenience there is also a tarball at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
>  
>  And the signature is at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
>  
>  Please send bug reports and test reports to xen-devel@lists.xenproject.org.
>  When sending bug reports, please CC relevant maintainers and me
> --- a/tools/misc/mktarball
> +++ b/tools/misc/mktarball
> @@ -31,4 +31,14 @@ git_archive_into $xen_root $tdir/xen-$de
>  
>  GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>  
> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
> +if [ -n "$(command -v xz)" ]
> +then
> +  tar c -C $tdir xen-$desc | $(command -v xz) -v -9 >$xen_root/dist/xen-$desc.tar.xz
> +fi
> +
> +if [ -n "$(command -v lzip)" ]
> +then
> +  tar c -C $tdir xen-$desc | $(command -v lzip) -v -9 >$xen_root/dist/xen-$desc.tar.lz
> +fi
> +
> +echo "Source tarball in $xen_root/dist/xen-$desc".tar.[glx]z

If we're deciding to use multiple compressions, they want to not be
optional here.  I'd far rather have a reminder to install the package,
than for it to simply be omitted.

We don't want to be re-tar-ing now that the qemu's are gone, so I think
the structure wants to end up as:

git archive --format=tar HEAD > tmp.tar
gzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.gz &
zx -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.xz &
lzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.lz &
wait

Might as well use the multiple cores better...

~Andrew
Re: [PATCH] releases: use newer compression methods for tarballs
Posted by Jan Beulich 2 months, 1 week ago 
On 20.08.2025 14:15, Andrew Cooper wrote:
> On 15/07/2025 7:33 am, Jan Beulich wrote:
>> It's unclear to me why we have git create a tarball, extract that, just
>> to then make a tarball again (without any special options, like to
>> override owner or timestamps;
> 
> That's because in c5be91eb8140 you deleted the intermediate step of
> merging qemus

Hmm, perhaps - I'm generally trying to keep the size of such changes down,
when working in an area I'm not overly familiar with.

>> in this context I notice that tarballs
>> created by Andrew have file ownership of andrew/andrew, while ones made
>> by Julien use root/root).
> 
> Ownership where exactly?  the tarball itself (which will be down to
> accounts on downloads.xenproject.org) or the tarball contents itself?

tarball contents.

>> lzip, unlike the other two tools, doesn't really show a progress
>> indicator with -v. Merely having final statistics may make the use of
>> the option here questionable.
> 
> I can't say I find any of the stats relevant.

Happy to drop the v options.

>> --- a/tools/misc/mktarball
>> +++ b/tools/misc/mktarball
>> @@ -31,4 +31,14 @@ git_archive_into $xen_root $tdir/xen-$de
>>  
>>  GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>>  
>> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
>> +if [ -n "$(command -v xz)" ]
>> +then
>> +  tar c -C $tdir xen-$desc | $(command -v xz) -v -9 >$xen_root/dist/xen-$desc.tar.xz
>> +fi
>> +
>> +if [ -n "$(command -v lzip)" ]
>> +then
>> +  tar c -C $tdir xen-$desc | $(command -v lzip) -v -9 >$xen_root/dist/xen-$desc.tar.lz
>> +fi
>> +
>> +echo "Source tarball in $xen_root/dist/xen-$desc".tar.[glx]z
> 
> If we're deciding to use multiple compressions, they want to not be
> optional here.  I'd far rather have a reminder to install the package,
> than for it to simply be omitted.
> 
> We don't want to be re-tar-ing now that the qemu's are gone, so I think
> the structure wants to end up as:
> 
> git archive --format=tar HEAD > tmp.tar
> gzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.gz &
> zx -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.xz &
> lzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.lz &
> wait
> 
> Might as well use the multiple cores better...

I, too, was thinking of that. Yet as per above, when fiddling with code I'm
not overly familiar with, I try to limit the amount of change done. As you
ask for a bigger change, I can certainly do so.

Jan
Re: [PATCH] releases: use newer compression methods for tarballs
Posted by Andrew Cooper 2 months, 1 week ago 
On 21/08/2025 8:26 am, Jan Beulich wrote:
> On 20.08.2025 14:15, Andrew Cooper wrote:
>> On 15/07/2025 7:33 am, Jan Beulich wrote:
>>> It's unclear to me why we have git create a tarball, extract that, just
>>> to then make a tarball again (without any special options, like to
>>> override owner or timestamps;
>> That's because in c5be91eb8140 you deleted the intermediate step of
>> merging qemus
> Hmm, perhaps - I'm generally trying to keep the size of such changes down,
> when working in an area I'm not overly familiar with.
>
>>> in this context I notice that tarballs
>>> created by Andrew have file ownership of andrew/andrew, while ones made
>>> by Julien use root/root).
>> Ownership where exactly?  the tarball itself (which will be down to
>> accounts on downloads.xenproject.org) or the tarball contents itself?
> tarball contents.
>
>>> lzip, unlike the other two tools, doesn't really show a progress
>>> indicator with -v. Merely having final statistics may make the use of
>>> the option here questionable.
>> I can't say I find any of the stats relevant.
> Happy to drop the v options.
>
>>> --- a/tools/misc/mktarball
>>> +++ b/tools/misc/mktarball
>>> @@ -31,4 +31,14 @@ git_archive_into $xen_root $tdir/xen-$de
>>>  
>>>  GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>>>  
>>> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
>>> +if [ -n "$(command -v xz)" ]
>>> +then
>>> +  tar c -C $tdir xen-$desc | $(command -v xz) -v -9 >$xen_root/dist/xen-$desc.tar.xz
>>> +fi
>>> +
>>> +if [ -n "$(command -v lzip)" ]
>>> +then
>>> +  tar c -C $tdir xen-$desc | $(command -v lzip) -v -9 >$xen_root/dist/xen-$desc.tar.lz
>>> +fi
>>> +
>>> +echo "Source tarball in $xen_root/dist/xen-$desc".tar.[glx]z
>> If we're deciding to use multiple compressions, they want to not be
>> optional here.  I'd far rather have a reminder to install the package,
>> than for it to simply be omitted.
>>
>> We don't want to be re-tar-ing now that the qemu's are gone, so I think
>> the structure wants to end up as:
>>
>> git archive --format=tar HEAD > tmp.tar
>> gzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.gz &
>> zx -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.xz &
>> lzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.lz &
>> wait
>>
>> Might as well use the multiple cores better...
> I, too, was thinking of that. Yet as per above, when fiddling with code I'm
> not overly familiar with, I try to limit the amount of change done. As you
> ask for a bigger change, I can certainly do so.

Would you like me to do the patch then?

~Andrew
Re: [PATCH] releases: use newer compression methods for tarballs
Posted by Jan Beulich 2 months, 1 week ago 
On 21.08.2025 11:31, Andrew Cooper wrote:
> On 21/08/2025 8:26 am, Jan Beulich wrote:
>> On 20.08.2025 14:15, Andrew Cooper wrote:
>>> On 15/07/2025 7:33 am, Jan Beulich wrote:
>>>> It's unclear to me why we have git create a tarball, extract that, just
>>>> to then make a tarball again (without any special options, like to
>>>> override owner or timestamps;
>>> That's because in c5be91eb8140 you deleted the intermediate step of
>>> merging qemus
>> Hmm, perhaps - I'm generally trying to keep the size of such changes down,
>> when working in an area I'm not overly familiar with.
>>
>>>> in this context I notice that tarballs
>>>> created by Andrew have file ownership of andrew/andrew, while ones made
>>>> by Julien use root/root).
>>> Ownership where exactly?  the tarball itself (which will be down to
>>> accounts on downloads.xenproject.org) or the tarball contents itself?
>> tarball contents.
>>
>>>> lzip, unlike the other two tools, doesn't really show a progress
>>>> indicator with -v. Merely having final statistics may make the use of
>>>> the option here questionable.
>>> I can't say I find any of the stats relevant.
>> Happy to drop the v options.
>>
>>>> --- a/tools/misc/mktarball
>>>> +++ b/tools/misc/mktarball
>>>> @@ -31,4 +31,14 @@ git_archive_into $xen_root $tdir/xen-$de
>>>>  
>>>>  GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>>>>  
>>>> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
>>>> +if [ -n "$(command -v xz)" ]
>>>> +then
>>>> +  tar c -C $tdir xen-$desc | $(command -v xz) -v -9 >$xen_root/dist/xen-$desc.tar.xz
>>>> +fi
>>>> +
>>>> +if [ -n "$(command -v lzip)" ]
>>>> +then
>>>> +  tar c -C $tdir xen-$desc | $(command -v lzip) -v -9 >$xen_root/dist/xen-$desc.tar.lz
>>>> +fi
>>>> +
>>>> +echo "Source tarball in $xen_root/dist/xen-$desc".tar.[glx]z
>>> If we're deciding to use multiple compressions, they want to not be
>>> optional here.  I'd far rather have a reminder to install the package,
>>> than for it to simply be omitted.
>>>
>>> We don't want to be re-tar-ing now that the qemu's are gone, so I think
>>> the structure wants to end up as:
>>>
>>> git archive --format=tar HEAD > tmp.tar
>>> gzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.gz &
>>> zx -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.xz &
>>> lzip -9 < tmp.tar > $xen_root/dist/xen-$desc.tar.lz &
>>> wait
>>>
>>> Might as well use the multiple cores better...
>> I, too, was thinking of that. Yet as per above, when fiddling with code I'm
>> not overly familiar with, I try to limit the amount of change done. As you
>> ask for a bigger change, I can certainly do so.
> 
> Would you like me to do the patch then?

No worries, I'll make a v2. It's just that in some cases I'm asked to limit changes
to what's strictly necessary, yet then (when I try to from the start) in others I'm
asked to make a bigger change.

Jan

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.