Other projects have long switched to xz and/or lzip.
Tidy things some as well: With the removal of qemu from the tarball,
intermediately extracting the tarball again has become wasteful. Drop
that. Invoke compressors using asynchronous lists, to reduce overall
latency. Drop the -v option from the (previously implicit) gzip
invocation.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
v2: Don't expand intermediate uncompressed tarball. Don't check for
commands' availablity. Don't request statistics. Use async lists.
--- a/docs/process/release-technician-checklist.txt
+++ b/docs/process/release-technician-checklist.txt
@@ -119,7 +119,7 @@ RELEASE TARBALL
make src-tarball # uses git-describe (best for RCs)
# ^find some way to add git-cache-proxy to this (done in ~iwj/.gitconfig)
mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v
- mv dist/xen-$v.tar.gz /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
+ mv dist/xen-$v.tar.[glx]z /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
# website-thing/xen.org is cvs -d mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org
cd /volatile/iwj/website-thing/xen.org
@@ -139,9 +139,12 @@ RELEASE TARBALL
cvs add -kb oss-xen/release/$v/
cd oss-xen/release/$v
- gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz
- cvs add -kb xen-$v.tar.gz
- cvs add -kb xen-$v.tar.gz.sig
+ for t in xen-$v.tar.[glx]z
+ do
+ gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t
+ cvs add -kb $t
+ cvs add -kb $t.sig
+ done
cd ../../..
cvs ci -m $v
@@ -152,6 +155,10 @@ RELEASE TARBALL
# should show something like
# U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz
# U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig
+ # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz
+ # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig
+ # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz
+ # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig
After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above).
--- a/docs/process/xen-release-management.pandoc
+++ b/docs/process/xen-release-management.pandoc
@@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
For your convenience there is also a tarball at:
-https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
+https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
And the signature is at:
-https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
+https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
Please send bug reports and test reports to xen-devel@lists.xenproject.org.
When sending bug reports, please CC relevant maintainers and me
--- a/tools/misc/mktarball
+++ b/tools/misc/mktarball
@@ -5,14 +5,6 @@
# Takes 2 arguments, the path to the dist directory and the version
set -ex
-function git_archive_into {
- mkdir -p "$2"
-
- git --git-dir="$1"/.git \
- archive --format=tar HEAD | \
- tar Cxf "$2" -
-}
-
if [[ -z "$1" || -z "$2" ]] ; then
echo "usage: $0 path-to-XEN_ROOT xen-version"
exit 1
@@ -21,14 +13,20 @@ fi
xen_root="$1"
desc="$2"
-tdir="$xen_root/dist/tmp.src-tarball"
+tdir="$xen_root/dist"
-rm -rf $tdir
+rm -f $tdir/xen-$desc.tar.[glx]z
mkdir -p $tdir
-git_archive_into $xen_root $tdir/xen-$desc
+git --git-dir="$xen_root/.git" archive --format=tar HEAD --prefix=xen-$desc/ \
+ >"$tdir/xen-$desc.tar"
+
+gzip -9k "$tdir/xen-$desc.tar" &
+xz -9k "$tdir/xen-$desc.tar" &
+lzip -9k "$tdir/xen-$desc.tar" &
+wait
-GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
+rm -f $tdir/xen-$desc.tar
-echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
+echo "Source tarball in" $tdir/xen-$desc.tar.[glx]zOn 25/08/2025 2:54 pm, Jan Beulich wrote:
> Other projects have long switched to xz and/or lzip.
>
> Tidy things some as well: With the removal of qemu from the tarball,
> intermediately extracting the tarball again has become wasteful. Drop
> that. Invoke compressors using asynchronous lists, to reduce overall
> latency. Drop the -v option from the (previously implicit) gzip
> invocation.
>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> ---
> v2: Don't expand intermediate uncompressed tarball. Don't check for
> commands' availablity. Don't request statistics. Use async lists.
>
> --- a/docs/process/release-technician-checklist.txt
> +++ b/docs/process/release-technician-checklist.txt
> @@ -119,7 +119,7 @@ RELEASE TARBALL
> make src-tarball # uses git-describe (best for RCs)
> # ^find some way to add git-cache-proxy to this (done in ~iwj/.gitconfig)
> mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v
> - mv dist/xen-$v.tar.gz /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
> + mv dist/xen-$v.tar.[glx]z /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
>
> # website-thing/xen.org is cvs -d mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org
> cd /volatile/iwj/website-thing/xen.org
> @@ -139,9 +139,12 @@ RELEASE TARBALL
> cvs add -kb oss-xen/release/$v/
>
> cd oss-xen/release/$v
> - gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz
> - cvs add -kb xen-$v.tar.gz
> - cvs add -kb xen-$v.tar.gz.sig
> + for t in xen-$v.tar.[glx]z
> + do
> + gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t
> + cvs add -kb $t
> + cvs add -kb $t.sig
> + done
> cd ../../..
>
> cvs ci -m $v
> @@ -152,6 +155,10 @@ RELEASE TARBALL
> # should show something like
> # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz
> # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig
> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz
> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig
> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz
> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig
>
> After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above).
>
> --- a/docs/process/xen-release-management.pandoc
> +++ b/docs/process/xen-release-management.pandoc
> @@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
> https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
>
> For your convenience there is also a tarball at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
>
> And the signature is at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
>
> Please send bug reports and test reports to xen-devel@lists.xenproject.org.
> When sending bug reports, please CC relevant maintainers and me
> --- a/tools/misc/mktarball
> +++ b/tools/misc/mktarball
> @@ -5,14 +5,6 @@
> # Takes 2 arguments, the path to the dist directory and the version
> set -ex
>
> -function git_archive_into {
> - mkdir -p "$2"
> -
> - git --git-dir="$1"/.git \
> - archive --format=tar HEAD | \
> - tar Cxf "$2" -
> -}
> -
> if [[ -z "$1" || -z "$2" ]] ; then
> echo "usage: $0 path-to-XEN_ROOT xen-version"
> exit 1
> @@ -21,14 +13,20 @@ fi
> xen_root="$1"
> desc="$2"
>
> -tdir="$xen_root/dist/tmp.src-tarball"
> +tdir="$xen_root/dist"
>
> -rm -rf $tdir
> +rm -f $tdir/xen-$desc.tar.[glx]z
This is asymmetric with the rm at the end. I'd remove
$tdir/xen-$desc.tar* here and remove the final rm.
Looking at the uncompressed tarball is part of my process, and it was
preserved previously.
With something along these lines, Reviewed-by: Andrew Cooper
<andrew.cooper3@citrix.com>
>
> mkdir -p $tdir
>
> -git_archive_into $xen_root $tdir/xen-$desc
> +git --git-dir="$xen_root/.git" archive --format=tar HEAD --prefix=xen-$desc/ \
> + >"$tdir/xen-$desc.tar"
> +
> +gzip -9k "$tdir/xen-$desc.tar" &
> +xz -9k "$tdir/xen-$desc.tar" &
> +lzip -9k "$tdir/xen-$desc.tar" &
> +wait
Interestingly, this wasn't fatal for not having lzip, but the error was
clear on the console given the reduced verbosity, and doing 3 at the
same time worked very nicely.
>
> -GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
> +rm -f $tdir/xen-$desc.tar
>
> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
> +echo "Source tarball in" $tdir/xen-$desc.tar.[glx]z
This was grammatically awkward to begin with, but is now pretty useless,
especially combined with the set -x so it gets printed twice.
Something like this:
echo "Source tarballs:"
ls -lah $tdir/xen-$desc.tar*
generates:
-rw-rw-r-- 1 andrew andrew 32M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar
-rw-rw-r-- 1 andrew andrew 6.8M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
-rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
-rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz
on my system and is rather more useful IMO.
~Andrew
On 02.09.2025 18:15, Andrew Cooper wrote:
> On 25/08/2025 2:54 pm, Jan Beulich wrote:
>> --- a/tools/misc/mktarball
>> +++ b/tools/misc/mktarball
>> @@ -5,14 +5,6 @@
>> # Takes 2 arguments, the path to the dist directory and the version
>> set -ex
>>
>> -function git_archive_into {
>> - mkdir -p "$2"
>> -
>> - git --git-dir="$1"/.git \
>> - archive --format=tar HEAD | \
>> - tar Cxf "$2" -
>> -}
>> -
>> if [[ -z "$1" || -z "$2" ]] ; then
>> echo "usage: $0 path-to-XEN_ROOT xen-version"
>> exit 1
>> @@ -21,14 +13,20 @@ fi
>> xen_root="$1"
>> desc="$2"
>>
>> -tdir="$xen_root/dist/tmp.src-tarball"
>> +tdir="$xen_root/dist"
>>
>> -rm -rf $tdir
>> +rm -f $tdir/xen-$desc.tar.[glx]z
>
> This is asymmetric with the rm at the end. I'd remove
> $tdir/xen-$desc.tar* here and remove the final rm.
Can do, but ...
> Looking at the uncompressed tarball is part of my process, and it was
> preserved previously.
... afaics none was ever generated previously. git_archive_into() piped git's
output into an "untar", and then the resulting tree was all that was ever
acted upon, with tar directly piping into gzip.
If we were retaining the uncompressed tarball, it wasn't quite clear to me
whether that might get in the way of the uploading process: We surely want to
upload only the compressed ones.
> With something along these lines, Reviewed-by: Andrew Cooper
> <andrew.cooper3@citrix.com>
Thanks, but the above first need clarifying (at the very least because I would
want to somehow mention the behavioral change in the description, yet then
there may be none if my observation was wrong).
>> mkdir -p $tdir
>>
>> -git_archive_into $xen_root $tdir/xen-$desc
>> +git --git-dir="$xen_root/.git" archive --format=tar HEAD --prefix=xen-$desc/ \
>> + >"$tdir/xen-$desc.tar"
>> +
>> +gzip -9k "$tdir/xen-$desc.tar" &
>> +xz -9k "$tdir/xen-$desc.tar" &
>> +lzip -9k "$tdir/xen-$desc.tar" &
>> +wait
>
> Interestingly, this wasn't fatal for not having lzip, but the error was
> clear on the console given the reduced verbosity, and doing 3 at the
> same time worked very nicely.
Well, obviously, because the exit status is effectively lost for async lists.
>> -GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>> +rm -f $tdir/xen-$desc.tar
>>
>> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
>> +echo "Source tarball in" $tdir/xen-$desc.tar.[glx]z
>
> This was grammatically awkward to begin with, but is now pretty useless,
> especially combined with the set -x so it gets printed twice.
>
> Something like this:
>
> echo "Source tarballs:"
> ls -lah $tdir/xen-$desc.tar*
>
> generates:
>
> -rw-rw-r-- 1 andrew andrew 32M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar
> -rw-rw-r-- 1 andrew andrew 6.8M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz
>
>
> on my system and is rather more useful IMO.
I was indeed wondering whether to do something like this, but didn't because
it again would have extended the scope of the patch. Now that you're asking
for it, I will. I won't pass 'a' to ls though, as the glob doesn't allow for
file names starting with '.'.
Jan
On 02/09/2025 5:15 pm, Andrew Cooper wrote:
> On 25/08/2025 2:54 pm, Jan Beulich wrote:
>> Other projects have long switched to xz and/or lzip.
>>
>> Tidy things some as well: With the removal of qemu from the tarball,
>> intermediately extracting the tarball again has become wasteful. Drop
>> that. Invoke compressors using asynchronous lists, to reduce overall
>> latency. Drop the -v option from the (previously implicit) gzip
>> invocation.
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>> ---
>> v2: Don't expand intermediate uncompressed tarball. Don't check for
>> commands' availablity. Don't request statistics. Use async lists.
>>
>> --- a/docs/process/release-technician-checklist.txt
>> +++ b/docs/process/release-technician-checklist.txt
>> @@ -119,7 +119,7 @@ RELEASE TARBALL
>> make src-tarball # uses git-describe (best for RCs)
>> # ^find some way to add git-cache-proxy to this (done in ~iwj/.gitconfig)
>> mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v
>> - mv dist/xen-$v.tar.gz /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
>> + mv dist/xen-$v.tar.[glx]z /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
>>
>> # website-thing/xen.org is cvs -d mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org
>> cd /volatile/iwj/website-thing/xen.org
>> @@ -139,9 +139,12 @@ RELEASE TARBALL
>> cvs add -kb oss-xen/release/$v/
>>
>> cd oss-xen/release/$v
>> - gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz
>> - cvs add -kb xen-$v.tar.gz
>> - cvs add -kb xen-$v.tar.gz.sig
>> + for t in xen-$v.tar.[glx]z
>> + do
>> + gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t
>> + cvs add -kb $t
>> + cvs add -kb $t.sig
>> + done
>> cd ../../..
>>
>> cvs ci -m $v
>> @@ -152,6 +155,10 @@ RELEASE TARBALL
>> # should show something like
>> # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz
>> # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig
>> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz
>> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig
>> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz
>> + # U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig
>>
>> After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above).
>>
>> --- a/docs/process/xen-release-management.pandoc
>> +++ b/docs/process/xen-release-management.pandoc
>> @@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
>> https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
>>
>> For your convenience there is also a tarball at:
>> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
>> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
>>
>> And the signature is at:
>> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
>> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
>>
>> Please send bug reports and test reports to xen-devel@lists.xenproject.org.
>> When sending bug reports, please CC relevant maintainers and me
>> --- a/tools/misc/mktarball
>> +++ b/tools/misc/mktarball
>> @@ -5,14 +5,6 @@
>> # Takes 2 arguments, the path to the dist directory and the version
>> set -ex
>>
>> -function git_archive_into {
>> - mkdir -p "$2"
>> -
>> - git --git-dir="$1"/.git \
>> - archive --format=tar HEAD | \
>> - tar Cxf "$2" -
>> -}
>> -
>> if [[ -z "$1" || -z "$2" ]] ; then
>> echo "usage: $0 path-to-XEN_ROOT xen-version"
>> exit 1
>> @@ -21,14 +13,20 @@ fi
>> xen_root="$1"
>> desc="$2"
>>
>> -tdir="$xen_root/dist/tmp.src-tarball"
>> +tdir="$xen_root/dist"
>>
>> -rm -rf $tdir
>> +rm -f $tdir/xen-$desc.tar.[glx]z
> This is asymmetric with the rm at the end. I'd remove
> $tdir/xen-$desc.tar* here and remove the final rm.
>
> Looking at the uncompressed tarball is part of my process, and it was
> preserved previously.
>
> With something along these lines, Reviewed-by: Andrew Cooper
> <andrew.cooper3@citrix.com>
>
>>
>> mkdir -p $tdir
>>
>> -git_archive_into $xen_root $tdir/xen-$desc
>> +git --git-dir="$xen_root/.git" archive --format=tar HEAD --prefix=xen-$desc/ \
>> + >"$tdir/xen-$desc.tar"
>> +
>> +gzip -9k "$tdir/xen-$desc.tar" &
>> +xz -9k "$tdir/xen-$desc.tar" &
>> +lzip -9k "$tdir/xen-$desc.tar" &
>> +wait
> Interestingly, this wasn't fatal for not having lzip, but the error was
> clear on the console given the reduced verbosity, and doing 3 at the
> same time worked very nicely.
>
>>
>> -GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>> +rm -f $tdir/xen-$desc.tar
>>
>> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
>> +echo "Source tarball in" $tdir/xen-$desc.tar.[glx]z
> This was grammatically awkward to begin with, but is now pretty useless,
> especially combined with the set -x so it gets printed twice.
>
> Something like this:
>
> echo "Source tarballs:"
> ls -lah $tdir/xen-$desc.tar*
>
> generates:
>
> -rw-rw-r-- 1 andrew andrew 32M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar
> -rw-rw-r-- 1 andrew andrew 6.8M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:13 /home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz
>
>
> on my system and is rather more useful IMO.
And just for reference, here is what zstd looks like:
-rw-rw-r-- 1 andrew andrew 32M Sep 2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar
-rw-rw-r-- 1 andrew andrew 6.8M Sep 2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
-rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
-rw-rw-r-- 1 andrew andrew 4.7M Sep 2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz
-rw-rw-r-- 1 andrew andrew 4.9M Sep 2 17:24
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.zst
using -19kq (zstd defaults to giving an interactive progress bar, and
needs quietening).
With a bit of `time` hacking, gzip is 2.4s, zx is 11.8s, zstd is 13s and
lzip is 17.3s on my system.
~Andrew
© 2016 - 2025 Red Hat, Inc.