[PATCH v3] releases: use newer compression methods for tarballs

Jan Beulich posted 1 patch 2 days, 19 hours ago
Failed in applying to current master (apply log)
[PATCH v3] releases: use newer compression methods for tarballs
Posted by Jan Beulich 2 days, 19 hours ago
Other projects have long switched to xz and/or lzip.

Tidy things some as well: With the removal of qemu from the tarball,
intermediately extracting the tarball again has become wasteful. Drop
that. Invoke compressors using asynchronous lists, to reduce overall
latency. Drop the -v option from the (previously implicit) gzip
invocation.

Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
v3: List tarballs using ls at end of script. (Don't alter the rm-s, due
    to lack of feedback there.)
v2: Don't expand intermediate uncompressed tarball. Don't check for
    commands' availablity. Don't request statistics. Use async lists.

--- a/docs/process/release-technician-checklist.txt
+++ b/docs/process/release-technician-checklist.txt
@@ -119,7 +119,7 @@ RELEASE TARBALL
        make src-tarball           # uses git-describe (best for RCs)
         # ^find some way to add git-cache-proxy to this (done in ~iwj/.gitconfig)
        mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v
-       mv dist/xen-$v.tar.gz /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
+       mv dist/xen-$v.tar.[glx]z /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
 
        # website-thing/xen.org is cvs -d mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org
 	cd /volatile/iwj/website-thing/xen.org
@@ -139,9 +139,12 @@ RELEASE TARBALL
 	cvs add -kb oss-xen/release/$v/
 
         cd oss-xen/release/$v
-        gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz
-	cvs add -kb xen-$v.tar.gz
-        cvs add -kb xen-$v.tar.gz.sig
+        for t in xen-$v.tar.[glx]z
+        do
+            gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t
+            cvs add -kb $t
+            cvs add -kb $t.sig
+        done
         cd ../../..
 
 	cvs ci -m $v
@@ -152,6 +155,10 @@ RELEASE TARBALL
 	# should show something like
 	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz
 	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz
+	#   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig
 
 After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above).
 
--- a/docs/process/xen-release-management.pandoc
+++ b/docs/process/xen-release-management.pandoc
@@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
 https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
 
 For your convenience there is also a tarball at:
-https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
+https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
 
 And the signature is at:
-https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
+https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
 
 Please send bug reports and test reports to xen-devel@lists.xenproject.org.
 When sending bug reports, please CC relevant maintainers and me
--- a/tools/misc/mktarball
+++ b/tools/misc/mktarball
@@ -5,14 +5,6 @@
 # Takes 2 arguments, the path to the dist directory and the version
 set -ex
 
-function git_archive_into {
-    mkdir -p "$2"
-
-    git --git-dir="$1"/.git \
-	archive --format=tar HEAD | \
-	tar Cxf "$2" -
-}
-
 if [[ -z "$1" || -z "$2" ]] ; then
   echo "usage: $0 path-to-XEN_ROOT xen-version"
   exit 1
@@ -21,14 +13,21 @@ fi
 xen_root="$1"
 desc="$2"
 
-tdir="$xen_root/dist/tmp.src-tarball"
+tdir="$xen_root/dist"
 
-rm -rf $tdir
+rm -f $tdir/xen-$desc.tar.[glx]z
 
 mkdir -p $tdir
 
-git_archive_into $xen_root $tdir/xen-$desc
+git --git-dir="$xen_root/.git" archive --format=tar HEAD --prefix=xen-$desc/ \
+    >"$tdir/xen-$desc.tar"
+
+gzip -9k "$tdir/xen-$desc.tar" &
+xz -9k "$tdir/xen-$desc.tar" &
+lzip -9k "$tdir/xen-$desc.tar" &
+wait
 
-GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
+rm -f $tdir/xen-$desc.tar
 
-echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
+echo "Source tarball(s):"
+ls -lh $tdir/xen-$desc.tar*
Re: [PATCH v3] releases: use newer compression methods for tarballs
Posted by Julien Grall 1 day, 6 hours ago
Hi Jan,

On 11/09/2025 09:14, Jan Beulich wrote:
> Other projects have long switched to xz and/or lzip.
> 
> Tidy things some as well: With the removal of qemu from the tarball,
> intermediately extracting the tarball again has become wasteful. Drop
> that. Invoke compressors using asynchronous lists, to reduce overall
> latency. Drop the -v option from the (previously implicit) gzip
> invocation.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

I have tested manually the steps and the correct tarballs have been 
produced. I will update my scripts to copy & sign all the tarballs once 
this is merged.

Acked-by: Julien Grall <jgrall@amazon.com>
Tested-by: Julien Grall <jgrall@amazon.com>

Is this intended for Xen 4.21?

Cheers,

-- 
Julien Grall