mktarball: Drop double-processing of the archive

[PATCH for-4.20] mktarball: Drop double-processing of the archive

Posted by Andrew Cooper 5 days, 11 hours ago

This is a partial backport of commit 63ebd0e9649e ("releases: use newer
compression methods for tarballs"), but keeping gz as the only compression
method.

In addition to efficiency, this causes the tarball to use root/root ownership,
rather than leak whomever produced the tarball.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Jan Beulich <JBeulich@suse.com>

I've made 4.20.2 using this, and the results look to be correct, and match
4.20.1 other than the ownership adjustment.

For 4.19 and earlier, I'm considering just passing --owner and --group to the
final tar invocation.  Thoughts?
---
 tools/misc/mktarball | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)

diff --git a/tools/misc/mktarball b/tools/misc/mktarball
index 7f9b61da4368..2f574f9bf0ee 100755
--- a/tools/misc/mktarball
+++ b/tools/misc/mktarball
@@ -5,14 +5,6 @@
 # Takes 2 arguments, the path to the dist directory and the version
 set -ex
 
-function git_archive_into {
-    mkdir -p "$2"
-
-    git --git-dir="$1"/.git \
-	archive --format=tar HEAD | \
-	tar Cxf "$2" -
-}
-
 if [[ -z "$1" || -z "$2" ]] ; then
   echo "usage: $0 path-to-XEN_ROOT xen-version"
   exit 1
@@ -21,14 +13,15 @@ fi
 xen_root="$1"
 desc="$2"
 
-tdir="$xen_root/dist/tmp.src-tarball"
+tdir="$xen_root/dist"
 
-rm -rf $tdir
+rm -f $tdir/xen-$desc.tar*
 
 mkdir -p $tdir
 
-git_archive_into $xen_root $tdir/xen-$desc
+git --git-dir="$xen_root/.git" archive --format=tar HEAD --prefix=xen-$desc/ \
+    >"$tdir/xen-$desc.tar"
 
-GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
+gzip -9k "$tdir/xen-$desc.tar"
 
 echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"

base-commit: 03299bb3ec817f47a608dc6080afc32453627fb4
-- 
2.39.5

Re: [PATCH for-4.20] mktarball: Drop double-processing of the archive

Posted by Jan Beulich 5 days, 11 hours ago

On 13.11.2025 14:01, Andrew Cooper wrote:
> This is a partial backport of commit 63ebd0e9649e ("releases: use newer
> compression methods for tarballs"), but keeping gz as the only compression
> method.
> 
> In addition to efficiency, this causes the tarball to use root/root ownership,
> rather than leak whomever produced the tarball.

I don't understand this part. Isn't the ownership whatever "git archive" reports?
I have to admit though ...

> --- a/tools/misc/mktarball
> +++ b/tools/misc/mktarball
> @@ -5,14 +5,6 @@
>  # Takes 2 arguments, the path to the dist directory and the version
>  set -ex
>  
> -function git_archive_into {
> -    mkdir -p "$2"
> -
> -    git --git-dir="$1"/.git \
> -	archive --format=tar HEAD | \
> -	tar Cxf "$2" -

... that I'm unaware of what the C here does. It can't be the same as -C, and the
--help output of the GNU tar that I checked doesn't mention anything else at all.

Jan

Re: [PATCH for-4.20] mktarball: Drop double-processing of the archive

Posted by Andrew Cooper 5 days, 10 hours ago

On 13/11/2025 1:12 pm, Jan Beulich wrote:
> On 13.11.2025 14:01, Andrew Cooper wrote:
>> This is a partial backport of commit 63ebd0e9649e ("releases: use newer
>> compression methods for tarballs"), but keeping gz as the only compression
>> method.
>>
>> In addition to efficiency, this causes the tarball to use root/root ownership,
>> rather than leak whomever produced the tarball.
> I don't understand this part. Isn't the ownership whatever "git archive" reports?

This is fixing the issue you noticed about internal ownership:

xen.org.cvs/oss-xen/release$ tar tf 4.20.1/xen-4.20.1.tar.gz --verbose | head
drwxrwxr-x andrew/andrew     0 2025-07-10 12:28 xen-4.20.1/
drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/
drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/workflows/
-rw-rw-r-- andrew/andrew  1362 2025-07-09 14:57 xen-4.20.1/.github/workflows/coverity.yml
-rw-rw-r-- andrew/andrew    96 2025-07-09 14:57 xen-4.20.1/.gitarchive-info
-rw-rw-r-- andrew/andrew  9668 2025-07-09 14:57 xen-4.20.1/Makefile
drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/
-rw-rw-r-- andrew/andrew 24220 2025-07-09 14:57 xen-4.20.1/stubdom/Makefile
drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/grub/
-rw-rw-r-- andrew/andrew  2252 2025-07-09 14:57 xen-4.20.1/stubdom/grub/Makefile

xen.org.cvs/oss-xen/release$ tar tf 4.20.2/xen-4.20.2.tar.gz --verbose | head
drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/
-rw-rw-r-- root/root      4781 2025-11-13 09:51 xen-4.20.2/.cirrus.yml
-rw-rw-r-- root/root        97 2025-11-13 09:51 xen-4.20.2/.gitarchive-info
-rw-rw-r-- root/root        30 2025-11-13 09:51 xen-4.20.2/.gitattributes
drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/
drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/workflows/
-rw-rw-r-- root/root      1362 2025-11-13 09:51 xen-4.20.2/.github/workflows/coverity.yml
-rw-rw-r-- root/root      7035 2025-11-13 09:51 xen-4.20.2/.gitignore
-rw-rw-r-- root/root       798 2025-11-13 09:51 xen-4.20.2/.gitlab-ci.yml
-rw-rw-r-- root/root     15298 2025-11-13 09:51 xen-4.20.2/CHANGELOG.md




> I have to admit though ...
>
>> --- a/tools/misc/mktarball
>> +++ b/tools/misc/mktarball
>> @@ -5,14 +5,6 @@
>>  # Takes 2 arguments, the path to the dist directory and the version
>>  set -ex
>>  
>> -function git_archive_into {
>> -    mkdir -p "$2"
>> -
>> -    git --git-dir="$1"/.git \
>> -	archive --format=tar HEAD | \
>> -	tar Cxf "$2" -
> ... that I'm unaware of what the C here does. It can't be the same as -C, and the
> --help output of the GNU tar that I checked doesn't mention anything else at all.

It is -C.  tar has dreadful cmdline syntax.

An equivalent would be tar -C "$2" xf -

~Andrew

Re: [PATCH for-4.20] mktarball: Drop double-processing of the archive

Posted by Jan Beulich 5 days, 10 hours ago

On 13.11.2025 14:29, Andrew Cooper wrote:
> On 13/11/2025 1:12 pm, Jan Beulich wrote:
>> On 13.11.2025 14:01, Andrew Cooper wrote:
>>> This is a partial backport of commit 63ebd0e9649e ("releases: use newer
>>> compression methods for tarballs"), but keeping gz as the only compression
>>> method.
>>>
>>> In addition to efficiency, this causes the tarball to use root/root ownership,
>>> rather than leak whomever produced the tarball.
>> I don't understand this part. Isn't the ownership whatever "git archive" reports?
> 
> This is fixing the issue you noticed about internal ownership:
> 
> xen.org.cvs/oss-xen/release$ tar tf 4.20.1/xen-4.20.1.tar.gz --verbose | head
> drwxrwxr-x andrew/andrew     0 2025-07-10 12:28 xen-4.20.1/
> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/
> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/workflows/
> -rw-rw-r-- andrew/andrew  1362 2025-07-09 14:57 xen-4.20.1/.github/workflows/coverity.yml
> -rw-rw-r-- andrew/andrew    96 2025-07-09 14:57 xen-4.20.1/.gitarchive-info
> -rw-rw-r-- andrew/andrew  9668 2025-07-09 14:57 xen-4.20.1/Makefile
> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/
> -rw-rw-r-- andrew/andrew 24220 2025-07-09 14:57 xen-4.20.1/stubdom/Makefile
> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/grub/
> -rw-rw-r-- andrew/andrew  2252 2025-07-09 14:57 xen-4.20.1/stubdom/grub/Makefile
> 
> xen.org.cvs/oss-xen/release$ tar tf 4.20.2/xen-4.20.2.tar.gz --verbose | head
> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/
> -rw-rw-r-- root/root      4781 2025-11-13 09:51 xen-4.20.2/.cirrus.yml
> -rw-rw-r-- root/root        97 2025-11-13 09:51 xen-4.20.2/.gitarchive-info
> -rw-rw-r-- root/root        30 2025-11-13 09:51 xen-4.20.2/.gitattributes
> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/
> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/workflows/
> -rw-rw-r-- root/root      1362 2025-11-13 09:51 xen-4.20.2/.github/workflows/coverity.yml
> -rw-rw-r-- root/root      7035 2025-11-13 09:51 xen-4.20.2/.gitignore
> -rw-rw-r-- root/root       798 2025-11-13 09:51 xen-4.20.2/.gitlab-ci.yml
> -rw-rw-r-- root/root     15298 2025-11-13 09:51 xen-4.20.2/CHANGELOG.md

I guess my reply was ambiguous. I did understand that's the effect, but I
wasn't able to tell why such a difference would result. It's all "git
archive", before and after the change.

>> I have to admit though ...
>>
>>> --- a/tools/misc/mktarball
>>> +++ b/tools/misc/mktarball
>>> @@ -5,14 +5,6 @@
>>>  # Takes 2 arguments, the path to the dist directory and the version
>>>  set -ex
>>>  
>>> -function git_archive_into {
>>> -    mkdir -p "$2"
>>> -
>>> -    git --git-dir="$1"/.git \
>>> -	archive --format=tar HEAD | \
>>> -	tar Cxf "$2" -
>> ... that I'm unaware of what the C here does. It can't be the same as -C, and the
>> --help output of the GNU tar that I checked doesn't mention anything else at all.
> 
> It is -C.  tar has dreadful cmdline syntax.
> 
> An equivalent would be tar -C "$2" xf -

Oh, wow. I would never have thought of writing that as Cxf.

Jan

Re: [PATCH for-4.20] mktarball: Drop double-processing of the archive

Posted by Andrew Cooper 5 days, 10 hours ago

On 13/11/2025 1:34 pm, Jan Beulich wrote:
> On 13.11.2025 14:29, Andrew Cooper wrote:
>> On 13/11/2025 1:12 pm, Jan Beulich wrote:
>>> On 13.11.2025 14:01, Andrew Cooper wrote:
>>>> This is a partial backport of commit 63ebd0e9649e ("releases: use newer
>>>> compression methods for tarballs"), but keeping gz as the only compression
>>>> method.
>>>>
>>>> In addition to efficiency, this causes the tarball to use root/root ownership,
>>>> rather than leak whomever produced the tarball.
>>> I don't understand this part. Isn't the ownership whatever "git archive" reports?
>> This is fixing the issue you noticed about internal ownership:
>>
>> xen.org.cvs/oss-xen/release$ tar tf 4.20.1/xen-4.20.1.tar.gz --verbose | head
>> drwxrwxr-x andrew/andrew     0 2025-07-10 12:28 xen-4.20.1/
>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/
>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/workflows/
>> -rw-rw-r-- andrew/andrew  1362 2025-07-09 14:57 xen-4.20.1/.github/workflows/coverity.yml
>> -rw-rw-r-- andrew/andrew    96 2025-07-09 14:57 xen-4.20.1/.gitarchive-info
>> -rw-rw-r-- andrew/andrew  9668 2025-07-09 14:57 xen-4.20.1/Makefile
>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/
>> -rw-rw-r-- andrew/andrew 24220 2025-07-09 14:57 xen-4.20.1/stubdom/Makefile
>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/grub/
>> -rw-rw-r-- andrew/andrew  2252 2025-07-09 14:57 xen-4.20.1/stubdom/grub/Makefile
>>
>> xen.org.cvs/oss-xen/release$ tar tf 4.20.2/xen-4.20.2.tar.gz --verbose | head
>> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/
>> -rw-rw-r-- root/root      4781 2025-11-13 09:51 xen-4.20.2/.cirrus.yml
>> -rw-rw-r-- root/root        97 2025-11-13 09:51 xen-4.20.2/.gitarchive-info
>> -rw-rw-r-- root/root        30 2025-11-13 09:51 xen-4.20.2/.gitattributes
>> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/
>> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/workflows/
>> -rw-rw-r-- root/root      1362 2025-11-13 09:51 xen-4.20.2/.github/workflows/coverity.yml
>> -rw-rw-r-- root/root      7035 2025-11-13 09:51 xen-4.20.2/.gitignore
>> -rw-rw-r-- root/root       798 2025-11-13 09:51 xen-4.20.2/.gitlab-ci.yml
>> -rw-rw-r-- root/root     15298 2025-11-13 09:51 xen-4.20.2/CHANGELOG.md
> I guess my reply was ambiguous. I did understand that's the effect, but I
> wasn't able to tell why such a difference would result. It's all "git
> archive", before and after the change.

Ah.  The first git archive does the right thing, but the result gets
expanded into the regular filesystem and takes local ownership.

Then, nothing is merged, and the result is re-tar'd using:

GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc

which retains local ownership into the resulting archive.

For 4.19 and earlier, I propose to add '--owner 0 --group 0' to this tar
invocation.

~Andrew

Re: [PATCH for-4.20] mktarball: Drop double-processing of the archive

Posted by Jan Beulich 5 days, 10 hours ago

On 13.11.2025 14:42, Andrew Cooper wrote:
> On 13/11/2025 1:34 pm, Jan Beulich wrote:
>> On 13.11.2025 14:29, Andrew Cooper wrote:
>>> On 13/11/2025 1:12 pm, Jan Beulich wrote:
>>>> On 13.11.2025 14:01, Andrew Cooper wrote:
>>>>> This is a partial backport of commit 63ebd0e9649e ("releases: use newer
>>>>> compression methods for tarballs"), but keeping gz as the only compression
>>>>> method.
>>>>>
>>>>> In addition to efficiency, this causes the tarball to use root/root ownership,
>>>>> rather than leak whomever produced the tarball.
>>>> I don't understand this part. Isn't the ownership whatever "git archive" reports?
>>> This is fixing the issue you noticed about internal ownership:
>>>
>>> xen.org.cvs/oss-xen/release$ tar tf 4.20.1/xen-4.20.1.tar.gz --verbose | head
>>> drwxrwxr-x andrew/andrew     0 2025-07-10 12:28 xen-4.20.1/
>>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/
>>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/.github/workflows/
>>> -rw-rw-r-- andrew/andrew  1362 2025-07-09 14:57 xen-4.20.1/.github/workflows/coverity.yml
>>> -rw-rw-r-- andrew/andrew    96 2025-07-09 14:57 xen-4.20.1/.gitarchive-info
>>> -rw-rw-r-- andrew/andrew  9668 2025-07-09 14:57 xen-4.20.1/Makefile
>>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/
>>> -rw-rw-r-- andrew/andrew 24220 2025-07-09 14:57 xen-4.20.1/stubdom/Makefile
>>> drwxrwxr-x andrew/andrew     0 2025-07-09 14:57 xen-4.20.1/stubdom/grub/
>>> -rw-rw-r-- andrew/andrew  2252 2025-07-09 14:57 xen-4.20.1/stubdom/grub/Makefile
>>>
>>> xen.org.cvs/oss-xen/release$ tar tf 4.20.2/xen-4.20.2.tar.gz --verbose | head
>>> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/
>>> -rw-rw-r-- root/root      4781 2025-11-13 09:51 xen-4.20.2/.cirrus.yml
>>> -rw-rw-r-- root/root        97 2025-11-13 09:51 xen-4.20.2/.gitarchive-info
>>> -rw-rw-r-- root/root        30 2025-11-13 09:51 xen-4.20.2/.gitattributes
>>> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/
>>> drwxrwxr-x root/root         0 2025-11-13 09:51 xen-4.20.2/.github/workflows/
>>> -rw-rw-r-- root/root      1362 2025-11-13 09:51 xen-4.20.2/.github/workflows/coverity.yml
>>> -rw-rw-r-- root/root      7035 2025-11-13 09:51 xen-4.20.2/.gitignore
>>> -rw-rw-r-- root/root       798 2025-11-13 09:51 xen-4.20.2/.gitlab-ci.yml
>>> -rw-rw-r-- root/root     15298 2025-11-13 09:51 xen-4.20.2/CHANGELOG.md
>> I guess my reply was ambiguous. I did understand that's the effect, but I
>> wasn't able to tell why such a difference would result. It's all "git
>> archive", before and after the change.
> 
> Ah.  The first git archive does the right thing, but the result gets
> expanded into the regular filesystem and takes local ownership.

Oh, I see - because of --no-same-owner being the default for non-root.

Acked-by: Jan Beulich <jbeulich@suse.com>

Jan

> Then, nothing is merged, and the result is re-tar'd using:
> 
> GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
> 
> which retains local ownership into the resulting archive.
> 
> For 4.19 and earlier, I propose to add '--owner 0 --group 0' to this tar
> invocation.
> 
> ~Andrew