The QEMU source archive is including the sources downloaded from crates.io
in both tarball form (in subprojects/packagecache) and expanded/patched
form (in the subprojects directory).  The former is the more authoritative
form, as it has a hash that can be verified in the wrap file and checked
against the download URL, so keep that one only.  This works also with
--disable-download; when building QEMU for the first time from the
tarball, Meson will print something like

    Using proc-macro2-1-rs source from cache.

for each subproject, and then go on to extract the tarball and apply the
overlay or the patches in subprojects/packagefiles.

Reported-by: Michael Tokarev <mjt@tls.msk.ru>
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2719
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 scripts/make-release | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/scripts/make-release b/scripts/make-release
index 8dc939124c4..2885e872109 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -10,6 +10,27 @@
 # This work is licensed under the terms of the GNU GPLv2 or later.
 # See the COPYING file in the top-level directory.
 
+function subproject_dir() {
+    if test ! -f "subprojects/$1.wrap"; then
+      error "scripts/archive-source.sh should only process wrap subprojects"
+    fi
+
+    # Print the directory key of the wrap file, defaulting to the
+    # subproject name.  The wrap file is in ini format and should
+    # have a single section only.  There should be only one section
+    # named "[wrap-*]", which helps keeping the script simple.
+    local dir
+    dir=$(sed -n \
+      -e '/^\[wrap-[a-z][a-z]*\]$/,/^\[/{' \
+      -e    '/^directory *= */!b' \
+      -e    's///p' \
+      -e    'q' \
+      -e '}' \
+      "subprojects/$1.wrap")
+
+    echo "${dir:-$1}"
+}
+
 if [ $# -ne 2 ]; then
     echo "Usage:"
     echo " $0 gitrepo version"
@@ -51,5 +72,13 @@ meson subprojects download $SUBPROJECTS
         CryptoPkg/Library/OpensslLib/openssl \
         MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
 popd
-tar --exclude=.git -cJf ${destination}.tar.xz ${destination}
+
+exclude=(--exclude=.git)
+# include the tarballs in subprojects/packagecache but not their expansion
+for sp in $SUBPROJECTS; do
+    if grep -xqF "[wrap-file]" subprojects/$sp.wrap; then
+      exclude+=(--exclude=subprojects/"$(subproject_dir $sp)")
+    fi
+done
+tar "${exclude[@]}" -cJf ${destination}.tar.xz ${destination}
 rm -rf ${destination}
-- 
2.47.1

19.12.2024 13:27, Paolo Bonzini wrote:
> The QEMU source archive is including the sources downloaded from crates.io
> in both tarball form (in subprojects/packagecache) and expanded/patched
> form (in the subprojects directory).  The former is the more authoritative
> form, as it has a hash that can be verified in the wrap file and checked
> against the download URL, so keep that one only.  This works also with
> --disable-download; when building QEMU for the first time from the
> tarball, Meson will print something like
> 
>      Using proc-macro2-1-rs source from cache.
> 
> for each subproject, and then go on to extract the tarball and apply the
> overlay or the patches in subprojects/packagefiles.
> 
> Reported-by: Michael Tokarev <mjt@tls.msk.ru>
> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2719
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>   scripts/make-release | 31 ++++++++++++++++++++++++++++++-
>   1 file changed, 30 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/make-release b/scripts/make-release
> index 8dc939124c4..2885e872109 100755
> --- a/scripts/make-release
> +++ b/scripts/make-release
> @@ -10,6 +10,27 @@
>   # This work is licensed under the terms of the GNU GPLv2 or later.
>   # See the COPYING file in the top-level directory.
>   
> +function subproject_dir() {
> +    if test ! -f "subprojects/$1.wrap"; then
> +      error "scripts/archive-source.sh should only process wrap subprojects"
> +    fi
> +
> +    # Print the directory key of the wrap file, defaulting to the
> +    # subproject name.  The wrap file is in ini format and should
> +    # have a single section only.  There should be only one section
> +    # named "[wrap-*]", which helps keeping the script simple.
> +    local dir
> +    dir=$(sed -n \
> +      -e '/^\[wrap-[a-z][a-z]*\]$/,/^\[/{' \
> +      -e    '/^directory *= */!b' \
> +      -e    's///p' \
> +      -e    'q' \
> +      -e '}' \
> +      "subprojects/$1.wrap")
> +
> +    echo "${dir:-$1}"
> +}
> +
>   if [ $# -ne 2 ]; then
>       echo "Usage:"
>       echo " $0 gitrepo version"
> @@ -51,5 +72,13 @@ meson subprojects download $SUBPROJECTS
>           CryptoPkg/Library/OpensslLib/openssl \
>           MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
>   popd
> -tar --exclude=.git -cJf ${destination}.tar.xz ${destination}
> +
> +exclude=(--exclude=.git)
> +# include the tarballs in subprojects/packagecache but not their expansion
> +for sp in $SUBPROJECTS; do
> +    if grep -xqF "[wrap-file]" subprojects/$sp.wrap; then
> +      exclude+=(--exclude=subprojects/"$(subproject_dir $sp)")
> +    fi
> +done
> +tar "${exclude[@]}" -cJf ${destination}.tar.xz ${destination}
>   rm -rf ${destination}

Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>

Nice simple and efficient sed stuff, haven't seen it in a while :)

Thanks,

/mjt

24.12.2024 22:41, Michael Tokarev wrote:
> 19.12.2024 13:27, Paolo Bonzini wrote:

>> -tar --exclude=.git -cJf ${destination}.tar.xz ${destination}
>> +
>> +exclude=(--exclude=.git)
>> +# include the tarballs in subprojects/packagecache but not their expansion
>> +for sp in $SUBPROJECTS; do
>> +    if grep -xqF "[wrap-file]" subprojects/$sp.wrap; then
>> +      exclude+=(--exclude=subprojects/"$(subproject_dir $sp)")
>> +    fi
>> +done
>> +tar "${exclude[@]}" -cJf ${destination}.tar.xz ${destination}
>>   rm -rf ${destination}

Actually there might be a better way here.  How about

  tar --exclude=.git $(sed -n 's|^/|subprojects/|' subprojects/.gitignore)

?

Thanks,

/mjt