From: Marc-André Lureau <marcandre.lureau@redhat.com>

Running meson on each subproject is quite slow.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/archive-source.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 035828c532..85cd0a63a0 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -77,9 +77,10 @@ function subproject_dir() {
 git archive --format tar "$(tree_ish)" > "$tar_file"
 test $? -ne 0 && error "failed to archive qemu"
 
+meson subprojects download $subprojects
+test $? -ne 0 && error "failed to download subprojects $subprojects"
+
 for sp in $subprojects; do
-    meson subprojects download $sp
-    test $? -ne 0 && error "failed to download subproject $sp"
     tar --append --file "$tar_file" --exclude=.git subprojects/"$(subproject_dir $sp)"
     test $? -ne 0 && error "failed to append subproject $sp to $tar_file"
 done
-- 
2.51.0

On Fri, Sep 19, 2025 at 05:32:56PM +0400, marcandre.lureau@redhat.com wrote:
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
> 
> Running meson on each subproject is quite slow.

Why is this faster ?  Does it make meson do parallel downloads
if you list then all at once ?

> 
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  scripts/archive-source.sh | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index 035828c532..85cd0a63a0 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -77,9 +77,10 @@ function subproject_dir() {
>  git archive --format tar "$(tree_ish)" > "$tar_file"
>  test $? -ne 0 && error "failed to archive qemu"
>  
> +meson subprojects download $subprojects
> +test $? -ne 0 && error "failed to download subprojects $subprojects"
> +
>  for sp in $subprojects; do
> -    meson subprojects download $sp
> -    test $? -ne 0 && error "failed to download subproject $sp"
>      tar --append --file "$tar_file" --exclude=.git subprojects/"$(subproject_dir $sp)"
>      test $? -ne 0 && error "failed to append subproject $sp to $tar_file"
>  done
> -- 
> 2.51.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Hi

On Fri, Sep 19, 2025 at 5:50 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Fri, Sep 19, 2025 at 05:32:56PM +0400, marcandre.lureau@redhat.com
> wrote:
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Running meson on each subproject is quite slow.
>
> Why is this faster ?  Does it make meson do parallel downloads
> if you list then all at once ?
>

It doesn't look like it does it in paralllel, yet it is x times faster...

time sh -c 'for s in keycodemapdb libvfio-user berkeley-softfloat-3
berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs bilge-0.2-rs
bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs libc-0.2-rs
proc-macro2-1-rs proc-macro-error-1-rs proc-macro-error-attr-1-rs
quote-1-rs syn-2-rs unicode-ident-1-rs; do
  meson subprojects download $s >/dev/null
done'
sh -c   5.27s user 0.33s system 99% cpu 5.613 total

time sh -c 'meson subprojects download keycodemapdb libvfio-user
berkeley-softfloat-3 berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs
bilge-0.2-rs bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs
libc-0.2-rs proc-macro2-1-rs proc-macro-error-1-rs
proc-macro-error-attr-1-rs quote-1-rs syn-2-rs unicode-ident-1-rs'
sh -c   0.32s user 0.01s system 99% cpu 0.339 total


>
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  scripts/archive-source.sh | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> > index 035828c532..85cd0a63a0 100755
> > --- a/scripts/archive-source.sh
> > +++ b/scripts/archive-source.sh
> > @@ -77,9 +77,10 @@ function subproject_dir() {
> >  git archive --format tar "$(tree_ish)" > "$tar_file"
> >  test $? -ne 0 && error "failed to archive qemu"
> >
> > +meson subprojects download $subprojects
> > +test $? -ne 0 && error "failed to download subprojects $subprojects"
> > +
> >  for sp in $subprojects; do
> > -    meson subprojects download $sp
> > -    test $? -ne 0 && error "failed to download subproject $sp"
> >      tar --append --file "$tar_file" --exclude=.git
> subprojects/"$(subproject_dir $sp)"
> >      test $? -ne 0 && error "failed to append subproject $sp to
> $tar_file"
> >  done
> > --
> > 2.51.0
> >
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

On Mon, Sep 22, 2025 at 10:59:11AM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Fri, Sep 19, 2025 at 5:50 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Fri, Sep 19, 2025 at 05:32:56PM +0400, marcandre.lureau@redhat.com
> > wrote:
> > > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> > >
> > > Running meson on each subproject is quite slow.
> >
> > Why is this faster ?  Does it make meson do parallel downloads
> > if you list then all at once ?
> >
> 
> It doesn't look like it does it in paralllel, yet it is x times faster...
> 
> time sh -c 'for s in keycodemapdb libvfio-user berkeley-softfloat-3
> berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs bilge-0.2-rs
> bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs libc-0.2-rs
> proc-macro2-1-rs proc-macro-error-1-rs proc-macro-error-attr-1-rs
> quote-1-rs syn-2-rs unicode-ident-1-rs; do
>   meson subprojects download $s >/dev/null
> done'
> sh -c   5.27s user 0.33s system 99% cpu 5.613 total
> 
> time sh -c 'meson subprojects download keycodemapdb libvfio-user
> berkeley-softfloat-3 berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs
> bilge-0.2-rs bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs
> libc-0.2-rs proc-macro2-1-rs proc-macro-error-1-rs
> proc-macro-error-attr-1-rs quote-1-rs syn-2-rs unicode-ident-1-rs'
> sh -c   0.32s user 0.01s system 99% cpu 0.339 total

Oh I bet that 0.32secs is the overhead of python importing all modules
that meson uses. Running python 18 times, gives the 5 second overhead.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 9/22/25 10:08, Daniel P. Berrangé wrote:
> On Mon, Sep 22, 2025 at 10:59:11AM +0400, Marc-André Lureau wrote:
>> Hi
>>
>> On Fri, Sep 19, 2025 at 5:50 PM Daniel P. Berrangé <berrange@redhat.com>
>> wrote:
>>
>>> On Fri, Sep 19, 2025 at 05:32:56PM +0400, marcandre.lureau@redhat.com
>>> wrote:
>>>> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>>>>
>>>> Running meson on each subproject is quite slow.
>>>
>>> Why is this faster ?  Does it make meson do parallel downloads
>>> if you list then all at once ?
>>>
>>
>> It doesn't look like it does it in paralllel, yet it is x times faster...
>>
>> time sh -c 'for s in keycodemapdb libvfio-user berkeley-softfloat-3
>> berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs bilge-0.2-rs
>> bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs libc-0.2-rs
>> proc-macro2-1-rs proc-macro-error-1-rs proc-macro-error-attr-1-rs
>> quote-1-rs syn-2-rs unicode-ident-1-rs; do
>>    meson subprojects download $s >/dev/null
>> done'
>> sh -c   5.27s user 0.33s system 99% cpu 5.613 total
>>
>> time sh -c 'meson subprojects download keycodemapdb libvfio-user
>> berkeley-softfloat-3 berkeley-testfloat-3 anyhow-1-rs arbitrary-int-1-rs
>> bilge-0.2-rs bilge-impl-0.2-rs either-1-rs foreign-0.3-rs itertools-0.11-rs
>> libc-0.2-rs proc-macro2-1-rs proc-macro-error-1-rs
>> proc-macro-error-attr-1-rs quote-1-rs syn-2-rs unicode-ident-1-rs'
>> sh -c   0.32s user 0.01s system 99% cpu 0.339 total
> 
> Oh I bet that 0.32secs is the overhead of python importing all modules
> that meson uses. Running python 18 times, gives the 5 second overhead.
Oh, it actually is parallel!

     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
     executor = ThreadPoolExecutor(options.num_processes)
     ...
     for wrap in wraps:
         dirname = Path(source_dir, subproject_dir, 
wrap.directory).as_posix()
         runner = Runner(logger, r, wrap, dirname, options)
         task = loop.run_in_executor(executor, runner.run)
         tasks.append(task)
         task_names.append(wrap.name)
     results = loop.run_until_complete(asyncio.gather(*tasks))

and then Runner.run -> Runner.run_method -> Runner.download