[v2] curl: locking cleanups/fixes, coroutine conversion, remove aio_poll

[Qemu-devel] [PATCH v2 0/7] curl: locking cleanups/fixes, coroutine conversion, remove aio_poll

Posted by Paolo Bonzini 8 years, 9 months ago

Since the last patch in v1 didn't work, I bit the bullet and converted
the whole thing to coroutines (patches 4-6).  This in turns allows a more
elegant solution to wait for CURLStates to get free (patch 7).

I tested this by lowering CURL_NUM_STATES to 2.  With this change, the
buggy case triggers a couple times while booting a Fedora netinst image.

Paolo

Paolo Bonzini (7):
  curl: strengthen assertion in curl_clean_state
  curl: never invoke callbacks with s->mutex held
  curl: avoid recursive locking of BDRVCURLState mutex
  curl: split curl_find_state/curl_init_state
  curl: convert CURLAIOCB to byte values
  curl: convert readv to coroutines
  curl: do not do aio_poll when waiting for a free CURLState

 block/curl.c              | 216 +++++++++++++++++++++++++---------------------
 1 files changed, 120 insertions(+), 96 deletions(-)

-- 
2.12.2

Re: [Qemu-devel] [PATCH v2 0/7] curl: locking cleanups/fixes, coroutine conversion, remove aio_poll

Posted by no-reply@patchew.org 8 years, 9 months ago

Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH v2 0/7] curl: locking cleanups/fixes, coroutine conversion, remove aio_poll
Type: series
Message-id: 20170510143205.32013-1-pbonzini@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
a91e9d4 curl: do not do aio_poll when waiting for a free CURLState
6c92680 curl: convert readv to coroutines
84c046c curl: convert CURLAIOCB to byte values
ab431ec curl: split curl_find_state/curl_init_state
ba309a5 curl: avoid recursive locking of BDRVCURLState mutex
a6b6d90 curl: never invoke callbacks with s->mutex held
6f11502 curl: strengthen assertion in curl_clean_state

=== OUTPUT BEGIN ===
Checking PATCH 1/7: curl: strengthen assertion in curl_clean_state...
ERROR: spaces required around that '=' (ctx:VxV)
#24: FILE: block/curl.c:536:
+    for (j=0; j<CURL_NUM_ACB; j++) {
           ^

ERROR: spaces required around that '<' (ctx:VxV)
#24: FILE: block/curl.c:536:
+    for (j=0; j<CURL_NUM_ACB; j++) {
                ^

total: 2 errors, 0 warnings, 11 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 2/7: curl: never invoke callbacks with s->mutex held...
Checking PATCH 3/7: curl: avoid recursive locking of BDRVCURLState mutex...
Checking PATCH 4/7: curl: split curl_find_state/curl_init_state...
ERROR: spaces required around that '=' (ctx:VxV)
#39: FILE: block/curl.c:463:
+    for (i=0; i<CURL_NUM_STATES; i++) {
           ^

ERROR: spaces required around that '<' (ctx:VxV)
#39: FILE: block/curl.c:463:
+    for (i=0; i<CURL_NUM_STATES; i++) {
                ^

total: 2 errors, 0 warnings, 92 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 5/7: curl: convert CURLAIOCB to byte values...
Checking PATCH 6/7: curl: convert readv to coroutines...
Checking PATCH 7/7: curl: do not do aio_poll when waiting for a free CURLState...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [PATCH v2 0/7] curl: locking cleanups/fixes, coroutine conversion, remove aio_poll

Posted by Richard W.M. Jones 8 years, 9 months ago

On Wed, May 10, 2017 at 04:31:58PM +0200, Paolo Bonzini wrote:
> Since the last patch in v1 didn't work, I bit the bullet and converted
> the whole thing to coroutines (patches 4-6).  This in turns allows a more
> elegant solution to wait for CURLStates to get free (patch 7).
> 
> I tested this by lowering CURL_NUM_STATES to 2.  With this change, the
> buggy case triggers a couple times while booting a Fedora netinst image.

This series fixes the original bug, so:

  Tested-by: Richard W.M. Jones <rjones@redhat.com>

I think the Reported-by in patch 3 should credit Kun Wei for finding
the bug, and we should probably mention the BZ too:

  Reported-by: Kun Wei <kuwei@redhat.com>
  Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1447590

A nit pick perhaps but in patch 5 you say "This was broken before for
disks > 2TB, but now it would break at 4GB.".
I understand after reading it a few times that you mean it would be
broken at 4GB, if you hadn't changed size_t -> uint64_t (on 32 bit
platforms).  Perhaps better to clarify that sentence.

---

I also ran some performance and stability testing.  I used virt-ls for
this.  The following command will iterate over every file in a remote
guest image and print an md5sum:

  LIBGUESTFS_BACKEND=direct \
  LIBGUESTFS_HV=~/d/qemu/x86_64-softmmu/qemu-system-x86_64 \
  virt-ls -a http://somehost/rhel-guest-image-7.1-20150224.0.x86_64.qcow2 \
          -lR --checksum /

I timed this with and without your patches, but there was no
significant difference (but note that virt-ls is a fundamentally
sequential program).

It didn't crash or hang at any time during my testing.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: [Qemu-devel] [Qemu-block] [PATCH v2 0/7] curl: locking cleanups/fixes, coroutine conversion, remove aio_poll

Posted by Max Reitz 8 years, 9 months ago

On 2017-05-10 17:57, Richard W.M. Jones wrote:
> On Wed, May 10, 2017 at 04:31:58PM +0200, Paolo Bonzini wrote:
>> Since the last patch in v1 didn't work, I bit the bullet and converted
>> the whole thing to coroutines (patches 4-6).  This in turns allows a more
>> elegant solution to wait for CURLStates to get free (patch 7).
>>
>> I tested this by lowering CURL_NUM_STATES to 2.  With this change, the
>> buggy case triggers a couple times while booting a Fedora netinst image.
> 
> This series fixes the original bug, so:
> 
>   Tested-by: Richard W.M. Jones <rjones@redhat.com>
> 
> I think the Reported-by in patch 3 should credit Kun Wei for finding
> the bug, and we should probably mention the BZ too:
> 
>   Reported-by: Kun Wei <kuwei@redhat.com>
>   Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1447590

This one is older, though:
https://bugzilla.redhat.com/show_bug.cgi?id=1437393

:-)

Max

> A nit pick perhaps but in patch 5 you say "This was broken before for
> disks > 2TB, but now it would break at 4GB.".
> I understand after reading it a few times that you mean it would be
> broken at 4GB, if you hadn't changed size_t -> uint64_t (on 32 bit
> platforms).  Perhaps better to clarify that sentence.
> 
> ---
> 
> I also ran some performance and stability testing.  I used virt-ls for
> this.  The following command will iterate over every file in a remote
> guest image and print an md5sum:
> 
>   LIBGUESTFS_BACKEND=direct \
>   LIBGUESTFS_HV=~/d/qemu/x86_64-softmmu/qemu-system-x86_64 \
>   virt-ls -a http://somehost/rhel-guest-image-7.1-20150224.0.x86_64.qcow2 \
>           -lR --checksum /
> 
> I timed this with and without your patches, but there was no
> significant difference (but note that virt-ls is a fundamentally
> sequential program).
> 
> It didn't crash or hang at any time during my testing.
> 
> Rich.
>

Re: [Qemu-devel] [Qemu-block] [PATCH v2 0/7] curl: locking cleanups/fixes, coroutine conversion, remove aio_poll

Posted by Richard W.M. Jones 8 years, 9 months ago

On Mon, May 15, 2017 at 09:12:54PM +0200, Max Reitz wrote:
> On 2017-05-10 17:57, Richard W.M. Jones wrote:
> > On Wed, May 10, 2017 at 04:31:58PM +0200, Paolo Bonzini wrote:
> >> Since the last patch in v1 didn't work, I bit the bullet and converted
> >> the whole thing to coroutines (patches 4-6).  This in turns allows a more
> >> elegant solution to wait for CURLStates to get free (patch 7).
> >>
> >> I tested this by lowering CURL_NUM_STATES to 2.  With this change, the
> >> buggy case triggers a couple times while booting a Fedora netinst image.
> > 
> > This series fixes the original bug, so:
> > 
> >   Tested-by: Richard W.M. Jones <rjones@redhat.com>
> > 
> > I think the Reported-by in patch 3 should credit Kun Wei for finding
> > the bug, and we should probably mention the BZ too:
> > 
> >   Reported-by: Kun Wei <kuwei@redhat.com>
> >   Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1447590
> 
> This one is older, though:
> https://bugzilla.redhat.com/show_bug.cgi?id=1437393

Fair enough, it does seem to be the same issue.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html