[Qemu-devel] [PULL v2 00/22] Docker and block patches

Fam Zheng posted 22 patches 8 years, 5 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20170602081120.3497-1-famz@redhat.com
Test checkpatch failed
Test docker passed
Test s390x passed
There is a newer version of this series
block.c                                 |   9 +-
block/accounting.c                      |  65 ++++++-----
block/block-backend.c                   |   5 +-
block/dirty-bitmap.c                    | 114 +++++++++++++++++--
block/io.c                              |  51 +++++----
block/mirror.c                          |  14 ++-
block/nfs.c                             |   4 +-
block/qapi.c                            |   2 +-
block/sheepdog.c                        |   3 +-
block/throttle-groups.c                 |  91 +++++++++++----
blockdev.c                              |  46 ++------
include/block/accounting.h              |   8 +-
include/block/block.h                   |   5 +-
include/block/block_int.h               |  65 +++++++----
include/block/dirty-bitmap.h            |  25 +++--
include/qemu/stats64.h                  | 193 ++++++++++++++++++++++++++++++++
include/sysemu/block-backend.h          |  10 +-
migration/block.c                       |  17 ++-
tests/docker/Makefile.include           |   2 +-
tests/docker/dockerfiles/centos6.docker |   2 +-
tests/docker/dockerfiles/fedora.docker  |   4 +-
util/Makefile.objs                      |   1 +
util/stats64.c                          | 137 +++++++++++++++++++++++
23 files changed, 688 insertions(+), 185 deletions(-)
create mode 100644 include/qemu/stats64.h
create mode 100644 util/stats64.c
[Qemu-devel] [PULL v2 00/22] Docker and block patches
Posted by Fam Zheng 8 years, 5 months ago
The following changes since commit 43771d5d92312504305c19abe29ec5bfabd55f01:

  Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-05-31' into staging (2017-06-01 16:39:16 +0100)

are available in the git repository at:

  git://github.com/famz/qemu.git tags/docker-and-block-pull-request

for you to fetch changes up to 4ab2cbc128f1938355a12970c9b6e419a63fcab6:

  block: make accounting thread-safe (2017-06-02 15:59:32 +0800)

----------------------------------------------------------------

v2: Fix building on OSX and BSD.

----------------------------------------------------------------

Fam Zheng (4):
  docker: Run tests with current user
  docker: Add bzip2 and hostname to fedora image
  docker: Add libaio to fedora image
  docker: Add flex and bison to centos6 image

Paolo Bonzini (18):
  block: access copy_on_read with atomic ops
  block: access quiesce_counter with atomic ops
  block: access io_limits_disabled with atomic ops
  block: access serialising_in_flight with atomic ops
  block: access wakeup with atomic ops
  block: access io_plugged with atomic ops
  throttle-groups: only start one coroutine from drained_begin
  throttle-groups: do not use qemu_co_enter_next
  throttle-groups: protect throttled requests with a CoMutex
  util: add stats64 module
  block: use Stat64 for wr_highest_offset
  block: access write_gen with atomics
  block: protect tracked_requests and flush_queue with reqs_lock
  block: introduce dirty_bitmap_mutex
  migration/block: reset dirty bitmap before reading
  block: protect modification of dirty bitmaps with a mutex
  block: introduce block_account_one_io
  block: make accounting thread-safe

 block.c                                 |   9 +-
 block/accounting.c                      |  65 ++++++-----
 block/block-backend.c                   |   5 +-
 block/dirty-bitmap.c                    | 114 +++++++++++++++++--
 block/io.c                              |  51 +++++----
 block/mirror.c                          |  14 ++-
 block/nfs.c                             |   4 +-
 block/qapi.c                            |   2 +-
 block/sheepdog.c                        |   3 +-
 block/throttle-groups.c                 |  91 +++++++++++----
 blockdev.c                              |  46 ++------
 include/block/accounting.h              |   8 +-
 include/block/block.h                   |   5 +-
 include/block/block_int.h               |  65 +++++++----
 include/block/dirty-bitmap.h            |  25 +++--
 include/qemu/stats64.h                  | 193 ++++++++++++++++++++++++++++++++
 include/sysemu/block-backend.h          |  10 +-
 migration/block.c                       |  17 ++-
 tests/docker/Makefile.include           |   2 +-
 tests/docker/dockerfiles/centos6.docker |   2 +-
 tests/docker/dockerfiles/fedora.docker  |   4 +-
 util/Makefile.objs                      |   1 +
 util/stats64.c                          | 137 +++++++++++++++++++++++
 23 files changed, 688 insertions(+), 185 deletions(-)
 create mode 100644 include/qemu/stats64.h
 create mode 100644 util/stats64.c

-- 
2.9.4


Re: [Qemu-devel] [PULL v2 00/22] Docker and block patches
Posted by no-reply@patchew.org 8 years, 5 months ago
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20170602081120.3497-1-famz@redhat.com
Subject: [Qemu-devel] [PULL v2 00/22] Docker and block patches

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
59415e4 block: make accounting thread-safe
38e95e5 block: introduce block_account_one_io
5040785 block: protect modification of dirty bitmaps with a mutex
98373d7 migration/block: reset dirty bitmap before reading
2a0e66f block: introduce dirty_bitmap_mutex
c7b2f63 block: protect tracked_requests and flush_queue with reqs_lock
cc0ef6c block: access write_gen with atomics
4dec7a7 block: use Stat64 for wr_highest_offset
a78b979 util: add stats64 module
2fe25d6 throttle-groups: protect throttled requests with a CoMutex
c3e4307 throttle-groups: do not use qemu_co_enter_next
5365455 throttle-groups: only start one coroutine from drained_begin
ed49954 block: access io_plugged with atomic ops
a7e767b block: access wakeup with atomic ops
adc5828 block: access serialising_in_flight with atomic ops
7601e2a block: access io_limits_disabled with atomic ops
0b323e7 block: access quiesce_counter with atomic ops
0a68858 block: access copy_on_read with atomic ops
e8f7751 docker: Add flex and bison to centos6 image
40c63fe docker: Add libaio to fedora image
a118858 docker: Add bzip2 and hostname to fedora image
1ab59e8 docker: Run tests with current user

=== OUTPUT BEGIN ===
Checking PATCH 1/22: docker: Run tests with current user...
Checking PATCH 2/22: docker: Add bzip2 and hostname to fedora image...
Checking PATCH 3/22: docker: Add libaio to fedora image...
Checking PATCH 4/22: docker: Add flex and bison to centos6 image...
Checking PATCH 5/22: block: access copy_on_read with atomic ops...
Checking PATCH 6/22: block: access quiesce_counter with atomic ops...
Checking PATCH 7/22: block: access io_limits_disabled with atomic ops...
Checking PATCH 8/22: block: access serialising_in_flight with atomic ops...
Checking PATCH 9/22: block: access wakeup with atomic ops...
Checking PATCH 10/22: block: access io_plugged with atomic ops...
Checking PATCH 11/22: throttle-groups: only start one coroutine from drained_begin...
Checking PATCH 12/22: throttle-groups: do not use qemu_co_enter_next...
Checking PATCH 13/22: throttle-groups: protect throttled requests with a CoMutex...
Checking PATCH 14/22: util: add stats64 module...
ERROR: memory barrier without comment
#332: FILE: util/stats64.c:102:
+        smp_wmb();

ERROR: memory barrier without comment
#361: FILE: util/stats64.c:131:
+        smp_wmb();

total: 2 errors, 0 warnings, 335 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 15/22: block: use Stat64 for wr_highest_offset...
Checking PATCH 16/22: block: access write_gen with atomics...
Checking PATCH 17/22: block: protect tracked_requests and flush_queue with reqs_lock...
Checking PATCH 18/22: block: introduce dirty_bitmap_mutex...
Checking PATCH 19/22: migration/block: reset dirty bitmap before reading...
Checking PATCH 20/22: block: protect modification of dirty bitmaps with a mutex...
WARNING: line over 80 characters
#306: FILE: migration/block.c:539:
+            bdrv_reset_dirty_bitmap_locked(bmds->dirty_bitmap, sector, nr_sectors);

total: 0 errors, 1 warnings, 272 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 21/22: block: introduce block_account_one_io...
Checking PATCH 22/22: block: make accounting thread-safe...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
Re: [Qemu-devel] [PULL v2 00/22] Docker and block patches
Posted by Fam Zheng 8 years, 5 months ago
On Fri, 06/02 04:38, no-reply@patchew.org wrote:
> Checking PATCH 14/22: util: add stats64 module...
> ERROR: memory barrier without comment
> #332: FILE: util/stats64.c:102:
> +        smp_wmb();
> 
> ERROR: memory barrier without comment
> #361: FILE: util/stats64.c:131:
> +        smp_wmb();
> 
> total: 2 errors, 0 warnings, 335 lines checked

As explained by Paolo in v1, the comments are just one more line above so it's a
false positive.

Fam

Re: [Qemu-devel] [PULL v2 00/22] Docker and block patches
Posted by Peter Maydell 8 years, 5 months ago
On 2 June 2017 at 09:10, Fam Zheng <famz@redhat.com> wrote:
> The following changes since commit 43771d5d92312504305c19abe29ec5bfabd55f01:
>
>   Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-05-31' into staging (2017-06-01 16:39:16 +0100)
>
> are available in the git repository at:
>
>   git://github.com/famz/qemu.git tags/docker-and-block-pull-request
>
> for you to fetch changes up to 4ab2cbc128f1938355a12970c9b6e419a63fcab6:
>
>   block: make accounting thread-safe (2017-06-02 15:59:32 +0800)
>
> ----------------------------------------------------------------
>
> v2: Fix building on OSX and BSD.
>
> ----------------------------------------------------------------
>
> Fam Zheng (4):
>   docker: Run tests with current user
>   docker: Add bzip2 and hostname to fedora image
>   docker: Add libaio to fedora image
>   docker: Add flex and bison to centos6 image
>
> Paolo Bonzini (18):
>   block: access copy_on_read with atomic ops
>   block: access quiesce_counter with atomic ops
>   block: access io_limits_disabled with atomic ops
>   block: access serialising_in_flight with atomic ops
>   block: access wakeup with atomic ops
>   block: access io_plugged with atomic ops
>   throttle-groups: only start one coroutine from drained_begin
>   throttle-groups: do not use qemu_co_enter_next
>   throttle-groups: protect throttled requests with a CoMutex
>   util: add stats64 module
>   block: use Stat64 for wr_highest_offset
>   block: access write_gen with atomics
>   block: protect tracked_requests and flush_queue with reqs_lock
>   block: introduce dirty_bitmap_mutex
>   migration/block: reset dirty bitmap before reading
>   block: protect modification of dirty bitmaps with a mutex
>   block: introduce block_account_one_io
>   block: make accounting thread-safe

Fails 'make check' on OSX:
  GTESTER tests/test-blockjob
qemu: qemu_mutex_destroy: Invalid argument

Backtrace:

(lldb) bt
* thread #1: tid = 0x442e6f, 0x00007fffc611ad42
libsystem_kernel.dylib`__pthread_kill + 10, queue =
'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fffc611ad42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fffc6208457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fffc6080420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001000c1335
test-blockjob`error_exit(err=<unavailable>, msg=<unavailable>) + 53 at
qemu-thread-posix.c:36
    frame #4: 0x00000001000c135d
test-blockjob`qemu_mutex_destroy(mutex=<unavailable>) + 29 at
qemu-thread-posix.c:54
    frame #5: 0x000000010004411d test-blockjob`blk_unref [inlined]
blk_delete(blk=0x0000000100c0c360) + 215 at block-backend.c:298
    frame #6: 0x0000000100044046
test-blockjob`blk_unref(blk=0x0000000100c0c360) + 38 at
block-backend.c:336
    frame #7: 0x0000000100005ffa
test-blockjob`bdrv_open_inherit(filename=<unavailable>,
reference=<unavailable>, options=0x000000010100c000,
flags=<unavailable>, parent=<unavailable>, child_role=<unavailable>,
errp=<unavailable>) + 4346 at block.c:2546
    frame #8: 0x0000000100006a07
test-blockjob`bdrv_open(filename=<unavailable>,
reference=<unavailable>, options=<unavailable>, flags=<unavailable>,
errp=<unavailable>) + 23 at block.c:2632
    frame #9: 0x0000000100001949
test-blockjob`create_blk(name=0x0000000000000000) + 73 at
test-blockjob.c:61
    frame #10: 0x00000001000016f5 test-blockjob`test_job_ids + 21 at
test-blockjob.c:92
    frame #11: 0x000000010070d91d
libglib-2.0.0.dylib`g_test_run_suite_internal + 626
    frame #12: 0x000000010070dae1
libglib-2.0.0.dylib`g_test_run_suite_internal + 1078
    frame #13: 0x000000010070d198 libglib-2.0.0.dylib`g_test_run_suite + 266
    frame #14: 0x00000001000016c8
test-blockjob`main(argc=<unavailable>, argv=<unavailable>) + 88 at
test-blockjob.c:152
    frame #15: 0x00007fffc5fec235 libdyld.dylib`start + 1
    frame #16: 0x00007fffc5fec235 libdyld.dylib`start + 1


thanks
-- PMM

Re: [Qemu-devel] [PULL v2 00/22] Docker and block patches
Posted by Fam Zheng 8 years, 5 months ago
Cc'ing Paolo.

On Fri, 06/02 16:50, Peter Maydell wrote:
> On 2 June 2017 at 09:10, Fam Zheng <famz@redhat.com> wrote:
> > The following changes since commit 43771d5d92312504305c19abe29ec5bfabd55f01:
> >
> >   Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-05-31' into staging (2017-06-01 16:39:16 +0100)
> >
> > are available in the git repository at:
> >
> >   git://github.com/famz/qemu.git tags/docker-and-block-pull-request
> >
> > for you to fetch changes up to 4ab2cbc128f1938355a12970c9b6e419a63fcab6:
> >
> >   block: make accounting thread-safe (2017-06-02 15:59:32 +0800)
> >
> > ----------------------------------------------------------------
> >
> > v2: Fix building on OSX and BSD.
> >
> > ----------------------------------------------------------------
> >
> > Fam Zheng (4):
> >   docker: Run tests with current user
> >   docker: Add bzip2 and hostname to fedora image
> >   docker: Add libaio to fedora image
> >   docker: Add flex and bison to centos6 image
> >
> > Paolo Bonzini (18):
> >   block: access copy_on_read with atomic ops
> >   block: access quiesce_counter with atomic ops
> >   block: access io_limits_disabled with atomic ops
> >   block: access serialising_in_flight with atomic ops
> >   block: access wakeup with atomic ops
> >   block: access io_plugged with atomic ops
> >   throttle-groups: only start one coroutine from drained_begin
> >   throttle-groups: do not use qemu_co_enter_next
> >   throttle-groups: protect throttled requests with a CoMutex
> >   util: add stats64 module
> >   block: use Stat64 for wr_highest_offset
> >   block: access write_gen with atomics
> >   block: protect tracked_requests and flush_queue with reqs_lock
> >   block: introduce dirty_bitmap_mutex
> >   migration/block: reset dirty bitmap before reading
> >   block: protect modification of dirty bitmaps with a mutex
> >   block: introduce block_account_one_io
> >   block: make accounting thread-safe
> 
> Fails 'make check' on OSX:
>   GTESTER tests/test-blockjob
> qemu: qemu_mutex_destroy: Invalid argument
> 
> Backtrace:
> 
> (lldb) bt
> * thread #1: tid = 0x442e6f, 0x00007fffc611ad42
> libsystem_kernel.dylib`__pthread_kill + 10, queue =
> 'com.apple.main-thread', stop reason = signal SIGABRT
>   * frame #0: 0x00007fffc611ad42 libsystem_kernel.dylib`__pthread_kill + 10
>     frame #1: 0x00007fffc6208457 libsystem_pthread.dylib`pthread_kill + 90
>     frame #2: 0x00007fffc6080420 libsystem_c.dylib`abort + 129
>     frame #3: 0x00000001000c1335
> test-blockjob`error_exit(err=<unavailable>, msg=<unavailable>) + 53 at
> qemu-thread-posix.c:36
>     frame #4: 0x00000001000c135d
> test-blockjob`qemu_mutex_destroy(mutex=<unavailable>) + 29 at
> qemu-thread-posix.c:54
>     frame #5: 0x000000010004411d test-blockjob`blk_unref [inlined]
> blk_delete(blk=0x0000000100c0c360) + 215 at block-backend.c:298
>     frame #6: 0x0000000100044046
> test-blockjob`blk_unref(blk=0x0000000100c0c360) + 38 at
> block-backend.c:336
>     frame #7: 0x0000000100005ffa
> test-blockjob`bdrv_open_inherit(filename=<unavailable>,
> reference=<unavailable>, options=0x000000010100c000,
> flags=<unavailable>, parent=<unavailable>, child_role=<unavailable>,
> errp=<unavailable>) + 4346 at block.c:2546
>     frame #8: 0x0000000100006a07
> test-blockjob`bdrv_open(filename=<unavailable>,
> reference=<unavailable>, options=<unavailable>, flags=<unavailable>,
> errp=<unavailable>) + 23 at block.c:2632
>     frame #9: 0x0000000100001949
> test-blockjob`create_blk(name=0x0000000000000000) + 73 at
> test-blockjob.c:61
>     frame #10: 0x00000001000016f5 test-blockjob`test_job_ids + 21 at
> test-blockjob.c:92
>     frame #11: 0x000000010070d91d
> libglib-2.0.0.dylib`g_test_run_suite_internal + 626
>     frame #12: 0x000000010070dae1
> libglib-2.0.0.dylib`g_test_run_suite_internal + 1078
>     frame #13: 0x000000010070d198 libglib-2.0.0.dylib`g_test_run_suite + 266
>     frame #14: 0x00000001000016c8
> test-blockjob`main(argc=<unavailable>, argv=<unavailable>) + 88 at
> test-blockjob.c:152
>     frame #15: 0x00007fffc5fec235 libdyld.dylib`start + 1
>     frame #16: 0x00007fffc5fec235 libdyld.dylib`start + 1

Re: [Qemu-devel] [PULL v2 00/22] Docker and block patches
Posted by Paolo Bonzini 8 years, 5 months ago

On 02/06/2017 23:30, Fam Zheng wrote:
> Cc'ing Paolo.
> 
> On Fri, 06/02 16:50, Peter Maydell wrote:
>> On 2 June 2017 at 09:10, Fam Zheng <famz@redhat.com> wrote:
>>> The following changes since commit 43771d5d92312504305c19abe29ec5bfabd55f01:
>>>
>>>   Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2017-05-31' into staging (2017-06-01 16:39:16 +0100)
>>>
>>> are available in the git repository at:
>>>
>>>   git://github.com/famz/qemu.git tags/docker-and-block-pull-request
>>>
>>> for you to fetch changes up to 4ab2cbc128f1938355a12970c9b6e419a63fcab6:
>>>
>>>   block: make accounting thread-safe (2017-06-02 15:59:32 +0800)
>>>
>>> ----------------------------------------------------------------
>>>
>>> v2: Fix building on OSX and BSD.

Seems to be a latent bug:

diff --git a/blockdev.c b/blockdev.c
index 34a20d3b9a..7b54d43896 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -564,6 +564,8 @@ static BlockBackend *blockdev_init(const char *file, QDict *bs_opts,
         blk_rs->read_only     = read_only;
         blk_rs->detect_zeroes = detect_zeroes;
 
+        block_acct_init(blk_get_stats(blk), account_invalid, account_failed);
+
         QDECREF(bs_opts);
     } else {
         if (file && !*file) {

I'll shortly send it as a separate patch.

Thanks,

Paolo