[Qemu-devel] [PATCH v3 0/9] qcow2: encryption threads

Vladimir Sementsov-Ogievskiy posted 9 patches 5 years, 3 months ago
Test docker-mingw@fedora failed
Test docker-clang@ubuntu passed
Test asan passed
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190108170655.29766-1-vsementsov@virtuozzo.com
Maintainers: Max Reitz <mreitz@redhat.com>, Kevin Wolf <kwolf@redhat.com>
There is a newer version of this series
block/qcow2.h          |  20 ++-
block/qcow2-bitmap.c   |   1 -
block/qcow2-cache.c    |   1 -
block/qcow2-cluster.c  |   8 +-
block/qcow2-refcount.c |   1 -
block/qcow2-snapshot.c |   1 -
block/qcow2-threads.c  | 268 +++++++++++++++++++++++++++++++++++++++
block/qcow2.c          | 277 ++++++++---------------------------------
block/Makefile.objs    |   2 +-
9 files changed, 343 insertions(+), 236 deletions(-)
create mode 100644 block/qcow2-threads.c
[Qemu-devel] [PATCH v3 0/9] qcow2: encryption threads
Posted by Vladimir Sementsov-Ogievskiy 5 years, 3 months ago
Hi all!

The series brings threads to qcow2 encryption/decryption path,
like it is already done for compression.
Now, based on master

v3:
01: drop 'include block_int.h' lines not needed more [Alberto]
02: fix comments style (open/close on separate line), add Alberto's r-b
03: improve commit message, add Alberto's r-b
04: new, follows Paolo's comment


v2: (not very careful change list)
    - multiple cipher inside QCryptoBlock instead of multiple
      blocks inside qcow2, as suggested by Daniel, and it is
      done in separate series
    - use threaded encryption in do_perform_cow_encrypt() too
    - some renaming and refactoring and simplifications


== Performance testing ==

Hmm, do we have something like tests/qemu-iotests, but for performance?

[root@kvm up-qcow2-encrypt-threads]# cat test.sh 
    #!/bin/bash

    size=1G
    src=/ssd/src.raw
    dst=/ssd/dst.enc.qcow2

    # create source for tests
    ./qemu-img create -f raw "$src" $size > /dev/null
    ./qemu-io -f raw -c "write -P 0xa 0 $size" "$src" > /dev/null

    for w in "" "-W"; do
        echo -e "Test with additional paramter for qemu-img: '$w'"

        # create target
        ./qemu-img create -f qcow2 --object secret,id=sec0,data=test -o encrypt.format=luks,encrypt.key-secret=sec0 "$dst" $size > /dev/null

        time ./qemu-img convert $w -f raw --object secret,id=sec0,data=test --target-image-opts -n "$src" "driver=qcow2,file.filename=$dst,encrypt.key-secret=sec0"
        echo
    done


before the series:
    Test with additional paramter for qemu-img: ''

    real    0m14.318s
    user    0m13.637s
    sys     0m0.881s

    Test with additional paramter for qemu-img: '-W'

    real    0m13.962s
    user    0m13.681s
    sys     0m1.156s


after the series:
    Test with additional paramter for qemu-img: ''

    real    0m14.382s
    user    0m13.735s
    sys     0m0.835s

    Test with additional paramter for qemu-img: '-W'

    real    0m5.696s
    user    0m15.931s
    sys     0m1.144s

Vladimir Sementsov-Ogievskiy (9):
  qcow2.h: add missing include
  qcow2: add separate file for threaded data processing functions
  qcow2-threads: use thread_pool_submit_co
  qcow2-threads: qcow2_co_do_compress: protect queuing by mutex
  qcow2-threads: split out generic path
  qcow2: qcow2_co_preadv: improve locking
  qcow2: qcow2_co_preadv: skip using hd_qiov when possible
  qcow2: bdrv_co_pwritev: move encryption code out of the lock
  qcow2: do encryption in threads

 block/qcow2.h          |  20 ++-
 block/qcow2-bitmap.c   |   1 -
 block/qcow2-cache.c    |   1 -
 block/qcow2-cluster.c  |   8 +-
 block/qcow2-refcount.c |   1 -
 block/qcow2-snapshot.c |   1 -
 block/qcow2-threads.c  | 268 +++++++++++++++++++++++++++++++++++++++
 block/qcow2.c          | 277 ++++++++---------------------------------
 block/Makefile.objs    |   2 +-
 9 files changed, 343 insertions(+), 236 deletions(-)
 create mode 100644 block/qcow2-threads.c

-- 
2.18.0


Re: [Qemu-devel] [PATCH v3 0/9] qcow2: encryption threads
Posted by no-reply@patchew.org 5 years, 3 months ago
Patchew URL: https://patchew.org/QEMU/20190108170655.29766-1-vsementsov@virtuozzo.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=14
=== TEST SCRIPT END ===

  CC      qapi/qapi-commands-crypto.o
  CC      qapi/qapi-commands-introspect.o
/tmp/qemu-test/src/block/sheepdog.c: In function 'find_vdi_name':
/tmp/qemu-test/src/block/sheepdog.c:1239:5: error: 'strncpy' specified bound 256 equals destination size [-Werror=stringop-truncation]
     strncpy(buf + SD_MAX_VDI_LEN, tag, SD_MAX_VDI_TAG_LEN);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors


The full log is available at
http://patchew.org/logs/20190108170655.29766-1-vsementsov@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [Qemu-devel] [PATCH v3 0/9] qcow2: encryption threads
Posted by Vladimir Sementsov-Ogievskiy 5 years, 3 months ago
23.01.2019 17:54, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20190108170655.29766-1-vsementsov@virtuozzo.com/
> 
> 
> 
> Hi,
> 
> This series failed the docker-mingw@fedora build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> time make docker-test-mingw@fedora SHOW_ENV=1 J=14
> === TEST SCRIPT END ===
> 
>    CC      qapi/qapi-commands-crypto.o
>    CC      qapi/qapi-commands-introspect.o
> /tmp/qemu-test/src/block/sheepdog.c: In function 'find_vdi_name':
> /tmp/qemu-test/src/block/sheepdog.c:1239:5: error: 'strncpy' specified bound 256 equals destination size [-Werror=stringop-truncation]
>       strncpy(buf + SD_MAX_VDI_LEN, tag, SD_MAX_VDI_TAG_LEN);
>       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> cc1: all warnings being treated as errors

unrelated to my series.

> 
> 
> The full log is available at
> http://patchew.org/logs/20190108170655.29766-1-vsementsov@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
> ---
> Email generated automatically by Patchew [http://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
> 


-- 
Best regards,
Vladimir
Re: [Qemu-devel] [PATCH v3 0/9] qcow2: encryption threads
Posted by Eric Blake 5 years, 3 months ago
On 1/23/19 9:07 AM, Vladimir Sementsov-Ogievskiy wrote:
> 23.01.2019 17:54, no-reply@patchew.org wrote:
>> Patchew URL: https://patchew.org/QEMU/20190108170655.29766-1-vsementsov@virtuozzo.com/
>>
>>
>>
>> Hi,
>>
>> This series failed the docker-mingw@fedora build test. Please find the testing commands and
>> their output below. If you have Docker installed, you can probably reproduce it
>> locally.
>>
>> === TEST SCRIPT BEGIN ===
>> #!/bin/bash
>> time make docker-test-mingw@fedora SHOW_ENV=1 J=14
>> === TEST SCRIPT END ===
>>
>>    CC      qapi/qapi-commands-crypto.o
>>    CC      qapi/qapi-commands-introspect.o
>> /tmp/qemu-test/src/block/sheepdog.c: In function 'find_vdi_name':
>> /tmp/qemu-test/src/block/sheepdog.c:1239:5: error: 'strncpy' specified bound 256 equals destination size [-Werror=stringop-truncation]
>>       strncpy(buf + SD_MAX_VDI_LEN, tag, SD_MAX_VDI_TAG_LEN);
>>       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> cc1: all warnings being treated as errors
> 
> unrelated to my series.
> 

Known false alarm; the patchew processing queue is backlogged, and is
testing patches against a version of qemu.git that does not yet have
patches for newer gcc compliance incorporated, while at the same time
having recently upgraded its testing environment to use the newer gcc
that warns.  Commit 97b583f4 will silence the noise, once patchew
catches up to using it.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[Qemu-devel] ping Re: [PATCH v3 0/9] qcow2: encryption threads
Posted by Vladimir Sementsov-Ogievskiy 5 years, 3 months ago
ping

Hi!

Mostly reviewed-by Alberto, "/* No sense in releasing the lock */" should be removed
from 06, should I resend just for this? I'd prefer to handle some other comments, if any.

08.01.2019 20:06, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> The series brings threads to qcow2 encryption/decryption path,
> like it is already done for compression.
> Now, based on master
> 
> v3:
> 01: drop 'include block_int.h' lines not needed more [Alberto]
> 02: fix comments style (open/close on separate line), add Alberto's r-b
> 03: improve commit message, add Alberto's r-b
> 04: new, follows Paolo's comment
> 
> 
> v2: (not very careful change list)
>      - multiple cipher inside QCryptoBlock instead of multiple
>        blocks inside qcow2, as suggested by Daniel, and it is
>        done in separate series
>      - use threaded encryption in do_perform_cow_encrypt() too
>      - some renaming and refactoring and simplifications
> 
> 
> == Performance testing ==
> 
> Hmm, do we have something like tests/qemu-iotests, but for performance?
> 
> [root@kvm up-qcow2-encrypt-threads]# cat test.sh
>      #!/bin/bash
> 
>      size=1G
>      src=/ssd/src.raw
>      dst=/ssd/dst.enc.qcow2
> 
>      # create source for tests
>      ./qemu-img create -f raw "$src" $size > /dev/null
>      ./qemu-io -f raw -c "write -P 0xa 0 $size" "$src" > /dev/null
> 
>      for w in "" "-W"; do
>          echo -e "Test with additional paramter for qemu-img: '$w'"
> 
>          # create target
>          ./qemu-img create -f qcow2 --object secret,id=sec0,data=test -o encrypt.format=luks,encrypt.key-secret=sec0 "$dst" $size > /dev/null
> 
>          time ./qemu-img convert $w -f raw --object secret,id=sec0,data=test --target-image-opts -n "$src" "driver=qcow2,file.filename=$dst,encrypt.key-secret=sec0"
>          echo
>      done
> 
> 
> before the series:
>      Test with additional paramter for qemu-img: ''
> 
>      real    0m14.318s
>      user    0m13.637s
>      sys     0m0.881s
> 
>      Test with additional paramter for qemu-img: '-W'
> 
>      real    0m13.962s
>      user    0m13.681s
>      sys     0m1.156s
> 
> 
> after the series:
>      Test with additional paramter for qemu-img: ''
> 
>      real    0m14.382s
>      user    0m13.735s
>      sys     0m0.835s
> 
>      Test with additional paramter for qemu-img: '-W'
> 
>      real    0m5.696s
>      user    0m15.931s
>      sys     0m1.144s
> 
> Vladimir Sementsov-Ogievskiy (9):
>    qcow2.h: add missing include
>    qcow2: add separate file for threaded data processing functions
>    qcow2-threads: use thread_pool_submit_co
>    qcow2-threads: qcow2_co_do_compress: protect queuing by mutex
>    qcow2-threads: split out generic path
>    qcow2: qcow2_co_preadv: improve locking
>    qcow2: qcow2_co_preadv: skip using hd_qiov when possible
>    qcow2: bdrv_co_pwritev: move encryption code out of the lock
>    qcow2: do encryption in threads
> 
>   block/qcow2.h          |  20 ++-
>   block/qcow2-bitmap.c   |   1 -
>   block/qcow2-cache.c    |   1 -
>   block/qcow2-cluster.c  |   8 +-
>   block/qcow2-refcount.c |   1 -
>   block/qcow2-snapshot.c |   1 -
>   block/qcow2-threads.c  | 268 +++++++++++++++++++++++++++++++++++++++
>   block/qcow2.c          | 277 ++++++++---------------------------------
>   block/Makefile.objs    |   2 +-
>   9 files changed, 343 insertions(+), 236 deletions(-)
>   create mode 100644 block/qcow2-threads.c
> 


-- 
Best regards,
Vladimir