[Qemu-devel] [PATCH v1 00/13] qcow2: space preallocation and COW improvements

Anton Nefedov posted 13 patches 6 years, 10 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/1495186480-114192-1-git-send-email-anton.nefedov@virtuozzo.com
Test checkpatch passed
Test docker passed
Test s390x passed
There is a newer version of this series
block/qcow2-cache.c                |   3 +
block/qcow2-cluster.c              | 216 +++++++++++++++++++++++-----
block/qcow2-refcount.c             |  21 +++
block/qcow2.c                      | 286 ++++++++++++++++++++++++++++++++++++-
block/qcow2.h                      |  26 ++++
tests/qemu-iotests/026.out         | 104 ++++++++++----
tests/qemu-iotests/026.out.nocache | 104 ++++++++++----
tests/qemu-iotests/029.out         |   5 +-
tests/qemu-iotests/046             |  38 ++++-
tests/qemu-iotests/046.out         |  23 +++
tests/qemu-iotests/060             |   2 +-
tests/qemu-iotests/060.out         |  13 +-
tests/qemu-iotests/061.out         |   5 +-
tests/qemu-iotests/066             |   2 +-
tests/qemu-iotests/066.out         |   9 +-
tests/qemu-iotests/098.out         |   7 +-
tests/qemu-iotests/108.out         |   5 +-
tests/qemu-iotests/112.out         |   5 +-
tests/qemu-iotests/154.out         |   4 +-
19 files changed, 769 insertions(+), 109 deletions(-)
[Qemu-devel] [PATCH v1 00/13] qcow2: space preallocation and COW improvements
Posted by Anton Nefedov 6 years, 10 months ago
This pull request is to address a few performance problems of qcow2 format:

  1. non cluster-aligned write requests (to unallocated clusters) explicitly
    pad data with zeroes if there is no backing data. This can be avoided
    and the whole clusters are preallocated and zeroed in a single
    efficient write_zeroes() operation, also providing better host file
    continuity

  2. moreover, efficient write_zeroes() operation can be used to preallocate
    space megabytes ahead which gives noticeable improvement on some storage
    types (e.g. distributed storages where space allocation operation is
    expensive)

  3. preallocating/zeroing the clusters in advance makes possible to enable
    simultaneous writes to the same unallocated cluster, which is beneficial
    for parallel sequential write operations which are not cluster-aligned

Performance test results are added to commit messages (see patch 3, 12)

Anton Nefedov (9):
  qcow2: is_zero_sectors(): return true if area is outside of backing
    file
  qcow2: do not COW the empty areas
  qcow2: set inactive flag
  qcow2: handle_prealloc(): find out if area zeroed by earlier
    preallocation
  qcow2: fix misleading comment about L2 linking
  qcow2-cluster: slightly refactor handle_dependencies()
  qcow2-cluster: make handle_dependencies() logic easier to follow
  qcow2: allow concurrent unaligned writes to the same clusters
  iotest 046: test simultaneous cluster write error case

Denis V. Lunev (3):
  qcow2: alloc space for COW in one chunk
  qcow2: preallocation at image expand
  qcow2: truncate preallocated space

Pavel Butsykin (1):
  qcow2: check space leak at the end of the image

 block/qcow2-cache.c                |   3 +
 block/qcow2-cluster.c              | 216 +++++++++++++++++++++++-----
 block/qcow2-refcount.c             |  21 +++
 block/qcow2.c                      | 286 ++++++++++++++++++++++++++++++++++++-
 block/qcow2.h                      |  26 ++++
 tests/qemu-iotests/026.out         | 104 ++++++++++----
 tests/qemu-iotests/026.out.nocache | 104 ++++++++++----
 tests/qemu-iotests/029.out         |   5 +-
 tests/qemu-iotests/046             |  38 ++++-
 tests/qemu-iotests/046.out         |  23 +++
 tests/qemu-iotests/060             |   2 +-
 tests/qemu-iotests/060.out         |  13 +-
 tests/qemu-iotests/061.out         |   5 +-
 tests/qemu-iotests/066             |   2 +-
 tests/qemu-iotests/066.out         |   9 +-
 tests/qemu-iotests/098.out         |   7 +-
 tests/qemu-iotests/108.out         |   5 +-
 tests/qemu-iotests/112.out         |   5 +-
 tests/qemu-iotests/154.out         |   4 +-
 19 files changed, 769 insertions(+), 109 deletions(-)

-- 
2.7.4


Re: [Qemu-devel] [PATCH v1 00/13] qcow2: space preallocation and COW improvements
Posted by Eric Blake 6 years, 10 months ago
On 05/19/2017 04:34 AM, Anton Nefedov wrote:
> This pull request is to address a few performance problems of qcow2 format:
> 
>   1. non cluster-aligned write requests (to unallocated clusters) explicitly
>     pad data with zeroes if there is no backing data. This can be avoided
>     and the whole clusters are preallocated and zeroed in a single
>     efficient write_zeroes() operation, also providing better host file
>     continuity
> 
>   2. moreover, efficient write_zeroes() operation can be used to preallocate
>     space megabytes ahead which gives noticeable improvement on some storage
>     types (e.g. distributed storages where space allocation operation is
>     expensive)
> 
>   3. preallocating/zeroing the clusters in advance makes possible to enable
>     simultaneous writes to the same unallocated cluster, which is beneficial
>     for parallel sequential write operations which are not cluster-aligned
> 
> Performance test results are added to commit messages (see patch 3, 12)

And now Berto has posted parallel patches. I'm not sure which ones to
focus on, or if you can work it out between you on the best approach
forward...

https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg05236.html

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v1 00/13] qcow2: space preallocation and COW improvements
Posted by Denis V. Lunev 6 years, 10 months ago
On 05/23/2017 05:35 PM, Eric Blake wrote:
> On 05/19/2017 04:34 AM, Anton Nefedov wrote:
>> This pull request is to address a few performance problems of qcow2 format:
>>
>>   1. non cluster-aligned write requests (to unallocated clusters) explicitly
>>     pad data with zeroes if there is no backing data. This can be avoided
>>     and the whole clusters are preallocated and zeroed in a single
>>     efficient write_zeroes() operation, also providing better host file
>>     continuity
>>
>>   2. moreover, efficient write_zeroes() operation can be used to preallocate
>>     space megabytes ahead which gives noticeable improvement on some storage
>>     types (e.g. distributed storages where space allocation operation is
>>     expensive)
>>
>>   3. preallocating/zeroing the clusters in advance makes possible to enable
>>     simultaneous writes to the same unallocated cluster, which is beneficial
>>     for parallel sequential write operations which are not cluster-aligned
>>
>> Performance test results are added to commit messages (see patch 3, 12)
> And now Berto has posted parallel patches. I'm not sure which ones to
> focus on, or if you can work it out between you on the best approach
> forward...
>
> https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg05236.html
>
I have seen those patches and will comment.

yes, they are intersect. He is concentrating on the fact of performing
operation in minimal read/write amount. We have made an attempt
to minimize amount of IO in the expand case.

Frankly speaking Berto patches are not that influenced. We definitely
should merge IO if write_zeroes are impossible. Thus both could
be merged in parallel.

What I want to note is the fact that there is another big problem
in QCOW2 layer. We have 64k block by default. Guest request
can be 4-5Mb in total. If the image is fragmented, most like
it will if it is old, we will have sequentially read/write the data.
Thus in ideal world we should switch to co-routine pool and the
same approach would be the best in COW too.

Pls note, that this also would very positively influence VM downtime
on migration as bdrv_drain_all is one the most important time hogs.

Den