[Qemu-devel] [PATCH v6 0/4] Add shrink image for qcow2

Pavel Butsykin posted 4 patches 6 years, 8 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20170714153749.25132-1-pbutsykin@virtuozzo.com
Test FreeBSD passed
Test checkpatch passed
Test docker passed
Test s390x passed
There is a newer version of this series
block/qcow2-cache.c        |  26 +++++++
block/qcow2-cluster.c      |  50 +++++++++++++
block/qcow2-refcount.c     | 140 ++++++++++++++++++++++++++++++++++++-
block/qcow2.c              |  43 +++++++++---
block/qcow2.h              |  17 +++++
qapi/block-core.json       |   3 +-
qemu-img-cmds.hx           |   4 +-
qemu-img.c                 |  23 ++++++
qemu-img.texi              |   6 +-
tests/qemu-iotests/102     |   4 +-
tests/qemu-iotests/163     | 170 +++++++++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/163.out |   5 ++
tests/qemu-iotests/group   |   1 +
13 files changed, 475 insertions(+), 17 deletions(-)
create mode 100644 tests/qemu-iotests/163
create mode 100644 tests/qemu-iotests/163.out
[Qemu-devel] [PATCH v6 0/4] Add shrink image for qcow2
Posted by Pavel Butsykin 6 years, 8 months ago
This patch add shrinking of the image file for qcow2. As a result, this allows
us to reduce the virtual image size and free up space on the disk without
copying the image. Image can be fragmented and shrink is done by punching holes
in the image file.

# ./qemu-img create -f qcow2 image.qcow2 4G
Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

# ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)

# ./qemu-img resize image.qcow2 512M
warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
error: qemu-img: Use the --shrink option to perform a shrink operation.

# ./qemu-img resize --shrink image.qcow2 128M
Image resized.

# ./qemu-img info image.qcow2
image: image.qcow2
file format: qcow2
virtual size: 128M (134217728 bytes)
disk size: 128M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# du -h image.qcow2
129M    image.qcow2

Changes from v1:
- add --shrink flag for qemu-img resize
- add qcow2_cache_discard
- simplify qcow2_shrink_l1_table() to reduce the likelihood of image corruption
- add new qemu-iotests for shrinking images

Changes from v2:
- replace qprintf() on error_report() (1)
- rewrite warning messages (1)
- enforce --shrink flag for all formats except raw (1)
- split qcow2_cache_discard() (2)
- minor fixes according to comments (3)
- rewrite the last part of qcow2_shrink_reftable() to avoid
  qcow2_free_clusters() calls inside (3)
- improve test for shrinking image (4)

Changes from v3:
- rebase on "Implement a warning_report function" Alistair's patch-set (1)
- spelling fixes (1)
- the man page fix according to the discussion (1)
- add call qcow2_signal_corruption() in case of image corruption (3)

Changes from v4:
- rebase on https://github.com/XanClic/qemu/commits/block Max's block branch

Changes from v5:
- the condition refcount == 0 should be enough to evict the l2/refcount cluster
  from the cache (2)
- overwrite the l1/refcount table in memory with zeros, even if overwriting the
  l1/refcount table on disk has failed (3)
- replace g_try_malloc() on g_malloc() for allocation reftable_tmp (3)

Pavel Butsykin (4):
  qemu-img: add --shrink flag for resize
  qcow2: add qcow2_cache_discard
  qcow2: add shrink image support
  qemu-iotests: add shrinking image test

 block/qcow2-cache.c        |  26 +++++++
 block/qcow2-cluster.c      |  50 +++++++++++++
 block/qcow2-refcount.c     | 140 ++++++++++++++++++++++++++++++++++++-
 block/qcow2.c              |  43 +++++++++---
 block/qcow2.h              |  17 +++++
 qapi/block-core.json       |   3 +-
 qemu-img-cmds.hx           |   4 +-
 qemu-img.c                 |  23 ++++++
 qemu-img.texi              |   6 +-
 tests/qemu-iotests/102     |   4 +-
 tests/qemu-iotests/163     | 170 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/163.out |   5 ++
 tests/qemu-iotests/group   |   1 +
 13 files changed, 475 insertions(+), 17 deletions(-)
 create mode 100644 tests/qemu-iotests/163
 create mode 100644 tests/qemu-iotests/163.out

-- 
2.13.0


Re: [Qemu-devel] [Qemu-block] [PATCH v6 0/4] Add shrink image for qcow2
Posted by John Snow 6 years, 7 months ago
Over a month with no replies and we're nearing the next QEMU release. If
this patchset is still applicable, can you rebase and resend for 2.11?

--js

On 07/14/2017 11:37 AM, Pavel Butsykin wrote:
> This patch add shrinking of the image file for qcow2. As a result, this allows
> us to reduce the virtual image size and free up space on the disk without
> copying the image. Image can be fragmented and shrink is done by punching holes
> in the image file.
> 
> # ./qemu-img create -f qcow2 image.qcow2 4G
> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
> 
> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
> wrote 1073741824/1073741824 bytes at offset 0
> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
> 
> # ./qemu-img resize image.qcow2 512M
> warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
> error: qemu-img: Use the --shrink option to perform a shrink operation.
> 
> # ./qemu-img resize --shrink image.qcow2 128M
> Image resized.
> 
> # ./qemu-img info image.qcow2
> image: image.qcow2
> file format: qcow2
> virtual size: 128M (134217728 bytes)
> disk size: 128M
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     lazy refcounts: false
>     refcount bits: 16
>     corrupt: false
> 
> # du -h image.qcow2
> 129M    image.qcow2
> 
> Changes from v1:
> - add --shrink flag for qemu-img resize
> - add qcow2_cache_discard
> - simplify qcow2_shrink_l1_table() to reduce the likelihood of image corruption
> - add new qemu-iotests for shrinking images
> 
> Changes from v2:
> - replace qprintf() on error_report() (1)
> - rewrite warning messages (1)
> - enforce --shrink flag for all formats except raw (1)
> - split qcow2_cache_discard() (2)
> - minor fixes according to comments (3)
> - rewrite the last part of qcow2_shrink_reftable() to avoid
>   qcow2_free_clusters() calls inside (3)
> - improve test for shrinking image (4)
> 
> Changes from v3:
> - rebase on "Implement a warning_report function" Alistair's patch-set (1)
> - spelling fixes (1)
> - the man page fix according to the discussion (1)
> - add call qcow2_signal_corruption() in case of image corruption (3)
> 
> Changes from v4:
> - rebase on https://github.com/XanClic/qemu/commits/block Max's block branch
> 
> Changes from v5:
> - the condition refcount == 0 should be enough to evict the l2/refcount cluster
>   from the cache (2)
> - overwrite the l1/refcount table in memory with zeros, even if overwriting the
>   l1/refcount table on disk has failed (3)
> - replace g_try_malloc() on g_malloc() for allocation reftable_tmp (3)
> 
> Pavel Butsykin (4):
>   qemu-img: add --shrink flag for resize
>   qcow2: add qcow2_cache_discard
>   qcow2: add shrink image support
>   qemu-iotests: add shrinking image test
> 
>  block/qcow2-cache.c        |  26 +++++++
>  block/qcow2-cluster.c      |  50 +++++++++++++
>  block/qcow2-refcount.c     | 140 ++++++++++++++++++++++++++++++++++++-
>  block/qcow2.c              |  43 +++++++++---
>  block/qcow2.h              |  17 +++++
>  qapi/block-core.json       |   3 +-
>  qemu-img-cmds.hx           |   4 +-
>  qemu-img.c                 |  23 ++++++
>  qemu-img.texi              |   6 +-
>  tests/qemu-iotests/102     |   4 +-
>  tests/qemu-iotests/163     | 170 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/163.out |   5 ++
>  tests/qemu-iotests/group   |   1 +
>  13 files changed, 475 insertions(+), 17 deletions(-)
>  create mode 100644 tests/qemu-iotests/163
>  create mode 100644 tests/qemu-iotests/163.out
> 

Re: [Qemu-devel] [Qemu-block] [PATCH v6 0/4] Add shrink image for qcow2
Posted by Pavel Butsykin 6 years, 7 months ago
On 17.08.2017 00:07, John Snow wrote:
> Over a month with no replies and we're nearing the next QEMU release. If
> this patchset is still applicable, can you rebase and resend for 2.11?

Thanks for digging up these patches :) I've sent the rebased version.

> --js
> 
> On 07/14/2017 11:37 AM, Pavel Butsykin wrote:
>> This patch add shrinking of the image file for qcow2. As a result, this allows
>> us to reduce the virtual image size and free up space on the disk without
>> copying the image. Image can be fragmented and shrink is done by punching holes
>> in the image file.
>>
>> # ./qemu-img create -f qcow2 image.qcow2 4G
>> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>
>> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
>> wrote 1073741824/1073741824 bytes at offset 0
>> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
>>
>> # ./qemu-img resize image.qcow2 512M
>> warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
>> error: qemu-img: Use the --shrink option to perform a shrink operation.
>>
>> # ./qemu-img resize --shrink image.qcow2 128M
>> Image resized.
>>
>> # ./qemu-img info image.qcow2
>> image: image.qcow2
>> file format: qcow2
>> virtual size: 128M (134217728 bytes)
>> disk size: 128M
>> cluster_size: 65536
>> Format specific information:
>>      compat: 1.1
>>      lazy refcounts: false
>>      refcount bits: 16
>>      corrupt: false
>>
>> # du -h image.qcow2
>> 129M    image.qcow2
>>
>> Changes from v1:
>> - add --shrink flag for qemu-img resize
>> - add qcow2_cache_discard
>> - simplify qcow2_shrink_l1_table() to reduce the likelihood of image corruption
>> - add new qemu-iotests for shrinking images
>>
>> Changes from v2:
>> - replace qprintf() on error_report() (1)
>> - rewrite warning messages (1)
>> - enforce --shrink flag for all formats except raw (1)
>> - split qcow2_cache_discard() (2)
>> - minor fixes according to comments (3)
>> - rewrite the last part of qcow2_shrink_reftable() to avoid
>>    qcow2_free_clusters() calls inside (3)
>> - improve test for shrinking image (4)
>>
>> Changes from v3:
>> - rebase on "Implement a warning_report function" Alistair's patch-set (1)
>> - spelling fixes (1)
>> - the man page fix according to the discussion (1)
>> - add call qcow2_signal_corruption() in case of image corruption (3)
>>
>> Changes from v4:
>> - rebase on https://github.com/XanClic/qemu/commits/block Max's block branch
>>
>> Changes from v5:
>> - the condition refcount == 0 should be enough to evict the l2/refcount cluster
>>    from the cache (2)
>> - overwrite the l1/refcount table in memory with zeros, even if overwriting the
>>    l1/refcount table on disk has failed (3)
>> - replace g_try_malloc() on g_malloc() for allocation reftable_tmp (3)
>>
>> Pavel Butsykin (4):
>>    qemu-img: add --shrink flag for resize
>>    qcow2: add qcow2_cache_discard
>>    qcow2: add shrink image support
>>    qemu-iotests: add shrinking image test
>>
>>   block/qcow2-cache.c        |  26 +++++++
>>   block/qcow2-cluster.c      |  50 +++++++++++++
>>   block/qcow2-refcount.c     | 140 ++++++++++++++++++++++++++++++++++++-
>>   block/qcow2.c              |  43 +++++++++---
>>   block/qcow2.h              |  17 +++++
>>   qapi/block-core.json       |   3 +-
>>   qemu-img-cmds.hx           |   4 +-
>>   qemu-img.c                 |  23 ++++++
>>   qemu-img.texi              |   6 +-
>>   tests/qemu-iotests/102     |   4 +-
>>   tests/qemu-iotests/163     | 170 +++++++++++++++++++++++++++++++++++++++++++++
>>   tests/qemu-iotests/163.out |   5 ++
>>   tests/qemu-iotests/group   |   1 +
>>   13 files changed, 475 insertions(+), 17 deletions(-)
>>   create mode 100644 tests/qemu-iotests/163
>>   create mode 100644 tests/qemu-iotests/163.out
>>