This patch add shrinking of the image file for qcow2. As a result, this allows
us to reduce the virtual image size and free up space on the disk without
copying the image. Image can be fragmented and shrink is done by punching holes
in the image file.

# ./qemu-img create -f qcow2 image.qcow2 4G
Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

# ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)

# ./qemu-img resize image.qcow2 512M
warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
error: qemu-img: Use the --shrink option to perform a shrink operation.

# ./qemu-img resize --shrink image.qcow2 128M
Image resized.

# ./qemu-img info image.qcow2
image: image.qcow2
file format: qcow2
virtual size: 128M (134217728 bytes)
disk size: 128M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# du -h image.qcow2
129M    image.qcow2

Changes from v1:
- add --shrink flag for qemu-img resize
- add qcow2_cache_discard
- simplify qcow2_shrink_l1_table() to reduce the likelihood of image corruption
- add new qemu-iotests for shrinking images

Changes from v2:
- replace qprintf() on error_report() (1)
- rewrite warning messages (1)
- enforce --shrink flag for all formats except raw (1)
- split qcow2_cache_discard() (2)
- minor fixes according to comments (3)
- rewrite the last part of qcow2_shrink_reftable() to avoid
  qcow2_free_clusters() calls inside (3)
- improve test for shrinking image (4)

Changes from v3:
- rebase on "Implement a warning_report function" Alistair's patch-set (1)
- spelling fixes (1)
- the man page fix according to the discussion (1)
- add call qcow2_signal_corruption() in case of image corruption (3)

Changes from v4:
- rebase on https://github.com/XanClic/qemu/commits/block Max's block branch

Changes from v5:
- the condition refcount == 0 should be enough to evict the l2/refcount cluster
  from the cache (2)
- overwrite the l1/refcount table in memory with zeros, even if overwriting the
  l1/refcount table on disk has failed (3)
- replace g_try_malloc() on g_malloc() for allocation reftable_tmp (3)

Changes from v6:
- rebase on master 1f29673387

Pavel Butsykin (4):
  qemu-img: add --shrink flag for resize
  qcow2: add qcow2_cache_discard
  qcow2: add shrink image support
  qemu-iotests: add shrinking image test

 block/qcow2-cache.c        |  26 +++++++
 block/qcow2-cluster.c      |  50 +++++++++++++
 block/qcow2-refcount.c     | 140 ++++++++++++++++++++++++++++++++++++-
 block/qcow2.c              |  43 +++++++++---
 block/qcow2.h              |  17 +++++
 qapi/block-core.json       |   3 +-
 qemu-img-cmds.hx           |   4 +-
 qemu-img.c                 |  23 ++++++
 qemu-img.texi              |   6 +-
 tests/qemu-iotests/102     |   4 +-
 tests/qemu-iotests/163     | 170 +++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/163.out |   5 ++
 tests/qemu-iotests/group   |   1 +
 13 files changed, 475 insertions(+), 17 deletions(-)
 create mode 100644 tests/qemu-iotests/163
 create mode 100644 tests/qemu-iotests/163.out

-- 
2.14.1

On 08/17/2017 05:15 AM, Pavel Butsykin wrote:
> This patch add shrinking of the image file for qcow2. As a result, this allows
> us to reduce the virtual image size and free up space on the disk without
> copying the image. Image can be fragmented and shrink is done by punching holes
> in the image file.
> 
> # ./qemu-img create -f qcow2 image.qcow2 4G
> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
> 
> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
> wrote 1073741824/1073741824 bytes at offset 0
> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
> 
> # ./qemu-img resize image.qcow2 512M
> warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
> error: qemu-img: Use the --shrink option to perform a shrink operation.
> 
> # ./qemu-img resize --shrink image.qcow2 128M
> Image resized.
> 
> # ./qemu-img info image.qcow2
> image: image.qcow2
> file format: qcow2
> virtual size: 128M (134217728 bytes)
> disk size: 128M
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     lazy refcounts: false
>     refcount bits: 16
>     corrupt: false
> 
> # du -h image.qcow2
> 129M    image.qcow2
> 

It looks sane to me, and already has a full set of R-Bs from Max. Are we
waiting for Kevin?

It looks like in Vladimir's series we sidestepped the problem of what
happens if we resize with persistent bitmaps: we deny the operation,
regardless of if we are shrinking, growing, or have bitmaps that are
read-only, frozen, or what-have-you.

It shouldn't be too hard to add soon, but this is fine for now.
(I think for adding it we just need to make sure the bitmaps aren't
frozen and are not read-only, and the implementation bitmap structure
can already handle truncation in either direction just fine.)

Anyway;

Reviewed-by: John Snow <jsnow@redhat.com>


> Changes from v1:
> - add --shrink flag for qemu-img resize
> - add qcow2_cache_discard
> - simplify qcow2_shrink_l1_table() to reduce the likelihood of image corruption
> - add new qemu-iotests for shrinking images
> 
> Changes from v2:
> - replace qprintf() on error_report() (1)
> - rewrite warning messages (1)
> - enforce --shrink flag for all formats except raw (1)
> - split qcow2_cache_discard() (2)
> - minor fixes according to comments (3)
> - rewrite the last part of qcow2_shrink_reftable() to avoid
>   qcow2_free_clusters() calls inside (3)
> - improve test for shrinking image (4)
> 
> Changes from v3:
> - rebase on "Implement a warning_report function" Alistair's patch-set (1)
> - spelling fixes (1)
> - the man page fix according to the discussion (1)
> - add call qcow2_signal_corruption() in case of image corruption (3)
> 
> Changes from v4:
> - rebase on https://github.com/XanClic/qemu/commits/block Max's block branch
> 
> Changes from v5:
> - the condition refcount == 0 should be enough to evict the l2/refcount cluster
>   from the cache (2)
> - overwrite the l1/refcount table in memory with zeros, even if overwriting the
>   l1/refcount table on disk has failed (3)
> - replace g_try_malloc() on g_malloc() for allocation reftable_tmp (3)
> 
> Changes from v6:
> - rebase on master 1f29673387
> 
> Pavel Butsykin (4):
>   qemu-img: add --shrink flag for resize
>   qcow2: add qcow2_cache_discard
>   qcow2: add shrink image support
>   qemu-iotests: add shrinking image test
> 
>  block/qcow2-cache.c        |  26 +++++++
>  block/qcow2-cluster.c      |  50 +++++++++++++
>  block/qcow2-refcount.c     | 140 ++++++++++++++++++++++++++++++++++++-
>  block/qcow2.c              |  43 +++++++++---
>  block/qcow2.h              |  17 +++++
>  qapi/block-core.json       |   3 +-
>  qemu-img-cmds.hx           |   4 +-
>  qemu-img.c                 |  23 ++++++
>  qemu-img.texi              |   6 +-
>  tests/qemu-iotests/102     |   4 +-
>  tests/qemu-iotests/163     | 170 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/163.out |   5 ++
>  tests/qemu-iotests/group   |   1 +
>  13 files changed, 475 insertions(+), 17 deletions(-)
>  create mode 100644 tests/qemu-iotests/163
>  create mode 100644 tests/qemu-iotests/163.out
> 

On 2017-08-22 01:31, John Snow wrote:
> 
> 
> On 08/17/2017 05:15 AM, Pavel Butsykin wrote:
>> This patch add shrinking of the image file for qcow2. As a result, this allows
>> us to reduce the virtual image size and free up space on the disk without
>> copying the image. Image can be fragmented and shrink is done by punching holes
>> in the image file.
>>
>> # ./qemu-img create -f qcow2 image.qcow2 4G
>> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>
>> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
>> wrote 1073741824/1073741824 bytes at offset 0
>> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
>>
>> # ./qemu-img resize image.qcow2 512M
>> warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
>> error: qemu-img: Use the --shrink option to perform a shrink operation.
>>
>> # ./qemu-img resize --shrink image.qcow2 128M
>> Image resized.
>>
>> # ./qemu-img info image.qcow2
>> image: image.qcow2
>> file format: qcow2
>> virtual size: 128M (134217728 bytes)
>> disk size: 128M
>> cluster_size: 65536
>> Format specific information:
>>     compat: 1.1
>>     lazy refcounts: false
>>     refcount bits: 16
>>     corrupt: false
>>
>> # du -h image.qcow2
>> 129M    image.qcow2
>>
> 
> It looks sane to me, and already has a full set of R-Bs from Max. Are we
> waiting for Kevin?

We were waiting for Kevin to handle 2.10 patches so he could go into PTO
and for me to come back from PTO. ;-)

I'm still sifting through my inbox... When I'm done, I'll take a look,
but I have no idea how long that might last.
(Since 2.10 isn't out yet, there might be more pressing issues still...?
I don't quite now yet, to be honest...)

> It looks like in Vladimir's series we sidestepped the problem of what
> happens if we resize with persistent bitmaps: we deny the operation,
> regardless of if we are shrinking, growing, or have bitmaps that are
> read-only, frozen, or what-have-you.
> 
> It shouldn't be too hard to add soon, but this is fine for now.
> (I think for adding it we just need to make sure the bitmaps aren't
> frozen and are not read-only, and the implementation bitmap structure
> can already handle truncation in either direction just fine.)
> 
> Anyway;
> 
> Reviewed-by: John Snow <jsnow@redhat.com>

Thanks, in any case. :-)

Max

On 2017-08-22 01:31, John Snow wrote:
> 
> 
> On 08/17/2017 05:15 AM, Pavel Butsykin wrote:
>> This patch add shrinking of the image file for qcow2. As a result, this allows
>> us to reduce the virtual image size and free up space on the disk without
>> copying the image. Image can be fragmented and shrink is done by punching holes
>> in the image file.
>>
>> # ./qemu-img create -f qcow2 image.qcow2 4G
>> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>
>> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
>> wrote 1073741824/1073741824 bytes at offset 0
>> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
>>
>> # ./qemu-img resize image.qcow2 512M
>> warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
>> error: qemu-img: Use the --shrink option to perform a shrink operation.
>>
>> # ./qemu-img resize --shrink image.qcow2 128M
>> Image resized.
>>
>> # ./qemu-img info image.qcow2
>> image: image.qcow2
>> file format: qcow2
>> virtual size: 128M (134217728 bytes)
>> disk size: 128M
>> cluster_size: 65536
>> Format specific information:
>>     compat: 1.1
>>     lazy refcounts: false
>>     refcount bits: 16
>>     corrupt: false
>>
>> # du -h image.qcow2
>> 129M    image.qcow2
>>
> 
> It looks sane to me, and already has a full set of R-Bs from Max. Are we
> waiting for Kevin?

Now I found that I was waiting for my own R-b for patch 3.  I hadn't yet
reviewed that patch in its current state (I had reviewed it for v5, but
it has changed quite a bit in v6).

Max

On 16.09.2017 17:56, Max Reitz wrote:
> On 2017-08-22 01:31, John Snow wrote:
>>
>>
>> On 08/17/2017 05:15 AM, Pavel Butsykin wrote:
>>> This patch add shrinking of the image file for qcow2. As a result, this allows
>>> us to reduce the virtual image size and free up space on the disk without
>>> copying the image. Image can be fragmented and shrink is done by punching holes
>>> in the image file.
>>>
>>> # ./qemu-img create -f qcow2 image.qcow2 4G
>>> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>>
>>> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
>>> wrote 1073741824/1073741824 bytes at offset 0
>>> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
>>>
>>> # ./qemu-img resize image.qcow2 512M
>>> warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
>>> error: qemu-img: Use the --shrink option to perform a shrink operation.
>>>
>>> # ./qemu-img resize --shrink image.qcow2 128M
>>> Image resized.
>>>
>>> # ./qemu-img info image.qcow2
>>> image: image.qcow2
>>> file format: qcow2
>>> virtual size: 128M (134217728 bytes)
>>> disk size: 128M
>>> cluster_size: 65536
>>> Format specific information:
>>>      compat: 1.1
>>>      lazy refcounts: false
>>>      refcount bits: 16
>>>      corrupt: false
>>>
>>> # du -h image.qcow2
>>> 129M    image.qcow2
>>>
>>
>> It looks sane to me, and already has a full set of R-Bs from Max. Are we
>> waiting for Kevin?
> 
> Now I found that I was waiting for my own R-b for patch 3.  I hadn't yet
> reviewed that patch in its current state (I had reviewed it for v5, but
> it has changed quite a bit in v6)

I decided that small changes are not considered :) I'm sorry for the
confusion with the R-b.


> Max
> 

On 17.08.2017 12:15, Pavel Butsykin wrote:
> This patch add shrinking of the image file for qcow2. As a result, this allows
> us to reduce the virtual image size and free up space on the disk without
> copying the image. Image can be fragmented and shrink is done by punching holes
> in the image file.
> 
> # ./qemu-img create -f qcow2 image.qcow2 4G
> Formatting 'image.qcow2', fmt=qcow2 size=4294967296 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
> 
> # ./qemu-io -c "write -P 0x22 0 1G" image.qcow2
> wrote 1073741824/1073741824 bytes at offset 0
> 1 GiB, 1 ops; 0:00:04.59 (222.886 MiB/sec and 0.2177 ops/sec)
> 
> # ./qemu-img resize image.qcow2 512M
> warning: qemu-img: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
> error: qemu-img: Use the --shrink option to perform a shrink operation.
> 
> # ./qemu-img resize --shrink image.qcow2 128M
> Image resized.
> 
> # ./qemu-img info image.qcow2
> image: image.qcow2
> file format: qcow2
> virtual size: 128M (134217728 bytes)
> disk size: 128M
> cluster_size: 65536
> Format specific information:
>      compat: 1.1
>      lazy refcounts: false
>      refcount bits: 16
>      corrupt: false
> 
> # du -h image.qcow2
> 129M    image.qcow2

ping


Hi guys, I would like to ask is there any plans to take this small patch
series? :) And thanks for the review.