[Qemu-devel] [RFC v2 0/8] qemu-img: add measure sub-command

Stefan Hajnoczi posted 8 patches 7 years, 1 month ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20170315092940.1367-1-stefanha@redhat.com
Test checkpatch passed
Test docker passed
There is a newer version of this series
qapi/block-core.json             |  19 +++
include/block/block.h            |   4 +
include/block/block_int.h        |   2 +
block.c                          |  33 +++++
block/qcow2.c                    | 296 ++++++++++++++++++++++++++-------------
block/raw-format.c               |  22 +++
qemu-img.c                       | 213 ++++++++++++++++++++++++++++
qemu-img-cmds.hx                 |   9 ++
tests/qemu-iotests/178           |  75 ++++++++++
tests/qemu-iotests/178.out.qcow2 |  33 +++++
tests/qemu-iotests/178.out.raw   |  33 +++++
tests/qemu-iotests/check         |   5 +
tests/qemu-iotests/group         |   1 +
13 files changed, 651 insertions(+), 94 deletions(-)
create mode 100755 tests/qemu-iotests/178
create mode 100644 tests/qemu-iotests/178.out.qcow2
create mode 100644 tests/qemu-iotests/178.out.raw
[Qemu-devel] [RFC v2 0/8] qemu-img: add measure sub-command
Posted by Stefan Hajnoczi 7 years, 1 month ago
RFCv2:
 * Publishing RFC again to discuss the new user-visible interfaces.  Code has
   changed quite a bit, I have not kept any Reviewed-by tags.
 * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir]
 * Report both "required bytes" and "fully allocated bytes" to handle the empty
   image file and prealloc use cases [Nir and Dan]
 * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto]
 * Rename "err" label "out" in qemu-img-cmds.c [Nir]
 * Add basic qcow2 support, doesn't support qemu-img convert from existing files yet

RFCv1:
 * Publishing patch series with just raw support, no qcow2 yet.  Please review
   the command-line interface and let me know if you are happy with this
   approach.

Users and management tools sometimes need to know the size required for a new
disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
Image formats like qcow2 have non-trivial metadata that makes it hard to
estimate the exact size without knowledge of file format internals.

This patch series introduces a new qemu-img sub-command that calculates the
required size for both image creation and conversion scenarios.

The conversion scenario is:

  $ qemu-img measure -f raw -O qcow2 input.img
  required bytes: 1327680
  fully allocated bytes: 1074069504

Here an existing image file is taken and the output includes the space required
for data from the input image file.

The creation scenario is:

  $ qemu-img measure -O qcow2 --size 5G
  required bytes: 327680
  fully allocated bytes: 1074069504

Stefan Hajnoczi (8):
  block: add bdrv_measure() API
  raw-format: add bdrv_measure() support
  qcow2: extract preallocation calculation function
  qcow2: extract image creation option parsing
  qcow2: add bdrv_measure() support
  qemu-img: add measure subcommand
  qemu-iotests: support per-format golden output files
  iotests: add test 178 for qemu-img measure

 qapi/block-core.json             |  19 +++
 include/block/block.h            |   4 +
 include/block/block_int.h        |   2 +
 block.c                          |  33 +++++
 block/qcow2.c                    | 296 ++++++++++++++++++++++++++-------------
 block/raw-format.c               |  22 +++
 qemu-img.c                       | 213 ++++++++++++++++++++++++++++
 qemu-img-cmds.hx                 |   9 ++
 tests/qemu-iotests/178           |  75 ++++++++++
 tests/qemu-iotests/178.out.qcow2 |  33 +++++
 tests/qemu-iotests/178.out.raw   |  33 +++++
 tests/qemu-iotests/check         |   5 +
 tests/qemu-iotests/group         |   1 +
 13 files changed, 651 insertions(+), 94 deletions(-)
 create mode 100755 tests/qemu-iotests/178
 create mode 100644 tests/qemu-iotests/178.out.qcow2
 create mode 100644 tests/qemu-iotests/178.out.raw

-- 
2.9.3


Re: [Qemu-devel] [RFC v2 0/8] qemu-img: add measure sub-command
Posted by Nir Soffer 7 years, 1 month ago
On Wed, Mar 15, 2017 at 11:29 AM Stefan Hajnoczi <stefanha@redhat.com>
wrote:

> RFCv2:
>  * Publishing RFC again to discuss the new user-visible interfaces.  Code
> has
>    changed quite a bit, I have not kept any Reviewed-by tags.
>  * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir]
>  * Report both "required bytes" and "fully allocated bytes" to handle the
> empty
>    image file and prealloc use cases [Nir and Dan]
>  * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto]
>  * Rename "err" label "out" in qemu-img-cmds.c [Nir]
>  * Add basic qcow2 support, doesn't support qemu-img convert from existing
> files yet
>
> RFCv1:
>  * Publishing patch series with just raw support, no qcow2 yet.  Please
> review
>    the command-line interface and let me know if you are happy with this
>    approach.
>
> Users and management tools sometimes need to know the size required for a
> new
> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of
> time.
> Image formats like qcow2 have non-trivial metadata that makes it hard to
> estimate the exact size without knowledge of file format internals.
>
> This patch series introduces a new qemu-img sub-command that calculates the
> required size for both image creation and conversion scenarios.


> The conversion scenario is:
>
>   $ qemu-img measure -f raw -O qcow2 input.img
>   required bytes: 1327680
>   fully allocated bytes: 1074069504


Other commands like qemu-img info are using "size" instead of "bytes", I
guess
we like to keep the term "size" for consistency.

Here an existing image file is taken and the output includes the space
> required
> for data from the input image file.
>
> The creation scenario is:
>
>   $ qemu-img measure -O qcow2 --size 5G
>   required bytes: 327680
>   fully allocated bytes: 1074069504
>

There is a third scenario that you may want to describe
here, extending a block device when it becomes full. In this case
we want to limit the block device to the fully allocated size, including
additional size needed for bitmaps. Currently ovirt is using virtual_size *
1.1
for this limit, which seems to work for images without additional bitmaps.


> Stefan Hajnoczi (8):
>   block: add bdrv_measure() API
>   raw-format: add bdrv_measure() support
>   qcow2: extract preallocation calculation function
>   qcow2: extract image creation option parsing
>   qcow2: add bdrv_measure() support
>   qemu-img: add measure subcommand
>   qemu-iotests: support per-format golden output files
>   iotests: add test 178 for qemu-img measure
>
>  qapi/block-core.json             |  19 +++
>  include/block/block.h            |   4 +
>  include/block/block_int.h        |   2 +
>  block.c                          |  33 +++++
>  block/qcow2.c                    | 296
> ++++++++++++++++++++++++++-------------
>  block/raw-format.c               |  22 +++
>  qemu-img.c                       | 213 ++++++++++++++++++++++++++++
>  qemu-img-cmds.hx                 |   9 ++
>  tests/qemu-iotests/178           |  75 ++++++++++
>  tests/qemu-iotests/178.out.qcow2 |  33 +++++
>  tests/qemu-iotests/178.out.raw   |  33 +++++
>  tests/qemu-iotests/check         |   5 +
>  tests/qemu-iotests/group         |   1 +
>  13 files changed, 651 insertions(+), 94 deletions(-)
>  create mode 100755 tests/qemu-iotests/178
>  create mode 100644 tests/qemu-iotests/178.out.qcow2
>  create mode 100644 tests/qemu-iotests/178.out.raw
>
> --
> 2.9.3
>
>
Re: [Qemu-devel] [RFC v2 0/8] qemu-img: add measure sub-command
Posted by Stefan Hajnoczi 7 years, 1 month ago
On Fri, Mar 17, 2017 at 11:45:20PM +0000, Nir Soffer wrote:
> On Wed, Mar 15, 2017 at 11:29 AM Stefan Hajnoczi <stefanha@redhat.com>
> wrote:
> 
> > RFCv2:
> >  * Publishing RFC again to discuss the new user-visible interfaces.  Code
> > has
> >    changed quite a bit, I have not kept any Reviewed-by tags.
> >  * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir]
> >  * Report both "required bytes" and "fully allocated bytes" to handle the
> > empty
> >    image file and prealloc use cases [Nir and Dan]
> >  * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto]
> >  * Rename "err" label "out" in qemu-img-cmds.c [Nir]
> >  * Add basic qcow2 support, doesn't support qemu-img convert from existing
> > files yet
> >
> > RFCv1:
> >  * Publishing patch series with just raw support, no qcow2 yet.  Please
> > review
> >    the command-line interface and let me know if you are happy with this
> >    approach.
> >
> > Users and management tools sometimes need to know the size required for a
> > new
> > disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of
> > time.
> > Image formats like qcow2 have non-trivial metadata that makes it hard to
> > estimate the exact size without knowledge of file format internals.
> >
> > This patch series introduces a new qemu-img sub-command that calculates the
> > required size for both image creation and conversion scenarios.
> 
> 
> > The conversion scenario is:
> >
> >   $ qemu-img measure -f raw -O qcow2 input.img
> >   required bytes: 1327680
> >   fully allocated bytes: 1074069504
> 
> 
> Other commands like qemu-img info are using "size" instead of "bytes", I
> guess
> we like to keep the term "size" for consistency.

Will fix.

> Here an existing image file is taken and the output includes the space
> > required
> > for data from the input image file.
> >
> > The creation scenario is:
> >
> >   $ qemu-img measure -O qcow2 --size 5G
> >   required bytes: 327680
> >   fully allocated bytes: 1074069504
> >
> 
> There is a third scenario that you may want to describe
> here, extending a block device when it becomes full. In this case
> we want to limit the block device to the fully allocated size, including
> additional size needed for bitmaps. Currently ovirt is using virtual_size *
> 1.1
> for this limit, which seems to work for images without additional bitmaps.

Please elaborate, I'm not sure which use case you mean:

1. The user wishes to grow the virtual disk size because the guest has
   filled up the disk.

2. The block device was thinly provisioned and needs to grow so that the
   guest can write more data.  In this case the virtual disk size stays
   constant.

Case #1 is the qemu-img convert case.  We have an existing file with
some data and the new image will have a larger virtual disk size.
(Nevermind that no actual new image needs to be created.)

Regarding case #2, the management tool doesn't need qemu-img measure.
It just increments the block device size by an arbitrary amount (e.g. 1
GB) whenever the write threshold is reached.

Stefan