[Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand

Stefan Hajnoczi posted 4 patches 7 years, 1 month ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20170303135150.12145-1-stefanha@redhat.com
Test checkpatch passed
Test docker passed
include/block/block.h      |   2 +
include/block/block_int.h  |   2 +
block.c                    |  37 +++++++++
block/raw-format.c         |  16 ++++
qemu-img.c                 | 196 +++++++++++++++++++++++++++++++++++++++++++++
qemu-img-cmds.hx           |   6 ++
tests/qemu-iotests/178     |  75 +++++++++++++++++
tests/qemu-iotests/178.out |  25 ++++++
tests/qemu-iotests/group   |   1 +
9 files changed, 360 insertions(+)
create mode 100755 tests/qemu-iotests/178
create mode 100644 tests/qemu-iotests/178.out
[Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Stefan Hajnoczi 7 years, 1 month ago
RFCv1:
 * Publishing patch series with just raw support, no qcow2 yet.  Please review
   the command-line interface and let me know if you are happy with this
   approach.

Users and management tools sometimes need to know the size required for a new
disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
Image formats like qcow2 have non-trivial metadata that makes it hard to
estimate the exact size without knowledge of file format internals.

This patch series introduces a new qemu-img subcommand that calculates the
required size for both image creation and conversion scenarios.

The conversion scenario is:

  $ qemu-img max-size -f raw -O qcow2 input.img
  107374184448

Here an existing image file is taken and the output includes the space required
for data from the input image file.

The creation scenario is:

  $ qemu-img max-size -O qcow2 --size 5G
  196688

Stefan Hajnoczi (4):
  block: add bdrv_max_size() API
  raw-format: add bdrv_max_size() support
  qemu-img: add max-size subcommand
  iotests: add test 178 for qemu-img max-size

 include/block/block.h      |   2 +
 include/block/block_int.h  |   2 +
 block.c                    |  37 +++++++++
 block/raw-format.c         |  16 ++++
 qemu-img.c                 | 196 +++++++++++++++++++++++++++++++++++++++++++++
 qemu-img-cmds.hx           |   6 ++
 tests/qemu-iotests/178     |  75 +++++++++++++++++
 tests/qemu-iotests/178.out |  25 ++++++
 tests/qemu-iotests/group   |   1 +
 9 files changed, 360 insertions(+)
 create mode 100755 tests/qemu-iotests/178
 create mode 100644 tests/qemu-iotests/178.out

-- 
2.9.3


Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by John Snow 7 years, 1 month ago

On 03/03/2017 08:51 AM, Stefan Hajnoczi wrote:
> RFCv1:
>  * Publishing patch series with just raw support, no qcow2 yet.  Please review
>    the command-line interface and let me know if you are happy with this
>    approach.
> 
> Users and management tools sometimes need to know the size required for a new
> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
> Image formats like qcow2 have non-trivial metadata that makes it hard to
> estimate the exact size without knowledge of file format internals.
> 
> This patch series introduces a new qemu-img subcommand that calculates the
> required size for both image creation and conversion scenarios.
> 
> The conversion scenario is:
> 
>   $ qemu-img max-size -f raw -O qcow2 input.img
>   107374184448
> 
> Here an existing image file is taken and the output includes the space required
> for data from the input image file.
> 
> The creation scenario is:
> 
>   $ qemu-img max-size -O qcow2 --size 5G
>   196688
> 

It looks sane to me, implementation looks clean enough.
Thanks!

> Stefan Hajnoczi (4):
>   block: add bdrv_max_size() API
>   raw-format: add bdrv_max_size() support
>   qemu-img: add max-size subcommand
>   iotests: add test 178 for qemu-img max-size
> 
>  include/block/block.h      |   2 +
>  include/block/block_int.h  |   2 +
>  block.c                    |  37 +++++++++
>  block/raw-format.c         |  16 ++++
>  qemu-img.c                 | 196 +++++++++++++++++++++++++++++++++++++++++++++
>  qemu-img-cmds.hx           |   6 ++
>  tests/qemu-iotests/178     |  75 +++++++++++++++++
>  tests/qemu-iotests/178.out |  25 ++++++
>  tests/qemu-iotests/group   |   1 +
>  9 files changed, 360 insertions(+)
>  create mode 100755 tests/qemu-iotests/178
>  create mode 100644 tests/qemu-iotests/178.out
> 

Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Nir Soffer 7 years, 1 month ago
On Fri, Mar 3, 2017 at 3:51 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> RFCv1:
>  * Publishing patch series with just raw support, no qcow2 yet.  Please review
>    the command-line interface and let me know if you are happy with this
>    approach.
>
> Users and management tools sometimes need to know the size required for a new
> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
> Image formats like qcow2 have non-trivial metadata that makes it hard to
> estimate the exact size without knowledge of file format internals.
>
> This patch series introduces a new qemu-img subcommand that calculates the
> required size for both image creation and conversion scenarios.
>
> The conversion scenario is:
>
>   $ qemu-img max-size -f raw -O qcow2 input.img
>   107374184448

Isn't this the minimal size required to convert input.img?

>
> Here an existing image file is taken and the output includes the space required
> for data from the input image file.
>
> The creation scenario is:
>
>   $ qemu-img max-size -O qcow2 --size 5G
>   196688

Again, this is the minimal size.

So maybe use min-size?

Or:

    qemu-img measure -f raw -O qcow2 input.img

Works nicely with other verbs like create, convert, check.

Now about the return value, do we want to return both the minimum size
and the maximum size?

For ovirt use case, we currently calculate the maximum size by multiplying
by 1.1. We use this when doing automatic extending of ovirt thin provisioned
disk. We start with 1G lv, and extend it each time it becomes full, stopping
when we reach virtual size * 1.1. Using more accurate calculation instead
can be nicer.

So we can retrun:

{
    "min-size": 196688,
    "max-size": 5905580032
}

Anyway thanks for working on this!

>
> Stefan Hajnoczi (4):
>   block: add bdrv_max_size() API
>   raw-format: add bdrv_max_size() support
>   qemu-img: add max-size subcommand
>   iotests: add test 178 for qemu-img max-size
>
>  include/block/block.h      |   2 +
>  include/block/block_int.h  |   2 +
>  block.c                    |  37 +++++++++
>  block/raw-format.c         |  16 ++++
>  qemu-img.c                 | 196 +++++++++++++++++++++++++++++++++++++++++++++
>  qemu-img-cmds.hx           |   6 ++
>  tests/qemu-iotests/178     |  75 +++++++++++++++++
>  tests/qemu-iotests/178.out |  25 ++++++
>  tests/qemu-iotests/group   |   1 +
>  9 files changed, 360 insertions(+)
>  create mode 100755 tests/qemu-iotests/178
>  create mode 100644 tests/qemu-iotests/178.out
>
> --
> 2.9.3
>

Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by John Snow 7 years, 1 month ago

On 03/03/2017 04:38 PM, Nir Soffer wrote:
> On Fri, Mar 3, 2017 at 3:51 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>
>> RFCv1:
>>  * Publishing patch series with just raw support, no qcow2 yet.  Please review
>>    the command-line interface and let me know if you are happy with this
>>    approach.
>>
>> Users and management tools sometimes need to know the size required for a new
>> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
>> Image formats like qcow2 have non-trivial metadata that makes it hard to
>> estimate the exact size without knowledge of file format internals.
>>
>> This patch series introduces a new qemu-img subcommand that calculates the
>> required size for both image creation and conversion scenarios.
>>
>> The conversion scenario is:
>>
>>   $ qemu-img max-size -f raw -O qcow2 input.img
>>   107374184448
> 
> Isn't this the minimal size required to convert input.img?
> 

It's an upper bound for the property being measured, which is current
allocation size, not maximum potential size after growth.

>>
>> Here an existing image file is taken and the output includes the space required
>> for data from the input image file.
>>
>> The creation scenario is:
>>
>>   $ qemu-img max-size -O qcow2 --size 5G
>>   196688
> 
> Again, this is the minimal size.
> 
> So maybe use min-size?
> 
> Or:
> 
>     qemu-img measure -f raw -O qcow2 input.img
> 
> Works nicely with other verbs like create, convert, check.
> 

Measure what? This is strictly less descriptive even if "max-size" isn't
a verb.

> Now about the return value, do we want to return both the minimum size
> and the maximum size?
> 
> For ovirt use case, we currently calculate the maximum size by multiplying
> by 1.1. We use this when doing automatic extending of ovirt thin provisioned
> disk. We start with 1G lv, and extend it each time it becomes full, stopping
> when we reach virtual size * 1.1. Using more accurate calculation instead
> can be nicer.
> 
> So we can retrun:
> 
> {
>     "min-size": 196688,
>     "max-size": 5905580032
> }
> 
> Anyway thanks for working on this!
> 

It sounds like you want something different from what was intuited by
Maor Lipchuck. There are two things to estimate:

(A) An estimate of the possible size of an image after conversion to a
different format, and
(B) An estimate of the possible size after full allocation.

I got the sense that Maor was asking for (A), but perhaps I am wrong
about that. However, both are "maximums" in different senses.

--js

>>
>> Stefan Hajnoczi (4):
>>   block: add bdrv_max_size() API
>>   raw-format: add bdrv_max_size() support
>>   qemu-img: add max-size subcommand
>>   iotests: add test 178 for qemu-img max-size
>>
>>  include/block/block.h      |   2 +
>>  include/block/block_int.h  |   2 +
>>  block.c                    |  37 +++++++++
>>  block/raw-format.c         |  16 ++++
>>  qemu-img.c                 | 196 +++++++++++++++++++++++++++++++++++++++++++++
>>  qemu-img-cmds.hx           |   6 ++
>>  tests/qemu-iotests/178     |  75 +++++++++++++++++
>>  tests/qemu-iotests/178.out |  25 ++++++
>>  tests/qemu-iotests/group   |   1 +
>>  9 files changed, 360 insertions(+)
>>  create mode 100755 tests/qemu-iotests/178
>>  create mode 100644 tests/qemu-iotests/178.out
>>
>> --
>> 2.9.3
>>

Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Nir Soffer 7 years, 1 month ago
On Sat, Mar 4, 2017 at 12:02 AM, John Snow <jsnow@redhat.com> wrote:
>
>
> On 03/03/2017 04:38 PM, Nir Soffer wrote:
>> On Fri, Mar 3, 2017 at 3:51 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>
>>> RFCv1:
>>>  * Publishing patch series with just raw support, no qcow2 yet.  Please review
>>>    the command-line interface and let me know if you are happy with this
>>>    approach.
>>>
>>> Users and management tools sometimes need to know the size required for a new
>>> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
>>> Image formats like qcow2 have non-trivial metadata that makes it hard to
>>> estimate the exact size without knowledge of file format internals.
>>>
>>> This patch series introduces a new qemu-img subcommand that calculates the
>>> required size for both image creation and conversion scenarios.
>>>
>>> The conversion scenario is:
>>>
>>>   $ qemu-img max-size -f raw -O qcow2 input.img
>>>   107374184448
>>
>> Isn't this the minimal size required to convert input.img?
>>
>
> It's an upper bound for the property being measured, which is current
> allocation size, not maximum potential size after growth.

From my point of view, this is the minimal size you must allocate if you
want to convert the image to logical volume.

>
>>>
>>> Here an existing image file is taken and the output includes the space required
>>> for data from the input image file.
>>>
>>> The creation scenario is:
>>>
>>>   $ qemu-img max-size -O qcow2 --size 5G
>>>   196688
>>
>> Again, this is the minimal size.
>>
>> So maybe use min-size?
>>
>> Or:
>>
>>     qemu-img measure -f raw -O qcow2 input.img
>>
>> Works nicely with other verbs like create, convert, check.
>>
>
> Measure what? This is strictly less descriptive even if "max-size" isn't
> a verb.

measure-size?

>> Now about the return value, do we want to return both the minimum size
>> and the maximum size?
>>
>> For ovirt use case, we currently calculate the maximum size by multiplying
>> by 1.1. We use this when doing automatic extending of ovirt thin provisioned
>> disk. We start with 1G lv, and extend it each time it becomes full, stopping
>> when we reach virtual size * 1.1. Using more accurate calculation instead
>> can be nicer.
>>
>> So we can retrun:
>>
>> {
>>     "min-size": 196688,
>>     "max-size": 5905580032
>> }
>>
>> Anyway thanks for working on this!
>>
>
> It sounds like you want something different from what was intuited by
> Maor Lipchuck. There are two things to estimate:
>
> (A) An estimate of the possible size of an image after conversion to a
> different format, and
> (B) An estimate of the possible size after full allocation.
>
> I got the sense that Maor was asking for (A), but perhaps I am wrong
> about that. However, both are "maximums" in different senses.

Both are minimum when you have to allocate the space.

Maor ask about A because he is working on fixing allocation when
converting existing files, but we also have other use cases like B.

Nir

>
> --js
>
>>>
>>> Stefan Hajnoczi (4):
>>>   block: add bdrv_max_size() API
>>>   raw-format: add bdrv_max_size() support
>>>   qemu-img: add max-size subcommand
>>>   iotests: add test 178 for qemu-img max-size
>>>
>>>  include/block/block.h      |   2 +
>>>  include/block/block_int.h  |   2 +
>>>  block.c                    |  37 +++++++++
>>>  block/raw-format.c         |  16 ++++
>>>  qemu-img.c                 | 196 +++++++++++++++++++++++++++++++++++++++++++++
>>>  qemu-img-cmds.hx           |   6 ++
>>>  tests/qemu-iotests/178     |  75 +++++++++++++++++
>>>  tests/qemu-iotests/178.out |  25 ++++++
>>>  tests/qemu-iotests/group   |   1 +
>>>  9 files changed, 360 insertions(+)
>>>  create mode 100755 tests/qemu-iotests/178
>>>  create mode 100644 tests/qemu-iotests/178.out
>>>
>>> --
>>> 2.9.3
>>>
>

Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Nir Soffer 7 years, 1 month ago
On Sat, Mar 4, 2017 at 12:15 AM, Nir Soffer <nirsof@gmail.com> wrote:
> On Sat, Mar 4, 2017 at 12:02 AM, John Snow <jsnow@redhat.com> wrote:
>>
>>
>> On 03/03/2017 04:38 PM, Nir Soffer wrote:
>>> On Fri, Mar 3, 2017 at 3:51 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>>
>>>> RFCv1:
>>>>  * Publishing patch series with just raw support, no qcow2 yet.  Please review
>>>>    the command-line interface and let me know if you are happy with this
>>>>    approach.
>>>>
>>>> Users and management tools sometimes need to know the size required for a new
>>>> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
>>>> Image formats like qcow2 have non-trivial metadata that makes it hard to
>>>> estimate the exact size without knowledge of file format internals.
>>>>
>>>> This patch series introduces a new qemu-img subcommand that calculates the
>>>> required size for both image creation and conversion scenarios.
>>>>
>>>> The conversion scenario is:
>>>>
>>>>   $ qemu-img max-size -f raw -O qcow2 input.img
>>>>   107374184448
>>>
>>> Isn't this the minimal size required to convert input.img?
>>>
>>
>> It's an upper bound for the property being measured, which is current
>> allocation size, not maximum potential size after growth.
>
> From my point of view, this is the minimal size you must allocate if you
> want to convert the image to logical volume.

Thinking more about this, max-size is correct for this use case, the
maximum size of the image, used as the minimal allocation.

>
>>
>>>>
>>>> Here an existing image file is taken and the output includes the space required
>>>> for data from the input image file.
>>>>
>>>> The creation scenario is:
>>>>
>>>>   $ qemu-img max-size -O qcow2 --size 5G
>>>>   196688
>>>
>>> Again, this is the minimal size.
>>>
>>> So maybe use min-size?
>>>
>>> Or:
>>>
>>>     qemu-img measure -f raw -O qcow2 input.img
>>>
>>> Works nicely with other verbs like create, convert, check.
>>>
>>
>> Measure what? This is strictly less descriptive even if "max-size" isn't
>> a verb.
>
> measure-size?
>
>>> Now about the return value, do we want to return both the minimum size
>>> and the maximum size?
>>>
>>> For ovirt use case, we currently calculate the maximum size by multiplying
>>> by 1.1. We use this when doing automatic extending of ovirt thin provisioned
>>> disk. We start with 1G lv, and extend it each time it becomes full, stopping
>>> when we reach virtual size * 1.1. Using more accurate calculation instead
>>> can be nicer.
>>>
>>> So we can retrun:
>>>
>>> {
>>>     "min-size": 196688,
>>>     "max-size": 5905580032
>>> }
>>>
>>> Anyway thanks for working on this!
>>>
>>
>> It sounds like you want something different from what was intuited by
>> Maor Lipchuck. There are two things to estimate:
>>
>> (A) An estimate of the possible size of an image after conversion to a
>> different format, and
>> (B) An estimate of the possible size after full allocation.
>>
>> I got the sense that Maor was asking for (A), but perhaps I am wrong
>> about that. However, both are "maximums" in different senses.
>
> Both are minimum when you have to allocate the space.
>
> Maor ask about A because he is working on fixing allocation when
> converting existing files, but we also have other use cases like B.
>
> Nir
>
>>
>> --js
>>
>>>>
>>>> Stefan Hajnoczi (4):
>>>>   block: add bdrv_max_size() API
>>>>   raw-format: add bdrv_max_size() support
>>>>   qemu-img: add max-size subcommand
>>>>   iotests: add test 178 for qemu-img max-size
>>>>
>>>>  include/block/block.h      |   2 +
>>>>  include/block/block_int.h  |   2 +
>>>>  block.c                    |  37 +++++++++
>>>>  block/raw-format.c         |  16 ++++
>>>>  qemu-img.c                 | 196 +++++++++++++++++++++++++++++++++++++++++++++
>>>>  qemu-img-cmds.hx           |   6 ++
>>>>  tests/qemu-iotests/178     |  75 +++++++++++++++++
>>>>  tests/qemu-iotests/178.out |  25 ++++++
>>>>  tests/qemu-iotests/group   |   1 +
>>>>  9 files changed, 360 insertions(+)
>>>>  create mode 100755 tests/qemu-iotests/178
>>>>  create mode 100644 tests/qemu-iotests/178.out
>>>>
>>>> --
>>>> 2.9.3
>>>>
>>

Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Stefan Hajnoczi 7 years, 1 month ago
On Sat, Mar 04, 2017 at 12:15:00AM +0200, Nir Soffer wrote:
> On Sat, Mar 4, 2017 at 12:02 AM, John Snow <jsnow@redhat.com> wrote:
> >
> >
> > On 03/03/2017 04:38 PM, Nir Soffer wrote:
> >> On Fri, Mar 3, 2017 at 3:51 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >>>
> >>> RFCv1:
> >>>  * Publishing patch series with just raw support, no qcow2 yet.  Please review
> >>>    the command-line interface and let me know if you are happy with this
> >>>    approach.
> >>>
> >>> Users and management tools sometimes need to know the size required for a new
> >>> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
> >>> Image formats like qcow2 have non-trivial metadata that makes it hard to
> >>> estimate the exact size without knowledge of file format internals.
> >>>
> >>> This patch series introduces a new qemu-img subcommand that calculates the
> >>> required size for both image creation and conversion scenarios.
> >>>
> >>> The conversion scenario is:
> >>>
> >>>   $ qemu-img max-size -f raw -O qcow2 input.img
> >>>   107374184448
> >>
> >> Isn't this the minimal size required to convert input.img?
> >>
> >
> > It's an upper bound for the property being measured, which is current
> > allocation size, not maximum potential size after growth.
> 
> From my point of view, this is the minimal size you must allocate if you
> want to convert the image to logical volume.
> 
> >
> >>>
> >>> Here an existing image file is taken and the output includes the space required
> >>> for data from the input image file.
> >>>
> >>> The creation scenario is:
> >>>
> >>>   $ qemu-img max-size -O qcow2 --size 5G
> >>>   196688
> >>
> >> Again, this is the minimal size.
> >>
> >> So maybe use min-size?
> >>
> >> Or:
> >>
> >>     qemu-img measure -f raw -O qcow2 input.img
> >>
> >> Works nicely with other verbs like create, convert, check.
> >>
> >
> > Measure what? This is strictly less descriptive even if "max-size" isn't
> > a verb.
> 
> measure-size?

You're right, the sub-command should be a verb.

> >> Now about the return value, do we want to return both the minimum size
> >> and the maximum size?
> >>
> >> For ovirt use case, we currently calculate the maximum size by multiplying
> >> by 1.1. We use this when doing automatic extending of ovirt thin provisioned
> >> disk. We start with 1G lv, and extend it each time it becomes full, stopping
> >> when we reach virtual size * 1.1. Using more accurate calculation instead
> >> can be nicer.
> >>
> >> So we can retrun:
> >>
> >> {
> >>     "min-size": 196688,
> >>     "max-size": 5905580032
> >> }
> >>
> >> Anyway thanks for working on this!
> >>
> >
> > It sounds like you want something different from what was intuited by
> > Maor Lipchuck. There are two things to estimate:
> >
> > (A) An estimate of the possible size of an image after conversion to a
> > different format, and
> > (B) An estimate of the possible size after full allocation.
> >
> > I got the sense that Maor was asking for (A), but perhaps I am wrong
> > about that. However, both are "maximums" in different senses.
> 
> Both are minimum when you have to allocate the space.
> 
> Maor ask about A because he is working on fixing allocation when
> converting existing files, but we also have other use cases like B.

Daniel Berrange is also interested in the size of a fully populated
image file.  I'll expand the patch series to report both sizes.

Stefan
Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Daniel P. Berrange 7 years, 1 month ago
On Fri, Mar 03, 2017 at 09:51:46PM +0800, Stefan Hajnoczi wrote:
> RFCv1:
>  * Publishing patch series with just raw support, no qcow2 yet.  Please review
>    the command-line interface and let me know if you are happy with this
>    approach.
> 
> Users and management tools sometimes need to know the size required for a new
> disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time.
> Image formats like qcow2 have non-trivial metadata that makes it hard to
> estimate the exact size without knowledge of file format internals.
> 
> This patch series introduces a new qemu-img subcommand that calculates the
> required size for both image creation and conversion scenarios.
> 
> The conversion scenario is:
> 
>   $ qemu-img max-size -f raw -O qcow2 input.img
>   107374184448
> 
> Here an existing image file is taken and the output includes the space required
> for data from the input image file.
> 
> The creation scenario is:
> 
>   $ qemu-img max-size -O qcow2 --size 5G
>   196688

Hmm, so that appears to be indicating the amount of physical space that
a qcow2 image would take up before any data has been written to it.

That's not actually what I was thinking. I would like to know the maximum
possible physical space that a 5G qcow2 image would take up once data is
written to every sector. Obviously this is impossible to say if you allow
for internal snapshots, but I think it is fine to say that we ignore
internal snapshots for purposes of this command.

IOW, I would expect  something like:

$ qemu-img max-size -O qcow2 --size 5G
5905580032

(I just blindly assumed qcow2 metadata is a 10% overhead for sake of
illustrating the number here)


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Alberto Garcia 7 years, 1 month ago
On Tue 07 Mar 2017 11:36:54 AM CET, Daniel P. Berrange wrote:
>> The creation scenario is:
>> 
>>   $ qemu-img max-size -O qcow2 --size 5G
>>   196688
>
> Hmm, so that appears to be indicating the amount of physical space
> that a qcow2 image would take up before any data has been written to
> it.
>
> That's not actually what I was thinking. I would like to know the
> maximum possible physical space that a 5G qcow2 image would take up
> once data is written to every sector. Obviously this is impossible to
> say if you allow for internal snapshots, but I think it is fine to say
> that we ignore internal snapshots for purposes of this command.

We have clearly two different use cases here, although I wonder how
useful the one that you are describing is. After all the maximum size of
a fully allocated imaged is always going to be the virtual size plus a
small overhead for the metadata. I haven't made the numbers for all
cases, but I'll take the risk and say it's always going to be really
small (the 10% you use to illustrate your point is way too much).

A fully allocated 1TB qcow2 image needs less than 200MB of metadata
(that's 0.02% of the total size). If we reduce the cluster size to its
minimum (512 bytes) then it's around 20GB (still ~2% of the total size).

Computing the size that you need to convert the data that is currently
allocated is a different matter, because it depends a lot on the
scenario: whether you are using preallocation, or you have backing
images, etc.

Berto

Re: [Qemu-devel] [RFC 0/4] qemu-img: add max-size subcommand
Posted by Daniel P. Berrange 7 years, 1 month ago
On Tue, Mar 07, 2017 at 01:11:44PM +0100, Alberto Garcia wrote:
> On Tue 07 Mar 2017 11:36:54 AM CET, Daniel P. Berrange wrote:
> >> The creation scenario is:
> >> 
> >>   $ qemu-img max-size -O qcow2 --size 5G
> >>   196688
> >
> > Hmm, so that appears to be indicating the amount of physical space
> > that a qcow2 image would take up before any data has been written to
> > it.
> >
> > That's not actually what I was thinking. I would like to know the
> > maximum possible physical space that a 5G qcow2 image would take up
> > once data is written to every sector. Obviously this is impossible to
> > say if you allow for internal snapshots, but I think it is fine to say
> > that we ignore internal snapshots for purposes of this command.
> 
> We have clearly two different use cases here, although I wonder how
> useful the one that you are describing is. After all the maximum size of
> a fully allocated imaged is always going to be the virtual size plus a
> small overhead for the metadata. I haven't made the numbers for all
> cases, but I'll take the risk and say it's always going to be really
> small (the 10% you use to illustrate your point is way too much).
> 
> A fully allocated 1TB qcow2 image needs less than 200MB of metadata
> (that's 0.02% of the total size). If we reduce the cluster size to its
> minimum (512 bytes) then it's around 20GB (still ~2% of the total size).

I know of several places where this info is useful. OpenStack stores
images in a filesystem. It knows what space is available in the filesystem
and wants to avoid creating qcow2 file that would overcommit that available
space. Knowing the qcow2 overhead would allow it to compute that without
having to resort to over-cautious guesswork which needlessly leaves
space unused. Similarly when creating a file that has a backing file,
qcow2 does not allow use of preallocation, and openstack thus uses
fallocate after qemu has created the file in order to reserve the
space needed to allow for maximum possible growth of the qcow2 file.
oVirt stores qcow2 images inside block devices and it would be useful
to know maximum possible size in order to correctly size the block
device without having to resort to guesswork.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|