[libvirt] [PATCH 0/2] Add 'l2-cache-size' property to specify maximum size of the L2 table cache for qcow2 image

Lin Ma posted 2 patches 6 years, 5 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/20171115015858.22158-1-lma@suse.com
docs/formatdomain.html.in                          | 11 +++++++
docs/schemas/domaincommon.rng                      |  5 ++++
src/conf/domain_conf.c                             | 11 +++++++
src/conf/domain_conf.h                             |  1 +
src/qemu/qemu_capabilities.c                       |  5 ++++
src/qemu/qemu_capabilities.h                       |  1 +
src/qemu/qemu_command.c                            | 18 ++++++++++++
.../caps_2.10.0-gicv2.aarch64.xml                  |  1 +
.../caps_2.10.0-gicv3.aarch64.xml                  |  1 +
tests/qemucapabilitiesdata/caps_2.10.0.ppc64.xml   |  1 +
tests/qemucapabilitiesdata/caps_2.10.0.s390x.xml   |  1 +
tests/qemucapabilitiesdata/caps_2.10.0.x86_64.xml  |  1 +
tests/qemucapabilitiesdata/caps_2.5.0.x86_64.xml   |  1 +
.../caps_2.6.0-gicv2.aarch64.xml                   |  1 +
.../caps_2.6.0-gicv3.aarch64.xml                   |  1 +
tests/qemucapabilitiesdata/caps_2.6.0.ppc64.xml    |  1 +
tests/qemucapabilitiesdata/caps_2.6.0.x86_64.xml   |  1 +
tests/qemucapabilitiesdata/caps_2.7.0.s390x.xml    |  1 +
tests/qemucapabilitiesdata/caps_2.7.0.x86_64.xml   |  1 +
tests/qemucapabilitiesdata/caps_2.8.0.s390x.xml    |  1 +
tests/qemucapabilitiesdata/caps_2.8.0.x86_64.xml   |  1 +
tests/qemucapabilitiesdata/caps_2.9.0.ppc64.xml    |  1 +
tests/qemucapabilitiesdata/caps_2.9.0.s390x.xml    |  1 +
tests/qemucapabilitiesdata/caps_2.9.0.x86_64.xml   |  1 +
.../qemuxml2argv-disk-drive-l2-cache-size.args     | 24 +++++++++++++++
.../qemuxml2argv-disk-drive-l2-cache-size.xml      | 34 ++++++++++++++++++++++
tests/qemuxml2argvtest.c                           |  2 ++
.../qemuxml2xmlout-disk-drive-l2-cache-size.xml    | 34 ++++++++++++++++++++++
tests/qemuxml2xmltest.c                            |  1 +
29 files changed, 164 insertions(+)
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-drive-l2-cache-size.args
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-drive-l2-cache-size.xml
create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-l2-cache-size.xml
[libvirt] [PATCH 0/2] Add 'l2-cache-size' property to specify maximum size of the L2 table cache for qcow2 image
Posted by Lin Ma 6 years, 5 months ago
Lin Ma (2):
  caps: Add capability for maximum size of the qcow2 L2 table cache
  qemu: Add support for 'l2-cache-size' property of qcow2

 docs/formatdomain.html.in                          | 11 +++++++
 docs/schemas/domaincommon.rng                      |  5 ++++
 src/conf/domain_conf.c                             | 11 +++++++
 src/conf/domain_conf.h                             |  1 +
 src/qemu/qemu_capabilities.c                       |  5 ++++
 src/qemu/qemu_capabilities.h                       |  1 +
 src/qemu/qemu_command.c                            | 18 ++++++++++++
 .../caps_2.10.0-gicv2.aarch64.xml                  |  1 +
 .../caps_2.10.0-gicv3.aarch64.xml                  |  1 +
 tests/qemucapabilitiesdata/caps_2.10.0.ppc64.xml   |  1 +
 tests/qemucapabilitiesdata/caps_2.10.0.s390x.xml   |  1 +
 tests/qemucapabilitiesdata/caps_2.10.0.x86_64.xml  |  1 +
 tests/qemucapabilitiesdata/caps_2.5.0.x86_64.xml   |  1 +
 .../caps_2.6.0-gicv2.aarch64.xml                   |  1 +
 .../caps_2.6.0-gicv3.aarch64.xml                   |  1 +
 tests/qemucapabilitiesdata/caps_2.6.0.ppc64.xml    |  1 +
 tests/qemucapabilitiesdata/caps_2.6.0.x86_64.xml   |  1 +
 tests/qemucapabilitiesdata/caps_2.7.0.s390x.xml    |  1 +
 tests/qemucapabilitiesdata/caps_2.7.0.x86_64.xml   |  1 +
 tests/qemucapabilitiesdata/caps_2.8.0.s390x.xml    |  1 +
 tests/qemucapabilitiesdata/caps_2.8.0.x86_64.xml   |  1 +
 tests/qemucapabilitiesdata/caps_2.9.0.ppc64.xml    |  1 +
 tests/qemucapabilitiesdata/caps_2.9.0.s390x.xml    |  1 +
 tests/qemucapabilitiesdata/caps_2.9.0.x86_64.xml   |  1 +
 .../qemuxml2argv-disk-drive-l2-cache-size.args     | 24 +++++++++++++++
 .../qemuxml2argv-disk-drive-l2-cache-size.xml      | 34 ++++++++++++++++++++++
 tests/qemuxml2argvtest.c                           |  2 ++
 .../qemuxml2xmlout-disk-drive-l2-cache-size.xml    | 34 ++++++++++++++++++++++
 tests/qemuxml2xmltest.c                            |  1 +
 29 files changed, 164 insertions(+)
 create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-drive-l2-cache-size.args
 create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-drive-l2-cache-size.xml
 create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-l2-cache-size.xml

-- 
2.9.2

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/2] Add 'l2-cache-size' property to specify maximum size of the L2 table cache for qcow2 image
Posted by John Ferlan 6 years, 5 months ago

On 11/14/2017 08:58 PM, Lin Ma wrote:
> Lin Ma (2):
>   caps: Add capability for maximum size of the qcow2 L2 table cache
>   qemu: Add support for 'l2-cache-size' property of qcow2
> 

There's already something on list about l2 cache, see

https://www.redhat.com/archives/libvir-list/2017-September/msg00553.html

it adds 3 related variables. IIRC, the "hold up" is how best to describe
how to use it or provide the feature in such a way that it's
"understandable" for general consumption without making a "policy" that
generates certain values based on expected usage/needs for the client.

John

>  docs/formatdomain.html.in                          | 11 +++++++
>  docs/schemas/domaincommon.rng                      |  5 ++++
>  src/conf/domain_conf.c                             | 11 +++++++
>  src/conf/domain_conf.h                             |  1 +
>  src/qemu/qemu_capabilities.c                       |  5 ++++
>  src/qemu/qemu_capabilities.h                       |  1 +
>  src/qemu/qemu_command.c                            | 18 ++++++++++++
>  .../caps_2.10.0-gicv2.aarch64.xml                  |  1 +
>  .../caps_2.10.0-gicv3.aarch64.xml                  |  1 +
>  tests/qemucapabilitiesdata/caps_2.10.0.ppc64.xml   |  1 +
>  tests/qemucapabilitiesdata/caps_2.10.0.s390x.xml   |  1 +
>  tests/qemucapabilitiesdata/caps_2.10.0.x86_64.xml  |  1 +
>  tests/qemucapabilitiesdata/caps_2.5.0.x86_64.xml   |  1 +
>  .../caps_2.6.0-gicv2.aarch64.xml                   |  1 +
>  .../caps_2.6.0-gicv3.aarch64.xml                   |  1 +
>  tests/qemucapabilitiesdata/caps_2.6.0.ppc64.xml    |  1 +
>  tests/qemucapabilitiesdata/caps_2.6.0.x86_64.xml   |  1 +
>  tests/qemucapabilitiesdata/caps_2.7.0.s390x.xml    |  1 +
>  tests/qemucapabilitiesdata/caps_2.7.0.x86_64.xml   |  1 +
>  tests/qemucapabilitiesdata/caps_2.8.0.s390x.xml    |  1 +
>  tests/qemucapabilitiesdata/caps_2.8.0.x86_64.xml   |  1 +
>  tests/qemucapabilitiesdata/caps_2.9.0.ppc64.xml    |  1 +
>  tests/qemucapabilitiesdata/caps_2.9.0.s390x.xml    |  1 +
>  tests/qemucapabilitiesdata/caps_2.9.0.x86_64.xml   |  1 +
>  .../qemuxml2argv-disk-drive-l2-cache-size.args     | 24 +++++++++++++++
>  .../qemuxml2argv-disk-drive-l2-cache-size.xml      | 34 ++++++++++++++++++++++
>  tests/qemuxml2argvtest.c                           |  2 ++
>  .../qemuxml2xmlout-disk-drive-l2-cache-size.xml    | 34 ++++++++++++++++++++++
>  tests/qemuxml2xmltest.c                            |  1 +
>  29 files changed, 164 insertions(+)
>  create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-drive-l2-cache-size.args
>  create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-drive-l2-cache-size.xml
>  create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-l2-cache-size.xml
> 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/2] Add 'l2-cache-size' property to specify maximum size of the L2 table cache for qcow2 image
Posted by Kashyap Chamarthy 6 years, 5 months ago
[Cc: Alberto Garcia of QEMU, who added the 'l2-cache-size' upstream]

On Wed, Nov 15, 2017 at 05:54:13AM -0500, John Ferlan wrote:
> 
> 
> On 11/14/2017 08:58 PM, Lin Ma wrote:
> > Lin Ma (2):
> >   caps: Add capability for maximum size of the qcow2 L2 table cache
> >   qemu: Add support for 'l2-cache-size' property of qcow2
> > 
> 
> There's already something on list about l2 cache, see
> 
> https://www.redhat.com/archives/libvir-list/2017-September/msg00553.html
> 
> it adds 3 related variables. IIRC, the "hold up" is how best to describe
> how to use it or provide the feature in such a way that it's
> "understandable" for general consumption without making a "policy" that
> generates certain values based on expected usage/needs for the client.

I recall Alberto saying that there's a default value of QCOW2
'l2-cache-size', but it is (intentionally?) low.  And that the value
depends on the workload / use case: size of the image, the amount of I/O
you want to do, etc.

There's also an RFE for OpenStack Nova to expose this.  (But I don't
know if it makes sense to expose it at that level, yet.)

[...]

-- 
/kashyap

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/2] Add 'l2-cache-size' property to specify maximum size of the L2 table cache for qcow2 image
Posted by Alberto Garcia 6 years, 5 months ago
On Wed 15 Nov 2017 12:21:24 PM CET, Kashyap Chamarthy wrote:
> [Cc: Alberto Garcia of QEMU, who added the 'l2-cache-size' upstream]

[...]

>> There's already something on list about l2 cache, see
>> 
>> https://www.redhat.com/archives/libvir-list/2017-September/msg00553.html
>> 
>> it adds 3 related variables. IIRC, the "hold up" is how best to
>> describe how to use it or provide the feature in such a way that it's
>> "understandable" for general consumption without making a "policy"
>> that generates certain values based on expected usage/needs for the
>> client.
>
> I recall Alberto saying that there's a default value of QCOW2
> 'l2-cache-size', but it is (intentionally?) low.  And that the value
> depends on the workload / use case: size of the image, the amount of
> I/O you want to do, etc.

I actually didn't add that parameter.

Looking at the history it seems that the default size hasn't changed
since Fabrice Bellard added it back in 2006, so I assume that it was
fine for most use cases back in the day and it was never discussed
afterwards.

This was probably mentioned already, but here's what I've written about
this topic:

   https://bugzilla.redhat.com/show_bug.cgi?id=1377735#c2

   https://www.redhat.com/archives/libvir-list/2017-October/msg00180.html

>From the user's point of view the simplest use cases are perhaps:

   1) I want to use at most xxx MB for the qcow2 L2 cache.
   2) I want to use as much cache as I need in order to get the best
      performance.

Those should be relatively straightforward and easy to understand for
the user, (1) is simply the value of l2-cache-size, and for (2) libvirt
would have to make the calculations itself.

Some things to take into account:

   - l2-refcount-size is set to be 1/4 of l2-cache-size (see "How to
     configure the cache sizes" in docs/qcow2-cache.txt in QEMU), but it
     probably makes sense to keep it artificially low (it's not so
     important for I/O perfomance, although I dont have numbers of its
     actual impact).
   - cache-clean-interval can also be useful, see the bugzilla link for
     more details.

Berto

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/2] Add 'l2-cache-size' property to specify maximum size of the L2 table cache for qcow2 image
Posted by Kashyap Chamarthy 6 years, 5 months ago
On Wed, Nov 15, 2017 at 04:09:28PM +0100, Alberto Garcia wrote:
> On Wed 15 Nov 2017 12:21:24 PM CET, Kashyap Chamarthy wrote:
> > [Cc: Alberto Garcia of QEMU, who added the 'l2-cache-size' upstream]
> 
> [...]
> 
> >> There's already something on list about l2 cache, see
> >> 
> >> https://www.redhat.com/archives/libvir-list/2017-September/msg00553.html
> >> 
> >> it adds 3 related variables. IIRC, the "hold up" is how best to
> >> describe how to use it or provide the feature in such a way that it's
> >> "understandable" for general consumption without making a "policy"
> >> that generates certain values based on expected usage/needs for the
> >> client.
> >
> > I recall Alberto saying that there's a default value of QCOW2
> > 'l2-cache-size', but it is (intentionally?) low.  And that the value
> > depends on the workload / use case: size of the image, the amount of
> > I/O you want to do, etc.
> 
> I actually didn't add that parameter.

Err, sorry for the misattribution, I already wondered while writing
that line that I should've checked. I meant to mention your blog post:

    https://blogs.igalia.com/berto/2015/12/17/improving-disk-io-performance-in-qemu-2-5-with-the-qcow2-l2-cache/
 
> Looking at the history it seems that the default size hasn't changed
> since Fabrice Bellard added it back in 2006, so I assume that it was
> fine for most use cases back in the day and it was never discussed
> afterwards.
> 
> This was probably mentioned already, but here's what I've written about
> this topic:
> 
>    https://bugzilla.redhat.com/show_bug.cgi?id=1377735#c2
> 
>    https://www.redhat.com/archives/libvir-list/2017-October/msg00180.html
> 
> From the user's point of view the simplest use cases are perhaps:
> 
>    1) I want to use at most xxx MB for the qcow2 L2 cache.
>    2) I want to use as much cache as I need in order to get the best
>       performance.
> 
> Those should be relatively straightforward and easy to understand for
> the user, (1) is simply the value of l2-cache-size, and for (2) libvirt
> would have to make the calculations itself.
> 
> Some things to take into account:
> 
>    - l2-refcount-size is set to be 1/4 of l2-cache-size (see "How to
>      configure the cache sizes" in docs/qcow2-cache.txt in QEMU), but it
>      probably makes sense to keep it artificially low (it's not so
>      important for I/O perfomance, although I dont have numbers of its
>      actual impact).
>    - cache-clean-interval can also be useful, see the bugzilla link for
>      more details.

Thanks for the explanation and pointers.

-- 
/kashyap

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] 答复: Re: [PATCH 0/2] Add 'l2-cache-size' property to specify maximum size of the L2 table cache for qcow2 image
Posted by Lin Ma 6 years, 5 months ago

>>> John Ferlan <jferlan@redhat.com> 2017/11/15 星期三 下午 6:54 >>>
>
>
>On 11/14/2017 08:58 PM, Lin Ma wrote:
>> Lin Ma (2):
>>   caps: Add capability for maximum size of the qcow2 L2 table cache
v>   qemu: Add support for 'l2-cache-size' property of qcow2
>> 
>
>There's already something on list about l2 cache, see
>
>https://www.redhat.com/archives/libvir-list/2017-September/msg00553.html
>
>it adds 3 related variables. IIRC, the "hold up" is how best to describe
>how to use it or provide the feature in such a way that it's
>"understandable" for general consumption without making a "policy" that
>generates certain values based on expected usage/needs for the client.
>
Thank you all for the information!
 
Lin
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list