[libvirt] [resend v2 0/7] Support cache tune in libvirt

Eli Qiao posted 7 patches 7 years, 1 month ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/1486347822-54305-1-git-send-email-liyong.qiao@intel.com
There is a newer version of this series
docs/schemas/domaincommon.rng |   41 ++
include/libvirt/virterror.h   |    1 +
src/Makefile.am               |    1 +
src/conf/capabilities.c       |   56 +++
src/conf/capabilities.h       |   23 +
src/conf/domain_conf.c        |  154 +++++++
src/conf/domain_conf.h        |   18 +
src/libvirt_private.syms      |   10 +
src/qemu/qemu_capabilities.c  |   68 +++
src/qemu/qemu_driver.c        |    4 +
src/qemu/qemu_process.c       |   11 +
src/util/virerror.c           |    1 +
src/util/virresctrl.c         | 1019 +++++++++++++++++++++++++++++++++++++++++
src/util/virresctrl.h         |  135 ++++++
14 files changed, 1542 insertions(+)
create mode 100644 src/util/virresctrl.c
create mode 100644 src/util/virresctrl.h
[libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Eli Qiao 7 years, 1 month ago
This series patches are for supportting CAT featues, which also
called cache tune in libvirt.

First to expose cache information which could be tuned in capabilites XML.
Then add new domain xml element support to add cacahe bank which will apply
on this libvirt domain.

This series patches add a util file `resctrl.c/h`, an interface to talk with
linux kernel's sys fs.

There are still some TODOs such as expose new public interface to get free
cache information.

Some discussion about this feature support can be found from:
https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html

Eli Qiao (7):
  Resctrl: Add some utils functions
  Resctrl: expose cache information to capabilities
  Resctrl: Add new xml element to support cache tune
  Resctrl: Add private interface to set cachebanks
  Qemu: Set cache banks
  Resctrl: enable l3code/l3data
  Resctrl: Make sure l3data/l3code are pairs

 docs/schemas/domaincommon.rng |   41 ++
 include/libvirt/virterror.h   |    1 +
 src/Makefile.am               |    1 +
 src/conf/capabilities.c       |   56 +++
 src/conf/capabilities.h       |   23 +
 src/conf/domain_conf.c        |  154 +++++++
 src/conf/domain_conf.h        |   18 +
 src/libvirt_private.syms      |   10 +
 src/qemu/qemu_capabilities.c  |   68 +++
 src/qemu/qemu_driver.c        |    4 +
 src/qemu/qemu_process.c       |   11 +
 src/util/virerror.c           |    1 +
 src/util/virresctrl.c         | 1019 +++++++++++++++++++++++++++++++++++++++++
 src/util/virresctrl.h         |  135 ++++++
 14 files changed, 1542 insertions(+)
 create mode 100644 src/util/virresctrl.c
 create mode 100644 src/util/virresctrl.h

--
1.9.1

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Marcelo Tosatti 7 years, 1 month ago
On Mon, Feb 06, 2017 at 10:23:35AM +0800, Eli Qiao wrote:
> This series patches are for supportting CAT featues, which also
> called cache tune in libvirt.
> 
> First to expose cache information which could be tuned in capabilites XML.
> Then add new domain xml element support to add cacahe bank which will apply
> on this libvirt domain.
> 
> This series patches add a util file `resctrl.c/h`, an interface to talk with
> linux kernel's sys fs.
> 
> There are still some TODOs such as expose new public interface to get free
> cache information.
> 
> Some discussion about this feature support can be found from:
> https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html

Two comments:

1) Please perform appropriate filesystem locking when accessing
resctrlfs, as described at:

http://www.spinics.net/lists/linux-tip-commits/msg36754.html

2) 

    <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>

[b4c270b5-e0f9-4106-a446-69032872ed7d]# cat tasks  
8654
[b4c270b5-e0f9-4106-a446-69032872ed7d]# pstree -p | grep qemu
           |-qemu-kvm(8654)-+-{qemu-kvm}(8688)
           |                |-{qemu-kvm}(8692)
           |                `-{qemu-kvm}(8693)

Should add individual vcpus to the "tasks" file, not the main QEMU
process.

The NFV usecase requires exclusive CAT allocation for the vcpu which
runs the sensitive workload.

Perhaps:

    <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/> 

Adds all vcpus that are pinned to the socket which cachebank with
host_id=1.

    <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB' vcpus=2,3/> 

Adds PID of vcpus 2 and 3 to resctrl directory created for this
allocation.





--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Marcelo Tosatti 7 years, 1 month ago
On Mon, Feb 06, 2017 at 01:33:09PM -0200, Marcelo Tosatti wrote:
> On Mon, Feb 06, 2017 at 10:23:35AM +0800, Eli Qiao wrote:
> > This series patches are for supportting CAT featues, which also
> > called cache tune in libvirt.
> > 
> > First to expose cache information which could be tuned in capabilites XML.
> > Then add new domain xml element support to add cacahe bank which will apply
> > on this libvirt domain.
> > 
> > This series patches add a util file `resctrl.c/h`, an interface to talk with
> > linux kernel's sys fs.
> > 
> > There are still some TODOs such as expose new public interface to get free
> > cache information.
> > 
> > Some discussion about this feature support can be found from:
> > https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html
> 
> Two comments:
> 
> 1) Please perform appropriate filesystem locking when accessing
> resctrlfs, as described at:
> 
> http://www.spinics.net/lists/linux-tip-commits/msg36754.html
> 
> 2) 
> 
>     <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>
> 
> [b4c270b5-e0f9-4106-a446-69032872ed7d]# cat tasks  
> 8654
> [b4c270b5-e0f9-4106-a446-69032872ed7d]# pstree -p | grep qemu
>            |-qemu-kvm(8654)-+-{qemu-kvm}(8688)
>            |                |-{qemu-kvm}(8692)
>            |                `-{qemu-kvm}(8693)
> 
> Should add individual vcpus to the "tasks" file, not the main QEMU
> process.
> 
> The NFV usecase requires exclusive CAT allocation for the vcpu which
> runs the sensitive workload.
> 
> Perhaps:
> 
>     <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/> 
> 
> Adds all vcpus that are pinned to the socket which cachebank with
> host_id=1.
> 
>     <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB' vcpus=2,3/> 
> 
> Adds PID of vcpus 2 and 3 to resctrl directory created for this
> allocation.

3) CDP / non-CDP convertion.

In case the size determination has been performed with non-CDP,
to emulate such allocation on a CDP host,
it would be good to allow both code and data allocations to share
the CBM space: 


     <cachetune id='10' host_id='1' type='l3data' size='3072' unit='KiB'/>
     <cachetune id='10' host_id='1' type='l3code' size='3072' unit='KiB'/>

Perhaps if using the same ID? 

Other than that, testing looks good.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Eli Qiao 7 years, 1 month ago

--  
Eli Qiao
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Tuesday, 7 February 2017 at 3:03 AM, Marcelo Tosatti wrote:

> On Mon, Feb 06, 2017 at 01:33:09PM -0200, Marcelo Tosatti wrote:
> > On Mon, Feb 06, 2017 at 10:23:35AM +0800, Eli Qiao wrote:
> > > This series patches are for supportting CAT featues, which also
> > > called cache tune in libvirt.
> > >  
> > > First to expose cache information which could be tuned in capabilites XML.
> > > Then add new domain xml element support to add cacahe bank which will apply
> > > on this libvirt domain.
> > >  
> > > This series patches add a util file `resctrl.c/h`, an interface to talk with
> > > linux kernel's sys fs.
> > >  
> > > There are still some TODOs such as expose new public interface to get free
> > > cache information.
> > >  
> > > Some discussion about this feature support can be found from:
> > > https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html
> > >  
> >  
> >  
> > Two comments:
> >  
> > 1) Please perform appropriate filesystem locking when accessing
> > resctrlfs, as described at:
> >  
> > http://www.spinics.net/lists/linux-tip-commits/msg36754.html

Sure.  
> >  
> > 2)  
> >  
> > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>
> >  
> > [b4c270b5-e0f9-4106-a446-69032872ed7d]# cat tasks  
> > 8654
> > [b4c270b5-e0f9-4106-a446-69032872ed7d]# pstree -p | grep qemu
> > |-qemu-kvm(8654)-+-{qemu-kvm}(8688)
> > | |-{qemu-kvm}(8692)
> > | `-{qemu-kvm}(8693)
> >  
> > Should add individual vcpus to the "tasks" file, not the main QEMU
> > process.
> >  
> > The NFV usecase requires exclusive CAT allocation for the vcpu which
> > runs the sensitive workload.
> >  
> > Perhaps:
> >  
> > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>  
> >  
> > Adds all vcpus that are pinned to the socket which cachebank with
> > host_id=1.
> >  
> > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB' vcpus=2,3/>  
> >  
> > Adds PID of vcpus 2 and 3 to resctrl directory created for this
> > allocation.
> >  
>  
>  
Hmm.. in this case, we need to figure out what’s the pid of vcpus=2 and vcpu=3 and added them to the resctrl directory.
currently, I create a resctrl directory(called resctrl domain) for a VM so just put all task ids to it.

this is my though:

let say the vm has vcpus=0 1 2 3, and you want to let 0, 1 benefit cache on host_id=0, and 2, 3 on host_id=1

you will do:

1)
    pin vcpus 0, 1 on the cpus of socket 0  
    pin vcpus 2, 3 on the cpus of socket 1
this can be done in vcputune

2) define cache tune like this:
<cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'/>
<cachetune id='1' host_id='1' type='l3' size='3072' unit='KiB'/>

in libvirt:
we create a resctrl directory naming with the VM’s uuid
and set schemata for each socket 0, and socket 1, put all qemu tasks ids into tasks file, this will work fine.  
please be note that in a resctrl directory, we can define schemata for each socket id separately.
  
  
> 3) CDP / non-CDP convertion.
>  
> In case the size determination has been performed with non-CDP,
> to emulate such allocation on a CDP host,
> it would be good to allow both code and data allocations to share
> the CBM space:  
>  
IOM, I don’t think it’s good to have this.
in libvirt capabilities xml, the application will get to know if the host support cdp or not.
  
>  
> <cachetune id='10' host_id='1' type='l3data' size='3072' unit='KiB'/>
> <cachetune id='10' host_id='1' type='l3code' size='3072' unit='KiB'/>
>  
> Perhaps if using the same ID?
I am open to hear about what other’s say,  
  
>  
>  
> Other than that, testing looks good.  
Thanks for the testing.  


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Daniel P. Berrange 7 years, 1 month ago
On Tue, Feb 07, 2017 at 02:43:13PM +0800, Eli Qiao wrote:
> > 3) CDP / non-CDP convertion.
> >  
> > In case the size determination has been performed with non-CDP,
> > to emulate such allocation on a CDP host,
> > it would be good to allow both code and data allocations to share
> > the CBM space:  
> >  
> IOM, I don’t think it’s good to have this.
> in libvirt capabilities xml, the application will get to know if the host support cdp or not.

Yep, as long as the capabilities XML provide info about the different
cache banks, IMHO it is better to have the application be explicit
about what they want for CDP & non-CDP scenarios.  Let the higher
level mgmt apps above libvirt apply specific policies if they desire


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Marcelo Tosatti 7 years, 1 month ago
On Tue, Feb 07, 2017 at 10:17:37AM +0000, Daniel P. Berrange wrote:
> On Tue, Feb 07, 2017 at 02:43:13PM +0800, Eli Qiao wrote:
> > > 3) CDP / non-CDP convertion.
> > >  
> > > In case the size determination has been performed with non-CDP,
> > > to emulate such allocation on a CDP host,
> > > it would be good to allow both code and data allocations to share
> > > the CBM space:  
> > >  
> > IOM, I don’t think it’s good to have this.
> > in libvirt capabilities xml, the application will get to know if the host support cdp or not.
> 
> Yep, as long as the capabilities XML provide info about the different
> cache banks, IMHO it is better to have the application be explicit
> about what they want for CDP & non-CDP scenarios.  Let the higher
> level mgmt apps above libvirt apply specific policies if they desire

There is no policy being applied here. What i mean is the following:

1) User determines that the type of the CAT allocation necessary
for his application is one which shares cache and data, that is non-CDP
(either because he didnt have a CDP machine at the time, or because
he had a CDP machine but sharing data and code cache turns out 
to be efficient for the application).
Say that this measured size is M.

2) A host with CDP enabled resctrl is used for this VM. How to create
a CAT allocation with shared code and data? Have to write to the
schemata file of the VM the following:


L3data:1=0x00ff;...
L3code:1=0x00ff;...

(that is, the data and code CBM masks use the same bits).

3) How is the user going to achieve that with this patchset today?
AFAIK, he can't. What he can do is the following:

  <cachetune id='1' host_id='0' type='l3code' size='M' unit='KiB'/>
  <cachetune id='2' host_id='0' type='l3data' size='M' unit='KiB'/>

But this will allocate the following schemata file:

	L3data:1=0xff00;...
	L3code:1=0x00ff;...

Which is not what is wanted.

Therefore the suggestion to _share the cbm bit space_ in case
a similar "cachetune id" is used.

(maybe have a different syntax, i don't care, as long as the user
can create code / data CAT allocations that share the CBM space).

Does that make sense now?


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Marcelo Tosatti 7 years, 1 month ago
On Tue, Feb 07, 2017 at 02:43:13PM +0800, Eli Qiao wrote:
> 
> 
> --  
> Eli Qiao
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> 
> 
> On Tuesday, 7 February 2017 at 3:03 AM, Marcelo Tosatti wrote:
> 
> > On Mon, Feb 06, 2017 at 01:33:09PM -0200, Marcelo Tosatti wrote:
> > > On Mon, Feb 06, 2017 at 10:23:35AM +0800, Eli Qiao wrote:
> > > > This series patches are for supportting CAT featues, which also
> > > > called cache tune in libvirt.
> > > >  
> > > > First to expose cache information which could be tuned in capabilites XML.
> > > > Then add new domain xml element support to add cacahe bank which will apply
> > > > on this libvirt domain.
> > > >  
> > > > This series patches add a util file `resctrl.c/h`, an interface to talk with
> > > > linux kernel's sys fs.
> > > >  
> > > > There are still some TODOs such as expose new public interface to get free
> > > > cache information.
> > > >  
> > > > Some discussion about this feature support can be found from:
> > > > https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html
> > > >  
> > >  
> > >  
> > > Two comments:
> > >  
> > > 1) Please perform appropriate filesystem locking when accessing
> > > resctrlfs, as described at:
> > >  
> > > http://www.spinics.net/lists/linux-tip-commits/msg36754.html
> 
> Sure.  
> > >  
> > > 2)  
> > >  
> > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>
> > >  
> > > [b4c270b5-e0f9-4106-a446-69032872ed7d]# cat tasks  
> > > 8654
> > > [b4c270b5-e0f9-4106-a446-69032872ed7d]# pstree -p | grep qemu
> > > |-qemu-kvm(8654)-+-{qemu-kvm}(8688)
> > > | |-{qemu-kvm}(8692)
> > > | `-{qemu-kvm}(8693)
> > >  
> > > Should add individual vcpus to the "tasks" file, not the main QEMU
> > > process.
> > >  
> > > The NFV usecase requires exclusive CAT allocation for the vcpu which
> > > runs the sensitive workload.
> > >  
> > > Perhaps:
> > >  
> > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>  
> > >  
> > > Adds all vcpus that are pinned to the socket which cachebank with
> > > host_id=1.
> > >  
> > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB' vcpus=2,3/>  
> > >  
> > > Adds PID of vcpus 2 and 3 to resctrl directory created for this
> > > allocation.
> > >  
> >  
> >  
> Hmm.. in this case, we need to figure out what’s the pid of vcpus=2 and vcpu=3 and added them to the resctrl directory.

Yes and only the pids of vcpus=2 and vcpus=3, not of any other vcpus.

> currently, I create a resctrl directory(called resctrl domain) for a VM so just put all task ids to it.
> 
> this is my though:
> 
> let say the vm has vcpus=0 1 2 3, and you want to let 0, 1 benefit cache on host_id=0, and 2, 3 on host_id=1
> 
> you will do:
> 
> 1)
>     pin vcpus 0, 1 on the cpus of socket 0  
>     pin vcpus 2, 3 on the cpus of socket 1
> this can be done in vcputune
> 
> 2) define cache tune like this:
> <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'/>
> <cachetune id='1' host_id='1' type='l3' size='3072' unit='KiB'/>
> 
> in libvirt:
> we create a resctrl directory naming with the VM’s uuid
> and set schemata for each socket 0, and socket 1, put all qemu tasks ids into tasks file, this will work fine.  

No, please don't do this.

> please be note that in a resctrl directory, we can define schemata for each socket id separately.

Please do not put vcpus automatically into the reservations.
Its necessary to have certain vcpus in a reservation and some not.

For example: 2 vcpu guest, vcpu0 pinned to socket 0 cpu0, 
  		           vcpu1 pinned to socket 0 cpu1.


<cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'/>

We want _only_ vcpu1 to be part of this reservation, and not vcpu0 
(want vcpu0 to use the default group, ie. schemata file at 

/sys/fs/resctrl/schemata

So please have the ability to add vcpus to the XML syntax:

<cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'
vcpus='1'/>

or

<cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'
vcpus='2,3'/>

This also allows different sizes to be specified.


> > 3) CDP / non-CDP convertion.
> >  
> > In case the size determination has been performed with non-CDP,
> > to emulate such allocation on a CDP host,
> > it would be good to allow both code and data allocations to share
> > the CBM space:  
> >  
> IOM, I don’t think it’s good to have this.
> in libvirt capabilities xml, the application will get to know if the host support cdp or not.
>   
> >  
> > <cachetune id='10' host_id='1' type='l3data' size='3072' unit='KiB'/>
> > <cachetune id='10' host_id='1' type='l3code' size='3072' unit='KiB'/>
> >  
> > Perhaps if using the same ID?
> I am open to hear about what other’s say,  
>   
> >  
> >  
> > Other than that, testing looks good.  
> Thanks for the testing.  
> 
> 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Eli Qiao 7 years, 1 month ago

--  
Eli Qiao
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Tuesday, 7 February 2017 at 7:56 PM, Marcelo Tosatti wrote:

> On Tue, Feb 07, 2017 at 02:43:13PM +0800, Eli Qiao wrote:
> >  
> >  
> > --  
> > Eli Qiao
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >  
> >  
> > On Tuesday, 7 February 2017 at 3:03 AM, Marcelo Tosatti wrote:
> >  
> > > On Mon, Feb 06, 2017 at 01:33:09PM -0200, Marcelo Tosatti wrote:
> > > > On Mon, Feb 06, 2017 at 10:23:35AM +0800, Eli Qiao wrote:
> > > > > This series patches are for supportting CAT featues, which also
> > > > > called cache tune in libvirt.
> > > > >  
> > > > > First to expose cache information which could be tuned in capabilites XML.
> > > > > Then add new domain xml element support to add cacahe bank which will apply
> > > > > on this libvirt domain.
> > > > >  
> > > > > This series patches add a util file `resctrl.c/h`, an interface to talk with
> > > > > linux kernel's sys fs.
> > > > >  
> > > > > There are still some TODOs such as expose new public interface to get free
> > > > > cache information.
> > > > >  
> > > > > Some discussion about this feature support can be found from:
> > > > > https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html
> > > > >  
> > > > >  
> > > >  
> > > >  
> > > >  
> > > > Two comments:
> > > >  
> > > > 1) Please perform appropriate filesystem locking when accessing
> > > > resctrlfs, as described at:
> > > >  
> > > > http://www.spinics.net/lists/linux-tip-commits/msg36754.html
> > > >  
> > >  
> > >  
> >  
> >  
> > Sure.  
> > > >  
> > > > 2)  
> > > >  
> > > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>
> > > >  
> > > > [b4c270b5-e0f9-4106-a446-69032872ed7d]# cat tasks  
> > > > 8654
> > > > [b4c270b5-e0f9-4106-a446-69032872ed7d]# pstree -p | grep qemu
> > > > |-qemu-kvm(8654)-+-{qemu-kvm}(8688)
> > > > | |-{qemu-kvm}(8692)
> > > > | `-{qemu-kvm}(8693)
> > > >  
> > > > Should add individual vcpus to the "tasks" file, not the main QEMU
> > > > process.
> > > >  
> > > > The NFV usecase requires exclusive CAT allocation for the vcpu which
> > > > runs the sensitive workload.
> > > >  
> > > > Perhaps:
> > > >  
> > > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>  
> > > >  
> > > > Adds all vcpus that are pinned to the socket which cachebank with
> > > > host_id=1.
> > > >  
> > > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB' vcpus=2,3/>  
> > > >  
> > > > Adds PID of vcpus 2 and 3 to resctrl directory created for this
> > > > allocation.
> > > >  
> > > >  
> > >  
> > >  
> > >  
> > >  
> >  
> > Hmm.. in this case, we need to figure out what’s the pid of vcpus=2 and vcpu=3 and added them to the resctrl directory.
> >  
>  
>  
> Yes and only the pids of vcpus=2 and vcpus=3, not of any other vcpus.
>  
> > currently, I create a resctrl directory(called resctrl domain) for a VM so just put all task ids to it.
> >  
> > this is my though:
> >  
> > let say the vm has vcpus=0 1 2 3, and you want to let 0, 1 benefit cache on host_id=0, and 2, 3 on host_id=1
> >  
> > you will do:
> >  
> > 1)
> > pin vcpus 0, 1 on the cpus of socket 0  
> > pin vcpus 2, 3 on the cpus of socket 1
> > this can be done in vcputune
> >  
> > 2) define cache tune like this:
> > <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'/>
> > <cachetune id='1' host_id='1' type='l3' size='3072' unit='KiB'/>
> >  
> > in libvirt:
> > we create a resctrl directory naming with the VM’s uuid
> > and set schemata for each socket 0, and socket 1, put all qemu tasks ids into tasks file, this will work fine.  
> >  
>  
>  
> No, please don't do this.
>  
> > please be note that in a resctrl directory, we can define schemata for each socket id separately.
>  
> Please do not put vcpus automatically into the reservations.
> Its necessary to have certain vcpus in a reservation and some not.
>  
> For example: 2 vcpu guest, vcpu0 pinned to socket 0 cpu0,  
> vcpu1 pinned to socket 0 cpu1.
>  
>  
> <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'/>
>  
> We want _only_ vcpu1 to be part of this reservation, and not vcpu0  
> (want vcpu0 to use the default group, ie. schemata file at  
>  
> /sys/fs/resctrl/schemata
>  
> So please have the ability to add vcpus to the XML syntax:
>  
> <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'
> vcpus='1'/>
>  
> or
>  
> <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'
> vcpus='2,3'/>
>  
Yep, get your mean, make sense.

if there’s exist method can get vcpus’ pid?  
>  
> This also allows different sizes to be specified.
>  
>  
> > > 3) CDP / non-CDP convertion.
> > >  
> > > In case the size determination has been performed with non-CDP,
> > > to emulate such allocation on a CDP host,
> > > it would be good to allow both code and data allocations to share
> > > the CBM space:  
> > >  
> > >  
> >  
> > IOM, I don’t think it’s good to have this.
> > in libvirt capabilities xml, the application will get to know if the host support cdp or not.
> >  
> > >  
> > > <cachetune id='10' host_id='1' type='l3data' size='3072' unit='KiB'/>
> > > <cachetune id='10' host_id='1' type='l3code' size='3072' unit='KiB'/>
> > >  
> > > Perhaps if using the same ID?
> > >  
> >  
> > I am open to hear about what other’s say,  
> >  
> > >  
> > >  
> > > Other than that, testing looks good.  
> > >  
> >  
> > Thanks for the testing.  
> >  
>  
>  
>  


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Marcelo Tosatti 7 years, 1 month ago
On Wed, Feb 08, 2017 at 10:19:07AM +0800, Eli Qiao wrote:
> 
> 
> --  
> Eli Qiao
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> 
> 
> On Tuesday, 7 February 2017 at 7:56 PM, Marcelo Tosatti wrote:
> 
> > On Tue, Feb 07, 2017 at 02:43:13PM +0800, Eli Qiao wrote:
> > >  
> > >  
> > > --  
> > > Eli Qiao
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >  
> > >  
> > > On Tuesday, 7 February 2017 at 3:03 AM, Marcelo Tosatti wrote:
> > >  
> > > > On Mon, Feb 06, 2017 at 01:33:09PM -0200, Marcelo Tosatti wrote:
> > > > > On Mon, Feb 06, 2017 at 10:23:35AM +0800, Eli Qiao wrote:
> > > > > > This series patches are for supportting CAT featues, which also
> > > > > > called cache tune in libvirt.
> > > > > >  
> > > > > > First to expose cache information which could be tuned in capabilites XML.
> > > > > > Then add new domain xml element support to add cacahe bank which will apply
> > > > > > on this libvirt domain.
> > > > > >  
> > > > > > This series patches add a util file `resctrl.c/h`, an interface to talk with
> > > > > > linux kernel's sys fs.
> > > > > >  
> > > > > > There are still some TODOs such as expose new public interface to get free
> > > > > > cache information.
> > > > > >  
> > > > > > Some discussion about this feature support can be found from:
> > > > > > https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html
> > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > Two comments:
> > > > >  
> > > > > 1) Please perform appropriate filesystem locking when accessing
> > > > > resctrlfs, as described at:
> > > > >  
> > > > > http://www.spinics.net/lists/linux-tip-commits/msg36754.html
> > > > >  
> > > >  
> > > >  
> > >  
> > >  
> > > Sure.  
> > > > >  
> > > > > 2)  
> > > > >  
> > > > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>
> > > > >  
> > > > > [b4c270b5-e0f9-4106-a446-69032872ed7d]# cat tasks  
> > > > > 8654
> > > > > [b4c270b5-e0f9-4106-a446-69032872ed7d]# pstree -p | grep qemu
> > > > > |-qemu-kvm(8654)-+-{qemu-kvm}(8688)
> > > > > | |-{qemu-kvm}(8692)
> > > > > | `-{qemu-kvm}(8693)
> > > > >  
> > > > > Should add individual vcpus to the "tasks" file, not the main QEMU
> > > > > process.
> > > > >  
> > > > > The NFV usecase requires exclusive CAT allocation for the vcpu which
> > > > > runs the sensitive workload.
> > > > >  
> > > > > Perhaps:
> > > > >  
> > > > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB'/>  
> > > > >  
> > > > > Adds all vcpus that are pinned to the socket which cachebank with
> > > > > host_id=1.
> > > > >  
> > > > > <cachetune id='10' host_id='1' type='l3' size='3072' unit='KiB' vcpus=2,3/>  
> > > > >  
> > > > > Adds PID of vcpus 2 and 3 to resctrl directory created for this
> > > > > allocation.
> > > > >  
> > > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > >  
> > > Hmm.. in this case, we need to figure out what’s the pid of vcpus=2 and vcpu=3 and added them to the resctrl directory.
> > >  
> >  
> >  
> > Yes and only the pids of vcpus=2 and vcpus=3, not of any other vcpus.
> >  
> > > currently, I create a resctrl directory(called resctrl domain) for a VM so just put all task ids to it.
> > >  
> > > this is my though:
> > >  
> > > let say the vm has vcpus=0 1 2 3, and you want to let 0, 1 benefit cache on host_id=0, and 2, 3 on host_id=1
> > >  
> > > you will do:
> > >  
> > > 1)
> > > pin vcpus 0, 1 on the cpus of socket 0  
> > > pin vcpus 2, 3 on the cpus of socket 1
> > > this can be done in vcputune
> > >  
> > > 2) define cache tune like this:
> > > <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'/>
> > > <cachetune id='1' host_id='1' type='l3' size='3072' unit='KiB'/>
> > >  
> > > in libvirt:
> > > we create a resctrl directory naming with the VM’s uuid
> > > and set schemata for each socket 0, and socket 1, put all qemu tasks ids into tasks file, this will work fine.  
> > >  
> >  
> >  
> > No, please don't do this.
> >  
> > > please be note that in a resctrl directory, we can define schemata for each socket id separately.
> >  
> > Please do not put vcpus automatically into the reservations.
> > Its necessary to have certain vcpus in a reservation and some not.
> >  
> > For example: 2 vcpu guest, vcpu0 pinned to socket 0 cpu0,  
> > vcpu1 pinned to socket 0 cpu1.
> >  
> >  
> > <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'/>
> >  
> > We want _only_ vcpu1 to be part of this reservation, and not vcpu0  
> > (want vcpu0 to use the default group, ie. schemata file at  
> >  
> > /sys/fs/resctrl/schemata
> >  
> > So please have the ability to add vcpus to the XML syntax:
> >  
> > <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'
> > vcpus='1'/>
> >  
> > or
> >  
> > <cachetune id='0' host_id=‘0' type='l3' size='3072' unit='KiB'
> > vcpus='2,3'/>
> >  
> Yep, get your mean, make sense.
> 
> if there’s exist method can get vcpus’ pid?  

qemu_process.c:    pid_t vcpupid = qemuDomainGetVcpuPid(vm, vcpuid);

> > This also allows different sizes to be specified.
> >  
> >  
> > > > 3) CDP / non-CDP convertion.
> > > >  
> > > > In case the size determination has been performed with non-CDP,
> > > > to emulate such allocation on a CDP host,
> > > > it would be good to allow both code and data allocations to share
> > > > the CBM space:  
> > > >  
> > > >  
> > >  
> > > IOM, I don’t think it’s good to have this.
> > > in libvirt capabilities xml, the application will get to know if the host support cdp or not.
> > >  
> > > >  
> > > > <cachetune id='10' host_id='1' type='l3data' size='3072' unit='KiB'/>
> > > > <cachetune id='10' host_id='1' type='l3code' size='3072' unit='KiB'/>
> > > >  
> > > > Perhaps if using the same ID?
> > > >  
> > >  
> > > I am open to hear about what other’s say,  
> > >  
> > > >  
> > > >  
> > > > Other than that, testing looks good.  
> > > >  
> > >  
> > > Thanks for the testing.  
> > >  
> >  
> >  
> >  
> 
> 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [resend v2 0/7] Support cache tune in libvirt
Posted by Martin Kletzander 7 years, 1 month ago
On Mon, Feb 06, 2017 at 10:23:35AM +0800, Eli Qiao wrote:
>This series patches are for supportting CAT featues, which also
>called cache tune in libvirt.
>
>First to expose cache information which could be tuned in capabilites XML.
>Then add new domain xml element support to add cacahe bank which will apply
>on this libvirt domain.
>
>This series patches add a util file `resctrl.c/h`, an interface to talk with
>linux kernel's sys fs.
>
>There are still some TODOs such as expose new public interface to get free
>cache information.
>
>Some discussion about this feature support can be found from:
>https://www.redhat.com/archives/libvir-list/2017-January/msg00644.html
>
>Eli Qiao (7):
>  Resctrl: Add some utils functions
>  Resctrl: expose cache information to capabilities
>  Resctrl: Add new xml element to support cache tune
>  Resctrl: Add private interface to set cachebanks
>  Qemu: Set cache banks
>  Resctrl: enable l3code/l3data
>  Resctrl: Make sure l3data/l3code are pairs
>
> docs/schemas/domaincommon.rng |   41 ++
> include/libvirt/virterror.h   |    1 +
> src/Makefile.am               |    1 +
> src/conf/capabilities.c       |   56 +++
> src/conf/capabilities.h       |   23 +
> src/conf/domain_conf.c        |  154 +++++++
> src/conf/domain_conf.h        |   18 +
> src/libvirt_private.syms      |   10 +
> src/qemu/qemu_capabilities.c  |   68 +++
> src/qemu/qemu_driver.c        |    4 +
> src/qemu/qemu_process.c       |   11 +
> src/util/virerror.c           |    1 +
> src/util/virresctrl.c         | 1019 +++++++++++++++++++++++++++++++++++++++++
> src/util/virresctrl.h         |  135 ++++++
> 14 files changed, 1542 insertions(+)
> create mode 100644 src/util/virresctrl.c
> create mode 100644 src/util/virresctrl.h
>

I see that Daniel didn't mention this, so just on a side note, we tend
to add documentation in the same patch as the parsing/formatting code,
so that it's consistent.  Also there could be at least xml2xml test case
added.

>--
>1.9.1
>
>--
>libvir-list mailing list
>libvir-list@redhat.com
>https://www.redhat.com/mailman/listinfo/libvir-list
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list