docs/formatdomain.html.in | 30 +- docs/schemas/domaincommon.rng | 14 +- src/conf/domain_conf.c | 310 ++++++++++-- src/conf/domain_conf.h | 12 + src/libvirt-domain.c | 9 + src/libvirt_private.syms | 11 + src/qemu/qemu_driver.c | 270 +++++++++- src/qemu/qemu_process.c | 69 ++- src/util/virresctrl.c | 560 +++++++++++++++++++-- src/util/virresctrl.h | 49 ++ tests/genericxml2xmlindata/cachetune-cdp.xml | 3 + .../cachetune-colliding-monitor.xml | 30 ++ tests/genericxml2xmlindata/cachetune-small.xml | 7 + tests/genericxml2xmltest.c | 2 + 14 files changed, 1283 insertions(+), 93 deletions(-) create mode 100644 tests/genericxml2xmlindata/cachetune-colliding-monitor.xml
This series of patches and the series already been merged introduce the x86 Cache Monitoring Technology (CMT) to libvirt by interacting with kernel resource control (resctrl) interface. CMT is one of the Intel(R) x86 CPU feature which belongs to the Resource Director Technology (RDT). CMT reports the occupancy of the last level cache, which is shared by all CPU cores. In the v1 series, an original and complete feature for CMT was introduced The v2 and v3 patches address the feature for the host capability of CMT. v4 is addressing the feature for monitoring VM vcpu thread set cache occupancy and reporting it through a virsh command. We have serval discussion about the enabling of CMT, please refer to following links for the RFCs. RFCv3 https://www.redhat.com/archives/libvir-list/2018-August/msg01213.html RFCv2 https://www.redhat.com/archives/libvir-list/2018-July/msg00409.html https://www.redhat.com/archives/libvir-list/2018-July/msg01241.html RFCv1 https://www.redhat.com/archives/libvir-list/2018-June/msg00674.html And the merged commits are list as below, for host capability of CMT. 6af8417415508c31f8ce71234b573b4999f35980 8f6887998bf63594ae26e3db18d4d5896c5f2cb4 58fcee6f3a2b7e89c21c1fb4ec21429c31a0c5b8 12093f1feaf8f5023dcd9d65dff111022842183d a5d293c18831dcf69ec6195798387fbb70c9f461 1. About reason why CMT is necessary in libvirt? The perf events of 'CMT, MBML, MBMT' have been phased out since Linux kernel commit c39a0e2c8850f08249383f2425dbd8dbe4baad69, in libvirt the perf based cmt,mbm will not work with the latest linux kernel. These patches add CMT feature to libvirt through kernel resctrlfs interface. 2 Create cache monitoring group (cache monitor). The main interface for creating monitoring group is through XML file. The proposed configuration is like: <cputune> <cachetune vcpus='1'> <cache id='0' level='3' type='code' size='7680' unit='KiB'/> <cache id='1' level='3' type='data' size='3840' unit='KiB'/> + <monitor level='3' vcpus='1'/> </cachetune> <cachetune vcpus='4-7'> + <monitor level='3' vcpus='4-6'/> </cachetune> </cputune> In above XML, created 2 cache resctrl allocation groups and 2 resctrl monitoring groups. The changes of cache monitor will be effective in next booting of VM. 2 Show CMT result through command 'domstats' Adding the interface in qemu to report this information for resource monitor group through command 'virsh domstats --cpu-total'. Below is a typical output: # virsh domstats 1 --cpu-total Domain: 'ubuntu16.04-base' ... cpu.cache.monitor.count=2 cpu.cache.monitor.0.name=vcpus_1 cpu.cache.monitor.0.vcpus=1 cpu.cache.monitor.0.bank.count=2 cpu.cache.monitor.0.bank.0.id=0 cpu.cache.monitor.0.bank.0.bytes=4505600 cpu.cache.monitor.0.bank.1.id=1 cpu.cache.monitor.0.bank.1.bytes=5586944 cpu.cache.monitor.1.name=vcpus_4-6 cpu.cache.monitor.1.vcpus=4,5,6 cpu.cache.monitor.1.bank.count=2 cpu.cache.monitor.1.bank.0.id=0 cpu.cache.monitor.1.bank.0.bytes=17571840 cpu.cache.monitor.1.bank.1.id=1 cpu.cache.monitor.1.bank.1.bytes=29106176 Changes in v7: - Add several lines removed by mistake. Changes in v6: - Addressing John's review comments for v5. - Removed and cleaned the concepts of 'default allocation' and 'default monitor'. - qemu: virsh domstats --cpu-total output for CMT, add identifier 'monitor' for each itm. Changes in v5: - qemu: Setting up vcpu and adding pids to resctrl monitor groups during re-connection. - Add the document for domain configuration related to resctrl monitor. Changes in v4: v4 is addressing the feature for monitoring VM vcpu thread set cache occupancy and reporting it through a virsh command. - Introduced resctrl default allocation - Introduced resctrl monitor and default monitor Changes in v3: - Addressed John Ferlan's review. - Typo fixed. - Removed VIR_ENUM_DECL(virMonitor); Changes in v2: - Introduced MBM capability. - Capability layout changed * Moved <monitor> from cahe <bank> to <cache> * Renamed <Threshold> to <reuseThreshold> - Document for 'reuseThreshold' changed. - Introduced API virResctrlInfoGetMonitorPrefix - Added more tests, covering standalone CMT, fake new feature. - Creating CMT resource control group will be subsequent job. Wang Huaqiang (18): docs,util: Refactor schemas and virresctrl to support optional cache util: Introduce resctrl monitor for CMT util: Refactor code for determining allocation path util: Add interface to determine monitor path util: Refactor code for adding PID to the resource group util: Add interface for adding PID to the monitor util: Refactor code for creating resctrl group util: Add interface for creating monitor group util: Add more interfaces for resctrl monitor conf: Remove virDomainResctrlAppend and introduce virDomainResctrlNew conf: move virResctrlAllocIsEmpty to a place a litter lower conf: Introduce cache monitor element in cachetune qemu: enable resctrl monitor in qemu util: Add function for checking if monitor is running conf: Add 'id' to virDomainResctrlDef qemu: refactor qemuDomainGetStatsCpu qemu: Report cache occupancy (CMT) with domstats qemu: Setting up vcpu and adding pids to resctrl monitor groups during reconnection docs/formatdomain.html.in | 30 +- docs/schemas/domaincommon.rng | 14 +- src/conf/domain_conf.c | 310 ++++++++++-- src/conf/domain_conf.h | 12 + src/libvirt-domain.c | 9 + src/libvirt_private.syms | 11 + src/qemu/qemu_driver.c | 270 +++++++++- src/qemu/qemu_process.c | 69 ++- src/util/virresctrl.c | 560 +++++++++++++++++++-- src/util/virresctrl.h | 49 ++ tests/genericxml2xmlindata/cachetune-cdp.xml | 3 + .../cachetune-colliding-monitor.xml | 30 ++ tests/genericxml2xmlindata/cachetune-small.xml | 7 + tests/genericxml2xmltest.c | 2 + 14 files changed, 1283 insertions(+), 93 deletions(-) create mode 100644 tests/genericxml2xmlindata/cachetune-colliding-monitor.xml -- 2.7.4 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 10/22/18 4:01 AM, Wang Huaqiang wrote: > This series of patches and the series already been merged introduce > the x86 Cache Monitoring Technology (CMT) to libvirt by interacting > with kernel resource control (resctrl) interface. CMT is one of the > Intel(R) x86 CPU feature which belongs to the Resource Director > Technology (RDT). CMT reports the occupancy of the last level cache, > which is shared by all CPU cores. > Rather than top or bottom post ;-) - as noted during the reviewing - many of the patches are fine; however, once we get to the meat in patch 12 and beyond I think more adjustment is necessary. I wouldn't want to introduce the XML from patch 12, but have no teeth behind it from the remaining patches. Getting closer, but not quite there yet. If you're good with my proposed adjustments for patches 1-11, then I can get those pushed so that we can make some progress. On your next round, please let's try to get a news.xml to summarize the changes added at the end. Saves me from chasing later on. I won't have a lot of cycles for the rest of the week to debate or provide feedback. I'm in class Wed -> Fri. I really don't think the comments I've provided in the later patches are that controversial. I do think we probably need a way to handle the Unplug with the existing usage of the *tasks file before we go too far and add monitor. You also need to figure a way to not add the same pid twice. I realize now that a *Remove operation will remove the *tasks file from a $path so some concerns I had were at least reduced. John > In the v1 series, an original and complete feature for CMT was introduced > The v2 and v3 patches address the feature for the host capability of CMT. > v4 is addressing the feature for monitoring VM vcpu thread set cache > occupancy and reporting it through a virsh command. > > We have serval discussion about the enabling of CMT, please refer to > following links for the RFCs. > RFCv3 > https://www.redhat.com/archives/libvir-list/2018-August/msg01213.html > RFCv2 > https://www.redhat.com/archives/libvir-list/2018-July/msg00409.html > https://www.redhat.com/archives/libvir-list/2018-July/msg01241.html > RFCv1 > https://www.redhat.com/archives/libvir-list/2018-June/msg00674.html > > And the merged commits are list as below, for host capability of CMT. > 6af8417415508c31f8ce71234b573b4999f35980 > 8f6887998bf63594ae26e3db18d4d5896c5f2cb4 > 58fcee6f3a2b7e89c21c1fb4ec21429c31a0c5b8 > 12093f1feaf8f5023dcd9d65dff111022842183d > a5d293c18831dcf69ec6195798387fbb70c9f461 > > > 1. About reason why CMT is necessary in libvirt? > The perf events of 'CMT, MBML, MBMT' have been phased out since Linux > kernel commit c39a0e2c8850f08249383f2425dbd8dbe4baad69, in libvirt > the perf based cmt,mbm will not work with the latest linux kernel. These > patches add CMT feature to libvirt through kernel resctrlfs interface. > > 2 Create cache monitoring group (cache monitor). > > The main interface for creating monitoring group is through XML file. The > proposed configuration is like: > > <cputune> > <cachetune vcpus='1'> > <cache id='0' level='3' type='code' size='7680' unit='KiB'/> > <cache id='1' level='3' type='data' size='3840' unit='KiB'/> > + <monitor level='3' vcpus='1'/> > </cachetune> > <cachetune vcpus='4-7'> > + <monitor level='3' vcpus='4-6'/> > </cachetune> > </cputune> > > In above XML, created 2 cache resctrl allocation groups and 2 resctrl > monitoring groups. > The changes of cache monitor will be effective in next booting of VM. > > 2 Show CMT result through command 'domstats' > > Adding the interface in qemu to report this information for resource > monitor group through command 'virsh domstats --cpu-total'. > Below is a typical output: > > # virsh domstats 1 --cpu-total > Domain: 'ubuntu16.04-base' > ... > cpu.cache.monitor.count=2 > cpu.cache.monitor.0.name=vcpus_1 > cpu.cache.monitor.0.vcpus=1 > cpu.cache.monitor.0.bank.count=2 > cpu.cache.monitor.0.bank.0.id=0 > cpu.cache.monitor.0.bank.0.bytes=4505600 > cpu.cache.monitor.0.bank.1.id=1 > cpu.cache.monitor.0.bank.1.bytes=5586944 > cpu.cache.monitor.1.name=vcpus_4-6 > cpu.cache.monitor.1.vcpus=4,5,6 > cpu.cache.monitor.1.bank.count=2 > cpu.cache.monitor.1.bank.0.id=0 > cpu.cache.monitor.1.bank.0.bytes=17571840 > cpu.cache.monitor.1.bank.1.id=1 > cpu.cache.monitor.1.bank.1.bytes=29106176 > > Changes in v7: > - Add several lines removed by mistake. > > Changes in v6: > - Addressing John's review comments for v5. > - Removed and cleaned the concepts of 'default allocation' and > 'default monitor'. > - qemu: virsh domstats --cpu-total output for CMT, add identifier > 'monitor' for each itm. > > Changes in v5: > - qemu: Setting up vcpu and adding pids to resctrl monitor groups during > re-connection. > - Add the document for domain configuration related to resctrl monitor. > > Changes in v4: > v4 is addressing the feature for monitoring VM vcpu > thread set cache occupancy and reporting it through a > virsh command. > - Introduced resctrl default allocation > - Introduced resctrl monitor and default monitor > > Changes in v3: > - Addressed John Ferlan's review. > - Typo fixed. > - Removed VIR_ENUM_DECL(virMonitor); > > Changes in v2: > - Introduced MBM capability. > - Capability layout changed > * Moved <monitor> from cahe <bank> to <cache> > * Renamed <Threshold> to <reuseThreshold> > - Document for 'reuseThreshold' changed. > - Introduced API virResctrlInfoGetMonitorPrefix > - Added more tests, covering standalone CMT, fake new > feature. > - Creating CMT resource control group will be > subsequent job. > > Wang Huaqiang (18): > docs,util: Refactor schemas and virresctrl to support optional cache > util: Introduce resctrl monitor for CMT > util: Refactor code for determining allocation path > util: Add interface to determine monitor path > util: Refactor code for adding PID to the resource group > util: Add interface for adding PID to the monitor > util: Refactor code for creating resctrl group > util: Add interface for creating monitor group > util: Add more interfaces for resctrl monitor > conf: Remove virDomainResctrlAppend and introduce virDomainResctrlNew > conf: move virResctrlAllocIsEmpty to a place a litter lower > conf: Introduce cache monitor element in cachetune > qemu: enable resctrl monitor in qemu > util: Add function for checking if monitor is running > conf: Add 'id' to virDomainResctrlDef > qemu: refactor qemuDomainGetStatsCpu > qemu: Report cache occupancy (CMT) with domstats > qemu: Setting up vcpu and adding pids to resctrl monitor groups during > reconnection > > docs/formatdomain.html.in | 30 +- > docs/schemas/domaincommon.rng | 14 +- > src/conf/domain_conf.c | 310 ++++++++++-- > src/conf/domain_conf.h | 12 + > src/libvirt-domain.c | 9 + > src/libvirt_private.syms | 11 + > src/qemu/qemu_driver.c | 270 +++++++++- > src/qemu/qemu_process.c | 69 ++- > src/util/virresctrl.c | 560 +++++++++++++++++++-- > src/util/virresctrl.h | 49 ++ > tests/genericxml2xmlindata/cachetune-cdp.xml | 3 + > .../cachetune-colliding-monitor.xml | 30 ++ > tests/genericxml2xmlindata/cachetune-small.xml | 7 + > tests/genericxml2xmltest.c | 2 + > 14 files changed, 1283 insertions(+), 93 deletions(-) > create mode 100644 tests/genericxml2xmlindata/cachetune-colliding-monitor.xml > -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 11/6/2018 4:19 AM, John Ferlan wrote: > > On 10/22/18 4:01 AM, Wang Huaqiang wrote: >> This series of patches and the series already been merged introduce >> the x86 Cache Monitoring Technology (CMT) to libvirt by interacting >> with kernel resource control (resctrl) interface. CMT is one of the >> Intel(R) x86 CPU feature which belongs to the Resource Director >> Technology (RDT). CMT reports the occupancy of the last level cache, >> which is shared by all CPU cores. >> > Rather than top or bottom post ;-) - as noted during the reviewing - > many of the patches are fine; however, once we get to the meat in patch > 12 and beyond I think more adjustment is necessary. I wouldn't want to > introduce the XML from patch 12, but have no teeth behind it from the > remaining patches. Really appreciate your review and comments, your comments not only pointed out my faults but also provide very reasonable suggestions. For patch 1-11, I also addressed your review opinions in my next version code, please make a review at your convenient time. For patches after 11, I also addressed your review suggestions by removing the patches for adding 'resctrl->id' and for 'virResctrlMonitorIsRunning', and made some code split for patch17. Please make a review. > > Getting closer, but not quite there yet. If you're good with my > proposed adjustments for patches 1-11, then I can get those pushed so > that we can make some progress. > > On your next round, please let's try to get a news.xml to summarize the > changes added at the end. Saves me from chasing later on. OK. A separate patch for change of news.xml is appended. > > I won't have a lot of cycles for the rest of the week to debate or > provide feedback. I'm in class Wed -> Fri. I really don't think the > comments I've provided in the later patches are that controversial. Agree. Your comments are very accurate. This time I have time to prepare the code along with the reply to your review emails. > > I do think we probably need a way to handle the Unplug with the existing > usage of the *tasks file before we go too far and add monitor. By analyzing the code handling vcpu unplug , it seems there is no problem to keep the *tasks file aligned with PIDs. Because the removal of PID is automatically done by kernel if kernel find some vcpu process is disappeared. I need more time to test on vcpu unplug cases for my code. I have not achieved a successful live removal of vcpus, even with code without my new resctrl monitor patches. I will update when I have some conclusion. > You also > need to figure a way to not add the same pid twice. I realize now that a > *Remove operation will remove the *tasks file from a $path so some > concerns I had were at least reduced. To add PID twice is because I thought the *tasks file could be removed/added in process of re-connection, when performing 'systemctl restart libvirtd', but it seems is does not. So it is not necessary to add same pid twice. The libvirtd daemon restart process works fine after the remove of relevant patch. And *tasks file is not removed or changed in process of re-connection. So patch18 and patch14 are removed. > > John Thanks for all for your review. Huaqiang > >> In the v1 series, an original and complete feature for CMT was introduced >> The v2 and v3 patches address the feature for the host capability of CMT. >> v4 is addressing the feature for monitoring VM vcpu thread set cache >> occupancy and reporting it through a virsh command. >> >> We have serval discussion about the enabling of CMT, please refer to >> following links for the RFCs. >> RFCv3 >> https://www.redhat.com/archives/libvir-list/2018-August/msg01213.html >> RFCv2 >> https://www.redhat.com/archives/libvir-list/2018-July/msg00409.html >> https://www.redhat.com/archives/libvir-list/2018-July/msg01241.html >> RFCv1 >> https://www.redhat.com/archives/libvir-list/2018-June/msg00674.html >> >> And the merged commits are list as below, for host capability of CMT. >> 6af8417415508c31f8ce71234b573b4999f35980 >> 8f6887998bf63594ae26e3db18d4d5896c5f2cb4 >> 58fcee6f3a2b7e89c21c1fb4ec21429c31a0c5b8 >> 12093f1feaf8f5023dcd9d65dff111022842183d >> a5d293c18831dcf69ec6195798387fbb70c9f461 >> >> >> 1. About reason why CMT is necessary in libvirt? >> The perf events of 'CMT, MBML, MBMT' have been phased out since Linux >> kernel commit c39a0e2c8850f08249383f2425dbd8dbe4baad69, in libvirt >> the perf based cmt,mbm will not work with the latest linux kernel. These >> patches add CMT feature to libvirt through kernel resctrlfs interface. >> >> 2 Create cache monitoring group (cache monitor). >> >> The main interface for creating monitoring group is through XML file. The >> proposed configuration is like: >> >> <cputune> >> <cachetune vcpus='1'> >> <cache id='0' level='3' type='code' size='7680' unit='KiB'/> >> <cache id='1' level='3' type='data' size='3840' unit='KiB'/> >> + <monitor level='3' vcpus='1'/> >> </cachetune> >> <cachetune vcpus='4-7'> >> + <monitor level='3' vcpus='4-6'/> >> </cachetune> >> </cputune> >> >> In above XML, created 2 cache resctrl allocation groups and 2 resctrl >> monitoring groups. >> The changes of cache monitor will be effective in next booting of VM. >> >> 2 Show CMT result through command 'domstats' >> >> Adding the interface in qemu to report this information for resource >> monitor group through command 'virsh domstats --cpu-total'. >> Below is a typical output: >> >> # virsh domstats 1 --cpu-total >> Domain: 'ubuntu16.04-base' >> ... >> cpu.cache.monitor.count=2 >> cpu.cache.monitor.0.name=vcpus_1 >> cpu.cache.monitor.0.vcpus=1 >> cpu.cache.monitor.0.bank.count=2 >> cpu.cache.monitor.0.bank.0.id=0 >> cpu.cache.monitor.0.bank.0.bytes=4505600 >> cpu.cache.monitor.0.bank.1.id=1 >> cpu.cache.monitor.0.bank.1.bytes=5586944 >> cpu.cache.monitor.1.name=vcpus_4-6 >> cpu.cache.monitor.1.vcpus=4,5,6 >> cpu.cache.monitor.1.bank.count=2 >> cpu.cache.monitor.1.bank.0.id=0 >> cpu.cache.monitor.1.bank.0.bytes=17571840 >> cpu.cache.monitor.1.bank.1.id=1 >> cpu.cache.monitor.1.bank.1.bytes=29106176 >> >> Changes in v7: >> - Add several lines removed by mistake. >> >> Changes in v6: >> - Addressing John's review comments for v5. >> - Removed and cleaned the concepts of 'default allocation' and >> 'default monitor'. >> - qemu: virsh domstats --cpu-total output for CMT, add identifier >> 'monitor' for each itm. >> >> Changes in v5: >> - qemu: Setting up vcpu and adding pids to resctrl monitor groups during >> re-connection. >> - Add the document for domain configuration related to resctrl monitor. >> >> Changes in v4: >> v4 is addressing the feature for monitoring VM vcpu >> thread set cache occupancy and reporting it through a >> virsh command. >> - Introduced resctrl default allocation >> - Introduced resctrl monitor and default monitor >> >> Changes in v3: >> - Addressed John Ferlan's review. >> - Typo fixed. >> - Removed VIR_ENUM_DECL(virMonitor); >> >> Changes in v2: >> - Introduced MBM capability. >> - Capability layout changed >> * Moved <monitor> from cahe <bank> to <cache> >> * Renamed <Threshold> to <reuseThreshold> >> - Document for 'reuseThreshold' changed. >> - Introduced API virResctrlInfoGetMonitorPrefix >> - Added more tests, covering standalone CMT, fake new >> feature. >> - Creating CMT resource control group will be >> subsequent job. >> >> Wang Huaqiang (18): >> docs,util: Refactor schemas and virresctrl to support optional cache >> util: Introduce resctrl monitor for CMT >> util: Refactor code for determining allocation path >> util: Add interface to determine monitor path >> util: Refactor code for adding PID to the resource group >> util: Add interface for adding PID to the monitor >> util: Refactor code for creating resctrl group >> util: Add interface for creating monitor group >> util: Add more interfaces for resctrl monitor >> conf: Remove virDomainResctrlAppend and introduce virDomainResctrlNew >> conf: move virResctrlAllocIsEmpty to a place a litter lower >> conf: Introduce cache monitor element in cachetune >> qemu: enable resctrl monitor in qemu >> util: Add function for checking if monitor is running >> conf: Add 'id' to virDomainResctrlDef >> qemu: refactor qemuDomainGetStatsCpu >> qemu: Report cache occupancy (CMT) with domstats >> qemu: Setting up vcpu and adding pids to resctrl monitor groups during >> reconnection >> >> docs/formatdomain.html.in | 30 +- >> docs/schemas/domaincommon.rng | 14 +- >> src/conf/domain_conf.c | 310 ++++++++++-- >> src/conf/domain_conf.h | 12 + >> src/libvirt-domain.c | 9 + >> src/libvirt_private.syms | 11 + >> src/qemu/qemu_driver.c | 270 +++++++++- >> src/qemu/qemu_process.c | 69 ++- >> src/util/virresctrl.c | 560 +++++++++++++++++++-- >> src/util/virresctrl.h | 49 ++ >> tests/genericxml2xmlindata/cachetune-cdp.xml | 3 + >> .../cachetune-colliding-monitor.xml | 30 ++ >> tests/genericxml2xmlindata/cachetune-small.xml | 7 + >> tests/genericxml2xmltest.c | 2 + >> 14 files changed, 1283 insertions(+), 93 deletions(-) >> create mode 100644 tests/genericxml2xmlindata/cachetune-colliding-monitor.xml >> -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
© 2016 - 2024 Red Hat, Inc.