Documentation/virt/kvm/api.rst | 24 ++++++++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 7 +++++++ arch/x86/kvm/pmu.c | 6 ++++++ arch/x86/kvm/pmu.h | 9 +++++++++ arch/x86/kvm/x86.c | 10 ++++++++++ include/uapi/linux/kvm.h | 1 + 6 files changed, 57 insertions(+)
On hosts that run guests with an *emulated* vPMU -- guest PMU MSR
accesses trap into KVM and each guest counter is backed by a host
perf_event -- the per-emulated-instruction PMU accounting added by commit
9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is
pathologically expensive.
Whenever a guest programs a counter for retired instructions (0xc0) or
retired branches (0xc2) -- routine for any profiler-style workload --
every instruction KVM emulates triggers a software increment plus a
batched KVM_REQ_PMU reprogram. The reprogram is drained on the next
VM-entry regardless of which exit preceded it, so ordinary exits (most
importantly the guest's 1kHz timer tick) absorb a full
reprogram_counter() cycle: ctx->mutex, ctx_resched(), and a burst of
serialized PMU-MSR writes. That inflates timer-interrupt handling into
the hundreds of microseconds and, under load, can escalate to a guest
CSD lockup.
This series adds KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING, a VM-scoped
capability that takes a bitmask of PERF_COUNT_HW_* event IDs and makes
KVM skip the software accounting of emulated instructions for those
events. It is opt-in and defaults off; the only cost to an opted-in
guest is that instructions-retired / branches-retired counters
undercount by the instructions KVM emulates in host context, i.e. the
behavior that predates that accounting. Hardware-executed guest
instructions are still counted by the backing perf_event and its
overflow/PMI path is untouched.
Patch 1 adds the capability and the accounting short-circuit.
Patch 2 documents it in Documentation/virt/kvm/api.rst.
Measured benefit
================
Host: AMD EPYC 7R13 "Milan", KVM, v7.2-rc4 + this series. Guest: QEMU,
-cpu host,pmu=on, 8 vCPUs. The guest workload keeps an instructions-
retired counter resident on every vCPU and periodically reprograms the
PMU, driving the accounting storm above, while the vCPUs are kept busy
so the 1kHz tick fires. The guest's local-APIC timer interrupt handler
(__sysvec_apic_timer_interrupt) is traced with ftrace function_graph
for 30s (~160k samples per run). The capability is toggled per-VM via
KVM_ENABLE_CAP; nothing else changes between runs.
Timer-interrupt handler duration:
metric accounting ON (baseline) accounting OFF (CAP) speedup
------ ------------------------ -------------------- -------
mean 209.8 us 3.16 us 66x
p50 256.4 us 3.14 us 82x
p99 463.3 us 4.45 us 104x
p99.9 915.2 us 5.03 us 182x
max 929.7 us 22.55 us 41x
Some workloads genuinely need the extra precision that the emulated-
instruction accounting provides, and for them the default behavior is
the right trade-off. But for many others the accuracy gained is not
worth the cost it imposes on every exit, and this capability lets those
users opt out of paying it.
Luka Absandze (2):
KVM: x86/pmu: Add CAP to disable SW accounting of emulated
instructions
KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING
Documentation/virt/kvm/api.rst | 24 ++++++++++++++++++++++++
arch/x86/include/asm/kvm_host.h | 7 +++++++
arch/x86/kvm/pmu.c | 6 ++++++
arch/x86/kvm/pmu.h | 9 +++++++++
arch/x86/kvm/x86.c | 10 ++++++++++
include/uapi/linux/kvm.h | 1 +
6 files changed, 57 insertions(+)
--
2.47.3
On Mon, 2026-07-20 at 19:22 +0000, Luka Absandze wrote:
> On hosts that run guests with an *emulated* vPMU -- guest PMU MSR
> accesses trap into KVM and each guest counter is backed by a host
> perf_event -- the per-emulated-instruction PMU accounting added by commit
> 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is
> pathologically expensive.
Given that the guest PMU MSR accesses are trapped... is there a reason
we can't fold a *separate* count of emulated instructions into the
value we return to the guest?
I guess the overflow would come late... but isn't that tolerable? And
we could probably impose a limit on how large the addend could be —
even if we only reprogram the PMU once every 100 times, that's still a
win?
On Tue, Jul 21, 2026, David Woodhouse wrote:
> On Mon, 2026-07-20 at 19:22 +0000, Luka Absandze wrote:
> > On hosts that run guests with an *emulated* vPMU -- guest PMU MSR
> > accesses trap into KVM and each guest counter is backed by a host
> > perf_event -- the per-emulated-instruction PMU accounting added by commit
> > 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is
> > pathologically expensive.
>
> Given that the guest PMU MSR accesses are trapped... is there a reason
> we can't fold a *separate* count of emulated instructions into the
> value we return to the guest?
>
> I guess the overflow would come late... but isn't that tolerable?
Like ignoring emulated instructions entirely, it probably depends on the guest.
> And we could probably impose a limit on how large the addend could be — even
> if we only reprogram the PMU once every 100 times, that's still a win?
Hmm, yeah, that'd be better than ignoring emulated instruction entirely, and
would give userspace far better control over how shoddy KVM's emulation is. Off
the cuff, the change would be fairly trivial too? E.g.
diff --git arch/x86/kvm/pmu.c arch/x86/kvm/pmu.c
index 7f777049d328..c5f436c6f83f 100644
--- arch/x86/kvm/pmu.c
+++ arch/x86/kvm/pmu.c
@@ -1084,7 +1084,8 @@ static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
*/
if (!kvm_vcpu_has_mediated_pmu(vcpu)) {
pmc->emulated_counter++;
- kvm_pmu_request_counter_reprogram(pmc);
+ if (pmc->emulated_counter > pmu_something_something_tolerance)
+ kvm_pmu_request_counter_reprogram(pmc);
return;
}
On 2026-07-21 09:47, Sean Christopherson wrote:
> Hmm, yeah, that'd be better than ignoring emulated instruction entirely, and
> would give userspace far better control over how shoddy KVM's emulation is. Off
> the cuff, the change would be fairly trivial too? E.g.
>
> diff --git arch/x86/kvm/pmu.c arch/x86/kvm/pmu.c
> index 7f777049d328..c5f436c6f83f 100644
> --- arch/x86/kvm/pmu.c
> +++ arch/x86/kvm/pmu.c
> @@ -1084,7 +1084,8 @@ static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
> */
> if (!kvm_vcpu_has_mediated_pmu(vcpu)) {
> pmc->emulated_counter++;
> - kvm_pmu_request_counter_reprogram(pmc);
> + if (pmc->emulated_counter > pmu_something_something_tolerance)
> + kvm_pmu_request_counter_reprogram(pmc);
> return;
> }
I believe this is fine, but perhaps the tolerance itself could be a knob
with <0 meaning no reprogram? While the contention on this path will be
much lower, we are still introducing more steal time, and the added
accuracy is probably not worth even that for guests who really only care
for the rate of change, rather than exact instruction counts.
On Tue, Jul 21, 2026, Luka Absandze wrote:
> On 2026-07-21 09:47, Sean Christopherson wrote:
> > Hmm, yeah, that'd be better than ignoring emulated instruction entirely, and
> > would give userspace far better control over how shoddy KVM's emulation is. Off
> > the cuff, the change would be fairly trivial too? E.g.
> >
> > diff --git arch/x86/kvm/pmu.c arch/x86/kvm/pmu.c
> > index 7f777049d328..c5f436c6f83f 100644
> > --- arch/x86/kvm/pmu.c
> > +++ arch/x86/kvm/pmu.c
> > @@ -1084,7 +1084,8 @@ static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
> > */
> > if (!kvm_vcpu_has_mediated_pmu(vcpu)) {
> > pmc->emulated_counter++;
> > - kvm_pmu_request_counter_reprogram(pmc);
> > + if (pmc->emulated_counter > pmu_something_something_tolerance)
> > + kvm_pmu_request_counter_reprogram(pmc);
> > return;
> > }
>
> I believe this is fine, but perhaps the tolerance itself could be a knob
Ya, I was envisioning a knob.
> with <0 meaning no reprogram?
Eh, I don't see a need to assign special meaning to <0. Make it a u32 (or u64),
and then a large value will provide the same behavior in practice, without needing
special handling in KVM.
> While the contention on this path will be much lower, we are still
> introducing more steal time, and the added accuracy is probably not worth
> even that for guests who really only care for the rate of change, rather than
> exact instruction counts.
On Tue, 2026-07-21 at 09:47 -0700, Sean Christopherson wrote:
> On Tue, Jul 21, 2026, David Woodhouse wrote:
> > On Mon, 2026-07-20 at 19:22 +0000, Luka Absandze wrote:
> > > On hosts that run guests with an *emulated* vPMU -- guest PMU MSR
> > > accesses trap into KVM and each guest counter is backed by a host
> > > perf_event -- the per-emulated-instruction PMU accounting added by commit
> > > 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is
> > > pathologically expensive.
> >
> > Given that the guest PMU MSR accesses are trapped... is there a reason
> > we can't fold a *separate* count of emulated instructions into the
> > value we return to the guest?
> >
> > I guess the overflow would come late... but isn't that tolerable?
>
> Like ignoring emulated instructions entirely, it probably depends on the guest.
>
> > And we could probably impose a limit on how large the addend could be — even
> > if we only reprogram the PMU once every 100 times, that's still a win?
>
> Hmm, yeah, that'd be better than ignoring emulated instruction entirely, and
> would give userspace far better control over how shoddy KVM's emulation is. Off
> the cuff, the change would be fairly trivial too? E.g.
>
> diff --git arch/x86/kvm/pmu.c arch/x86/kvm/pmu.c
> index 7f777049d328..c5f436c6f83f 100644
> --- arch/x86/kvm/pmu.c
> +++ arch/x86/kvm/pmu.c
> @@ -1084,7 +1084,8 @@ static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
> */
> if (!kvm_vcpu_has_mediated_pmu(vcpu)) {
> pmc->emulated_counter++;
> - kvm_pmu_request_counter_reprogram(pmc);
> + if (pmc->emulated_counter > pmu_something_something_tolerance)
> + kvm_pmu_request_counter_reprogram(pmc);
> return;
> }
>
We also want to add the same delta when the the guest reads the
counter?
On Tue, Jul 21, 2026, David Woodhouse wrote:
> On Tue, 2026-07-21 at 09:47 -0700, Sean Christopherson wrote:
> > On Tue, Jul 21, 2026, David Woodhouse wrote:
> > > On Mon, 2026-07-20 at 19:22 +0000, Luka Absandze wrote:
> > > > On hosts that run guests with an *emulated* vPMU -- guest PMU MSR
> > > > accesses trap into KVM and each guest counter is backed by a host
> > > > perf_event -- the per-emulated-instruction PMU accounting added by commit
> > > > 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is
> > > > pathologically expensive.
> > >
> > > Given that the guest PMU MSR accesses are trapped... is there a reason
> > > we can't fold a *separate* count of emulated instructions into the
> > > value we return to the guest?
> > >
> > > I guess the overflow would come late... but isn't that tolerable?
> >
> > Like ignoring emulated instructions entirely, it probably depends on the guest.
> >
> > > And we could probably impose a limit on how large the addend could be — even
> > > if we only reprogram the PMU once every 100 times, that's still a win?
> >
> > Hmm, yeah, that'd be better than ignoring emulated instruction entirely, and
> > would give userspace far better control over how shoddy KVM's emulation is. Off
> > the cuff, the change would be fairly trivial too? E.g.
> >
> > diff --git arch/x86/kvm/pmu.c arch/x86/kvm/pmu.c
> > index 7f777049d328..c5f436c6f83f 100644
> > --- arch/x86/kvm/pmu.c
> > +++ arch/x86/kvm/pmu.c
> > @@ -1084,7 +1084,8 @@ static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
> > */
> > if (!kvm_vcpu_has_mediated_pmu(vcpu)) {
> > pmc->emulated_counter++;
> > - kvm_pmu_request_counter_reprogram(pmc);
> > + if (pmc->emulated_counter > pmu_something_something_tolerance)
> > + kvm_pmu_request_counter_reprogram(pmc);
> > return;
> > }
> >
>
> We also want to add the same delta when the the guest reads the counter?
Isn't that already handled by pmc_read_counter()? Or am I misunderstanding the
concern?
static inline u64 pmc_read_counter(struct kvm_pmc *pmc)
{
u64 counter, enabled, running;
if (kvm_vcpu_has_mediated_pmu(pmc->vcpu))
return pmc->counter & pmc_bitmask(pmc);
counter = pmc->counter + pmc->emulated_counter; <===============
if (pmc->perf_event && !pmc->is_paused)
counter += perf_event_read_value(pmc->perf_event,
&enabled, &running);
/* FIXME: Scaling needed? */
return counter & pmc_bitmask(pmc);
}
© 2016 - 2026 Red Hat, Inc.