On a host with an emulated vPMU (guest PMU MSR accesses trap and each
guest counter is backed by a host perf_event), the per-emulated-
instruction PMU accounting added by commit 9cd803d496e7 ("KVM: x86:
Update vPMCs when retiring instructions") is pathologically expensive.

kvm_pmu_incr_counter() requests a counter reprogram (KVM_REQ_PMU) on
every emulated instruction that matches a programmed counter. The
reprogram is drained on the vCPU's next VM-entry, where
reprogram_counter() runs the full pmc_pause_counter() +
perf_event_period() + perf_event_enable() sequence: ctx->mutex, a
ctx_resched() of the PMU context, and a burst of serialized PMU-MSR
writes. Because the batched reprogram is serviced on the next VM-entry
regardless of which exit preceded it, ordinary exits -- notably the
guest's 1kHz timer tick -- absorb the cost, inflating timer interrupts
into the hundreds of microseconds and, under some workloads, escalating
to a guest CSD lockup.

Add emulated_counter_reprogram_tolerance, a module parameter bounding
how many emulated instructions may accumulate on a counter before KVM
forces a reprogram. 0 (default) preserves today's behavior of
reprogramming on every emulated instruction. A larger value batches the
reprograms.

Batching does not lose counts: pmc_read_counter() and
reprogram_counter() already fold pmc->emulated_counter into the value
observed on guest counter reads, so RDPMC/RDMSR remain accurate. The
only effect is that an overflow-driven PMI may be delivered up to
"tolerance" instructions late -- acceptable given the overflow PMI is
not cycle-accurate to begin with. The knob only affects the emulated
vPMU; a mediated vPMU increments its counter directly and is unchanged.

Suggested-by: David Woodhouse <dwmw2@infradead.org>
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Luka Absandze <absandze@amazon.de>
---
v1 (RFC): https://lore.kernel.org/kvm/20260720192221.72912-1-absandze@amazon.de/

Changes since v1:
 - Drop the new KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING capability and its
   Documentation, i.e. add no new uAPI.
 - Instead of disabling emulated-instruction accounting outright, batch the
   counter reprograms behind a tolerance threshold, per David's suggestion
   to bound the reprogram rate rather than skip accounting [1]. This keeps
   guest counter reads accurate and only bounds how late an overflow PMI is
   delivered.
 - Expose the threshold as the emulated_counter_reprogram_tolerance module
   param (uint, 0644); 0 preserves current behavior.

[1] https://lore.kernel.org/kvm/f5552576fc749b4b453514f52921332cc104c6a6.camel@infradead.org/

Results (single run, not averaged over multiple runs):

Host: AMD EPYC 7R13 (Milan), emulated vPMU. Guest: 8 vCPUs under an
8-thread busy load, running a binary that toggles a PMU counter group
cross-vCPU once per second while an instructions-retired counter is
resident. Latency is the duration of the guest's local-APIC timer
interrupt handler (__sysvec_apic_timer_interrupt), measured in-guest with
ftrace function_graph over 30s (~160k ticks per pass). The module param
was changed live on one running guest between passes.

  tolerance      mean      p50      p99    p99.9      max
  ----------  --------  -------  -------  -------  --------
  0 (default)  148.973   11.080  464.200  469.569  919.690   (us)
  1000           3.194    2.950    7.251    9.831  470.100
  100000         2.974    2.940    3.869    4.400    8.060
  10000000       3.054    2.940    7.129    9.211  209.880

These are numbers from a single run and are meant only to show
the order of magnitude, not to be precise.

 arch/x86/kvm/pmu.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
index dd1c57593f48..33a73271186a 100644
--- a/arch/x86/kvm/pmu.c
+++ b/arch/x86/kvm/pmu.c
@@ -39,6 +39,14 @@ bool __read_mostly enable_pmu = true;
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_pmu);
 module_param(enable_pmu, bool, 0444);
 
+/*
+ * Number of KVM-emulated instructions that may accumulate on an emulated
+ * (perf-based) vPMU counter before KVM forces a counter reprogram to fold the
+ * emulated count into the backing perf_event
+ */
+static uint __read_mostly emulated_counter_reprogram_tolerance;
+module_param(emulated_counter_reprogram_tolerance, uint, 0644);
+
 /* Enable/disabled mediated PMU virtualization. */
 bool __read_mostly enable_mediated_pmu;
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_mediated_pmu);
@@ -1072,7 +1080,15 @@ static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
 	 */
 	if (!kvm_vcpu_has_mediated_pmu(vcpu)) {
 		pmc->emulated_counter++;
-		kvm_pmu_request_counter_reprogram(pmc);
+
+		/*
+		 * Batch reprograms: only force one once the accumulated
+		 * emulated count exceeds the tolerance. The count is still
+		 * reflected in guest counter reads via pmc->emulated_counter;
+		 * this only bounds how late an overflow-driven PMI arrives.
+		 */
+		if (pmc->emulated_counter > emulated_counter_reprogram_tolerance)
+			kvm_pmu_request_counter_reprogram(pmc);
 		return;
 	}
 

base-commit: 1590cf0329716306e948a8fc29f1d3ee87d3989f
-- 
2.47.3