[PATCH v7 00/36] Cleaning up the KVM clock mess

David Woodhouse posted 36 patches 3 weeks, 6 days ago
Failed in applying to current master (apply log)
There is a newer version of this series
Documentation/virt/kvm/api.rst                     |   37 +
Documentation/virt/kvm/devices/vcpu.rst            |  120 ++-
MAINTAINERS                                        |    4 +-
arch/x86/include/asm/kvm_host.h                    |   16 +-
arch/x86/include/uapi/asm/kvm.h                    |    7 +
arch/x86/include/{ => uapi}/asm/pvclock-abi.h      |   27 +-
arch/x86/kvm/cpuid.c                               |   16 -
arch/x86/kvm/msrs.c                                |    9 +-
arch/x86/kvm/svm/svm.c                             |    3 +-
arch/x86/kvm/vmx/vmx.c                             |   10 +
arch/x86/kvm/x86.c                                 | 1089 ++++++++++++--------
arch/x86/kvm/x86.h                                 |    3 +-
arch/x86/kvm/xen.c                                 |   28 +-
arch/x86/kvm/xen.h                                 |   13 -
include/uapi/linux/kvm.h                           |    3 +
scripts/xen-hypercalls.sh                          |    2 +-
tools/testing/selftests/kvm/Makefile.kvm           |    5 +
.../selftests/kvm/x86/masterclock_offset_test.c    |  180 ++++
.../selftests/kvm/x86/pvclock_migration_test.c     |  383 +++++++
tools/testing/selftests/kvm/x86/pvclock_test.c     |  443 ++++++++
.../selftests/kvm/x86/xen_cpuid_timing_test.c      |  230 +++++
.../testing/selftests/kvm/x86/xen_migration_test.c |  205 ++++
tools/testing/selftests/kvm/x86/xen_shinfo_test.c  |   17 +-
23 files changed, 2325 insertions(+), 525 deletions(-)
[PATCH v7 00/36] Cleaning up the KVM clock mess
Posted by David Woodhouse 3 weeks, 6 days ago
This is v7 of the series to clean up the KVM clock, rebased onto
kvm-x86/next (patch 1 of v6 is already merged there as commit
710b3a30f407).

The KVM clock has historically suffered from three problems:

 1. Imprecision: get_kvmclock_ns() computed the clock from the *host*
    TSC without applying guest TSC scaling, causing systemic drift from
    the values the guest computes from its own TSC.

 2. Unnecessary discontinuities: gratuitous KVM_REQ_MASTERCLOCK_UPDATE
    requests caused the master clock reference point to be re-snapshotted,
    yanking the guest's clock due to arithmetic precision differences.

 3. No precise migration API: the existing KVM_[GS]ET_CLOCK only allows
    setting the clock at a given UTC reference time, which is necessarily
    imprecise. There was no way to preserve the exact arithmetic
    relationship between guest TSC and KVM clock across live migration.

This series addresses all three, and adds new APIs for precise clock
migration and TSC frequency reporting. As an added bonus, it now rips
out the whole pvclock_gtod_data hack which was shadowing the kernel's
timekeeping, and uses ktime snapshots as $DEITY (well, Thomas) intended.

The series is now ordered with the fixes and cleanups first; the new
uAPI (the pvclock-abi UAPI move, KVM_[GS]ET_CLOCK_GUEST and
KVM_VCPU_TSC_SCALE) and all the new selftests come at the end, so that
the former are not gated on review of the latter.

v7: https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/kvmclock7

 - Rebased onto kvm-x86/next; dropped the already-merged first patch.
 - Reordered so that all the new uAPI (pvclock-abi UAPI move,
   KVM_[GS]ET_CLOCK_GUEST, KVM_VCPU_TSC_SCALE) and the new selftests
   come at the end of the series, after the fixes and cleanups which
   do not depend on them (Sean).
 - Split "Restructure get_kvmclock()" into two: dropping the CPU
   pinning is now a separate preliminary patch (Sean).
 - compute_guest_tsc(): use abs() and a conditional negation instead
   of duplicated scaling logic, and note in the changelog that the
   negative delta case is believed unreachable in practice (Sean).
 - Use HZ_PER_KHZ instead of literal 1000 for TSC unit conversions
   (Sean).
 - xen_migration_test: transfer the guest TSC via KVM_VCPU_TSC_OFFSET
   instead of a no-op TSC_ADJUST save/restore, making the runstate
   restore and steal-time assertions deterministic instead of racing
   VM-creation latency.
 - Reference the kvmclock definitions taxonomy by its now-stable
   (hopefully) commit ID 710b3a30f407.

v6: https://lore.kernel.org/all/20260703212145.343527-1-dwmw2@infradead.org/
    https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/kvmclock6

David Woodhouse (33):
      KVM: x86: Improve accuracy of KVM clock when TSC scaling is in force
      KVM: x86: Explicitly disable TSC scaling without CONSTANT_TSC
      KVM: x86: Activate master clock immediately on vCPU creation
      KVM: x86: Avoid NTP frequency skew for KVM clock on 32-bit host
      KVM: x86: Fold __get_kvmclock() into get_kvmclock()
      KVM: x86: Drop CPU pinning in get_kvmclock()
      KVM: x86: Restructure get_kvmclock()
      KVM: x86: Fix KVM clock precision in get_kvmclock() with TSC scaling
      KVM: x86: Use get_kvmclock() in kvm_get_wall_clock_epoch()
      KVM: x86: Fix compute_guest_tsc() to handle negative time deltas
      KVM: x86: Restructure kvm_guest_time_update() for TSC upscaling
      KVM: x86: Simplify and comment kvm_get_time_scale()
      KVM: x86: Remove implicit rdtsc() from kvm_compute_l1_tsc_offset()
      KVM: x86: Improve synchronization in kvm_synchronize_tsc()
      KVM: x86: Kill last_tsc_{nsec,write,offset} fields
      KVM: x86: Replace nr_vcpus_matched_tsc count with all_vcpus_matched_tsc bool
      KVM: x86: Allow KVM master clock mode when TSCs are offset from each other
      KVM: x86: Factor out kvm_use_master_clock()
      KVM: x86: Avoid gratuitous global clock updates
      KVM: x86/xen: Prevent runstate times from becoming negative
      KVM: x86: Avoid redundant masterclock updates from multiple vCPUs
      KVM: x86: Remove runtime Xen TSC frequency CPUID update
      KVM: x86: Re-synchronize TSC after KVM_SET_TSC_KHZ
      KVM: x86: Use ktime_get_snapshot_id() for master clock
      KVM: x86: Compute kvmclock base without pvclock_gtod_data
      KVM: x86: Cache host vclock_mode for masterclock eligibility checks
      KVM: x86: Remove pvclock_gtod_data and private timekeeping code
      KVM: x86: Activate master clock from kvm_arch_init_vm()
      KVM: selftests: Use UAPI pvclock-abi.h in xen_shinfo_test
      KVM: x86: Add KVM_VCPU_TSC_SCALE and fix the documentation on TSC migration
      KVM: selftests: Add master clock offset test
      KVM: selftests: Add Xen/generic CPUID timing leaf test
      KVM: selftests: Add Xen runstate migration test

Jack Allister (3):
      UAPI: x86: Move pvclock-abi to UAPI for x86 platforms
      KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration
      KVM: selftests: Add KVM/PV clock selftest to prove timer correction

 Documentation/virt/kvm/api.rst                     |   37 +
 Documentation/virt/kvm/devices/vcpu.rst            |  120 ++-
 MAINTAINERS                                        |    4 +-
 arch/x86/include/asm/kvm_host.h                    |   16 +-
 arch/x86/include/uapi/asm/kvm.h                    |    7 +
 arch/x86/include/{ => uapi}/asm/pvclock-abi.h      |   27 +-
 arch/x86/kvm/cpuid.c                               |   16 -
 arch/x86/kvm/msrs.c                                |    9 +-
 arch/x86/kvm/svm/svm.c                             |    3 +-
 arch/x86/kvm/vmx/vmx.c                             |   10 +
 arch/x86/kvm/x86.c                                 | 1089 ++++++++++++--------
 arch/x86/kvm/x86.h                                 |    3 +-
 arch/x86/kvm/xen.c                                 |   28 +-
 arch/x86/kvm/xen.h                                 |   13 -
 include/uapi/linux/kvm.h                           |    3 +
 scripts/xen-hypercalls.sh                          |    2 +-
 tools/testing/selftests/kvm/Makefile.kvm           |    5 +
 .../selftests/kvm/x86/masterclock_offset_test.c    |  180 ++++
 .../selftests/kvm/x86/pvclock_migration_test.c     |  383 +++++++
 tools/testing/selftests/kvm/x86/pvclock_test.c     |  443 ++++++++
 .../selftests/kvm/x86/xen_cpuid_timing_test.c      |  230 +++++
 .../testing/selftests/kvm/x86/xen_migration_test.c |  205 ++++
 tools/testing/selftests/kvm/x86/xen_shinfo_test.c  |   17 +-
 23 files changed, 2325 insertions(+), 525 deletions(-)
Re: [PATCH v7 00/36] Cleaning up the KVM clock mess
Posted by Sean Christopherson 3 weeks, 6 days ago
On Tue, Jul 28, 2026, David Woodhouse wrote:
> This is v7 of the series to clean up the KVM clock, rebased onto
> kvm-x86/next (patch 1 of v6 is already merged there as commit
> 710b3a30f407).

FYI, I'm going to rebase that commit again, to put all of this in a dedicated
topic branch.  But I'll also take care of updating references in commit messages.
More below.

> David Woodhouse (33):
>       KVM: x86: Improve accuracy of KVM clock when TSC scaling is in force
>       KVM: x86: Explicitly disable TSC scaling without CONSTANT_TSC
>       KVM: x86: Activate master clock immediately on vCPU creation
>       KVM: x86: Avoid NTP frequency skew for KVM clock on 32-bit host
>       KVM: x86: Fold __get_kvmclock() into get_kvmclock()
>       KVM: x86: Drop CPU pinning in get_kvmclock()
>       KVM: x86: Restructure get_kvmclock()
>       KVM: x86: Fix KVM clock precision in get_kvmclock() with TSC scaling
>       KVM: x86: Use get_kvmclock() in kvm_get_wall_clock_epoch()
>       KVM: x86: Fix compute_guest_tsc() to handle negative time deltas
>       KVM: x86: Restructure kvm_guest_time_update() for TSC upscaling
>       KVM: x86: Simplify and comment kvm_get_time_scale()

I'll post a new version up through here, get it applied soon-ish, and then get
back to chipping away at the rest of the series.  It'd obviously be nice to get
all the cleanups (and the tests) into 7.3, but I suspect the rest will take a
bit of back and worth (with Sashiko, if not me :-D).

I may punt on "Restructure kvm_guest_time_update() for TSC upscaling" if I have
trouble sorting through Sashiko's feedback, but I'll give it a shot.  I need to
page that all back in anyways.

>       KVM: x86: Remove implicit rdtsc() from kvm_compute_l1_tsc_offset()
>       KVM: x86: Improve synchronization in kvm_synchronize_tsc()
>       KVM: x86: Kill last_tsc_{nsec,write,offset} fields
>       KVM: x86: Replace nr_vcpus_matched_tsc count with all_vcpus_matched_tsc bool
>       KVM: x86: Allow KVM master clock mode when TSCs are offset from each other
>       KVM: x86: Factor out kvm_use_master_clock()
>       KVM: x86: Avoid gratuitous global clock updates
>       KVM: x86/xen: Prevent runstate times from becoming negative
>       KVM: x86: Avoid redundant masterclock updates from multiple vCPUs
>       KVM: x86: Remove runtime Xen TSC frequency CPUID update
>       KVM: x86: Re-synchronize TSC after KVM_SET_TSC_KHZ
>       KVM: x86: Use ktime_get_snapshot_id() for master clock
>       KVM: x86: Compute kvmclock base without pvclock_gtod_data
>       KVM: x86: Cache host vclock_mode for masterclock eligibility checks
>       KVM: x86: Remove pvclock_gtod_data and private timekeeping code
>       KVM: x86: Activate master clock from kvm_arch_init_vm()
>       KVM: selftests: Use UAPI pvclock-abi.h in xen_shinfo_test
>       KVM: x86: Add KVM_VCPU_TSC_SCALE and fix the documentation on TSC migration
>       KVM: selftests: Add master clock offset test
>       KVM: selftests: Add Xen/generic CPUID timing leaf test
>       KVM: selftests: Add Xen runstate migration test
> 
> Jack Allister (3):
>       UAPI: x86: Move pvclock-abi to UAPI for x86 platforms
>       KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration
>       KVM: selftests: Add KVM/PV clock selftest to prove timer correction
Re: [PATCH v7 00/36] Cleaning up the KVM clock mess
Posted by David Woodhouse 3 weeks, 5 days ago
On Tue, 2026-07-28 at 16:18 -0700, Sean Christopherson wrote:
> I'll post a new version up through here, get it applied soon-ish, and
> then get back to chipping away at the rest of the series.

Thanks. I'll leave that first part alone now, and rebase the remainder
on top of what lands. 

Not sure why Sashiko didn't reply to v6, but I've addressed the two
real bugs it found in v7:

 - "Re-synchronize TSC after KVM_SET_TSC_KHZ" made kvm_set_tsc_khz()
   call kvm_synchronize_tsc() with a non-NULL value, which set
   kvm->arch.user_set_tsc from the kernel's own vCPU-creation path and
   thus enabled the 1-second slop heuristic before userspace ever
   wrote a TSC — breaking restore of a guest with <1s of runtime.
   Fixed by making "user initiated" an explicit parameter instead of
   inferring it from the pointer being non-NULL. I've also added a
   selftest which fails on v7 and passes with the fix (first user
   write of a TSC value ~500ms of cycles from the kernel default gets
   snapped back to the default generation by the bug).

 - The S4 backwards-TSC adjustment in "Kill last_tsc_{nsec,write,
   offset} fields" adjusted cur_tsc_offset by the raw host-cycle delta
   while each vCPU's offset moves by the guest-scaled delta (via
   adjust_tsc_offset_host()), so they disagreed for scaled guests. And
   the cur_tsc_write adjustment shouldn't have existed at all — the
   guest TSC is kept continuous across S4 and the {cur_tsc_nsec,
   cur_tsc_write} pair lives in the wall-clock/guest-cycle domain.

Also fixed the unused-function breakage you hit in "Use
ktime_get_snapshot_id() for master clock" (the private readers are now
deleted in that patch, where their last callers go away), a stale
paragraph in the kvm_synchronize_tsc() changelog, the ΔT*freq unit
confusion in the TSC migration documentation, and a handful of
legitimate selftest nits from the same review.

All pushed to a kvmclock8 branch (which is identical up to commit
dcbe8dc4ca "Remove implicit rdtsc()"):
https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/kvmclock8
Re: [PATCH v7 00/36] Cleaning up the KVM clock mess
Posted by Sean Christopherson 2 weeks ago
On Tue, 28 Jul 2026 15:39:40 +0100, David Woodhouse wrote:
> This is v7 of the series to clean up the KVM clock, rebased onto
> kvm-x86/next (patch 1 of v6 is already merged there as commit
> 710b3a30f407).
> 
> The KVM clock has historically suffered from three problems:
> 
>  1. Imprecision: get_kvmclock_ns() computed the clock from the *host*
>     TSC without applying guest TSC scaling, causing systemic drift from
>     the values the guest computes from its own TSC.
> 
> [...]

Applied two more Xen patches to kvm-x86 clocks.

Given that KVM was updating the wrong sub-leaf (which amused me greatly), I
didn't see any point in waiting for the new uAPI updates to yank out the CPUID
updates.  Holler if you feel strongly that that change in particular should be
bundled with the kernel that adds the TSC scaling uAPI, omitting these changes
from 7.3 and waiting for 7.4 is trivial.  I'm just trying to reduce the number
of in-flight patches we need to keep track of.

Thanks!

[20/36] KVM: x86/xen: Prevent runstate times from becoming negative
        https://github.com/kvm-x86/linux/commit/36a85200643e

...

[22/36] KVM: x86: Remove runtime Xen TSC frequency CPUID update
        https://github.com/kvm-x86/linux/commit/7d3bd21e457b

--
https://github.com/kvm-x86/linux/tree/next
Re: [PATCH v7 00/36] Cleaning up the KVM clock mess
Posted by David Woodhouse 2 weeks ago
On Mon, 2026-08-10 at 09:42 -0700, Sean Christopherson wrote:
> 
> Applied two more Xen patches to kvm-x86 clocks.
> 
> Given that KVM was updating the wrong sub-leaf (which amused me greatly), I
> didn't see any point in waiting for the new uAPI updates to yank out the CPUID
> updates.  Holler if you feel strongly that that change in particular should be
> bundled with the kernel that adds the TSC scaling uAPI, omitting these changes
> from 7.3 and waiting for 7.4 is trivial.  I'm just trying to reduce the number
> of in-flight patches we need to keep track of.

Sounds good. Let me know when the dust has settled, and I can look at
rebasing the remaining parts on top of part 1. In the meantime I shall
continue to polish the GPC/RCU stuff, which looks like it's working
quite nicely now.