From nobody Tue Oct 7 16:53:00 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EAF45270571 for ; Wed, 9 Jul 2025 07:05:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044705; cv=none; b=Nv/Xfu3r2iGME6RysYJaEM8O8tWTi6C7gk4PdsQvB9mmt/kMBKYpIDsF9vE70gX2vD/u14oe/y+wrtrzipWagaUq74fiSxZoT6RD9HZ23uV7YehLExie2nsMijd7bpQPpfnFnNGyVDHy6tngTzNH1XpMHMd1Tg26szqQMUVK1Ik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044705; c=relaxed/simple; bh=8acnn84F2EPwPo/2bUKI9y8xzYUhx0M85P0qcx2e/FM=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Cc:Content-Type; b=q+I9mkZUWQ19BQybBBfN9RtuJstXkAyFVBnYLUfowEzG1glaM3iaK6cjhWj2N0nBXMlBKZ8786Pcn9pYirR5iZF1hd3yilK4u5rZ8K4F9uJKU8w7tHM7y80dPVQvmhKNxu8mDHCj2Cg0E5Pib4z222yJeGeueRMaBmhQaY7Zu3s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AQx3t1PB; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AQx3t1PB" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-70f841fb19dso65537047b3.1 for ; Wed, 09 Jul 2025 00:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1752044703; x=1752649503; darn=vger.kernel.org; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8hUdo8ARlQkzGI2qXgVk3yoTg5wBj9RNjVmfVyEYsuY=; b=AQx3t1PBdqBcHVVibM34HQUd7PMnwcfc377Cuue613zxiv6KGDFvFYiefSSc5Aanon yZ8UeafT+Ic3HW50/nEVdwtenrRULjtZcTkPbFA+JoqAiBCaUJ1Oo6eb7XDD+uXYMTqP dHqCjElSGxIGHtF/H3S9z3lcimsNih9EuI75DbVYQYQEQnw1MbnsTR9MH8rd2ZZ3zaVp b9lj7STUCTJlBixoWp3tHBPRhfJTA+3yL/ZXht4PeEJ7bazmh6Dt/8NXh99KBzfu+FXB 4mL0//2/GuvienP3HZvZtnCuEeP2B7OuJEBAgwKuNw6QudM6PwDKALrzJnpoNFPGmICg yFSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752044703; x=1752649503; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8hUdo8ARlQkzGI2qXgVk3yoTg5wBj9RNjVmfVyEYsuY=; b=PnJEgxFAMRCkurfR84kSVQWwk++Wt/jip2ZEM6g7u+ZpRMGeCCPJeYbt8YNjbe0FRN 4LESbTE0TtpjE9oE3/6TWaVi24GGPya5qTXSaNTHr3T9Zhm0G5IRkjW+EuL9nx8VS/GK cT43l41CmjRPOdD3pXsQ9OnFLammI0uvm/LdGs7jxdYw4O48bhyWtjUvnC/QGiurNifZ AHF/Mpr0QXeTqRylSmtV+1Ibr1j7SwzAKcHFv+PAGEl+xjPD5EWGpT34gyHzIdftZga9 KFrr0s/WrdC2LOp4Px8cOu49+4rJTwRhQ2+8ljOkYHxd67Zlq/4sSni67AOKybLj4Va0 pKVQ== X-Forwarded-Encrypted: i=1; AJvYcCXQ8xOQUZJCXmceLC+F1TyOM5b/mk+k6u3FFC2KmJLC/bvQzmGYQkjKbasyaYtCeKUnE1wPsDXB5HKYDHw=@vger.kernel.org X-Gm-Message-State: AOJu0Yyq0/vZ85qmXqBUlgdSERpN+8qqMa9iETsc6w7CPMt7fbNR2Viy 17fGoiQy1F2gUTPc5Uek5jhRaW2/MWpV6BuDrWzJtw8nQFGBp7H62vs5YM9iPZCcHtQORzOZVyO AjxuQJYZRkuLupA== X-Google-Smtp-Source: AGHT+IGEfiidPbWbXfAEZ47CmcdUs9ZPOOEtK0ixwyyLTU/yNC0ka1UYfmzV8mQy6TonmuKtwlqV6SEZTjJqyg== X-Received: from suleiman1.tok.corp.google.com ([2401:fa00:8f:203:a92c:694f:82fe:62a]) (user=suleiman job=sendgmr) by 2002:a25:d344:0:b0:e7d:6afc:71da with SMTP id 3f1490d57ef6-e8b6e10d3a8mr709276.2.1752044702827; Wed, 09 Jul 2025 00:05:02 -0700 (PDT) Date: Wed, 9 Jul 2025 16:04:49 +0900 In-Reply-To: <20250709070450.473297-1-suleiman@google.com> Message-Id: <20250709070450.473297-3-suleiman@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250709070450.473297-1-suleiman@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Subject: [PATCH v6 2/3] KVM: x86: Include host suspended duration in steal time From: Suleiman Souhlal To: Paolo Bonzini , Sean Christopherson Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Chao Gao , David Woodhouse , Sergey Senozhatsky , Konrad Rzeszutek Wilk , Tzung-Bi Shih , John Stultz , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, ssouhlal@freebsd.org, Suleiman Souhlal Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce MSR_KVM_SUSPEND_STEAL which controls whether or not a guest wants the duration of host suspend to be included in steal time. This lets guests subtract the duration during which the host was suspended from the runtime of tasks that were running over the suspend, in order to prevent cases where host suspend causes long runtimes in guest tasks, even though their effective runtime was much shorter. Signed-off-by: Suleiman Souhlal --- Documentation/virt/kvm/x86/cpuid.rst | 4 ++ Documentation/virt/kvm/x86/msr.rst | 15 ++++++ arch/x86/include/asm/kvm_host.h | 3 ++ arch/x86/include/uapi/asm/kvm_para.h | 2 + arch/x86/kvm/cpuid.c | 4 +- arch/x86/kvm/x86.c | 80 ++++++++++++++++++++++++++-- 6 files changed, 102 insertions(+), 6 deletions(-) diff --git a/Documentation/virt/kvm/x86/cpuid.rst b/Documentation/virt/kvm/= x86/cpuid.rst index bda3e3e737d71f..71b42b6499733b 100644 --- a/Documentation/virt/kvm/x86/cpuid.rst +++ b/Documentation/virt/kvm/x86/cpuid.rst @@ -103,6 +103,10 @@ KVM_FEATURE_HC_MAP_GPA_RANGE 16 guest c= hecks this feature bit bef KVM_FEATURE_MIGRATION_CONTROL 17 guest checks this feature b= it before using MSR_KVM_MIGRATION_CON= TROL =20 +KVM_FEATURE_SUSPEND_STEAL 18 guest checks this feature b= it + before using + MSR_KVM_SUSPEND_STEAL. + KVM_FEATURE_CLOCKSOURCE_STABLE_BIT 24 host will warn if no guest-= side per-cpu warps are expected = in kvmclock diff --git a/Documentation/virt/kvm/x86/msr.rst b/Documentation/virt/kvm/x8= 6/msr.rst index 3aecf2a70e7b43..8c31b3994e733d 100644 --- a/Documentation/virt/kvm/x86/msr.rst +++ b/Documentation/virt/kvm/x86/msr.rst @@ -296,6 +296,12 @@ data: the amount of time in which this vCPU did not run, in nanoseconds. Time during which the vcpu is idle, will not be reported as steal time. + If the guest set the enable bit in MSR_KVM_SUSPEND_STEAL, + steal time includes the duration during which the host is + suspended. The case where the host suspends during a VM + migration might not be accounted if VCPUs aren't entered + post-resume. A workaround would be for the VMM to ensure that + the guest is entered with KVM_RUN after resuming from suspend. =20 preempted: indicate the vCPU who owns this struct is running or @@ -388,3 +394,12 @@ data: guest is communicating page encryption status to the host using the ``KVM_HC_MAP_GPA_RANGE`` hypercall, it can set bit 0 in this MSR to allow live migration of the guest. + +MSR_KVM_SUSPEND_STEAL: + 0x4b564d09 + +data: + This MSR is available if KVM_FEATURE_SUSPEND_STEAL is present in + CPUID. Bit 0 controls whether the host should include the duration it + has been suspended in steal time (1), or not (0). + diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 5c465bdd6d088a..b35099260aa625 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -932,6 +932,8 @@ struct kvm_vcpu_arch { u8 preempted; u64 msr_val; u64 last_steal; + u64 suspend_ts; + atomic64_t suspend_ns; struct gfn_to_hva_cache cache; } st; =20 @@ -1028,6 +1030,7 @@ struct kvm_vcpu_arch { } pv_eoi; =20 u64 msr_kvm_poll_control; + u64 msr_kvm_suspend_steal; =20 /* pv related host specific info */ struct { diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/a= sm/kvm_para.h index a1efa7907a0b10..678ebc3d7eeb55 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -36,6 +36,7 @@ #define KVM_FEATURE_MSI_EXT_DEST_ID 15 #define KVM_FEATURE_HC_MAP_GPA_RANGE 16 #define KVM_FEATURE_MIGRATION_CONTROL 17 +#define KVM_FEATURE_SUSPEND_STEAL 18 =20 #define KVM_HINTS_REALTIME 0 =20 @@ -58,6 +59,7 @@ #define MSR_KVM_ASYNC_PF_INT 0x4b564d06 #define MSR_KVM_ASYNC_PF_ACK 0x4b564d07 #define MSR_KVM_MIGRATION_CONTROL 0x4b564d08 +#define MSR_KVM_SUSPEND_STEAL 0x4b564d09 =20 struct kvm_steal_time { __u64 steal; diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index b2d006756e02a6..983867f243cafc 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -1614,8 +1614,10 @@ static inline int __do_cpuid_func(struct kvm_cpuid_a= rray *array, u32 function) (1 << KVM_FEATURE_PV_SCHED_YIELD) | (1 << KVM_FEATURE_ASYNC_PF_INT); =20 - if (sched_info_on()) + if (sched_info_on()) { entry->eax |=3D (1 << KVM_FEATURE_STEAL_TIME); + entry->eax |=3D (1 << KVM_FEATURE_SUSPEND_STEAL); + } =20 entry->ebx =3D 0; entry->ecx =3D 0; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e66bab1a1f56e2..691f46b5e50f27 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3753,6 +3753,8 @@ static void record_steal_time(struct kvm_vcpu *vcpu) steal +=3D current->sched_info.run_delay - vcpu->arch.st.last_steal; vcpu->arch.st.last_steal =3D current->sched_info.run_delay; + if (unlikely(atomic64_read(&vcpu->arch.st.suspend_ns))) + steal +=3D atomic64_xchg(&vcpu->arch.st.suspend_ns, 0); unsafe_put_user(steal, &st->steal, out); =20 version +=3D 1; @@ -4058,6 +4060,17 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct= msr_data *msr_info) vcpu->arch.msr_kvm_poll_control =3D data; break; =20 + case MSR_KVM_SUSPEND_STEAL: + if (!guest_pv_has(vcpu, KVM_FEATURE_SUSPEND_STEAL) || + !guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) + return 1; + + if (!(data & KVM_MSR_ENABLED)) + return 1; + + vcpu->arch.msr_kvm_suspend_steal =3D data; + break; + case MSR_IA32_MCG_CTL: case MSR_IA32_MCG_STATUS: case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: @@ -4404,6 +4417,11 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct= msr_data *msr_info) =20 msr_info->data =3D vcpu->arch.msr_kvm_poll_control; break; + case MSR_KVM_SUSPEND_STEAL: + if (!guest_pv_has(vcpu, KVM_FEATURE_SUSPEND_STEAL)) + return 1; + msr_info->data =3D vcpu->arch.msr_kvm_suspend_steal; + break; case MSR_IA32_P5_MC_ADDR: case MSR_IA32_P5_MC_TYPE: case MSR_IA32_MCG_CAP: @@ -7006,13 +7024,52 @@ static int kvm_arch_suspend_notifier(struct kvm *kv= m) { struct kvm_vcpu *vcpu; unsigned long i; + bool kick_vcpus =3D false; =20 - /* - * Ignore the return, marking the guest paused only "fails" if the vCPU - * isn't using kvmclock; continuing on is correct and desirable. - */ - kvm_for_each_vcpu(i, vcpu, kvm) + kvm_for_each_vcpu(i, vcpu, kvm) { + if (vcpu->arch.msr_kvm_suspend_steal & KVM_MSR_ENABLED) { + kick_vcpus =3D true; + WRITE_ONCE(vcpu->arch.st.suspend_ts, + ktime_get_boottime_ns()); + } + /* + * Ignore the return, marking the guest paused only "fails" if + * the vCPU isn't using kvmclock; continuing on is correct and + * desirable. + */ (void)kvm_set_guest_paused(vcpu); + } + + if (kick_vcpus) + kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); + + return NOTIFY_DONE; +} + +static int +kvm_arch_resume_notifier(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + + kvm_for_each_vcpu(i, vcpu, kvm) { + u64 suspend_ns =3D ktime_get_boottime_ns() - + vcpu->arch.st.suspend_ts; + + WRITE_ONCE(vcpu->arch.st.suspend_ts, 0); + + /* + * Only accumulate the suspend time if suspend steal-time is + * enabled, but always clear suspend_ts and kick the vCPU as + * the vCPU could have disabled suspend steal-time after the + * suspend notifier grabbed suspend_ts. + */ + if (vcpu->arch.msr_kvm_suspend_steal & KVM_MSR_ENABLED) + atomic64_add(suspend_ns, &vcpu->arch.st.suspend_ns); + + kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); + kvm_vcpu_kick(vcpu); + } =20 return NOTIFY_DONE; } @@ -7023,6 +7080,9 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned lo= ng state) case PM_HIBERNATION_PREPARE: case PM_SUSPEND_PREPARE: return kvm_arch_suspend_notifier(kvm); + case PM_POST_HIBERNATION: + case PM_POST_SUSPEND: + return kvm_arch_resume_notifier(kvm); } =20 return NOTIFY_DONE; @@ -11212,6 +11272,16 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) =20 static bool kvm_vcpu_running(struct kvm_vcpu *vcpu) { + /* + * During host SUSPEND/RESUME tasks get frozen after SUSPEND notifiers + * run, and thawed before RESUME notifiers, i.e. vCPUs can be actively + * running when KVM sees the system as suspended. Block the vCPU if + * KVM sees the vCPU as suspended to ensure the suspend steal time is + * accounted before the guest can run, and to the correct guest task. + */ + if (READ_ONCE(vcpu->arch.st.suspend_ts)) + return false; + return (vcpu->arch.mp_state =3D=3D KVM_MP_STATE_RUNNABLE && !vcpu->arch.apf.halted); } --=20 2.50.0.727.gbf7dc18ff4-goog