From nobody Tue Oct 7 05:43:01 2025 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F80C1E7C23 for ; Mon, 14 Jul 2025 03:37:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752464238; cv=none; b=Zi9PnNc1OchTK9KklRB4CYsas2qAtQlkNji8Y/Nl8sUNwdWn7wUXzBrxNdT9J8iAuCypFlseozfGm0XhcoiGGLQMTyNBTbu57Bdad46UuE/UWRzZ21kgi6sHp8iwrKxQ5hfFK4WPDEDC6RT52uTHxSz9RdpgVNkuqYCDhObfmE0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752464238; c=relaxed/simple; bh=mw5FigrZgTtmJOz9CWwbj+XpPfjGVB57QW5wwlLEhXg=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Cc:Content-Type; b=hFfCrf5xFtOhYpxAwpSmDFp83QzqZA5GnUKevcJD4wjZHMX65gwNFt2WS2Pqsl39aLskYArKxbwrhvMQTS2w4HGN3dGCRd0NcirHJqOjp41PRkpftkxcBttQn/fIbFnJjVOfWaXT8Lu/BuBqSD+Skse2Ku0vugyl/Yr3A1tUk24= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lTgs8mId; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lTgs8mId" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e81a449767aso4392310276.0 for ; Sun, 13 Jul 2025 20:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1752464234; x=1753069034; darn=vger.kernel.org; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oxJhV9NlBlE0YNzyyXRMKN2+XBXwbAmlO7pRJnVb+ZM=; b=lTgs8mIdhtNptbilVGOYeIB+jbRsvUWTPZk3v2pE2EzBNqC9uMF+2BZo2z3yN/lisT +J4+7W0L8tN12EeXzWyyuej1sxEuVekDjpl1ru4VJGlY1lSBNtZeIIbE1ozi/JF8Anlk /nfQbr5wzlKvSHoJOQox/YfAxkVwLk8SnWqw8hNfumkx0RjuQjNz1Un7MYyjopO7Hykc nz1wW2SovZt6LFFQDDnGgYhDP4KW+jAvr912JYPnlyBgvPHWSRT6cmKMtj0Ewd9KOW6/ PYQGpN9KI9fsT9WyQ0YQzpiX0EjxfzncHYCROb1/ly6SbzIXmKaTUcn5fDat4osIBSgp lt7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752464234; x=1753069034; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oxJhV9NlBlE0YNzyyXRMKN2+XBXwbAmlO7pRJnVb+ZM=; b=G3CSpAIu7ZIWFCajDFHpoy9jcr2/B+3b1D2sbXhuC7HKFumREsruQgfzQ/TJo+MsWa UrQ+OI2P33DROZbWuMyTzM+Tzvmj8c1+QDZJyPEEwb43DaLfSbwDLMsZDdy0VjYUDv0R tHBlyIpUu2PGckYG0we9W8p7ze9RSdQtaJ7WoLvY9Pn9ZjHO4bu0dVediIO0XgnmbfYv hrAJiukVfB1Nw857a2L5kISmAHQN4V0V9CZew+mE4H3qrXsLtcPySjvY6hJFyjuDaIqN wkl0pVShOqHGiU+Ah7/FMYiHMGJqwpkcsrq10ILR6DS12Ck6b3UbmHB1OOuSceL/Xphp GjAQ== X-Forwarded-Encrypted: i=1; AJvYcCWZS4DidovCCtuCx5TI+i6k5puIcpSFPefaKQiIg7qnlZoAc/uTle4jQy9H0xOUa46dXfsN9iR0PBjn4vg=@vger.kernel.org X-Gm-Message-State: AOJu0Yyf/uCEMVVeHNLbpkIDTLzchvhNqARYSm+Poq2U6wVkl1wCNiCI bnpT+tiHbBBibxWGYtMAnhTyNg7++822jsQpd7neMaTt+m7LqAw9Rd+T8EJMrlaryk0G4okVDw/ 2f0h8QNxvEAXe0A== X-Google-Smtp-Source: AGHT+IHKbV9ka5oFdT+zXuHEvk8R4LEZJ8+tvfQz+p5fpWhzt6Vy5KjvxcDRWPaoyblR40rIUlihiynW9uQsFg== X-Received: from suleiman1.tok.corp.google.com ([2401:fa00:8f:203:4f90:1ee3:198b:1fe4]) (user=suleiman job=sendgmr) by 2002:a25:b10f:0:b0:e86:1c5:3e2 with SMTP id 3f1490d57ef6-e8b85a39fd7mr5176276.3.1752464234542; Sun, 13 Jul 2025 20:37:14 -0700 (PDT) Date: Mon, 14 Jul 2025 12:36:47 +0900 In-Reply-To: <20250714033649.4024311-1-suleiman@google.com> Message-Id: <20250714033649.4024311-2-suleiman@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250714033649.4024311-1-suleiman@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Subject: [PATCH v7 1/3] KVM: x86: Advance guest TSC after deep suspend. From: Suleiman Souhlal To: Paolo Bonzini , Sean Christopherson Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Chao Gao , David Woodhouse , Sergey Senozhatsky , Konrad Rzeszutek Wilk , Tzung-Bi Shih , John Stultz , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, ssouhlal@freebsd.org, Suleiman Souhlal Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Try to advance guest TSC to current time after suspend when the host TSCs went backwards. This makes the behavior consistent between suspends where host TSC resets and suspends where it doesn't, such as suspend-to-idle, where in the former case if the host TSC resets, the guests' would previously be "frozen" due to KVM's backwards TSC prevention, while in the latter case they would advance. Suggested-by: Sean Christopherson Signed-off-by: Suleiman Souhlal Tested-by: Tzung-Bi Shih --- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/x86.c | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 7b9ccdd99f32..3650a513ba19 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1414,6 +1414,9 @@ struct kvm_arch { u64 cur_tsc_offset; u64 cur_tsc_generation; int nr_vcpus_matched_tsc; +#ifdef CONFIG_X86_64 + bool host_was_suspended; +#endif =20 u32 default_tsc_khz; bool user_set_tsc; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e21f5f2fe059..6539af701016 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5035,7 +5035,36 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int c= pu) =20 /* Apply any externally detected TSC adjustments (due to suspend) */ if (unlikely(vcpu->arch.tsc_offset_adjustment)) { +#ifdef CONFIG_X86_64 + unsigned long flags; + struct kvm *kvm; + bool advance; + u64 kernel_ns, l1_tsc, offset, tsc_now; + + kvm =3D vcpu->kvm; + advance =3D kvm_get_time_and_clockread(&kernel_ns, &tsc_now); + raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags); + /* + * Advance the guest's TSC to current time instead of only + * preventing it from going backwards, while making sure + * all the vCPUs use the same offset. + */ + if (kvm->arch.host_was_suspended && advance) { + l1_tsc =3D nsec_to_cycles(vcpu, + kvm->arch.kvmclock_offset + kernel_ns); + offset =3D kvm_compute_l1_tsc_offset(vcpu, l1_tsc); + kvm->arch.cur_tsc_offset =3D offset; + kvm_vcpu_write_tsc_offset(vcpu, offset); + } else if (advance) { + kvm_vcpu_write_tsc_offset(vcpu, kvm->arch.cur_tsc_offset); + } else { + adjust_tsc_offset_host(vcpu, vcpu->arch.tsc_offset_adjustment); + } + kvm->arch.host_was_suspended =3D false; + raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags); +#else adjust_tsc_offset_host(vcpu, vcpu->arch.tsc_offset_adjustment); +#endif /* CONFIG_X86_64 */ vcpu->arch.tsc_offset_adjustment =3D 0; kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); } @@ -12729,6 +12758,9 @@ int kvm_arch_enable_virtualization_cpu(void) kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); } =20 +#ifdef CONFIG_X86_64 + kvm->arch.host_was_suspended =3D true; +#endif /* * We have to disable TSC offset matching.. if you were * booting a VM while issuing an S4 host suspend.... --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Tue Oct 7 05:43:01 2025 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4CCE1F7910 for ; Mon, 14 Jul 2025 03:37:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752464241; cv=none; b=iyTWPwUnDX5Ktxzm2W6rsGkGL2RRdiP0s48Rfjf48atArFp8SaWL9ZLkCpur2eEeb8LKJffFiw/P0PPig6UKB6sPLIJxL3nUbSxzyBTYkJ0A+4ii2/ugrgCXj2NYrac8ttPwHvPemvPPOpqjvbdzdTr3XQtdxWv0L1QzBcmOR+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752464241; c=relaxed/simple; bh=bQh6lNTZFxebjSpM1cDa2wcwFwoS/xV7qOP5vuezGjc=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Cc:Content-Type; b=Cq7stM04lgzE29BPm9nUBXFFxmVVQlUB93zW3Q81aZHq+afCTLdCs3khm9Tuc/RUnRxmALMVbhn5hQJHXBcWAWNUmk7Rgn8byNOwe16NV1+ObwomjqOTlT7TNP5yB4n1eyO1GJNinLYKfpsM7n8Deje+XiyoBGPziGstrhpMXyo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=F8cihbEj; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="F8cihbEj" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e73290d75a8so5050867276.1 for ; Sun, 13 Jul 2025 20:37:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1752464239; x=1753069039; darn=vger.kernel.org; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t62AOuxhdPL2QOkp9lsjeQTVTW6/wnmvAqRNuUhfa18=; b=F8cihbEj/+EHfQ2bTRBbAi7kA91F7D0loGBWSUrBvbPF7l1igenpVcVTYCJJnZEGhb qO9f1F7aJ48WeWsxppuyAHsREcI/hVHI9l16e824MX2oR6V1HQwWHeqAjglLHNdpGBYt kIxw/U8Rl/3WX29fh2Vn/rHpCwolvUuzCKwB258sIjZKZvfa3kG3Vdq/Zwx+OqtIPDL3 07h/cBzLzqjp84UmKUoIGcbM6N13xksGKXGPNJZNLal/n8M7QAmsM9XW8z/6Cu+C/lhO n6fHml2+ThhM0nAZYAUmsc10Q1MFVJMRf9+7aPmwPJLDnI7SDg3WJdZcyHqtXOtqWTYU AoHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752464239; x=1753069039; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t62AOuxhdPL2QOkp9lsjeQTVTW6/wnmvAqRNuUhfa18=; b=ZT+3mHAsRZmZZfJ5idAMfSwoY3odlg0fyEnwxby1vP55N9+qnk0N+nxsEYQ+QHZlZT H/zHUl35f4cHhLmXfbLfrxP3RGHe4umrH/ZkY0p0DmD0OkqItFxfFFljjbb36IvXIc1T 091TYARLeyBjZrClUWj3H95A8gE0hoCMwnmhEmcdCF8D0e9gKFOifVmceF2qIUtDNTRV h7EWiBYg6CEHfvIYhYrluURzaZSj2jSauTLJHxE/wpfS7Ay/jRHMzhiW/BMuSlVIFbZL D4ey9nBXC1KI02tqKupTccAsCDF0ny6NJIzB1OLm1WkrmhXqeCgGdU1ZwZKo7hrcOmhs WwNg== X-Forwarded-Encrypted: i=1; AJvYcCUwsuxqJqfuM22C3uXxCx/WVuHGp6QMErZNfWE+Z3wNu1DGE3lnMeX6pRlx3DReNCxC86I9fU8C/wOV2GM=@vger.kernel.org X-Gm-Message-State: AOJu0YxZ6L2RxDv0BSfDMMB78VJ8aDBvZFYfxabiT9oTFDnXmIBMmml7 XxSppcAfIh7SrxNcbt9EFiyp01iRAeWn4G2DsiSdyjKuWAhCtRFbgSLJPdqmLq8Hf1iPiQJQz5n wUM9W4a+wXYB9ZQ== X-Google-Smtp-Source: AGHT+IEy0cYHEfq24aMv4LtmBYySF/u3vZZaiTjH0mJ/mH40FuntnizKzhahMzHk6OD8VcWQug97V1xgogNOag== X-Received: from suleiman1.tok.corp.google.com ([2401:fa00:8f:203:4f90:1ee3:198b:1fe4]) (user=suleiman job=sendgmr) by 2002:a25:dd41:0:b0:e85:fa6e:f58b with SMTP id 3f1490d57ef6-e8b8744f0f1mr2731276.9.1752464238272; Sun, 13 Jul 2025 20:37:18 -0700 (PDT) Date: Mon, 14 Jul 2025 12:36:48 +0900 In-Reply-To: <20250714033649.4024311-1-suleiman@google.com> Message-Id: <20250714033649.4024311-3-suleiman@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250714033649.4024311-1-suleiman@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Subject: [PATCH v7 2/3] KVM: x86: Include host suspended duration in steal time From: Suleiman Souhlal To: Paolo Bonzini , Sean Christopherson Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Chao Gao , David Woodhouse , Sergey Senozhatsky , Konrad Rzeszutek Wilk , Tzung-Bi Shih , John Stultz , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, ssouhlal@freebsd.org, Suleiman Souhlal Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce MSR_KVM_SUSPEND_STEAL which controls whether or not a guest wants the duration of host suspend to be included in steal time. This lets guests subtract the duration during which the host was suspended from the runtime of tasks that were running over the suspend, in order to prevent cases where host suspend causes long runtimes in guest tasks, even though their effective runtime was much shorter. Signed-off-by: Suleiman Souhlal --- Documentation/virt/kvm/x86/cpuid.rst | 4 ++ Documentation/virt/kvm/x86/msr.rst | 14 +++++ arch/x86/include/asm/kvm_host.h | 3 ++ arch/x86/include/uapi/asm/kvm_para.h | 2 + arch/x86/kvm/cpuid.c | 4 +- arch/x86/kvm/x86.c | 80 ++++++++++++++++++++++++++-- 6 files changed, 101 insertions(+), 6 deletions(-) diff --git a/Documentation/virt/kvm/x86/cpuid.rst b/Documentation/virt/kvm/= x86/cpuid.rst index bda3e3e737d7..71b42b649973 100644 --- a/Documentation/virt/kvm/x86/cpuid.rst +++ b/Documentation/virt/kvm/x86/cpuid.rst @@ -103,6 +103,10 @@ KVM_FEATURE_HC_MAP_GPA_RANGE 16 guest c= hecks this feature bit bef KVM_FEATURE_MIGRATION_CONTROL 17 guest checks this feature b= it before using MSR_KVM_MIGRATION_CON= TROL =20 +KVM_FEATURE_SUSPEND_STEAL 18 guest checks this feature b= it + before using + MSR_KVM_SUSPEND_STEAL. + KVM_FEATURE_CLOCKSOURCE_STABLE_BIT 24 host will warn if no guest-= side per-cpu warps are expected = in kvmclock diff --git a/Documentation/virt/kvm/x86/msr.rst b/Documentation/virt/kvm/x8= 6/msr.rst index 3aecf2a70e7b..7c33f9ee11f5 100644 --- a/Documentation/virt/kvm/x86/msr.rst +++ b/Documentation/virt/kvm/x86/msr.rst @@ -296,6 +296,12 @@ data: the amount of time in which this vCPU did not run, in nanoseconds. Time during which the vcpu is idle, will not be reported as steal time. + If the guest set the enable bit in MSR_KVM_SUSPEND_STEAL, + steal time includes the duration during which the host is + suspended. The case where the host suspends during a VM + migration might not be accounted if VCPUs aren't entered + post-resume. A workaround would be for the VMM to ensure that + the guest is entered with KVM_RUN after resuming from suspend. =20 preempted: indicate the vCPU who owns this struct is running or @@ -388,3 +394,11 @@ data: guest is communicating page encryption status to the host using the ``KVM_HC_MAP_GPA_RANGE`` hypercall, it can set bit 0 in this MSR to allow live migration of the guest. + +MSR_KVM_SUSPEND_STEAL: + 0x4b564d09 + +data: + This MSR is available if KVM_FEATURE_SUSPEND_STEAL is present in + CPUID. Bit 0 controls whether the host should include the duration it + has been suspended in steal time (1), or not (0). diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 3650a513ba19..015cf86b4e63 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -932,6 +932,8 @@ struct kvm_vcpu_arch { u8 preempted; u64 msr_val; u64 last_steal; + u64 suspend_ts; + atomic64_t suspend_ns; struct gfn_to_hva_cache cache; } st; =20 @@ -1028,6 +1030,7 @@ struct kvm_vcpu_arch { } pv_eoi; =20 u64 msr_kvm_poll_control; + u64 msr_kvm_suspend_steal; =20 /* pv related host specific info */ struct { diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/a= sm/kvm_para.h index a1efa7907a0b..678ebc3d7eeb 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -36,6 +36,7 @@ #define KVM_FEATURE_MSI_EXT_DEST_ID 15 #define KVM_FEATURE_HC_MAP_GPA_RANGE 16 #define KVM_FEATURE_MIGRATION_CONTROL 17 +#define KVM_FEATURE_SUSPEND_STEAL 18 =20 #define KVM_HINTS_REALTIME 0 =20 @@ -58,6 +59,7 @@ #define MSR_KVM_ASYNC_PF_INT 0x4b564d06 #define MSR_KVM_ASYNC_PF_ACK 0x4b564d07 #define MSR_KVM_MIGRATION_CONTROL 0x4b564d08 +#define MSR_KVM_SUSPEND_STEAL 0x4b564d09 =20 struct kvm_steal_time { __u64 steal; diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index b2d006756e02..983867f243ca 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -1614,8 +1614,10 @@ static inline int __do_cpuid_func(struct kvm_cpuid_a= rray *array, u32 function) (1 << KVM_FEATURE_PV_SCHED_YIELD) | (1 << KVM_FEATURE_ASYNC_PF_INT); =20 - if (sched_info_on()) + if (sched_info_on()) { entry->eax |=3D (1 << KVM_FEATURE_STEAL_TIME); + entry->eax |=3D (1 << KVM_FEATURE_SUSPEND_STEAL); + } =20 entry->ebx =3D 0; entry->ecx =3D 0; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 6539af701016..1535f653f942 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3753,6 +3753,8 @@ static void record_steal_time(struct kvm_vcpu *vcpu) steal +=3D current->sched_info.run_delay - vcpu->arch.st.last_steal; vcpu->arch.st.last_steal =3D current->sched_info.run_delay; + if (unlikely(atomic64_read(&vcpu->arch.st.suspend_ns))) + steal +=3D atomic64_xchg(&vcpu->arch.st.suspend_ns, 0); unsafe_put_user(steal, &st->steal, out); =20 version +=3D 1; @@ -4058,6 +4060,17 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct= msr_data *msr_info) vcpu->arch.msr_kvm_poll_control =3D data; break; =20 + case MSR_KVM_SUSPEND_STEAL: + if (!guest_pv_has(vcpu, KVM_FEATURE_SUSPEND_STEAL) || + !guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) + return 1; + + if (!(data & KVM_MSR_ENABLED)) + return 1; + + vcpu->arch.msr_kvm_suspend_steal =3D data; + break; + case MSR_IA32_MCG_CTL: case MSR_IA32_MCG_STATUS: case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: @@ -4404,6 +4417,11 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct= msr_data *msr_info) =20 msr_info->data =3D vcpu->arch.msr_kvm_poll_control; break; + case MSR_KVM_SUSPEND_STEAL: + if (!guest_pv_has(vcpu, KVM_FEATURE_SUSPEND_STEAL)) + return 1; + msr_info->data =3D vcpu->arch.msr_kvm_suspend_steal; + break; case MSR_IA32_P5_MC_ADDR: case MSR_IA32_P5_MC_TYPE: case MSR_IA32_MCG_CAP: @@ -7010,13 +7028,52 @@ static int kvm_arch_suspend_notifier(struct kvm *kv= m) { struct kvm_vcpu *vcpu; unsigned long i; + bool kick_vcpus =3D false; =20 - /* - * Ignore the return, marking the guest paused only "fails" if the vCPU - * isn't using kvmclock; continuing on is correct and desirable. - */ - kvm_for_each_vcpu(i, vcpu, kvm) + kvm_for_each_vcpu(i, vcpu, kvm) { + if (vcpu->arch.msr_kvm_suspend_steal & KVM_MSR_ENABLED) { + kick_vcpus =3D true; + WRITE_ONCE(vcpu->arch.st.suspend_ts, + ktime_get_boottime_ns()); + } + /* + * Ignore the return, marking the guest paused only "fails" if + * the vCPU isn't using kvmclock; continuing on is correct and + * desirable. + */ (void)kvm_set_guest_paused(vcpu); + } + + if (kick_vcpus) + kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); + + return NOTIFY_DONE; +} + +static int +kvm_arch_resume_notifier(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + + kvm_for_each_vcpu(i, vcpu, kvm) { + u64 suspend_ns =3D ktime_get_boottime_ns() - + vcpu->arch.st.suspend_ts; + + WRITE_ONCE(vcpu->arch.st.suspend_ts, 0); + + /* + * Only accumulate the suspend time if suspend steal-time is + * enabled, but always clear suspend_ts and kick the vCPU as + * the vCPU could have disabled suspend steal-time after the + * suspend notifier grabbed suspend_ts. + */ + if (vcpu->arch.msr_kvm_suspend_steal & KVM_MSR_ENABLED) + atomic64_add(suspend_ns, &vcpu->arch.st.suspend_ns); + + kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); + kvm_vcpu_kick(vcpu); + } =20 return NOTIFY_DONE; } @@ -7027,6 +7084,9 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned lo= ng state) case PM_HIBERNATION_PREPARE: case PM_SUSPEND_PREPARE: return kvm_arch_suspend_notifier(kvm); + case PM_POST_HIBERNATION: + case PM_POST_SUSPEND: + return kvm_arch_resume_notifier(kvm); } =20 return NOTIFY_DONE; @@ -11216,6 +11276,16 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) =20 static bool kvm_vcpu_running(struct kvm_vcpu *vcpu) { + /* + * During host SUSPEND/RESUME tasks get frozen after SUSPEND notifiers + * run, and thawed before RESUME notifiers, i.e. vCPUs can be actively + * running when KVM sees the system as suspended. Block the vCPU if + * KVM sees the vCPU as suspended to ensure the suspend steal time is + * accounted before the guest can run, and to the correct guest task. + */ + if (READ_ONCE(vcpu->arch.st.suspend_ts)) + return false; + return (vcpu->arch.mp_state =3D=3D KVM_MP_STATE_RUNNABLE && !vcpu->arch.apf.halted); } --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Tue Oct 7 05:43:01 2025 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CED11FE47B for ; Mon, 14 Jul 2025 03:37:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752464246; cv=none; b=FU6TwOXOSsydoF8b3RQRI+nz2BxcgeB5P6D1KfCi/zcfNp4z7LWwGoW//V5GO6XhUvJU+TN6wH53qs/BmxU4q0Fw++yXvzIuu0/yonvzzTKWb5m19gujRnCFNqGjAeeTJLJHm7LI5cUGqNc6qYny6ynShbJxlwALE4qsz2LMGrA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752464246; c=relaxed/simple; bh=Iw8UgECeSh4MOBXDjEL7rz+cVwwQWB07OSoxUeNK10o=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Cc:Content-Type; b=rtWf7Fo2xeH7kraA2XnhvDTz+0qgNN63SBcAGdvZqwCKMp8NDRn5KZ6c7XySMG0gvc4FYqzk2/NQ+qWeuhW/b6ZZdUH8unZjW/s6cYU7GjeEaAlvqHiksJDzdxgTNT6CsCtlN0iUTTSgRsqRiR2DE3IctjJRlj7olVNE6olDcWc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ilgu4MnR; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ilgu4MnR" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-236725af87fso64848085ad.3 for ; Sun, 13 Jul 2025 20:37:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1752464243; x=1753069043; darn=vger.kernel.org; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IuCOV0BBhXpldFkF7SjoypukvpXxsnclSm2cFgamVis=; b=ilgu4MnRFiqrR5o4Iqtwhst7MJsLhRAyzNxSu13FLX/rFXwEPKqM7NqvPyaGjHofnp DBxn8O2ezQ7X/azijNos0qUcezREd/2BKcx1tLA3nWSE5rfjGarxB8BGZoWPRvWv5FJx yqgKIvpLgRPb4W/b3YqHI1886i6M3pU51VAhTOO8pUtBuNBGi4Zt3f1JK+Ak+eGTcv4K Wp8iBTCAvsHQM1ykdd0ZUv95MPG2ciIEa2LRrE6eT1nZ9+XcvvcYZoxIYfKPRLiz/OL8 ajbDnfDyi3i2Zl8LfrCk/Ag2ov6GPAHsAmzCL42vfZtZ4m/sxcYNGHY3uL30PBxYzBI7 gXaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752464243; x=1753069043; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IuCOV0BBhXpldFkF7SjoypukvpXxsnclSm2cFgamVis=; b=GR+1BucLrN62vlotTBxgCoSnHZBmG1M9EQZPZ9g+ogaAr/3GukJNQHLZaN/QV3FsmA kfX/AbmsRDu4GwFwsIrgrEQL3PZTQDGApugqv4bkGjE/HopTuMJApwtCNSfNOf/55gR1 0bYhWWauQJ5/5AAftejjgC+kw4o+VEFmMtNiaLH355fLDoVwQ2OT1TNPFAhfsd9lwA3k KpHRcodEo5/dxkaAiWoxvgffegnunkDm2++RUBq+eEWjIivw+eTGrvNSW0yNhRby25jv CUNz+x1Au4TYr/WwwH58ZkA8rZgyoGuqbTUXTOnzHUkaSdY5YBFm2qY9iBBfQt+uYd53 0Wbg== X-Forwarded-Encrypted: i=1; AJvYcCVBXOAzt17VYFUXhVQgO1ASTgdNOU+IDkMx/CBavAGsAaocYONxJuXq+e8Bu9tS5wWPWIghKdsdgxxSDbk=@vger.kernel.org X-Gm-Message-State: AOJu0YxdUeA6xfGudC/vcdfNURa8tynJHE/vQxtFdW0Ma4Eh28Eg2kwa JmjWnF4Yi6YGMcZcihLOsp13FIHH24R8f9MMfN7y7KOGbZSs/5bB93jDyIrknc4QTAR99UZ5Nxh 312mH3XOt9FmLCQ== X-Google-Smtp-Source: AGHT+IFrBZOkLURtmxf4lvSzOVmE0ai7SJHbgn0xbbIb3ggEOAP+Yzu5GmOIjRZRMCD8thfwgdWkLGsD3AzRyw== X-Received: from suleiman1.tok.corp.google.com ([2401:fa00:8f:203:4f90:1ee3:198b:1fe4]) (user=suleiman job=sendgmr) by 2002:a17:903:348d:b0:234:ddd7:5c16 with SMTP id d9443c01a7336-23dede708ddmr9195ad.4.1752464242498; Sun, 13 Jul 2025 20:37:22 -0700 (PDT) Date: Mon, 14 Jul 2025 12:36:49 +0900 In-Reply-To: <20250714033649.4024311-1-suleiman@google.com> Message-Id: <20250714033649.4024311-4-suleiman@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250714033649.4024311-1-suleiman@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Subject: [PATCH v7 3/3] KVM: x86: Add "suspendsteal" cmdline to request host to add suspend duration in steal time From: Suleiman Souhlal To: Paolo Bonzini , Sean Christopherson Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Chao Gao , David Woodhouse , Sergey Senozhatsky , Konrad Rzeszutek Wilk , Tzung-Bi Shih , John Stultz , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, ssouhlal@freebsd.org, Suleiman Souhlal Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a new command line parameter, "suspendsteal", enabling the guest to use MSR_KVM_SUSPEND_STEAL, which tells the host that it would like host suspend duration to be included in steal time. Signed-off-by: Suleiman Souhlal --- Documentation/admin-guide/kernel-parameters.txt | 5 +++++ arch/x86/kernel/kvm.c | 15 +++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index f1f2c0874da9..9f5758ca8fad 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7074,6 +7074,11 @@ improve throughput, but will also increase the amount of memory reserved for use by the client. =20 + suspendsteal + [X86,PV_OPS] + Enable requesting the host to include the duration the + host was suspended in steal time. Disabled by default. + suspend.pm_test_delay=3D [SUSPEND] Sets the number of seconds to remain in a suspend test diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 921c1c783bc1..35d1bb2283c2 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -320,6 +320,18 @@ static void __init paravirt_ops_setup(void) #endif } =20 +static bool suspend_steal; + +static int __init suspendsteal_setup(char *s) +{ + if (kvm_para_has_feature(KVM_FEATURE_SUSPEND_STEAL)) + suspend_steal =3D true; + + return 0; +} + +early_param("suspendsteal", suspendsteal_setup); + static void kvm_register_steal_time(void) { int cpu =3D smp_processor_id(); @@ -331,6 +343,9 @@ static void kvm_register_steal_time(void) wrmsrq(MSR_KVM_STEAL_TIME, (slow_virt_to_phys(st) | KVM_MSR_ENABLED)); pr_debug("stealtime: cpu %d, msr %llx\n", cpu, (unsigned long long) slow_virt_to_phys(st)); + + if (suspend_steal) + wrmsrl(MSR_KVM_SUSPEND_STEAL, KVM_MSR_ENABLED); } =20 static DEFINE_PER_CPU_DECRYPTED(unsigned long, kvm_apic_eoi) =3D KVM_PV_EO= I_DISABLED; --=20 2.50.0.727.gbf7dc18ff4-goog