From nobody Tue Oct 7 14:57:04 2025 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45D3526CE3B for ; Wed, 9 Jul 2025 07:05:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044701; cv=none; b=T6+aiyq7HcBb8gJKpSqxEes5tkv9zNHFCGgCpGGPjCxXSWLUIZDKctL0dUsvrO6Nop395yK4EHTivSuQ0MMeJrOYPdvEr4WBed8m9bpkIOx7v8/UxmXPXs8PdPiXwkcPk5py0j6NHo6/fBqD8ycRyTE3k4mnEdbkMazkG3wr/44= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044701; c=relaxed/simple; bh=b1ROLBUa4PfoqSlkIa9V4t2N2ShKCwEmh/fAeQ9vc1A=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Cc:Content-Type; b=uLT0swuGCHN9gHU0jCg4pjaare8ugAZji/A3x0fosndAgEZoF3xusIsNrjmMsE38LO7SqtT384lNZq15xBD+3Kg/VthGke/urKsz1OoqSNG2eGMkPUarQRc+MWtJSAfTQq7XKBp4MbeBU3ZVh1EKoyhxPeXU85+AS0OuKCvhPOg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=wU0tuLLR; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="wU0tuLLR" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e8973c943d6so1133283276.0 for ; Wed, 09 Jul 2025 00:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1752044699; x=1752649499; darn=vger.kernel.org; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=A2BQcTBupd9j0a9HC05i7mP4BWhaEQOOI0kqaY+fjtk=; b=wU0tuLLRkVAg/IsAzEgIHe+wOuRLtmt15BlJ7vF7nrXyWwaprADj06rUkOBnQycogI UN+s459AXOrGVT7Zm7JTaE2syJMAjm8ivbqthtLhNTKxPExb3fwzqigw3lT6bqZAOEA1 2yaGlOV7kdZZ65loKF8yZkzvn3KdzLAg/DGn05Z2DAyLcwLtfEG8foKxgnd2Gv6WW6V9 jPZ7TD0SiaZbuMKo2UGXh04NF4ruOxHEFuPDLd/aLLBAgnOrzjZ+ZrDP6UZKfXiX1mQb 5lsvOxooEooGdrbYUxX+Fmn4rWWG+SvlQ9cmFF9Yo2QUj7KxnIGHDo+NY2oAcn9XzbWw mJlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752044699; x=1752649499; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=A2BQcTBupd9j0a9HC05i7mP4BWhaEQOOI0kqaY+fjtk=; b=UjPyiLDLON/PsJXkQWHbsBOzmO9zHYGm6rLhyHvxocjyvFotlCrQOtn4TY8mVKD2TR RF1YIgjxbIkdcxFqrKCHaJJy9ca64wV0uK5HerhlFrSp88P8uhvTcibL+0yuKH9et8Zr t+sQCkEvuiGogqdncKUGZaFF0YBDdadHjtDWzfdYDSQ4NeYdjDltwmTmeBWW49z+ZTA2 30gMdSMBOpZWkZOlMKdUYlwSGmOALTw6OZAPBBztltpmhp/y20u2VRIRcAFdj/jJD+pF ec7IrazN/VsDwhrzQ2qMfpxBZbnM2E5Zv6e27cQFfG0M40MWqBrTezcCTdSJJZMOnsJ4 vJUg== X-Forwarded-Encrypted: i=1; AJvYcCU/IRAQ6yW01wtoBcRPGHeJDCbzcJPctUaZfzi2Dy8bcp0FtB9C9vucCP2TvjLof139NEyAtlxzTr9GTEM=@vger.kernel.org X-Gm-Message-State: AOJu0YxQGV07XIQJSTyqOLq17C8NHMo79d1IusOtSzePafF5VJPPZnTg Eh1Uyky5hXtaUuNa6hxJOoQS/tnIVPeQSO/mWGzCAIFOOadgyEWQke0sUlpRjUH4E9RUT0OffBE r89DTqPY38ucosw== X-Google-Smtp-Source: AGHT+IFq26cqQyA00Q/GYkwS611peRZ2ZRuPlGXXHtSsrI7go/c7F2mBXe5ybETtjGiZXawkvn9U7II0Kz+zeQ== X-Received: from suleiman1.tok.corp.google.com ([2401:fa00:8f:203:a92c:694f:82fe:62a]) (user=suleiman job=sendgmr) by 2002:a25:3d81:0:b0:e87:9e04:4c30 with SMTP id 3f1490d57ef6-e8b6f4db84fmr865276.4.1752044699070; Wed, 09 Jul 2025 00:04:59 -0700 (PDT) Date: Wed, 9 Jul 2025 16:04:48 +0900 In-Reply-To: <20250709070450.473297-1-suleiman@google.com> Message-Id: <20250709070450.473297-2-suleiman@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250709070450.473297-1-suleiman@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Subject: [PATCH v6 1/3] KVM: x86: Advance guest TSC after deep suspend. From: Suleiman Souhlal To: Paolo Bonzini , Sean Christopherson Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Chao Gao , David Woodhouse , Sergey Senozhatsky , Konrad Rzeszutek Wilk , Tzung-Bi Shih , John Stultz , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, ssouhlal@freebsd.org, Suleiman Souhlal Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Advance guest TSC to current time after suspend when the host TSCs went backwards. This makes the behavior consistent between suspends where host TSC resets and suspends where it doesn't, such as suspend-to-idle, where in the former case if the host TSC resets, the guests' would previously be "frozen" due to KVM's backwards TSC prevention, while in the latter case they would advance. Suggested-by: Sean Christopherson Signed-off-by: Suleiman Souhlal --- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 28 +++++++++++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 639d9bcee8424d..5c465bdd6d088a 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1414,6 +1414,7 @@ struct kvm_arch { u64 cur_tsc_offset; u64 cur_tsc_generation; int nr_vcpus_matched_tsc; + bool host_was_suspended; =20 u32 default_tsc_khz; bool user_set_tsc; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index a9d992d5652fa0..e66bab1a1f56e2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5035,7 +5035,32 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int c= pu) =20 /* Apply any externally detected TSC adjustments (due to suspend) */ if (unlikely(vcpu->arch.tsc_offset_adjustment)) { - adjust_tsc_offset_host(vcpu, vcpu->arch.tsc_offset_adjustment); + unsigned long flags; + struct kvm *kvm; + bool advance; + u64 kernel_ns, l1_tsc, offset, tsc_now; + + kvm =3D vcpu->kvm; + advance =3D kvm_get_time_and_clockread(&kernel_ns, &tsc_now); + raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags); + /* + * Advance the guest's TSC to current time instead of only + * preventing it from going backwards, while making sure + * all the vCPUs use the same offset. + */ + if (kvm->arch.host_was_suspended && advance) { + l1_tsc =3D nsec_to_cycles(vcpu, + kvm->arch.kvmclock_offset + kernel_ns); + offset =3D kvm_compute_l1_tsc_offset(vcpu, l1_tsc); + kvm->arch.cur_tsc_offset =3D offset; + kvm_vcpu_write_tsc_offset(vcpu, offset); + } else if (advance) + kvm_vcpu_write_tsc_offset(vcpu, kvm->arch.cur_tsc_offset); + } else { + adjust_tsc_offset_host(vcpu, vcpu->arch.tsc_offset_adjustment); + } + kvm->arch.host_was_suspended =3D false; + raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags); vcpu->arch.tsc_offset_adjustment =3D 0; kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); } @@ -12729,6 +12754,7 @@ int kvm_arch_enable_virtualization_cpu(void) kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); } =20 + kvm->arch.host_was_suspended =3D true; /* * We have to disable TSC offset matching.. if you were * booting a VM while issuing an S4 host suspend.... --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Tue Oct 7 14:57:04 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EAF45270571 for ; Wed, 9 Jul 2025 07:05:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044705; cv=none; b=Nv/Xfu3r2iGME6RysYJaEM8O8tWTi6C7gk4PdsQvB9mmt/kMBKYpIDsF9vE70gX2vD/u14oe/y+wrtrzipWagaUq74fiSxZoT6RD9HZ23uV7YehLExie2nsMijd7bpQPpfnFnNGyVDHy6tngTzNH1XpMHMd1Tg26szqQMUVK1Ik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044705; c=relaxed/simple; bh=8acnn84F2EPwPo/2bUKI9y8xzYUhx0M85P0qcx2e/FM=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Cc:Content-Type; b=q+I9mkZUWQ19BQybBBfN9RtuJstXkAyFVBnYLUfowEzG1glaM3iaK6cjhWj2N0nBXMlBKZ8786Pcn9pYirR5iZF1hd3yilK4u5rZ8K4F9uJKU8w7tHM7y80dPVQvmhKNxu8mDHCj2Cg0E5Pib4z222yJeGeueRMaBmhQaY7Zu3s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AQx3t1PB; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AQx3t1PB" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-70f841fb19dso65537047b3.1 for ; Wed, 09 Jul 2025 00:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1752044703; x=1752649503; darn=vger.kernel.org; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8hUdo8ARlQkzGI2qXgVk3yoTg5wBj9RNjVmfVyEYsuY=; b=AQx3t1PBdqBcHVVibM34HQUd7PMnwcfc377Cuue613zxiv6KGDFvFYiefSSc5Aanon yZ8UeafT+Ic3HW50/nEVdwtenrRULjtZcTkPbFA+JoqAiBCaUJ1Oo6eb7XDD+uXYMTqP dHqCjElSGxIGHtF/H3S9z3lcimsNih9EuI75DbVYQYQEQnw1MbnsTR9MH8rd2ZZ3zaVp b9lj7STUCTJlBixoWp3tHBPRhfJTA+3yL/ZXht4PeEJ7bazmh6Dt/8NXh99KBzfu+FXB 4mL0//2/GuvienP3HZvZtnCuEeP2B7OuJEBAgwKuNw6QudM6PwDKALrzJnpoNFPGmICg yFSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752044703; x=1752649503; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8hUdo8ARlQkzGI2qXgVk3yoTg5wBj9RNjVmfVyEYsuY=; b=PnJEgxFAMRCkurfR84kSVQWwk++Wt/jip2ZEM6g7u+ZpRMGeCCPJeYbt8YNjbe0FRN 4LESbTE0TtpjE9oE3/6TWaVi24GGPya5qTXSaNTHr3T9Zhm0G5IRkjW+EuL9nx8VS/GK cT43l41CmjRPOdD3pXsQ9OnFLammI0uvm/LdGs7jxdYw4O48bhyWtjUvnC/QGiurNifZ AHF/Mpr0QXeTqRylSmtV+1Ibr1j7SwzAKcHFv+PAGEl+xjPD5EWGpT34gyHzIdftZga9 KFrr0s/WrdC2LOp4Px8cOu49+4rJTwRhQ2+8ljOkYHxd67Zlq/4sSni67AOKybLj4Va0 pKVQ== X-Forwarded-Encrypted: i=1; AJvYcCXQ8xOQUZJCXmceLC+F1TyOM5b/mk+k6u3FFC2KmJLC/bvQzmGYQkjKbasyaYtCeKUnE1wPsDXB5HKYDHw=@vger.kernel.org X-Gm-Message-State: AOJu0Yyq0/vZ85qmXqBUlgdSERpN+8qqMa9iETsc6w7CPMt7fbNR2Viy 17fGoiQy1F2gUTPc5Uek5jhRaW2/MWpV6BuDrWzJtw8nQFGBp7H62vs5YM9iPZCcHtQORzOZVyO AjxuQJYZRkuLupA== X-Google-Smtp-Source: AGHT+IGEfiidPbWbXfAEZ47CmcdUs9ZPOOEtK0ixwyyLTU/yNC0ka1UYfmzV8mQy6TonmuKtwlqV6SEZTjJqyg== X-Received: from suleiman1.tok.corp.google.com ([2401:fa00:8f:203:a92c:694f:82fe:62a]) (user=suleiman job=sendgmr) by 2002:a25:d344:0:b0:e7d:6afc:71da with SMTP id 3f1490d57ef6-e8b6e10d3a8mr709276.2.1752044702827; Wed, 09 Jul 2025 00:05:02 -0700 (PDT) Date: Wed, 9 Jul 2025 16:04:49 +0900 In-Reply-To: <20250709070450.473297-1-suleiman@google.com> Message-Id: <20250709070450.473297-3-suleiman@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250709070450.473297-1-suleiman@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Subject: [PATCH v6 2/3] KVM: x86: Include host suspended duration in steal time From: Suleiman Souhlal To: Paolo Bonzini , Sean Christopherson Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Chao Gao , David Woodhouse , Sergey Senozhatsky , Konrad Rzeszutek Wilk , Tzung-Bi Shih , John Stultz , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, ssouhlal@freebsd.org, Suleiman Souhlal Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce MSR_KVM_SUSPEND_STEAL which controls whether or not a guest wants the duration of host suspend to be included in steal time. This lets guests subtract the duration during which the host was suspended from the runtime of tasks that were running over the suspend, in order to prevent cases where host suspend causes long runtimes in guest tasks, even though their effective runtime was much shorter. Signed-off-by: Suleiman Souhlal --- Documentation/virt/kvm/x86/cpuid.rst | 4 ++ Documentation/virt/kvm/x86/msr.rst | 15 ++++++ arch/x86/include/asm/kvm_host.h | 3 ++ arch/x86/include/uapi/asm/kvm_para.h | 2 + arch/x86/kvm/cpuid.c | 4 +- arch/x86/kvm/x86.c | 80 ++++++++++++++++++++++++++-- 6 files changed, 102 insertions(+), 6 deletions(-) diff --git a/Documentation/virt/kvm/x86/cpuid.rst b/Documentation/virt/kvm/= x86/cpuid.rst index bda3e3e737d71f..71b42b6499733b 100644 --- a/Documentation/virt/kvm/x86/cpuid.rst +++ b/Documentation/virt/kvm/x86/cpuid.rst @@ -103,6 +103,10 @@ KVM_FEATURE_HC_MAP_GPA_RANGE 16 guest c= hecks this feature bit bef KVM_FEATURE_MIGRATION_CONTROL 17 guest checks this feature b= it before using MSR_KVM_MIGRATION_CON= TROL =20 +KVM_FEATURE_SUSPEND_STEAL 18 guest checks this feature b= it + before using + MSR_KVM_SUSPEND_STEAL. + KVM_FEATURE_CLOCKSOURCE_STABLE_BIT 24 host will warn if no guest-= side per-cpu warps are expected = in kvmclock diff --git a/Documentation/virt/kvm/x86/msr.rst b/Documentation/virt/kvm/x8= 6/msr.rst index 3aecf2a70e7b43..8c31b3994e733d 100644 --- a/Documentation/virt/kvm/x86/msr.rst +++ b/Documentation/virt/kvm/x86/msr.rst @@ -296,6 +296,12 @@ data: the amount of time in which this vCPU did not run, in nanoseconds. Time during which the vcpu is idle, will not be reported as steal time. + If the guest set the enable bit in MSR_KVM_SUSPEND_STEAL, + steal time includes the duration during which the host is + suspended. The case where the host suspends during a VM + migration might not be accounted if VCPUs aren't entered + post-resume. A workaround would be for the VMM to ensure that + the guest is entered with KVM_RUN after resuming from suspend. =20 preempted: indicate the vCPU who owns this struct is running or @@ -388,3 +394,12 @@ data: guest is communicating page encryption status to the host using the ``KVM_HC_MAP_GPA_RANGE`` hypercall, it can set bit 0 in this MSR to allow live migration of the guest. + +MSR_KVM_SUSPEND_STEAL: + 0x4b564d09 + +data: + This MSR is available if KVM_FEATURE_SUSPEND_STEAL is present in + CPUID. Bit 0 controls whether the host should include the duration it + has been suspended in steal time (1), or not (0). + diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 5c465bdd6d088a..b35099260aa625 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -932,6 +932,8 @@ struct kvm_vcpu_arch { u8 preempted; u64 msr_val; u64 last_steal; + u64 suspend_ts; + atomic64_t suspend_ns; struct gfn_to_hva_cache cache; } st; =20 @@ -1028,6 +1030,7 @@ struct kvm_vcpu_arch { } pv_eoi; =20 u64 msr_kvm_poll_control; + u64 msr_kvm_suspend_steal; =20 /* pv related host specific info */ struct { diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/a= sm/kvm_para.h index a1efa7907a0b10..678ebc3d7eeb55 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -36,6 +36,7 @@ #define KVM_FEATURE_MSI_EXT_DEST_ID 15 #define KVM_FEATURE_HC_MAP_GPA_RANGE 16 #define KVM_FEATURE_MIGRATION_CONTROL 17 +#define KVM_FEATURE_SUSPEND_STEAL 18 =20 #define KVM_HINTS_REALTIME 0 =20 @@ -58,6 +59,7 @@ #define MSR_KVM_ASYNC_PF_INT 0x4b564d06 #define MSR_KVM_ASYNC_PF_ACK 0x4b564d07 #define MSR_KVM_MIGRATION_CONTROL 0x4b564d08 +#define MSR_KVM_SUSPEND_STEAL 0x4b564d09 =20 struct kvm_steal_time { __u64 steal; diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index b2d006756e02a6..983867f243cafc 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -1614,8 +1614,10 @@ static inline int __do_cpuid_func(struct kvm_cpuid_a= rray *array, u32 function) (1 << KVM_FEATURE_PV_SCHED_YIELD) | (1 << KVM_FEATURE_ASYNC_PF_INT); =20 - if (sched_info_on()) + if (sched_info_on()) { entry->eax |=3D (1 << KVM_FEATURE_STEAL_TIME); + entry->eax |=3D (1 << KVM_FEATURE_SUSPEND_STEAL); + } =20 entry->ebx =3D 0; entry->ecx =3D 0; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e66bab1a1f56e2..691f46b5e50f27 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3753,6 +3753,8 @@ static void record_steal_time(struct kvm_vcpu *vcpu) steal +=3D current->sched_info.run_delay - vcpu->arch.st.last_steal; vcpu->arch.st.last_steal =3D current->sched_info.run_delay; + if (unlikely(atomic64_read(&vcpu->arch.st.suspend_ns))) + steal +=3D atomic64_xchg(&vcpu->arch.st.suspend_ns, 0); unsafe_put_user(steal, &st->steal, out); =20 version +=3D 1; @@ -4058,6 +4060,17 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct= msr_data *msr_info) vcpu->arch.msr_kvm_poll_control =3D data; break; =20 + case MSR_KVM_SUSPEND_STEAL: + if (!guest_pv_has(vcpu, KVM_FEATURE_SUSPEND_STEAL) || + !guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) + return 1; + + if (!(data & KVM_MSR_ENABLED)) + return 1; + + vcpu->arch.msr_kvm_suspend_steal =3D data; + break; + case MSR_IA32_MCG_CTL: case MSR_IA32_MCG_STATUS: case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: @@ -4404,6 +4417,11 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct= msr_data *msr_info) =20 msr_info->data =3D vcpu->arch.msr_kvm_poll_control; break; + case MSR_KVM_SUSPEND_STEAL: + if (!guest_pv_has(vcpu, KVM_FEATURE_SUSPEND_STEAL)) + return 1; + msr_info->data =3D vcpu->arch.msr_kvm_suspend_steal; + break; case MSR_IA32_P5_MC_ADDR: case MSR_IA32_P5_MC_TYPE: case MSR_IA32_MCG_CAP: @@ -7006,13 +7024,52 @@ static int kvm_arch_suspend_notifier(struct kvm *kv= m) { struct kvm_vcpu *vcpu; unsigned long i; + bool kick_vcpus =3D false; =20 - /* - * Ignore the return, marking the guest paused only "fails" if the vCPU - * isn't using kvmclock; continuing on is correct and desirable. - */ - kvm_for_each_vcpu(i, vcpu, kvm) + kvm_for_each_vcpu(i, vcpu, kvm) { + if (vcpu->arch.msr_kvm_suspend_steal & KVM_MSR_ENABLED) { + kick_vcpus =3D true; + WRITE_ONCE(vcpu->arch.st.suspend_ts, + ktime_get_boottime_ns()); + } + /* + * Ignore the return, marking the guest paused only "fails" if + * the vCPU isn't using kvmclock; continuing on is correct and + * desirable. + */ (void)kvm_set_guest_paused(vcpu); + } + + if (kick_vcpus) + kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); + + return NOTIFY_DONE; +} + +static int +kvm_arch_resume_notifier(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + + kvm_for_each_vcpu(i, vcpu, kvm) { + u64 suspend_ns =3D ktime_get_boottime_ns() - + vcpu->arch.st.suspend_ts; + + WRITE_ONCE(vcpu->arch.st.suspend_ts, 0); + + /* + * Only accumulate the suspend time if suspend steal-time is + * enabled, but always clear suspend_ts and kick the vCPU as + * the vCPU could have disabled suspend steal-time after the + * suspend notifier grabbed suspend_ts. + */ + if (vcpu->arch.msr_kvm_suspend_steal & KVM_MSR_ENABLED) + atomic64_add(suspend_ns, &vcpu->arch.st.suspend_ns); + + kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); + kvm_vcpu_kick(vcpu); + } =20 return NOTIFY_DONE; } @@ -7023,6 +7080,9 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned lo= ng state) case PM_HIBERNATION_PREPARE: case PM_SUSPEND_PREPARE: return kvm_arch_suspend_notifier(kvm); + case PM_POST_HIBERNATION: + case PM_POST_SUSPEND: + return kvm_arch_resume_notifier(kvm); } =20 return NOTIFY_DONE; @@ -11212,6 +11272,16 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) =20 static bool kvm_vcpu_running(struct kvm_vcpu *vcpu) { + /* + * During host SUSPEND/RESUME tasks get frozen after SUSPEND notifiers + * run, and thawed before RESUME notifiers, i.e. vCPUs can be actively + * running when KVM sees the system as suspended. Block the vCPU if + * KVM sees the vCPU as suspended to ensure the suspend steal time is + * accounted before the guest can run, and to the correct guest task. + */ + if (READ_ONCE(vcpu->arch.st.suspend_ts)) + return false; + return (vcpu->arch.mp_state =3D=3D KVM_MP_STATE_RUNNABLE && !vcpu->arch.apf.halted); } --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Tue Oct 7 14:57:04 2025 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64093273D8E for ; Wed, 9 Jul 2025 07:05:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044708; cv=none; b=RJce0rOyM37R1sWoilaup8quL3NU3q9cGnnI5obJPxxtAP5Yi/s/Bypl/8mrYW6OotpBABmXZeBb8ww7yAbIz6lRlBHPo69oOToCbglt2SOjH2lta+mQ24b7mqgt56moTWI0CBqgfRcGVHPIVdm9ilTot9kO3eXtpNshTjhNVRM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752044708; c=relaxed/simple; bh=bqDfiVKFuR+e8N7emCK6GdbPPlTbsBsyYl3PH3KjyJw=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Cc:Content-Type; b=lW//EAhFeRQ+q4AfHV9L6FmC7+uS7251jvfS/WEqaPQO6s7QhCKwJXITWWE30vRfz/iWOV8iGEXskLFPY7swHih/QPRDdCxfjjTAGJmgTGrlvVBw2zn7f/B4ycWpzygKuE3kuJgCP4G5x+bYl8j33016V/5MBppXRabLGqIzqVI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lGa5dMb4; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--suleiman.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lGa5dMb4" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e819e8eb985so6723034276.2 for ; Wed, 09 Jul 2025 00:05:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1752044706; x=1752649506; darn=vger.kernel.org; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HaDQLHJuamQG+Oz/rXsc1nJQc3j6887WnfJ3FWY6wPY=; b=lGa5dMb4/QCskSc/99m9viQEHFXRqBqRbuSxe8xbbz1YtNKIQIHcB3SWi2d6gc03NM yjMjJYjLpL0efYbj12ikl3F9Vt1vpd40ZbT4SQEUyFezNUsZuxbxOO4foCxR9M6eeRDJ 0ddqiQYvYZsttyrpxusJBc1Lk8t31SSk11tb7LaEAQGqTgEeEUdjrQKl2FmJjeFNYvlx ZQx3s3C0sOEOxfUt+NQeDSIvkXR1lQGvJs7RuI4fVOKa7zt/5mQkLHPa+eI0mAKQuPkm PC5ZUj4H62nz3hRyxxrxayLIkoMo8SV9au9VXzGrTmwxQGpk0vde0OVl5yOU90R5uDLE Q7pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752044706; x=1752649506; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HaDQLHJuamQG+Oz/rXsc1nJQc3j6887WnfJ3FWY6wPY=; b=YQof20no55mSfQRzCDj3jcdBEg1QRT8okt/xZkN0pxvncxUwvf388SctZHAgugPm/Z 2xsPwXV72k0tu7jX0qgFXTA5xPdNgcFbYrdvBjW15L3Y6ZNthkRF+gVKwBAg4BOn5/eq ze6dpMxVKqu7NYQwBXhp+ywjOHESuera3LjVJ4iAJH++yItwHKyFzXmqrsCN8/4iBiOV F1+AUfkD0UT2geOVV+o0ofdvmn5pf2eD14uVBdgzxug/b2uIa4/uvjmuKeLR29xpG0VI o1U74rgt5HVKoSlT5Y9OwaG/Lm8ElXcznpl4iCgZ0sbAPkru2cRysLuVOCQSeheUciXN tbGQ== X-Forwarded-Encrypted: i=1; AJvYcCVue7V/3kHRg9uYfKX1E0+7A9wOVFskBoPmdT/KkjWPUUlO4cb+uHnIe91jrkcE5wOzPlLrGO+tv9HrS84=@vger.kernel.org X-Gm-Message-State: AOJu0YzEtqm38ZYbyaxIw5TT7Ysyb9e2PbXL/uusdqAN1s06nWkx7D9p 0PcOIy2LMdmXvywYQqXuxTY7ZIed5esAnfTp8IYoa2Jf/zAKg3SDqmfObCv+AH7WecWCeNqfwZB iit3DjmsW0mvrag== X-Google-Smtp-Source: AGHT+IE5ZkJJrfChwxmjelv9TbY/xNBu5uRt03nGE+i4WAPwg6JLhbF8sllzldrGvtvYqXjNVHB2+Nx7G6uM6w== X-Received: from suleiman1.tok.corp.google.com ([2401:fa00:8f:203:a92c:694f:82fe:62a]) (user=suleiman job=sendgmr) by 2002:a05:6902:6c0e:b0:e89:883d:dd33 with SMTP id 3f1490d57ef6-e8b6e19fdcamr688276.5.1752044706399; Wed, 09 Jul 2025 00:05:06 -0700 (PDT) Date: Wed, 9 Jul 2025 16:04:50 +0900 In-Reply-To: <20250709070450.473297-1-suleiman@google.com> Message-Id: <20250709070450.473297-4-suleiman@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250709070450.473297-1-suleiman@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Subject: [PATCH v6 3/3] KVM: x86: Add "suspendsteal" cmdline to request host to add suspend duration in steal time From: Suleiman Souhlal To: Paolo Bonzini , Sean Christopherson Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Chao Gao , David Woodhouse , Sergey Senozhatsky , Konrad Rzeszutek Wilk , Tzung-Bi Shih , John Stultz , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, ssouhlal@freebsd.org, Suleiman Souhlal Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a new command line parameter, "suspendsteal", enabling the guest to use MSR_KVM_SUSPEND_STEAL, which tells the host that it would like host suspend duration to be included in steal time. Signed-off-by: Suleiman Souhlal --- Documentation/admin-guide/kernel-parameters.txt | 5 +++++ arch/x86/kernel/kvm.c | 15 +++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index f1f2c0874da9dd..9f5758ca8fadd5 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7074,6 +7074,11 @@ improve throughput, but will also increase the amount of memory reserved for use by the client. =20 + suspendsteal + [X86,PV_OPS] + Enable requesting the host to include the duration the + host was suspended in steal time. Disabled by default. + suspend.pm_test_delay=3D [SUSPEND] Sets the number of seconds to remain in a suspend test diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 921c1c783bc187..35d1bb2283c2c0 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -320,6 +320,18 @@ static void __init paravirt_ops_setup(void) #endif } =20 +static bool suspend_steal; + +static int __init suspendsteal_setup(char *s) +{ + if (kvm_para_has_feature(KVM_FEATURE_SUSPEND_STEAL)) + suspend_steal =3D true; + + return 0; +} + +early_param("suspendsteal", suspendsteal_setup); + static void kvm_register_steal_time(void) { int cpu =3D smp_processor_id(); @@ -331,6 +343,9 @@ static void kvm_register_steal_time(void) wrmsrq(MSR_KVM_STEAL_TIME, (slow_virt_to_phys(st) | KVM_MSR_ENABLED)); pr_debug("stealtime: cpu %d, msr %llx\n", cpu, (unsigned long long) slow_virt_to_phys(st)); + + if (suspend_steal) + wrmsrl(MSR_KVM_SUSPEND_STEAL, KVM_MSR_ENABLED); } =20 static DEFINE_PER_CPU_DECRYPTED(unsigned long, kvm_apic_eoi) =3D KVM_PV_EO= I_DISABLED; --=20 2.50.0.727.gbf7dc18ff4-goog