From nobody Wed Jun 17 07:15:20 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE05C37DE9B for ; Thu, 23 Apr 2026 16:26:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776961594; cv=none; b=XfGSboQUWlp02VqBE8vSAItn72pVBL2LZWYDPyckApxGn/hVl+bhrOR9rcYRIw1oReHYhRyJC2urDAL+MtdGy7YOrY2iJhBwTkYXpxDu4Q9nwfQvDnUHFKgELPfh6s5jCpfYrH1NJVNCTXf2BR2GtVd3vPXOpXGs8E7HjlWxVV0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776961594; c=relaxed/simple; bh=+sHcNeW5LGYxEMASXp78quDxR4fsZAKAoVzQNC5fElk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HrVQsyYPTNFpd+N2a25oJzl01uHVeEWe9psMDQbCqebS+jWD9ok+n/MAtwa4H/w+otM+/CJMFnnXqz0CPAkkEIJoWdJ4ZLcwmTwsIvaN7u9iS0PzWy3XKemgRtJRr5lgmBM3RaaVhvPZZO+CreuNxYi8EngCaotO9DfKN7IREDM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tryILKMV; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tryILKMV" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2b249975139so126452215ad.0 for ; Thu, 23 Apr 2026 09:26:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1776961592; x=1777566392; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=178Sl2Pz7vfzNqL8EoR9/QLtRB+UOblPI2eBUX1bVH4=; b=tryILKMVmUeG0pdYY7kwI38WCrAjle9lSQRnmnChxwd7zVOG1VcFvGT6ebHOj/Ov6N a8I+ltheq4y03wnm59eU6LpKhFA+fqHmi0O+Z7tay3NnlMkMlOleZe1W90+CvkM+q6g9 mmJ6NhbZ0JA7kTxnUY0b85waPNwSXkIaIudy4B4IpYG+3XlgaLhwAZYCnYTzKWA1zud2 9q98AAbyXSvy/adECTVJSNQTsEd4oWSkZH66wJgbVFflB7B+szwjvxMhlb+PWAJF8CYm iiCG2v8mdNzItHGmVjwGvrPsvI+Z3gOhszRlUph7Hw7J/f/jfpAUX3tGD+5bbIM44iTQ uZxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776961592; x=1777566392; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=178Sl2Pz7vfzNqL8EoR9/QLtRB+UOblPI2eBUX1bVH4=; b=NZgJPMuKyKBU4Vh4r8uYRN/BO28iwz+MFp5UGdiSj1bICDAWF069qouisEt44sqmgO Pn/agtHQs4iiEPp94ezmu9AjUZNhlv5l7RksJnpJaLx8UrWx2+YPS4ZCkv2y+3SNBI28 k1PqYfSd+a4+RBjxSL24jzY+ANj0aMEp0lJSOViDZnIZzhYVMUVRGVyd6rG/5e+0uDkY KA10HdULUsjBMMeoc6Sh0jhRkJnahIjays45wyVCy770/SfBQcOVOyMvi9kAjm2iu1bE o8F3WlOiqPL7ib4yRID6+jsHnkWXjBOBehOcXWln2wuKwOSiGmlbhMw1Depf1WPB4mRT lSkQ== X-Forwarded-Encrypted: i=1; AFNElJ9TOLuiSHJjbuVAsdfXE2eNzNuyZQOxOsFQ7qEmu7yZfvtd/iccDFU769wP0gLRk6v///ekIOO/yVhqnUk=@vger.kernel.org X-Gm-Message-State: AOJu0Yxo3XULTA4egIa77Qul4E+3KvZEJ4PX5vukgNZ6lTOwMNjgGN2T v4B0zzEvTCyKwS7USdJ6cMWSLZ5ZAIFClCjLs5TKUwOq3VhIb6/9t5yUM5Iv5kq4gebrZqQrdd2 AZHLgiw== X-Received: from plhx4.prod.google.com ([2002:a17:903:2c04:b0:2b0:ad22:aebd]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:3c24:b0:2b7:a3bf:b2a0 with SMTP id d9443c01a7336-2b7a3bfb415mr42481585ad.5.1776961591975; Thu, 23 Apr 2026 09:26:31 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 23 Apr 2026 09:26:27 -0700 In-Reply-To: <20260423162628.490962-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260423162628.490962-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260423162628.490962-2-seanjc@google.com> Subject: [PATCH 1/2] KVM: x86: Ensure vendor's exit handler runs before fastpath userspace exits From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, "Nikunj A . Dadhania" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the handling of fastpath userspace exits into vendor code to ensure KVM runs vendor specific operations that need to run before userspace gains control of the vCPU. E.g. for VMX (and soon to be for SVM as well), KVM needs to flush the PML buffer prior to exiting to userspace, otherwise any memory written by the final KVM_RUN might never be flagged as dirty. Note, waiting to snapshot CR0 and CR3 until svm_handle_exit() is flawed in general, as that risks consuming stale state in a fastpath handler. That will be addressed in a future change. Fixes: f7f39c50edb9 ("KVM: x86: Exit to userspace if fastpath triggers one = on instruction skip") Cc: stable@vger.kernel.org Cc: Nikunj A. Dadhania Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Nikunj A. Dadhania --- arch/x86/kvm/svm/svm.c | 3 +++ arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.c | 3 --- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index e7fdd7a9c280..eb351ca4dd82 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3652,6 +3652,9 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fas= tpath_t exit_fastpath) vcpu->arch.cr3 =3D svm->vmcb->save.cr3; } =20 + if (unlikely(exit_fastpath =3D=3D EXIT_FASTPATH_EXIT_USERSPACE)) + return 0; + if (is_guest_mode(vcpu)) { int vmexit; =20 diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a29896a9ef14..4cb355ecfe46 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6687,6 +6687,9 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, f= astpath_t exit_fastpath) if (enable_pml && !is_guest_mode(vcpu)) vmx_flush_pml_buffer(vcpu); =20 + if (unlikely(exit_fastpath =3D=3D EXIT_FASTPATH_EXIT_USERSPACE)) + return 0; + /* * KVM should never reach this point with a pending nested VM-Enter. * More specifically, short-circuiting VM-Entry to emulate L2 due to diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 0a1b63c63d1a..9ad7ec3bf0f1 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11602,9 +11602,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (vcpu->arch.apic_attention) kvm_lapic_sync_from_vapic(vcpu); =20 - if (unlikely(exit_fastpath =3D=3D EXIT_FASTPATH_EXIT_USERSPACE)) - return 0; - r =3D kvm_x86_call(handle_exit)(vcpu, exit_fastpath); return r; =20 --=20 2.54.0.545.g6539524ca2-goog From nobody Wed Jun 17 07:15:20 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5168A3806D1 for ; Thu, 23 Apr 2026 16:26:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776961595; cv=none; b=eyNEFZnHCxiJ7RNFg/Zw0pqvTzp1HdwjpAc5zjDaf6Nb9P2borokrU3o5cVuvX33t/3t9VMdcRFexYrMTbnBAPz1Uiqi5H6hpKLISB/xb3bVO7UdV21EveCt2xF4JR6MCIvmFgfRID87JryGvuyQiIU1d5+9WmL7TFC5/k0m8R8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776961595; c=relaxed/simple; bh=mLYv8WZqZbpdK7gb+Jc9DDETesX8Ne4WD9MXv4JEr9Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ElEAQUCyTnpsKqL9qwAkGlhZRwhc0SoFVWWZ7aHHLnO6pmHY1C0E/RWwetFrPu5ZelprIne9wPJy+9ZIXH9DLQodV4HrlQOVSh4m36TeT2fX/cCu4BKcOLYEn8ZZYxzVIKN4P8BjwAlIt8eMAlpBBSEe4ae9M5tUYalbHKv7Vag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ubJePkEO; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ubJePkEO" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-35845fcf0f5so7437778a91.0 for ; Thu, 23 Apr 2026 09:26:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1776961594; x=1777566394; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=4t9k99Hk+Nxw6lMwotfPJxna6r0dcmYdL1wJ+beGW5I=; b=ubJePkEO1AHEnwv+NUROYzYOj1rvXS3WbwYSbnLHTlKIgp+AVWHhAW0qMaAxRaTIsT OV0LmVBXGR3KK7YC5sloX3x9fICzaYBXXehnWc9W5vwbbauMGHAXCdIklRt6DJIU80uB DmPeXTA0UTANVM9Dh+RFwQWOEsyJxz7aCV51jtywFzq0kDOVoP03MtZmYhGxhFdvOmvM 4v8blJ0TITaHKB8K789yjbIPCFOv6L/NRFbg9z8P4m9vUcPha9jzinXeKpBwPNa4IB4f IqpzMj76C223KwrmJ6XQf6QCBsNqrBnP4/Xggbup4fp4IHLJaTNYaP0KgkA0rd/fBm4a +yug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776961594; x=1777566394; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4t9k99Hk+Nxw6lMwotfPJxna6r0dcmYdL1wJ+beGW5I=; b=D1FTdTdakanTZFPY6WOFrLVfdTm26V2dcbyogpPm9tSewLKlj06LtkeHK0pjKULmbr 43oE/DCXm3/eMTw6m0OAO1GUK8vZP04fsRrs2hJXLtxEgPBNq5QnXNWkmF+DoKGU7yG5 9Vr24xtnwneQ7dZoaQM5Jbn8nUdOhZNrLzorkB7Pcc1cwtESmm7/7hOaXSsrIPx36fJk zE2mzcpexBRE4vllEWUKgXnLwkyWYHWFx7CfXTu3WgU2tlRHEoZqS/q8qNpEPw3BjBl1 8Kiyz3N8NoIm/dhQGuuBnO5K56bamotMax0wa1k8nZw+/UXAKhEghMzyPbh/IJfTEDDn Quzw== X-Forwarded-Encrypted: i=1; AFNElJ8dLyiIwA44gcLIT99ZdOpa58kJ8YUREKXsnylPaKDHIHwbr9lEZqhZxkNLlpML1dr3gsLuGnofYbeaR98=@vger.kernel.org X-Gm-Message-State: AOJu0YyA1QijHZIyjhsdA/2yCbAYqsndFhBzYRfA+W/pFEEwaqeyv1M2 vGTnC85IKDyphPB5MYST725LuQw8njRZX18rVxljTR0vqUwHHuikKPB3ChzHEQYactdK1XV9BG1 73SJ8Ow== X-Received: from plte11.prod.google.com ([2002:a17:902:744b:b0:2b4:6527:7f6]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2c8d:b0:359:84a3:1942 with SMTP id 98e67ed59e1d1-361402eaf94mr26630578a91.13.1776961593615; Thu, 23 Apr 2026 09:26:33 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 23 Apr 2026 09:26:28 -0700 In-Reply-To: <20260423162628.490962-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260423162628.490962-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260423162628.490962-3-seanjc@google.com> Subject: [PATCH 2/2] KVM: SVM: Refresh vcpu->arch.cr{0,3} prior to invoking fastpath handler From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, "Nikunj A . Dadhania" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refresh KVM's copies of CR0 and CR3 from the VMCB prior to (potentially) invoking a fastpath handler to ensure that KVM doesn't consume stale state. While it's unlikely KVM will ever consume CR3 or CR0.{TS,MP} in the fastpath, grabbing the values from the VMCB is inexpensive, i.e. the risk of subtle bugs far outweighs the reward of deferring reads for a small subset of VM-Exits. Note, KVM doesn't currently consume CR3 or CR0.{TS,MP} in the fastpath, as KVM requires next_rip to be valid (i.e. KVM doesn't read CR3 to decode the instruction), CR0.MP is never consumed, and CR0.TS is only consumed by the full emulator. Signed-off-by: Sean Christopherson Reviewed-by: Nikunj A. Dadhania --- arch/x86/kvm/svm/svm.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index eb351ca4dd82..df0bd132edf7 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3644,14 +3644,6 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fa= stpath_t exit_fastpath) struct vcpu_svm *svm =3D to_svm(vcpu); struct kvm_run *kvm_run =3D vcpu->run; =20 - /* SEV-ES guests must use the CR write traps to track CR registers. */ - if (!is_sev_es_guest(vcpu)) { - if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE)) - vcpu->arch.cr0 =3D svm->vmcb->save.cr0; - if (npt_enabled) - vcpu->arch.cr3 =3D svm->vmcb->save.cr3; - } - if (unlikely(exit_fastpath =3D=3D EXIT_FASTPATH_EXIT_USERSPACE)) return 0; =20 @@ -4505,11 +4497,17 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kv= m_vcpu *vcpu, u64 run_flags) if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL)) x86_spec_ctrl_restore_host(svm->virt_spec_ctrl); =20 + /* SEV-ES guests must use the CR write traps to track CR registers. */ if (!is_sev_es_guest(vcpu)) { vcpu->arch.cr2 =3D svm->vmcb->save.cr2; vcpu->arch.regs[VCPU_REGS_RAX] =3D svm->vmcb->save.rax; vcpu->arch.regs[VCPU_REGS_RSP] =3D svm->vmcb->save.rsp; vcpu->arch.regs[VCPU_REGS_RIP] =3D svm->vmcb->save.rip; + + if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE)) + vcpu->arch.cr0 =3D svm->vmcb->save.cr0; + if (npt_enabled) + vcpu->arch.cr3 =3D svm->vmcb->save.cr3; } vcpu->arch.regs_dirty =3D 0; =20 --=20 2.54.0.545.g6539524ca2-goog