From nobody Wed Sep 30 20:13:29 2026 Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D083423E99 for ; Thu, 6 Aug 2026 21:40:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052455; cv=none; b=OC/jHKhjgbszL0HNDU/zzSNuWfix0y1uBPub0dhkX6jL9vHaK04sFDRTNh/UXZnqzJi8w1nUtdDXuLi1yRn/8HPloCe0fEC+pzI8EKt3aUkZiKK+wUsvQ2co7/7OobQ82FrjJvhRhAUf+9mFZlxO1qyC8Rj8Pxlsdlalp8YHktw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052455; c=relaxed/simple; bh=fcJaKiC+PLAI3Uhl19cXGBBLAU5ESuVMNz27IIUYbzU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YmTpSRbu7h5+VuLzNCmLJATVri4BJcIguZ0YAdB/On06O5udZ8Y+AdCwK/+L/aWrnlVTNEYsy7NJJXPKH9/ZflLY8PbOmSeBKsx3SiG4FndPvWkpj6YW9CDHch10N78rXx6hQnQ5kbofhnYQ8tALiOr8SHGlqUXN4sXbHDoBNzc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vU5WyDI8; arc=none smtp.client-ip=209.85.214.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vU5WyDI8" Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-2cfca8558d2so37244855ad.2 for ; Thu, 06 Aug 2026 14:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786052454; x=1786657254; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:from:to:cc:subject:date:message-id :reply-to:content-type; bh=XaYcUb4i48pvUQtoL/N+DHfkbu37vszwR9D06c2Pljg=; b=vU5WyDI8lCUhfzX8cw9y8DnCq7irbw2T9cGlir2qyojewJtyU/BvW2YJl64EghF9M/ Is7Z4u/Z7ox7xPImVnlcG2V0OWuOM10RdN9YYtMQIUJ4iryWuMOdYbQMG8Axwkv7zEwf o/CyvqDFLKsqtaJRV9Ulnk6pH2Maq/A6ns20uuXvg50eBatEx5wsrf0CRaj+Y5E0/Z/N XCyNhNe0KtJezo/7eNP6ulewAKOmkEaNnkGhouLCO1sSMMMhl3VfA9aWAz4SGUSP9RSo VgYIdklJZATu3+GBbD6laW3EO4Ufxm/eQC7t7OZyWuF3y+BAlYF8GgxREpRVXdB1biQz uoNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786052454; x=1786657254; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to:content-type; bh=XaYcUb4i48pvUQtoL/N+DHfkbu37vszwR9D06c2Pljg=; b=cHXUXt5W+ykRNuYcOHFCW55BTnSGwsjEsrIMMsZXnh4arCkei5YAjZ8Vp354ZLtm0K IyuV06F5sJMVwy03UxUF38gqEBl2cwKWe5g6mqf1j68iGmlP9/9dMy21nZ7epigtDPpG Mauu/Vyil6wtkN/mid9eFM5kBmP6tcKoFGam37pU2i0HFdemaf6LbLISt8GtlcaPR0ZK 4vSTrp1JxoY8o4yUMq2krt1Cm4/n3rUoiADyhVtiSawztCAhYEQVB1OeNqUNJQID+vJb AhJf1MxSn/V56g+MYG4TelWF8sr8LeNXVCrKEYcXC5I/0cQCn8chSwivVgu1G5MJcT9+ qXvg== X-Forwarded-Encrypted: i=1; AHgh+Rr/mzwr91YbKV5K9atI9L09eZwA8bfO/luu1fLf4ZCW2r8BkWq/YsdeR6i1kICFPZPlyOIxkpsKuy2/Vj4=@vger.kernel.org X-Gm-Message-State: AOJu0YzC8tBMHH1qS0wk7rxH8GEcaGjVN3yEqLc63PhiwkKGeyQqOzMt 4jcTlgoERtGB6+BCSK4EidcDaGWAvSvCY0lxe0/qKYkdJkxkvDn+FDxqfFF1rDPGoLPD+3rUDKA NZMG17w== X-Received: from plblo4.prod.google.com ([2002:a17:903:4344:b0:2cf:7d3e:e383]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:298f:b0:2c7:f12d:5d37 with SMTP id d9443c01a7336-2d0caa02fd3mr219970215ad.17.1786052453781; Thu, 06 Aug 2026 14:40:53 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 6 Aug 2026 14:40:47 -0700 In-Reply-To: <20260806214050.78058-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260806214050.78058-1-seanjc@google.com> X-Mailer: git-send-email 2.55.0.679.g6767b8d81c-goog Message-ID: <20260806214050.78058-2-seanjc@google.com> Subject: [PATCH 1/4] KVM: x86/mmu: Reload MMU on *every* page pre-fault attempt/iteration From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Kai Huang , Yan Zhao , Rick Edgecombe , Sashiko Bot Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Reload the MMU (which is a nop if the MMU doesn't need to be reloaded) on every attempt to pre-fault a guest page, i.e. when the page fault path signals that the caller should retry. If the synchronize_srcu_expedited() in kvm_invalidate_memslot() completes before kvm_vcpu_pre_fault_memory() grabs SRCU, but kvm_mmu_reload() in the pre-fault path completes before kvm_invalidate_memslot() triggers x86's "fast zap all", then the pre-fault task will reach kvm_tdp_page_prefault() with an invalid root. Attempting to fault-in memory with an invalid root ultimately puts kvm_tdp_page_prefault() into an infinite (breakable) retry loop, which manifests most obviously as a hang in the pre_fault_memory_test selftest, but also eventually causes RCU (SRCU?) to complain. INFO: rcu_tasks detected stalls on tasks: 000000000cda47bd: .. nvcsw: 6/6 holdout: 1 idle_cpu: -1/25 task:pre_fault_memor state:R running task stack:12696 pid:95588 tgid:95588 ppid:95584 task_flags:0x400000 flags:0x00080801 Call Trace: lock_release+0x4e/0x320 __get_user_pages+0x546/0xcd0 up_read+0x1b/0x30 get_user_pages_unlocked+0xee/0x350 hva_to_pfn+0xd3/0x3d0 [kvm] lock_release+0x4e/0x320 xa_load+0x5c/0x170 xa_load+0x14c/0x170 __kvm_faultin_pfn+0xd9/0x130 [kvm] lock_acquire+0x65/0x2b0 lock_release+0x4e/0x320 kvm_mmu_faultin_pfn+0x1e1/0x690 [kvm] gup_fast_fallback+0x63e/0xdf0 kvm_tdp_page_fault+0xeb/0x140 [kvm] kvm_mmu_do_page_fault+0x12e/0x200 [kvm] kvm_arch_vcpu_pre_fault_memory+0x16e/0x200 [kvm] kvm_vcpu_pre_fault_memory+0xc1/0x1f0 [kvm] kvm_vcpu_pre_fault_memory+0x116/0x1f0 [kvm] kvm_vcpu_ioctl+0x3a4/0x6b0 [kvm] clockevents_program_event+0x5d/0x170 __se_sys_ioctl+0x6d/0xb0 entry_SYSCALL_64_after_hwframe+0x4b/0x53 do_syscall_64+0x10a/0x480 __irq_exit_rcu+0x8e/0x140 entry_SYSCALL_64_after_hwframe+0x4b/0x53 Fixes: 6e01b7601dfe ("KVM: x86: Implement kvm_arch_vcpu_pre_fault_memory()") Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Rick Edgecombe --- arch/x86/kvm/mmu/mmu.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c519e8e8d646..621b0a42f2a1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5036,6 +5036,10 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vc= pu, gpa_t gpa, if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu)) return -EIO; =20 + r =3D kvm_mmu_reload(vcpu); + if (r) + return r; + cond_resched(); r =3D kvm_mmu_do_page_fault(vcpu, gpa, error_code, true, NULL, level); } while (r =3D=3D RET_PF_RETRY); @@ -5076,14 +5080,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu = *vcpu, if (kvm_is_gfn_alias(vcpu->kvm, gpa_to_gfn(range->gpa))) return -EINVAL; =20 - /* - * reload is efficient when called repeatedly, so we can do it on - * every iteration. - */ - r =3D kvm_mmu_reload(vcpu); - if (r) - return r; - direct_bits =3D 0; if (kvm_arch_has_private_mem(vcpu->kvm) && kvm_mem_is_private(vcpu->kvm, gpa_to_gfn(range->gpa))) --=20 2.55.0.679.g6767b8d81c-goog From nobody Wed Sep 30 20:13:29 2026 Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E82D942902C for ; Thu, 6 Aug 2026 21:40:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052458; cv=none; b=khO+wghbIMsJ1ffAJqNBm+c8fLrqw7Y6Vc6rnGA3+3n7q7RiU69A6kbuGszMMERnswN3ClBs948qBpHzSWq8GEJHUjG30tY91q6PlHceiXwFpTA1FjCxwunDr6dQNJtdxSTq3ZUADMdAv01LJviqv4YnKGm76yHIEKz7Gcv9vuw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052458; c=relaxed/simple; bh=YXvxBbhrwBIqcRqu7EKMvZk9hvjNwBNYInFYiCi5wRU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uC8qzGru0SuL2OqHhWHUEXn8caXDTEzXO0y4vZD7/wxuKk3w+FFqowZte1rnuWqUN+3WcapRKp+ELD9QIUeb3IxvVC7dmMGJtrVhI+85FwF4dvnJb30mAfUlH2DBUJ55SUT4CLRU6CGPPqiTl20aJCGK//XSBfM2Ks1h8YE27g4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=e/2TIBff; arc=none smtp.client-ip=209.85.210.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="e/2TIBff" Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-84e048a801dso3547115b3a.3 for ; Thu, 06 Aug 2026 14:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786052456; x=1786657256; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:from:to:cc:subject:date:message-id :reply-to:content-type; bh=rEEwTcg7bv7FkGEAqkUfC5yUVJEh4P3MKarrm6QtyZY=; b=e/2TIBffTAf1F/sIz6Vmg4Ta/254Zztm2NGBoJzFXzHopCtue/wZP0KHMFEd5seYoU MKEqG5/ipPrZzoTflckM0Rj3AjgZczv/6lEZIv4dAc7kW+/K0fcfeb1crfbNwN2UO70R +0jIHLZyktYs91+ah5ThuUJoeObVraPBy8Ep2XQV3l38x1B9QtHh/dCZOBcyVF+2RLoi kgz7sr5WWIeB0/2xnS+DS/ELJH0fm+Ibp6MAbweByXkpO8fgb56y+JS3xkxN0iTJWWFz wkR95b9fSe5Ka1MOqh1fRUfEfUa8FL+uoBXs1IBTDFcdHda3Ju2FjtSlB+pxD9Mf0j5s wPJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786052456; x=1786657256; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to:content-type; bh=rEEwTcg7bv7FkGEAqkUfC5yUVJEh4P3MKarrm6QtyZY=; b=McdfWs28zqagNB9F2H36uzDD3kg+tEnQIY/HW1RmsVoUR2YkdqSqBsTM4kAhCRMWWG rM+W0MuHZp4khnRTGHN1bpWvsxWnxylT1mM0S4fkqt+0eV3bpTL0Egb8FKtK5bOFYOGD ODT5nBefXDuGcfe6+vABLGrsRDgheSPUxv0yZom707bS6/iYWW8sZdanO+mdWZfIk9Pa lZQr5uemJ8Ucpjvwp72wl0yYZt4LsBDzFFC9elbjCME2o5/if+VcGMugNyU65L/E+QDC yyKx4QqVNNPM0tGVGArDfE4NSaCbNJYz0CTSxgx2GvpEaI6vTXAsK1akJttIMzibRKvW P/Zw== X-Forwarded-Encrypted: i=1; AHgh+RqKd7GrDnqW4mUyj/z5IfRqokOnuJTrJpGteanImegUW5gsNggsTeJ6hraaVoHONU4mggttQhb7g3iQEdc=@vger.kernel.org X-Gm-Message-State: AOJu0YwGxxmCAoAIL1zpK9+e7luCK9YpOUlk8E71zwv6rSSTQtKRCHy2 w25rrM32QnZekYZf7w9XStBea2OWU+qNI7DUXaflQ5KTQzHXN8aFENLvo//Z+2l12C4TiUA6Oie SBKn0fA== X-Received: from pfn34.prod.google.com ([2002:a05:6a00:a222:b0:84a:31b0:7544]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:c81:b0:845:cdc1:a803 with SMTP id d2e1a72fcca58-84f2dfc4763mr19978858b3a.11.1786052455960; Thu, 06 Aug 2026 14:40:55 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 6 Aug 2026 14:40:48 -0700 In-Reply-To: <20260806214050.78058-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260806214050.78058-1-seanjc@google.com> X-Mailer: git-send-email 2.55.0.679.g6767b8d81c-goog Message-ID: <20260806214050.78058-3-seanjc@google.com> Subject: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Kai Huang , Yan Zhao , Rick Edgecombe , Sashiko Bot Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move kvm_tdp_mmu_map_private_pfn()'s reload of the MMU into its tight loop so that an unexpected root invalidation has a better chance of being handled gracefully, even though it should be impossible for the vCPU's root to be invalidated after the initial reload. As is, encountering an invalid root is *guaranteed* to put the task into an infinite loop (albeit a breakable loop that honors NEED_RESCHED). Add a WARN to try and detect bugs that break KVM's expectations, along with a comment to explain why it should be impossible for the root to be invalidated. Cc: Kai Huang Cc: Yan Zhao Cc: Rick Edgecombe Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 621b0a42f2a1..c6cac893cbad 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcp= u, gfn_t gfn, kvm_pfn_t pfn) if (kvm_gfn_is_write_tracked(kvm, fault.slot, fault.gfn)) return -EPERM; =20 - r =3D kvm_mmu_reload(vcpu); - if (r) - return r; - r =3D mmu_topup_memory_caches(vcpu, false); if (r) return r; @@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vc= pu, gfn_t gfn, kvm_pfn_t pfn) if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) return -EIO; =20 + r =3D kvm_mmu_reload(vcpu); + if (r) + return r; + cond_resched(); =20 guard(read_lock)(&kvm->mmu_lock); =20 + /* + * Because slots_lock is held, it should be impossible for roots + * to be invalidated after the initial MMU reload. WARN, but + * continue and re-reload the MMU to try and keep the VM alive. + */ + WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)); + r =3D kvm_tdp_mmu_map(vcpu, &fault); } while (r =3D=3D RET_PF_RETRY); =20 --=20 2.55.0.679.g6767b8d81c-goog From nobody Wed Sep 30 20:13:29 2026 Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44D6542A142 for ; Thu, 6 Aug 2026 21:40:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052459; cv=none; b=UIDTtY5O7eIXiq/2P93cXTMrmxpFQiBG541BYpo4oK+ukii8J2n0MwZyXPfh2nRt95w3V6xYFJo8Pkh/j9GAiTPqiFJRvQD7Vp28ITy9hXQg5CjZsr5Wag5tTEgM/xXZFr4rfxK1JgrpIMHfLhQCwoJx8k8sriwfkRC+xoJeC2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052459; c=relaxed/simple; bh=vERuOJjKIwts7JqEB+ZkH11Qq/PmOdDg/7G+G2HrLeg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XzQT3z4A/D/+DB64Erup368ySL4xf4M8CVibKvI5ytGgXEOA1orUV4QrM7Cgzh7wrNIW8DsbkcYM14Q1j1ju113cviGXjfH3mOs5VmEWt6qvaXfZPGPZDT0ozjuhbzynOpD432uNyIJyKunsRpQieZEj8hqCrAXv+y+3GXY8MJ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sFwWUEGh; arc=none smtp.client-ip=209.85.210.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sFwWUEGh" Received: by mail-pf1-f200.google.com with SMTP id d2e1a72fcca58-848544a8496so2969582b3a.0 for ; Thu, 06 Aug 2026 14:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786052458; x=1786657258; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:from:to:cc:subject:date:message-id :reply-to:content-type; bh=DBP0D1Kh/01rtZWlznil3VMqkDCrOu4cEL69wfep54g=; b=sFwWUEGhs9CQN8ZEp7wgoDigZAp8Y66tmx1vxcMGzYGUnxJdIsVD7yHkuVa5euf4Vw 1RAQBuV9/eKWIzCSohjdbb6J2BRBQAZ7/NbndPpP+TUcMWpWaE3p5tAXQE/cxOSdsXVq DrqG1UnJfuZWOjkH+nyH5qPom7SGTPDDmUbKLk8MFFW4P4p6oJX1/ubwaPf7kzbsPYGd z3K1fPqd02JhpKSOAsmZHhPynO/f6FybvtkRqklVhsuMzlmYk30OlJVLUGnfRrUvE9u5 fZ92ZMsH446PxZ0sFb5R52+aCLfA3MknardJ58rMg4pf3c1oIkyJF3IDqe8yPnKbIxAo W/6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786052458; x=1786657258; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to:content-type; bh=DBP0D1Kh/01rtZWlznil3VMqkDCrOu4cEL69wfep54g=; b=DI9MUj5Tj0e0vGnzMdff6PoVnR0oQSeybm9tfqcXqCQdTVLqOexxP7vjZHM2QMrpyu Cmn7OfS+H1S1t1rXC2T9308NdlKEVva+BGsA5n8WATEK19VQN5P0zh7QnnxDoJKja50J cW/8RJJn8nR2NeW8Q7Zy7x3/6hqAaNovMBXmV2eNLfn4ye3RfvMIvmjSNMeTIXH12yKM yjpoHraKJmc1fLCEH9NkQK+HOagd6GPZvTc+wb/xGKAt427y3wgGf1Spebd8IhSmB14g Y4J3Ih2RigoFLYQnXFqnKQSLzhvHfAhfzSTIKAGfuCwD1KVwLAuWP5uRmWwEYqYWVTU6 ZPdA== X-Forwarded-Encrypted: i=1; AHgh+RqcbgvIvYGdy4zgOVAPhobti/lqw/CFBgR84esQOvz6xUeHOhepTU8RNhlMX3U6CDaU0ihEAv3cyV9BV7w=@vger.kernel.org X-Gm-Message-State: AOJu0YznhBP8v6I3crSZV8CSgW0VIlVNufmJTcAFqLIIG0ZKK6/zYrW+ J1yCR7TXFOxzKIpZhTerKSS2N71Pd5cfassIdkKgjttc1l25K4AnEJyHci/0axCSoO0hJ3xFZ9o t5G7Vkg== X-Received: from pfbde1.prod.google.com ([2002:a05:6a00:4681:b0:84a:3b69:a287]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:8d1:b0:845:ba94:a56 with SMTP id d2e1a72fcca58-84f4fd9f82amr4931166b3a.5.1786052457310; Thu, 06 Aug 2026 14:40:57 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 6 Aug 2026 14:40:49 -0700 In-Reply-To: <20260806214050.78058-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260806214050.78058-1-seanjc@google.com> X-Mailer: git-send-email 2.55.0.679.g6767b8d81c-goog Message-ID: <20260806214050.78058-4-seanjc@google.com> Subject: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN" From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Kai Huang , Yan Zhao , Rick Edgecombe , Sashiko Bot Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When mapping a private PFN in TDX's post-populate callback, top-up the memory caches on every attempt to map the PFN to harden against bugs in the map flow that could consume cache entries even if mapping ultimately fails. E.g. as pointed out by Sashiko, the in-progress Dynamic PAMT support could consume PAMT cache entries on TDX-Module lock contention. Harden KVM even though consuming an entry on failure is considered a KVM bug, as retry is uncommon, top-up is "free" if there's no work to be done, and populating a TDX guest's memory is a slow path, i.e. there's no meaningful downside to the hardening. Reported-by: Sashiko Bot Closes: https://lore.kernel.org/all/20260718061050.E17B01F000E9@smtp.kernel= .org Signed-off-by: Sean Christopherson Reviewed-by: Rick Edgecombe --- arch/x86/kvm/mmu/mmu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c6cac893cbad..379f570ef04f 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcp= u, gfn_t gfn, kvm_pfn_t pfn) if (kvm_gfn_is_write_tracked(kvm, fault.slot, fault.gfn)) return -EPERM; =20 - r =3D mmu_topup_memory_caches(vcpu, false); - if (r) - return r; - do { if (signal_pending(current)) return -EINTR; @@ -5199,6 +5195,10 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcp= u, gfn_t gfn, kvm_pfn_t pfn) if (r) return r; =20 + r =3D mmu_topup_memory_caches(vcpu, false); + if (r) + return r; + cond_resched(); =20 guard(read_lock)(&kvm->mmu_lock); --=20 2.55.0.679.g6767b8d81c-goog From nobody Wed Sep 30 20:13:29 2026 Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E2C842BC30 for ; Thu, 6 Aug 2026 21:41:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052461; cv=none; b=r7ZgsS+jeIXfQDvrgtnYYdJqEnFaNr3LVTcGysAVLB3dTNSuBFHWrfDxkSWC67fddspwj+Mi9JuIy4nn5LlHIUvbUI0qkf4FnjhhEGZmuFv7hTgWLVPjTJG8tc0NpSribhGxy4wjRu9fyf3olpUZH2FSDq0xciGQc9OHWjIfq2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786052461; c=relaxed/simple; bh=+O/AF4uYXNVy99eduE1BlXJm9BGh+j38uEA3Ic0iCPI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RRIWdE8Xi3lcGHFJI8DjuXzdvm1UlZS6wnxLC+LWSqTEUvDDRaP9qUC3TGX7FdbKE7gtRXmDhIwpFXFEnq+p82QY2mFesLmyTKz2/XSGYbO0RFQTa1xqAUlsOLrizt8erIY/BnIHqZUU1dVyvPHxWQpu2MTvUezr3Jbyv5QWnLA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HEZ2akTh; arc=none smtp.client-ip=209.85.214.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HEZ2akTh" Received: by mail-pl1-f198.google.com with SMTP id d9443c01a7336-2ccb6823efcso24564095ad.0 for ; Thu, 06 Aug 2026 14:41:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786052459; x=1786657259; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:from:to:cc:subject:date:message-id :reply-to:content-type; bh=RmJui4amYNqqPOSkRmfgD6DH/shD7OhU51lO1MsWx68=; b=HEZ2akThSvdZbk+7aA8vIV0ZBUMB17Xyb89fWg61TuLBrOYONMZvU2k8kSJ9tn6abU UjIwyMBqjmOLklWbLuhXuo+5fL2L3W+j/J7vRn41Yr3uFSlDA2hGFdl76jRa/DgsXkqk hDK+MhZWa1NgeTTMUIbvRiVymXVt7bglEYWHn0NjbCqQX6/QMw5WVogm62KxGNqXIzCZ I2z3EUeA+YN1eIYr1+cn/E9J4yQwRyXNddhfwOwveHsiuK+0fjQ3mY3Sa1bjyyLYqjQO jEXxw0491NUO657j7ZuZ1YWDPR0fyDxN0XMPFV+Vp4o+VXL48z/XVQygEwS87VngIt0z 3OEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786052459; x=1786657259; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:reply-to:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to:content-type; bh=RmJui4amYNqqPOSkRmfgD6DH/shD7OhU51lO1MsWx68=; b=O3rtE0xjzaqg4r8T7kdCpr7P8+Rjp9N3EPDFZ4Gi0to2e9VcReCMJ2aIImOpSch955 27yIWoLYQyCvEASUcJ9S2UNJiTRQcXbCYLRBT+N38UTczgLMUFNjuo75BlVeBqHY0/HO wkiB+RWIMbJscyXIEkq6d/SynQ6Ps/Kw0z0vBLa3KnIalzRpLb+ko+RtyKkwQfmcpAPh 6t1LNRLkIq24SUmU5iidriFNTjAsowCCEYHLvmQj+ifis8q1PxMd0EQUgEG5QmzUq0HX WO1ZHEUY+0BTD4Gdo0Hf3S48lLMayO9dN3JL99gMFtAZ3mVE3hnPBAdNl3yTW+PQDaUt 2uFA== X-Forwarded-Encrypted: i=1; AHgh+RpyxXKHcY6/LmEVLPCkIXce6NrvoM8HB+bc366r4pqeMKjOhYfVKluQIF0jOYUfnVoVr63mprNGrMGIaJc=@vger.kernel.org X-Gm-Message-State: AOJu0YxgEM/Bb4DRJ7CVnEVKxOCoHMbJjMm2CCSVMYjIbke9yxiZ+Ksq pclUgbK3Znq6650BTKhMOzjPl79Fw2Rec6VHkAjsgvXFdDrRgcJyQHMEyLNpWiYJyJkY5sJwufA kc2oH1A== X-Received: from plbmm14.prod.google.com ([2002:a17:903:a0e:b0:2ca:cf09:91ca]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:46c7:b0:2c9:b48c:fdec with SMTP id d9443c01a7336-2d0ca75d2admr231460015ad.12.1786052459403; Thu, 06 Aug 2026 14:40:59 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 6 Aug 2026 14:40:50 -0700 In-Reply-To: <20260806214050.78058-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260806214050.78058-1-seanjc@google.com> X-Mailer: git-send-email 2.55.0.679.g6767b8d81c-goog Message-ID: <20260806214050.78058-5-seanjc@google.com> Subject: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN" From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Kai Huang , Yan Zhao , Rick Edgecombe , Sashiko Bot Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Harden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free. Snapshot the invalidation sequence after acquiring mmu_lock to avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point of kvm_tdp_mmu_map_private_pfn() is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 379f570ef04f..76e3cd717324 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcp= u, gfn_t gfn, kvm_pfn_t pfn) */ WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)); =20 + /* + * Snapshot the invalidation sequence counter after acquiring + * mmu_lock, as guest_memfd guarantees the validity of the pfn, + * i.e. any concurrent invalidations are guaranteed to be + * irrelevant. + */ + fault.mmu_seq =3D vcpu->kvm->mmu_invalidate_seq; + if (is_page_fault_stale(vcpu, &fault)) + continue; + r =3D kvm_tdp_mmu_map(vcpu, &fault); } while (r =3D=3D RET_PF_RETRY); =20 --=20 2.55.0.679.g6767b8d81c-goog