From nobody Fri Sep 25 11:05:44 2026 Received: from www5210.sakura.ne.jp (www5210.sakura.ne.jp [133.167.8.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 636E2368D62; Sun, 13 Sep 2026 15:56:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=133.167.8.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789314964; cv=none; b=NEhFTCl5Mv8EJ38C5ULPrnNR63iZIxTrIxfLh2OQshYzBSs1wi9BzuRZ2bq7MXS+nd82PWtg3R6w7YrkWV74hzolYzIvRiqLtCOHD55v1qWiFVHuUprA7VJNC884Ml59IUicytKQ4MfR1pRbYECLxtXpBuQ7qMomI5RiL1qBlmw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789314964; c=relaxed/simple; bh=1iSs/vrWHDlC2vp1ysdAiuMR483Lav+FKygNYu6jyBU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XgQJDR1wQMsqTB4XqNjlTvnEP2piVMn2MVnuceQWzNjHbKxBdrqMsRBkszg3wazod7hcg+rYNURwnM0o5GitNu4i3ohNypaXatg4SCY1hFiJjFf7j5Ra/v8boqSZWfYkJK0GDddTNo0bNFo+F/ojDvdDpXy40hbGnspQWJu0Enc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me; spf=pass smtp.mailfrom=mgml.me; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b=byjqTuJW; arc=none smtp.client-ip=133.167.8.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mgml.me Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b="byjqTuJW" Received: from fedora (p3233127-ipxg00b01tokaisakaetozai.aichi.ocn.ne.jp [180.17.90.127]) (authenticated bits=0) by www5210.sakura.ne.jp (8.16.1/8.16.1) with ESMTPSA id 68DFtZE6020593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Sep 2026 00:55:47 +0900 (JST) (envelope-from k@mgml.me) DKIM-Signature: a=rsa-sha256; bh=qljtLoYmPRL1jB9MR4rFR4VrJjKYSmEFjcRWwNcZwss=; c=relaxed/relaxed; d=mgml.me; h=From:Message-ID:To:Subject:Date; s=rs20250315; t=1789314948; v=1; b=byjqTuJWOuLY43AbZVSjKW18D9GZPOEMK9zk4Qjb3qPfUJhBbm+bEE0DVSxTVoZd VK8IKqPWJvvbHg7bOMy5/zywxD4jE/+TLWVyYNFDNidcqgg/xcyqyi+Iv7hwuDqt LmfqMfqb+t5OG1jFkTf6tE5Lj4jCXOxSjD30dx0LLSwvIapQ9sAy56RmW927kI1x DCcTOfMseLwXbMVOexNwQh2f2oM/o+MXSjSdIXYxQNREE0umThbAHbVoiW/BTvTV R535VIYO8AzgXQHn5JsO0SoYtbkqyz7ll5P+5YdcztoRvEaQAetOlV7uHiECzn2o rS8fgz93pRZNLycPA7nOPQ== From: Kenta Akagi To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org Cc: Paolo Bonzini , David Matlack , Lai Jiangshan , Sean Christopherson , Kenta Akagi Subject: [PATCH 5.15.y v3 1/7] KVM: x86/mmu: Use a bool for direct Date: Mon, 14 Sep 2026 00:55:17 +0900 Message-ID: <20260913155523.7423-2-k@mgml.me> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260913155523.7423-1-k@mgml.me> References: <20260913155523.7423-1-k@mgml.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Matlack commit 27a59d57f073f21f029df1517c2c0a1abea5b0ce upstream. The parameter "direct" can either be true or false, and all of the callers pass in a bool variable or true/false literal, so just use the type bool. No functional change intended. Reviewed-by: Lai Jiangshan Reviewed-by: Sean Christopherson Signed-off-by: David Matlack Message-Id: <20220516232138.1783324-3-dmatlack@google.com> Signed-off-by: Paolo Bonzini Signed-off-by: Kenta Akagi --- arch/x86/kvm/mmu/mmu.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c0257bbb25c3..f5cfbf973e85 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1738,7 +1738,7 @@ static void drop_parent_pte(struct kvm_mmu_page *sp, mmu_spte_clear_no_track(parent_pte); } =20 -static struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, int = direct) +static struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, bool= direct) { struct kvm_mmu_page *sp; =20 @@ -2079,7 +2079,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct k= vm_vcpu *vcpu, gfn_t gfn, gva_t gaddr, unsigned level, - int direct, + bool direct, unsigned int access) { bool direct_mmu =3D vcpu->arch.mmu->direct_map; --=20 2.53.0 From nobody Fri Sep 25 11:05:44 2026 Received: from www5210.sakura.ne.jp (www5210.sakura.ne.jp [133.167.8.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07CF2366048; Sun, 13 Sep 2026 16:37:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=133.167.8.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317470; cv=none; b=aSrBtutgymxjix6DNX0uURq8Tj65+4z0xk4Zw1lbWxtGIi8N7puLzX3iyo6wFZpt8XqXXabC1YPdpSYpMEyXD9dmjPRG0ZdHahq8f/N9Vlr2Q7jPe8DtF5asdUbOos2eCRGmnPpDlqxx5+j7dtpPdVgLTvFfIKSxyT4x3KrvEWg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317470; c=relaxed/simple; bh=fF6BrxNabZ8uYURYJMaZdKHpbdY2dl7YO/LwAZZNXE4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YotNdtZxZ7Cj7BuxIjOE+nxNqpPLmd6L/EXbmA18bsHlnmqDXsOwxjSvAkN6ONv6CxWA+/Aj94JCYwApGaUILbNZBUN9R3ddxLzVjNovvMPlPGcaXSlE/9rUlROeanQBGX8uAxCDFWYCuQgibmNuix8jGFobKJwGTw+yzfxsBgg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me; spf=pass smtp.mailfrom=mgml.me; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b=W2qQG+77; arc=none smtp.client-ip=133.167.8.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mgml.me Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b="W2qQG+77" Received: from fedora (p3233127-ipxg00b01tokaisakaetozai.aichi.ocn.ne.jp [180.17.90.127]) (authenticated bits=0) by www5210.sakura.ne.jp (8.16.1/8.16.1) with ESMTPSA id 68DFtZE7020593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Sep 2026 00:55:48 +0900 (JST) (envelope-from k@mgml.me) DKIM-Signature: a=rsa-sha256; bh=ONwTd27hGGlJTVKeH/5etwyHj4/yEw8xYsr/hYa0hl0=; c=relaxed/relaxed; d=mgml.me; h=From:Message-ID:To:Subject:Date; s=rs20250315; t=1789314948; v=1; b=W2qQG+77hKKEw+pjoo2QxESJ7S3WxU4j6BskQ6iqO2DZRyYg04Z0ahTg1L1UCW+h 6H/DXYYt+2gY8QXa7a3Xm8N4ygiVBR590Fs3Jhx+ZKU/Ds+rPNhYb2BTLILBbm4e JxGWK9IvicCZPLs9OJ18Nscs6GYf0y52CCuEzgcUCywNJ//SFuFXl5Tgr9u1TWe7 4MTyhVZENFhZUvi5LcsF6u35si/aO3AnbXFmSQRW3qCh/JhhzZ6PU7IxF+WTIr/w 5IONLLvMxI7vJRwNxwkIqWwpx9RK9eRoS1MTEdX9tNcwd2b8S1sRouU9/1Xj4/vU X+AwU5tvw5j4eruvm6TtrQ== From: Kenta Akagi To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org Cc: Paolo Bonzini , David Matlack , Lai Jiangshan , Kenta Akagi Subject: [PATCH 5.15.y v3 2/7] KVM: x86/mmu: Stop passing "direct" to mmu_alloc_root() Date: Mon, 14 Sep 2026 00:55:18 +0900 Message-ID: <20260913155523.7423-3-k@mgml.me> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260913155523.7423-1-k@mgml.me> References: <20260913155523.7423-1-k@mgml.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Matlack commit 86938ab6925b8fe174ca6abf397e6ea9d3c054a4 upstream. The "direct" argument is vcpu->arch.mmu->root_role.direct, because unlike non-root page tables, it's impossible to have a direct root in an indirect MMU. So just use that. Suggested-by: Lai Jiangshan Signed-off-by: David Matlack Message-Id: <20220516232138.1783324-4-dmatlack@google.com> Signed-off-by: Paolo Bonzini Signed-off-by: Kenta Akagi --- arch/x86/kvm/mmu/mmu.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f5cfbf973e85..9f193b1620a9 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3439,8 +3439,9 @@ static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_= t root_gfn) } =20 static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, gva_t gva, - u8 level, bool direct) + u8 level) { + bool direct =3D vcpu->arch.mmu->mmu_role.base.direct; struct kvm_mmu_page *sp; =20 sp =3D kvm_mmu_get_page(vcpu, gfn, gva, level, direct, ACC_ALL); @@ -3466,7 +3467,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vc= pu) root =3D kvm_tdp_mmu_get_vcpu_root_hpa(vcpu); mmu->root_hpa =3D root; } else if (shadow_root_level >=3D PT64_ROOT_4LEVEL) { - root =3D mmu_alloc_root(vcpu, 0, 0, shadow_root_level, true); + root =3D mmu_alloc_root(vcpu, 0, 0, shadow_root_level); mmu->root_hpa =3D root; } else if (shadow_root_level =3D=3D PT32E_ROOT_LEVEL) { if (WARN_ON_ONCE(!mmu->pae_root)) { @@ -3478,7 +3479,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vc= pu) WARN_ON_ONCE(IS_VALID_PAE_ROOT(mmu->pae_root[i])); =20 root =3D mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), - i << 30, PT32_ROOT_LEVEL, true); + i << 30, PT32_ROOT_LEVEL); mmu->pae_root[i] =3D root | PT_PRESENT_MASK | shadow_me_mask; } @@ -3541,7 +3542,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vc= pu) */ if (mmu->root_level >=3D PT64_ROOT_4LEVEL) { root =3D mmu_alloc_root(vcpu, root_gfn, 0, - mmu->shadow_root_level, false); + mmu->shadow_root_level); mmu->root_hpa =3D root; goto set_root_pgd; } @@ -3587,7 +3588,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vc= pu) } =20 root =3D mmu_alloc_root(vcpu, root_gfn, i << 30, - PT32_ROOT_LEVEL, false); + PT32_ROOT_LEVEL); mmu->pae_root[i] =3D root | pm_mask; } =20 --=20 2.53.0 From nobody Fri Sep 25 11:05:44 2026 Received: from www5210.sakura.ne.jp (www5210.sakura.ne.jp [133.167.8.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDB2634EF07; Sun, 13 Sep 2026 15:56:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=133.167.8.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789314963; cv=none; b=Hkx4mp6Mow+ZketEfnSLA9Na/DcAVZSzMHGdUd+WCIgnnxYEJ44Qimf/xwjx/yPUYH+aV6XEKSUqJYst8wZP4HqodWo30EML9u6SKU/0LhNDsJWS4L8cS+CjGPnWmnM773pezLBtF8zEaXJHn2Nz7DMlINZ+FsndyKyJBIUEIW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789314963; c=relaxed/simple; bh=pEfW/VbURYIZsZ9HlQySG8Xf/is+jgd7kBTjwCkD7QY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bD9fhmi8KsqdNc5AYsfOUFe9rl9T4zDI8IBLZznopJJk5Z6PpXpwe7yj+h8uNHQ3qGTSVivCr/F3GSsn+gzo2fSWRgaoWn3E1jRRy0nucxJVq1E7UKnVAQS/sMU6oldfFVRVtDrkdM6VRZ0BFXI7QO8Y9058/lHwtLeq5sEO0bw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me; spf=pass smtp.mailfrom=mgml.me; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b=op50g2uf; arc=none smtp.client-ip=133.167.8.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mgml.me Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b="op50g2uf" Received: from fedora (p3233127-ipxg00b01tokaisakaetozai.aichi.ocn.ne.jp [180.17.90.127]) (authenticated bits=0) by www5210.sakura.ne.jp (8.16.1/8.16.1) with ESMTPSA id 68DFtZE8020593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Sep 2026 00:55:48 +0900 (JST) (envelope-from k@mgml.me) DKIM-Signature: a=rsa-sha256; bh=YaEIAsiwhSGLcvDX/PKpeIusk3/ck+HYp/Kt6VDFGOU=; c=relaxed/relaxed; d=mgml.me; h=From:Message-ID:To:Subject:Date; s=rs20250315; t=1789314948; v=1; b=op50g2ufd4frQQDgZVkqVDJTjaTFaafx+G0OjrUFOOrjdB7etMvwqzBKSrT/plJN 1fcr5OrPqVJ7e4/MuVA50EX1slB8ZhesMjexskbwBReq41e4OIQWi3u/uak49/WG 1cWT4TnycSF59sVDNhj05OewT7k+1eAR5fS/Tov9ljEc41tGwoOfQH+BJZQRn3+I jD3bjwUF1BNOlhDSLKKYNFLPD4px7lAVccqwNTneeLaWI9QUjNEImhQruNrzeDz/ cSwyFnHBYxFdBIsknQbkK8/+rRfO5/AK7le7jrFnNopqO0hZM8fsC3CTYTxgohat BVKwN8EI9uZ+5+nBXnDTkA== From: Kenta Akagi To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , David Matlack , Kenta Akagi Subject: [PATCH 5.15.y v3 3/7] KVM: x86/mmu: Derive shadow MMU page role from parent Date: Mon, 14 Sep 2026 00:55:19 +0900 Message-ID: <20260913155523.7423-4-k@mgml.me> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260913155523.7423-1-k@mgml.me> References: <20260913155523.7423-1-k@mgml.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini commit 2e65e842c57d72e9a573ba42bc2055b7f626ea1f upstream. Instead of computing the shadow page role from scratch for every new page, derive most of the information from the parent shadow page. This eliminates the dependency on the vCPU root role to allocate shadow page tables, and reduces the number of parameters to kvm_mmu_get_page(). Preemptively split out the role calculation to a separate function for use in a following commit. Note that when calculating the MMU root role, we can take @role.passthrough, @role.direct, and @role.access directly from @vcpu->arch.mmu->root_role. Only @role.level and @role.quadrant still must be overridden for PAE page directories, when shadowing 32-bit guest page tables with PAE page tables. No functional change intended. Reviewed-by: Peter Xu Signed-off-by: David Matlack Message-Id: <20220516232138.1783324-5-dmatlack@google.com> Signed-off-by: Paolo Bonzini Signed-off-by: Kenta Akagi --- arch/x86/kvm/mmu/mmu.c | 99 ++++++++++++++++++++++------------ arch/x86/kvm/mmu/paging_tmpl.h | 9 ++-- 2 files changed, 71 insertions(+), 37 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 9f193b1620a9..b0504d40ac8c 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2075,33 +2075,15 @@ static void clear_sp_write_flooding_count(u64 *spte) __clear_sp_write_flooding_count(sptep_to_sp(spte)); } =20 -static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, - gfn_t gfn, - gva_t gaddr, - unsigned level, - bool direct, - unsigned int access) +static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, gfn_t = gfn, + union kvm_mmu_page_role role) { bool direct_mmu =3D vcpu->arch.mmu->direct_map; - union kvm_mmu_page_role role; struct hlist_head *sp_list; - unsigned quadrant; struct kvm_mmu_page *sp; int collisions =3D 0; LIST_HEAD(invalid_list); =20 - role =3D vcpu->arch.mmu->mmu_role.base; - role.level =3D level; - role.direct =3D direct; - if (role.direct) - role.gpte_is_8_bytes =3D true; - role.access =3D access; - if (!direct_mmu && vcpu->arch.mmu->root_level <=3D PT32_ROOT_LEVEL) { - quadrant =3D gaddr >> (PAGE_SHIFT + (PT64_PT_BITS * level)); - quadrant &=3D (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1; - role.quadrant =3D quadrant; - } - sp_list =3D &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)]; for_each_valid_sp(vcpu->kvm, sp, sp_list) { if (sp->gfn !=3D gfn) { @@ -2119,7 +2101,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct k= vm_vcpu *vcpu, * Unsync pages must not be left as is, because the new * upper-level page will be write-protected. */ - if (level > PG_LEVEL_4K && sp->unsync) + if (role.level > PG_LEVEL_4K && sp->unsync) kvm_mmu_prepare_zap_page(vcpu->kvm, sp, &invalid_list); continue; @@ -2157,14 +2139,14 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct= kvm_vcpu *vcpu, =20 ++vcpu->kvm->stat.mmu_cache_miss; =20 - sp =3D kvm_mmu_alloc_page(vcpu, direct); + sp =3D kvm_mmu_alloc_page(vcpu, role.direct); =20 sp->gfn =3D gfn; sp->role =3D role; hlist_add_head(&sp->hash_link, sp_list); - if (!direct) { + if (!role.direct) { account_shadowed(vcpu->kvm, sp); - if (level =3D=3D PG_LEVEL_4K && rmap_write_protect(vcpu, gfn)) + if (role.level =3D=3D PG_LEVEL_4K && rmap_write_protect(vcpu, gfn)) kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn, 1); } trace_kvm_mmu_get_page(sp, true); @@ -2176,6 +2158,54 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct = kvm_vcpu *vcpu, return sp; } =20 +static union kvm_mmu_page_role kvm_mmu_child_role(u64 *sptep, bool direct,= unsigned int access) +{ + struct kvm_mmu_page *parent_sp =3D sptep_to_sp(sptep); + union kvm_mmu_page_role role; + + role =3D parent_sp->role; + role.level--; + role.access =3D access; + role.direct =3D direct; + + /* + * If the guest has 4-byte PTEs then that means it's using 32-bit, + * 2-level, non-PAE paging. KVM shadows such guests with PAE paging + * (i.e. 8-byte PTEs). The difference in PTE size means that KVM must + * shadow each guest page table with multiple shadow page tables, which + * requires extra bookkeeping in the role. + * + * Specifically, to shadow the guest's page directory (which covers a + * 4GiB address space), KVM uses 4 PAE page directories, each mapping + * 1GiB of the address space. @role.quadrant encodes which quarter of + * the address space each maps. + * + * To shadow the guest's page tables (which each map a 4MiB region), KVM + * uses 2 PAE page tables, each mapping a 2MiB region. For these, + * @role.quadrant encodes which half of the region they map. + * + * Note, the 4 PAE page directories are pre-allocated and the quadrant + * assigned in mmu_alloc_root(). So only page tables need to be handled + * here. + */ + if (!role.gpte_is_8_bytes) { + WARN_ON_ONCE(role.level !=3D PG_LEVEL_4K); + role.quadrant =3D (sptep - parent_sp->spt) % 2; + } + + return role; +} + +static struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu, + u64 *sptep, gfn_t gfn, + bool direct, unsigned int access) +{ + union kvm_mmu_page_role role; + + role =3D kvm_mmu_child_role(sptep, direct, access); + return kvm_mmu_get_page(vcpu, gfn, role); +} + static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *i= terator, struct kvm_vcpu *vcpu, hpa_t root, u64 addr) @@ -3043,8 +3073,7 @@ static int direct_map(struct kvm_vcpu *vcpu, gpa_t gp= a, u32 error_code, if (is_shadow_present_pte(*it.sptep)) continue; =20 - sp =3D kvm_mmu_get_page(vcpu, base_gfn, it.addr, - it.level - 1, true, ACC_ALL); + sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, base_gfn, true, ACC_ALL); =20 link_shadow_page(vcpu, it.sptep, sp); if (is_tdp && huge_page_disallowed && @@ -3438,13 +3467,18 @@ static int mmu_check_root(struct kvm_vcpu *vcpu, gf= n_t root_gfn) return ret; } =20 -static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, gva_t gva, +static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level) { - bool direct =3D vcpu->arch.mmu->mmu_role.base.direct; + union kvm_mmu_page_role role =3D vcpu->arch.mmu->mmu_role.base; struct kvm_mmu_page *sp; =20 - sp =3D kvm_mmu_get_page(vcpu, gfn, gva, level, direct, ACC_ALL); + role.level =3D level; + + if (!role.gpte_is_8_bytes) + role.quadrant =3D quadrant; + + sp =3D kvm_mmu_get_page(vcpu, gfn, role); ++sp->root_count; =20 return __pa(sp->spt); @@ -3478,8 +3512,8 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vc= pu) for (i =3D 0; i < 4; ++i) { WARN_ON_ONCE(IS_VALID_PAE_ROOT(mmu->pae_root[i])); =20 - root =3D mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), - i << 30, PT32_ROOT_LEVEL); + root =3D mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), i, + PT32_ROOT_LEVEL); mmu->pae_root[i] =3D root | PT_PRESENT_MASK | shadow_me_mask; } @@ -3587,8 +3621,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vc= pu) root_gfn =3D pdptrs[i] >> PAGE_SHIFT; } =20 - root =3D mmu_alloc_root(vcpu, root_gfn, i << 30, - PT32_ROOT_LEVEL); + root =3D mmu_alloc_root(vcpu, root_gfn, i, PT32_ROOT_LEVEL); mmu->pae_root[i] =3D root | pm_mask; } =20 diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index fd3f2c5bb7ed..003383e2a309 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -704,8 +704,9 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, gpa_t ad= dr, if (!is_shadow_present_pte(*it.sptep)) { table_gfn =3D gw->table_gfn[it.level - 2]; access =3D gw->pt_access[it.level - 2]; - sp =3D kvm_mmu_get_page(vcpu, table_gfn, addr, - it.level-1, false, access); + sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, table_gfn, + false, access); + /* * We must synchronize the pagetable before linking it * because the guest doesn't need to flush tlb when @@ -763,8 +764,8 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, gpa_t ad= dr, drop_large_spte(vcpu, it.sptep); =20 if (!is_shadow_present_pte(*it.sptep)) { - sp =3D kvm_mmu_get_page(vcpu, base_gfn, addr, - it.level - 1, true, direct_access); + sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, base_gfn, + true, direct_access); link_shadow_page(vcpu, it.sptep, sp); if (huge_page_disallowed && req_level >=3D it.level) account_huge_nx_page(vcpu->kvm, sp); --=20 2.53.0 From nobody Fri Sep 25 11:05:44 2026 Received: from www5210.sakura.ne.jp (www5210.sakura.ne.jp [133.167.8.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D605237DE9B; Sun, 13 Sep 2026 16:37:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=133.167.8.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317458; cv=none; b=Jt33oPCrUi79HbqeLBQdYJP8tQYvIsskfh7iBr6j92fV0+KdRGA5gc8tIkr0nQG75eX5WcAvSzSpxcyMg5t8chvdfhw9ABs3o0dAazLFZF2HFUL+k158z2qFsWLxYr8oWe+hP8sdFsJAHUKsc9j65NW1jNzo4RLxauSOYAme43Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317458; c=relaxed/simple; bh=5C2SbOtkyUoqeQdkPA3Ebp0XAKFZct1yG6BK+sDxttE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QwFhc+de/NwEqpDp5FR/A1v+Q/Th0QCem77p4rNF3OKC6MIZ+5PUDH4C7x2mnDDDgwpg8dv5ceRwucik1dJhGj/dqINqVSuKF+4coA4tlHc+k8Xgwk4PDeJpTRenYBwNWSeIuM1UGq69IBG3MKIhzGUAJMIN10C3/vdlC/Cq88w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me; spf=pass smtp.mailfrom=mgml.me; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b=kXBChM8p; arc=none smtp.client-ip=133.167.8.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mgml.me Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b="kXBChM8p" Received: from fedora (p3233127-ipxg00b01tokaisakaetozai.aichi.ocn.ne.jp [180.17.90.127]) (authenticated bits=0) by www5210.sakura.ne.jp (8.16.1/8.16.1) with ESMTPSA id 68DFtZE9020593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Sep 2026 00:55:48 +0900 (JST) (envelope-from k@mgml.me) DKIM-Signature: a=rsa-sha256; bh=pNPAfQf6/2IBhXvxqMcBuNLv8oIzKY2JGuQKmXs1nPY=; c=relaxed/relaxed; d=mgml.me; h=From:Message-ID:To:Subject:Date; s=rs20250315; t=1789314948; v=1; b=kXBChM8pyrwasQjtrtj9Y2S1kWbGxA5dz5I/hZ5HgBg9MI81bAx1jw+YyXAIz8/w taPQFaFoAFFSKB84KiMuvk/7RH1y/wcQswu5Upu3W5IG0ckxqHnEc7cWOOAWf7na VhFcfInckRv+IWG6I/lzci+F4W4t3vPim8zDtHM/XJShAHGqvKAyQ0HF/O2plt9N rwyH7TXiBUBwk/wYxmN/6jWtTMbFsjsNsc41N8kfZrB5M7kGnu1Cl37HzQ/0PVkk ECBw6fdHXk1lMdqT1/3ZRv1h0ClUUHFp2FU9D7uP+l9HlTo+48TskWXqHWcOgq6T 51hTRPKZKpGY8cwhG9okUQ== From: Kenta Akagi To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org Cc: Paolo Bonzini , David Matlack , Kenta Akagi Subject: [PATCH 5.15.y v3 4/7] KVM: x86/mmu: Always pass 0 for @quadrant when gptes are 8 bytes Date: Mon, 14 Sep 2026 00:55:20 +0900 Message-ID: <20260913155523.7423-5-k@mgml.me> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260913155523.7423-1-k@mgml.me> References: <20260913155523.7423-1-k@mgml.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini commit 7f49777550e55a7d6832cbb0873f48f91c175b9c upstream. The quadrant is only used when gptes are 4 bytes, but mmu_alloc_{direct,shadow}_roots() pass in a non-zero quadrant for PAE page directories regardless. Make this less confusing by only passing in a non-zero quadrant when it is actually necessary. Signed-off-by: David Matlack Message-Id: <20220516232138.1783324-6-dmatlack@google.com> Signed-off-by: Paolo Bonzini Signed-off-by: Kenta Akagi --- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b0504d40ac8c..b7801690ea17 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3474,9 +3474,10 @@ static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, g= fn_t gfn, int quadrant, struct kvm_mmu_page *sp; =20 role.level =3D level; + role.quadrant =3D quadrant; =20 - if (!role.gpte_is_8_bytes) - role.quadrant =3D quadrant; + WARN_ON_ONCE(quadrant && role.gpte_is_8_bytes); + WARN_ON_ONCE(role.direct && !role.gpte_is_8_bytes); =20 sp =3D kvm_mmu_get_page(vcpu, gfn, role); ++sp->root_count; @@ -3512,7 +3513,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vc= pu) for (i =3D 0; i < 4; ++i) { WARN_ON_ONCE(IS_VALID_PAE_ROOT(mmu->pae_root[i])); =20 - root =3D mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), i, + root =3D mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), 0, PT32_ROOT_LEVEL); mmu->pae_root[i] =3D root | PT_PRESENT_MASK | shadow_me_mask; @@ -3536,9 +3537,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vc= pu) struct kvm_mmu *mmu =3D vcpu->arch.mmu; u64 pdptrs[4], pm_mask; gfn_t root_gfn, root_pgd; + int quadrant, i, r; hpa_t root; - unsigned i; - int r; =20 root_pgd =3D mmu->get_guest_pgd(vcpu); root_gfn =3D root_pgd >> PAGE_SHIFT; @@ -3621,7 +3621,15 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *v= cpu) root_gfn =3D pdptrs[i] >> PAGE_SHIFT; } =20 - root =3D mmu_alloc_root(vcpu, root_gfn, i, PT32_ROOT_LEVEL); + /* + * If shadowing 32-bit non-PAE page tables, each PAE page + * directory maps one quarter of the guest's non-PAE page + * directory. Othwerise each PAE page direct shadows one guest + * PAE page directory so that quadrant should be 0. + */ + quadrant =3D !mmu->mmu_role.base.gpte_is_8_bytes ? i : 0; + + root =3D mmu_alloc_root(vcpu, root_gfn, quadrant, PT32_ROOT_LEVEL); mmu->pae_root[i] =3D root | pm_mask; } =20 --=20 2.53.0 From nobody Fri Sep 25 11:05:44 2026 Received: from www5210.sakura.ne.jp (www5210.sakura.ne.jp [133.167.8.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87B26374E6D; Sun, 13 Sep 2026 16:37:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=133.167.8.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317468; cv=none; b=WnqVeHjt5+5r5oGFZh0jitwxD9nER9+rpOW3BwXpFwJcYxCJ9l4v9n33jlO/9JA3PBlaxbEPmqjXfj0vZfs1okkIhZM50ZgOfMGtA98FV7y9K9kx1Xjz7Co2k266sy//OIo3s6D6wLNjTNHC+jSnZG6Tsu/6buAUAum26SuKsJ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317468; c=relaxed/simple; bh=8fjRi2R+1cVXeHL2jiDyWQgVzHtJyUWsiopOwcBNSHA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nxYyrup6sit4KFFvWSLjk4grRbOrcro7xAMSYROJNdiTLH+vnYb2jkavCK84R5DW4HY/Sbw7ev88xXDlpagMO0d3bJJXftSycAYmJFsQC3IhtCIFanauK+xq1O6tYNGatjfnjZfR0N8sNJcxI05tpGZfzLL6lSl6VKRu2XnXitc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me; spf=pass smtp.mailfrom=mgml.me; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b=d7DJEgDv; arc=none smtp.client-ip=133.167.8.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mgml.me Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b="d7DJEgDv" Received: from fedora (p3233127-ipxg00b01tokaisakaetozai.aichi.ocn.ne.jp [180.17.90.127]) (authenticated bits=0) by www5210.sakura.ne.jp (8.16.1/8.16.1) with ESMTPSA id 68DFtZEA020593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Sep 2026 00:55:48 +0900 (JST) (envelope-from k@mgml.me) DKIM-Signature: a=rsa-sha256; bh=L6aTFnd5HwgNh95Yx+fpQUD5wbe0WLrQfvUznwq7/nE=; c=relaxed/relaxed; d=mgml.me; h=From:Message-ID:To:Subject:Date; s=rs20250315; t=1789314948; v=1; b=d7DJEgDvODBEy+2BlWJ1bQZkNykY59/DIdIHtHPfSf05aCZbhNB6a48Cg9JTd5cN Jnb78s/VLqrONxin/norGa3vwatwGVzQOebc1qrP9I+haXIZPbb/yBxu3hpQ/pwB h1ta6cUmtCEd0QoSuu9yIAmP0v289y2gRw15dMfVmRlAtHFeW8AP3w7BQ87oVZ1i cMpsCuOTXF2DT2EKS3YRtnNvWuqlHEX0rahtUoYs9tb6uXTLXp2DTzwyIOIFclzK uLHNo14EqwSrE0chClGex8ki0chtib8McY1//imEZrQzP8v8oB5ukhCDSdkx6HKV DTIuvGXmri36esh8AAv2ag== From: Kenta Akagi To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , Kenta Akagi Subject: [PATCH 5.15.y v3 5/7] KVM: x86/mmu: pull call to drop_large_spte() into __link_shadow_page() Date: Mon, 14 Sep 2026 00:55:21 +0900 Message-ID: <20260913155523.7423-6-k@mgml.me> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260913155523.7423-1-k@mgml.me> References: <20260913155523.7423-1-k@mgml.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini commit 0cd8dc739833080aa0813cbd94d907a93e3a14c3 upstream. Before allocating a child shadow page table, all callers check whether the parent already points to a huge page and, if so, they drop that SPTE. This is done by drop_large_spte(). However, dropping the large SPTE is really only necessary before the sp is installed. While the sp is returned by kvm_mmu_get_child_sp(), installing it happens later in __link_shadow_page(). Move the call there instead of having it in each and every caller. To ensure that the shadow page is not linked twice if it was present, do _not_ opportunistically make kvm_mmu_get_child_sp() idempotent: instead, return an error value if the shadow page already existed. This is a bit more verbose, but clearer than NULL. Finally, now that the drop_large_spte() name is not taken anymore, remove the two underscores in front of __drop_large_spte(). Reviewed-by: Sean Christopherson Signed-off-by: Paolo Bonzini Signed-off-by: Kenta Akagi --- arch/x86/kvm/mmu/mmu.c | 49 +++++++++++++++++++--------------- arch/x86/kvm/mmu/paging_tmpl.h | 29 +++++++++----------- 2 files changed, 40 insertions(+), 38 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b7801690ea17..78c7e77bfdd0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1180,26 +1180,16 @@ static void drop_spte(struct kvm *kvm, u64 *sptep) rmap_remove(kvm, sptep); } =20 - -static bool __drop_large_spte(struct kvm *kvm, u64 *sptep) +static void drop_large_spte(struct kvm *kvm, u64 *sptep) { - if (is_large_pte(*sptep)) { - WARN_ON(sptep_to_sp(sptep)->role.level =3D=3D PG_LEVEL_4K); - drop_spte(kvm, sptep); - return true; - } - - return false; -} + struct kvm_mmu_page *sp; =20 -static void drop_large_spte(struct kvm_vcpu *vcpu, u64 *sptep) -{ - if (__drop_large_spte(vcpu->kvm, sptep)) { - struct kvm_mmu_page *sp =3D sptep_to_sp(sptep); + sp =3D sptep_to_sp(sptep); + WARN_ON(sp->role.level =3D=3D PG_LEVEL_4K); =20 - kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn, + drop_spte(kvm, sptep); + kvm_flush_remote_tlbs_with_address(kvm, sp->gfn, KVM_PAGES_PER_HPAGE(sp->role.level)); - } } =20 /* @@ -2202,6 +2192,9 @@ static struct kvm_mmu_page *kvm_mmu_get_child_sp(stru= ct kvm_vcpu *vcpu, { union kvm_mmu_page_role role; =20 + if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep)) + return ERR_PTR(-EEXIST); + role =3D kvm_mmu_child_role(sptep, direct, access); return kvm_mmu_get_page(vcpu, gfn, role); } @@ -2269,13 +2262,21 @@ static void shadow_walk_next(struct kvm_shadow_walk= _iterator *iterator) __shadow_walk_next(iterator, *iterator->sptep); } =20 -static void link_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep, - struct kvm_mmu_page *sp) +static void __link_shadow_page(struct kvm_vcpu *vcpu, + struct kvm_mmu_memory_cache *cache, u64 *sptep, + struct kvm_mmu_page *sp) { u64 spte; =20 BUILD_BUG_ON(VMX_EPT_WRITABLE_MASK !=3D PT_WRITABLE_MASK); =20 + /* + * If an SPTE is present already, it must be a leaf and therefore + * a large one. Drop it and flush the TLB before installing sp. + */ + if (is_shadow_present_pte(*sptep)) + drop_large_spte(vcpu->kvm, sptep); + spte =3D make_nonleaf_spte(sp->spt, sp_ad_disabled(sp)); =20 mmu_spte_set(sptep, spte); @@ -2286,6 +2287,12 @@ static void link_shadow_page(struct kvm_vcpu *vcpu, = u64 *sptep, mark_unsync(sptep); } =20 +static void link_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep, + struct kvm_mmu_page *sp) +{ + __link_shadow_page(vcpu, &vcpu->arch.mmu_pte_list_desc_cache, sptep, sp); +} + static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, unsigned direct_access) { @@ -3069,11 +3076,9 @@ static int direct_map(struct kvm_vcpu *vcpu, gpa_t g= pa, u32 error_code, if (it.level =3D=3D level) break; =20 - drop_large_spte(vcpu, it.sptep); - if (is_shadow_present_pte(*it.sptep)) - continue; - sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, base_gfn, true, ACC_ALL); + if (sp =3D=3D ERR_PTR(-EEXIST)) + continue; =20 link_shadow_page(vcpu, it.sptep, sp); if (is_tdp && huge_page_disallowed && diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 003383e2a309..0d50270e4f2f 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -698,15 +698,13 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, gpa_t = addr, gfn_t table_gfn; =20 clear_sp_write_flooding_count(it.sptep); - drop_large_spte(vcpu, it.sptep); =20 - sp =3D NULL; - if (!is_shadow_present_pte(*it.sptep)) { - table_gfn =3D gw->table_gfn[it.level - 2]; - access =3D gw->pt_access[it.level - 2]; - sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, table_gfn, - false, access); + table_gfn =3D gw->table_gfn[it.level - 2]; + access =3D gw->pt_access[it.level - 2]; + sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, table_gfn, + false, access); =20 + if (sp !=3D ERR_PTR(-EEXIST)) { /* * We must synchronize the pagetable before linking it * because the guest doesn't need to flush tlb when @@ -735,7 +733,7 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, gpa_t ad= dr, if (FNAME(gpte_changed)(vcpu, gw, it.level - 1)) goto out_gpte_changed; =20 - if (sp) + if (sp !=3D ERR_PTR(-EEXIST)) link_shadow_page(vcpu, it.sptep, sp); } =20 @@ -761,15 +759,14 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, gpa_t = addr, =20 validate_direct_spte(vcpu, it.sptep, direct_access); =20 - drop_large_spte(vcpu, it.sptep); + sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, base_gfn, + true, direct_access); + if (sp =3D=3D ERR_PTR(-EEXIST)) + continue; =20 - if (!is_shadow_present_pte(*it.sptep)) { - sp =3D kvm_mmu_get_child_sp(vcpu, it.sptep, base_gfn, - true, direct_access); - link_shadow_page(vcpu, it.sptep, sp); - if (huge_page_disallowed && req_level >=3D it.level) - account_huge_nx_page(vcpu->kvm, sp); - } + link_shadow_page(vcpu, it.sptep, sp); + if (huge_page_disallowed && req_level >=3D it.level) + account_huge_nx_page(vcpu->kvm, sp); } =20 ret =3D mmu_set_spte(vcpu, it.sptep, gw->pte_access, write_fault, --=20 2.53.0 From nobody Fri Sep 25 11:05:44 2026 Received: from www5210.sakura.ne.jp (www5210.sakura.ne.jp [133.167.8.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84F4B37AA65; Sun, 13 Sep 2026 15:56:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=133.167.8.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789314968; cv=none; b=KG7gOr+9ULRtggb1KY3Gn0s7GECx5LFEksLNxIOQiaUmJk7xDqCtiSvhuKXTu15KqSi8llfYMtagkzz7LHca10Vwlq2xijY6SFelv9HMq1FXo1IcM6hFGaqQ2Ku/6PAsfGov26LHOa/qO4/qtUjgBVW6CYQu2gjokcJGzK1mdGE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789314968; c=relaxed/simple; bh=foVvBTZGZiEy5Y1kEdmNf1VrmnTJmUtwTC9xMqYUlio=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JQXr+ioN0pwhWAlfeNRaev5ZtpSJ/ywp1UzmPbyq4Z5D35ogx0R99VdAUl8ZWLhjfdktPcgMZG/aS6K3nBHLdjJmjtIjV1P+3jtNoRQdUb67GLRSTEH1jTnqUgrLE/U+Q82sy96ctsvmjErY+NWemNpq7y2xpsDIyWATy4Q79Cw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me; spf=pass smtp.mailfrom=mgml.me; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b=gzHO04Ae; arc=none smtp.client-ip=133.167.8.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mgml.me Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b="gzHO04Ae" Received: from fedora (p3233127-ipxg00b01tokaisakaetozai.aichi.ocn.ne.jp [180.17.90.127]) (authenticated bits=0) by www5210.sakura.ne.jp (8.16.1/8.16.1) with ESMTPSA id 68DFtZEB020593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Sep 2026 00:55:48 +0900 (JST) (envelope-from k@mgml.me) DKIM-Signature: a=rsa-sha256; bh=0meUkPycroKNsH/2KAUZCp1jRFq4WmTTXqEz0it/ReU=; c=relaxed/relaxed; d=mgml.me; h=From:Message-ID:To:Subject:Date; s=rs20250315; t=1789314948; v=1; b=gzHO04AeVgHcRnYldIG2qzeH6jYLxJsgR1Hhh0i1tgIIpOuuZN5dqzZvfB/n3Ea7 3NinJLuGAZvEwglgNJ01X01Z0l0CSnDZeeV8JRAWqceWMWSU+/wtg/zZKboFzMll KzzYAUU0X1veZBsrZXQIDLa+4+6OdjVEs0qTX5C2KeUizwlqsIQ3MeYB3X2o5vA9 X3TTSm2rXAiL9VsQ7uNH/j9UvgkXoEVN7WlQXVAjMLOGzIHiQwzW84mQqCVjl7Mw kGANPf9LANNSV173EqD2QSibz4jR5S1Gxe63nMQ1PEPZP+phej0k52H5ES71+ix0 Oie9szYxfkfCBSDx+E1uVw== From: Kenta Akagi To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , Alexander Bulekov , Fred Griffoul , Kenta Akagi Subject: [PATCH 5.15.y v3 6/7] KVM: x86: Fix shadow paging use-after-free due to unexpected GFN Date: Mon, 14 Sep 2026 00:55:22 +0900 Message-ID: <20260913155523.7423-7-k@mgml.me> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260913155523.7423-1-k@mgml.me> References: <20260913155523.7423-1-k@mgml.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Sean Christopherson commit 0cb2af2ea66ad8ff195c156ea690f11216285bdf upstream. The shadow MMU computes GFNs for direct shadow pages using sp->gfn plus the SPTE index. This assumption breaks for shadow paging if the guest page tables are modified between VM entries (similar to commit aad885e77496, "KVM: x86/mmu: Drop/zap existing present SPTE even when creating an MMIO SPTE", 2026-03-27). The flow is as follows: - a PDE is installed for a 2MB mapping, and a page in that area is accessed. KVM creates a kvm_mmu_page consisting of 512 4KB pages; the kvm_mmu_page is marked by FNAME(fetch) as direct-mapped because the guest's mapping is a huge page (and thus contiguous). - the PDE mapping is changed from outside the guest. - the guest accesses another page in the same 2MB area. KVM installs a new leaf SPTE and rmap entry; the SPTE uses the "correct" GFN (i.e. based on the new mapping, as changed in the previous step) but that GFN is outside of the [sp->gfn, sp->gfn + 511] range; therefore the rmap entry cannot be found and removed when the kvm_mmu_page is zapped. - the memslot that covers the first 2MB mapping is deleted, and the kvm_mmu_page for the now-invalid GPA is zapped. However, rmap_remove() only looks at the [sp->gfn, sp->gfn + 511] range established in step 1, and fails to find the rmap entry that was recorded by step 3. - any operation that causes an rmap walk for the same page accessed by step 3 then walks a stale rmap and dereferences a freed kvm_mmu_page. This includes dirty logging or MMU notifier invalidations (e.g., from MADV_DONTNEED). The underlying issue is that KVM's walking of shadow PTEs assumes that if a SPTE is present when KVM wants to install a non-leaf SPTE, then the existing kvm_mmu_page must be for the correct gfn. Because the only way for the gfn to be wrong is if KVM messed up and failed to zap a SPTE... which shouldn't happen, but *actually* only happens in response to a guest write. That bug dates back literally forever, as even the first version of KVM assumes that the GFN matches and walks into the "wrong" shadow page. However, that was only an imprecision until 2032a93d66fa ("KVM: MMU: Don't allocate gfns page for direct mmu pages") came along. Fix it by checking for a target gfn mismatch and zapping the existing SPTE. That way the old SP and rmap entries are gone, KVM installs the rmap in the right location, and everyone is happy. Fixes: 2032a93d66fa ("KVM: MMU: Don't allocate gfns page for direct mmu pag= es") Fixes: 6aa8b732ca01 ("kvm: userspace interface") Reported-by: Alexander Bulekov Reported-by: Fred Griffoul Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Link: https://patch.msgid.link/20260503201029.106481-1-pbonzini@redhat.com/ Signed-off-by: Paolo Bonzini Signed-off-by: Kenta Akagi --- arch/x86/kvm/mmu/mmu.c | 33 ++++++++++++++------------------- arch/x86/kvm/mmu/spte.h | 5 +++++ 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 78c7e77bfdd0..0610198b1f74 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -188,6 +188,8 @@ static struct percpu_counter kvm_total_used_mmu_pages; static void mmu_spte_set(u64 *sptep, u64 spte); static union kvm_mmu_page_role kvm_mmu_calc_root_page_role(struct kvm_vcpu *vcpu); +static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, + u64 *spte, struct list_head *invalid_list); =20 struct kvm_mmu_role_regs { const unsigned long cr0; @@ -1180,18 +1182,6 @@ static void drop_spte(struct kvm *kvm, u64 *sptep) rmap_remove(kvm, sptep); } =20 -static void drop_large_spte(struct kvm *kvm, u64 *sptep) -{ - struct kvm_mmu_page *sp; - - sp =3D sptep_to_sp(sptep); - WARN_ON(sp->role.level =3D=3D PG_LEVEL_4K); - - drop_spte(kvm, sptep); - kvm_flush_remote_tlbs_with_address(kvm, sp->gfn, - KVM_PAGES_PER_HPAGE(sp->role.level)); -} - /* * Write-protect on the specified @sptep, @pt_protect indicates whether * spte write-protection is caused by protecting shadow page table. @@ -2192,7 +2182,8 @@ static struct kvm_mmu_page *kvm_mmu_get_child_sp(stru= ct kvm_vcpu *vcpu, { union kvm_mmu_page_role role; =20 - if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep)) + if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep) && + spte_to_child_sp(*sptep) && spte_to_child_sp(*sptep)->gfn =3D=3D gfn) return ERR_PTR(-EEXIST); =20 role =3D kvm_mmu_child_role(sptep, direct, access); @@ -2270,12 +2261,16 @@ static void __link_shadow_page(struct kvm_vcpu *vcp= u, =20 BUILD_BUG_ON(VMX_EPT_WRITABLE_MASK !=3D PT_WRITABLE_MASK); =20 - /* - * If an SPTE is present already, it must be a leaf and therefore - * a large one. Drop it and flush the TLB before installing sp. - */ - if (is_shadow_present_pte(*sptep)) - drop_large_spte(vcpu->kvm, sptep); + if (is_shadow_present_pte(*sptep)) { + struct kvm_mmu_page *parent_sp; + LIST_HEAD(invalid_list); + + parent_sp =3D sptep_to_sp(sptep); + WARN_ON_ONCE(parent_sp->role.level =3D=3D PG_LEVEL_4K); + + mmu_page_zap_pte(vcpu->kvm, parent_sp, sptep, &invalid_list); + kvm_mmu_remote_flush_or_zap(vcpu->kvm, &invalid_list, true); + } =20 spte =3D make_nonleaf_spte(sp->spt, sp_ad_disabled(sp)); =20 diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 31d6456d8ac3..31d03d15415c 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -267,6 +267,11 @@ static inline bool is_executable_pte(u64 spte) return (spte & (shadow_x_mask | shadow_nx_mask)) =3D=3D shadow_x_mask; } =20 +static inline struct kvm_mmu_page *spte_to_child_sp(u64 spte) +{ + return to_shadow_page(spte & PT64_BASE_ADDR_MASK); +} + static inline kvm_pfn_t spte_to_pfn(u64 pte) { return (pte & PT64_BASE_ADDR_MASK) >> PAGE_SHIFT; --=20 2.53.0 From nobody Fri Sep 25 11:05:44 2026 Received: from www5210.sakura.ne.jp (www5210.sakura.ne.jp [133.167.8.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CD1537F306; Sun, 13 Sep 2026 16:37:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=133.167.8.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317466; cv=none; b=luyRvyauE8LNWA56fPU9sURhwlSoJckpgKx4vQMlEOFKLLHg0UqQE4LUdm29jh893FL5amW/D02/T+MgeNzgG/KbHPPMxaA+MUH18sEXPrcYw67Wl+s/BhVQk6zzDg9PxHr4zoo/ZAKRNJHJk+we3URFKf6m2KgFcLPBHuPaj6g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789317466; c=relaxed/simple; bh=+qras4rwVzl2jsJaGMk0tKVesW5WPg9MF47fqYbvlVw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rMfAkdR/ZkDypjAdF9UmsuGaMhUxHiQSyP8kAv6sNOzlcrBepQiCZxUjnQwhW69xw8T1ro4wC5JmMwvsyzcPXPGXp0chF4HUwS3uWdKhewPI2buZGqLdw4uH4y/bnXtesQT3KNmfZFqf3PL25/mOYacXZL0xsY/eTc8jkx3t/jQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me; spf=pass smtp.mailfrom=mgml.me; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b=n3lKEz+3; arc=none smtp.client-ip=133.167.8.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=mgml.me Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=mgml.me Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=mgml.me header.i=@mgml.me header.b="n3lKEz+3" Received: from fedora (p3233127-ipxg00b01tokaisakaetozai.aichi.ocn.ne.jp [180.17.90.127]) (authenticated bits=0) by www5210.sakura.ne.jp (8.16.1/8.16.1) with ESMTPSA id 68DFtZEC020593 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 14 Sep 2026 00:55:48 +0900 (JST) (envelope-from k@mgml.me) DKIM-Signature: a=rsa-sha256; bh=pVwcP1e/me71DbSJVFHXPWArLsQKJ4GrRhXHd9EJWSk=; c=relaxed/relaxed; d=mgml.me; h=From:Message-ID:To:Subject:Date; s=rs20250315; t=1789314948; v=1; b=n3lKEz+399metEt7v7wGwPUNuIt+KxpZ3+EMaTuI9e6bvrLE58VuAndtgjzvi+aS 6ekAy6Wv3ku57Laof+cpXlz8WbU9Swqc1OfHQihIUMnLTWVDAT6qxctvD0J9BBsw HVEIHq5zSdaI3I2qLuv50DCoRPrQCqmXdjzwR4hCmbwmwpzid95Uz8dJPhOA8tqv uZOl/Eav1dQSAyoY4HopasQm0Y1rmPhNs8aTYKON+YGb3c5ZtcDhac3aNqzGzSy3 ZZst5oaE4h27MKVGBuvo5/IHn7APFXnliPBfTAx87fUVG7aDCnTVlrcgHArsJyAV CTI+HuwGsjk7dbUH8/9yyw== From: Kenta Akagi To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, stable@vger.kernel.org Cc: Paolo Bonzini , Hyunwoo Kim , Kenta Akagi Subject: [PATCH 5.15.y v3 7/7] KVM: x86: Fix shadow paging use-after-free due to unexpected role Date: Mon, 14 Sep 2026 00:55:23 +0900 Message-ID: <20260913155523.7423-8-k@mgml.me> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260913155523.7423-1-k@mgml.me> References: <20260913155523.7423-1-k@mgml.me> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini [ Upstream commit 81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb ] Commit 0cb2af2ea66ad ("KVM: x86: Fix shadow paging use-after-free due to unexpected GFN") fixed a shadow paging mismatch between stored and computed GFNs; the bug could be triggered by changing a PDE mapping from outside the guest, and then deleting a memslot. The rmap_remove() call would miss entries created after the PDE change because the GFN of the leaf SPTE does not match the GFN of the struct kvm_mmu_page. A similar hole however remains if the modified PDE points to a non-leaf page. In this case the gfn can be made to match, but the role does not match: the original large 2MB page creates a kvm_mmu_page with direct=3D1, while the new 4KB needs a kvm_mmu_page with direct=3D0. However, kvm_mmu_get_child_sp() does not compare the role, and therefore reuses the page. The next step is installing a leaf (4KB) SPTE on the new path which records an rmap entry under the gfn resolved by the walk. But when that child is zapped its parent kvm_mmu_page has direct=3D1 and kvm_mmu_page_get_gfn() computes the gfn for the 4KB page as sp->gfn + index instead of using sp->shadowed_translation[] (or sp->gfns[] in older kernels). It therefore fails to remove the recorded entry. When the memslot is dropped the shadow page is freed but the rmap entry survives, as in the scenario that was already fixed. Code that later walks that gfn (dirty logging, MMU notifier invalidation, and so on) dereferences an sptep that lies in the freed page, causing the use-after-free. Fixes: 2032a93d66fa ("KVM: MMU: Don't allocate gfns page for direct mmu pag= es") Reported-by: Hyunwoo Kim Signed-off-by: Paolo Bonzini Signed-off-by: Kenta Akagi --- arch/x86/kvm/mmu/mmu.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 0610198b1f74..cc0dfacab65c 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2180,13 +2180,15 @@ static struct kvm_mmu_page *kvm_mmu_get_child_sp(st= ruct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, bool direct, unsigned int access) { - union kvm_mmu_page_role role; + union kvm_mmu_page_role role =3D kvm_mmu_child_role(sptep, direct, access= ); =20 - if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep) && - spte_to_child_sp(*sptep) && spte_to_child_sp(*sptep)->gfn =3D=3D gfn) + if (is_shadow_present_pte(*sptep) && + !is_large_pte(*sptep) && + spte_to_child_sp(*sptep) && + spte_to_child_sp(*sptep)->gfn =3D=3D gfn && + spte_to_child_sp(*sptep)->role.word =3D=3D role.word) return ERR_PTR(-EEXIST); =20 - role =3D kvm_mmu_child_role(sptep, direct, access); return kvm_mmu_get_page(vcpu, gfn, role); } =20 --=20 2.53.0