From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-67.mta0.migadu.com [91.218.175.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E598459AE1 for ; Mon, 14 Sep 2026 12:30:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.67 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389047; cv=none; b=n8pGUQZnLMq6TFyzMsquWPzh1aTBmm0lQLCpzu2s1/dHvTUufWguZWthr1TJYmgiDtopVce0yVJLZf764svwiIMJ7YkHo7o6SqrkjxTyKWG0SyjhQ0zhImzKlJIbK9LuPmn6Y1TBEfZ43sgh5Zl8Ikm371Bi/CdIV5vyUv8XP7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389047; c=relaxed/simple; bh=tuISWPJ3M4ynpVPzFp7v67YsNxfpcndtSeOMAgkqq08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O2MbyaSF8BiLr5zwCaouLb+TR2vZuQqvO6zj0uUa/7GZu/wJZ7oB2m87hSoK8104LUgjw6W54Shz1SdYyAwBjudEEjXUV6RnckDE2PDw8BLjFay6tA140n0pq73UYb8ZtJrw4Bjrh2n7tje8/8/1uaZE5coeBePNW/F85Y7TcoA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=UHTip5QV; arc=none smtp.client-ip=91.218.175.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="UHTip5QV" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=tuISWPJ3M4ynpVPzFp7v67YsNxfpcndtSeOMAgkqq08=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389039; v=1; x=1789993839; b=UHTip5QVLb49ioch3GdXPfvpGZwkxa52B/FRIS5AswMbM8y6lJHM+qweWecLL+u2FNjDT5Oo 9Us0srk9Fb/bnHvSZL2zh5RlSrM0WC3WotX1muaA5466AIRrAzeec7NsriNFqWaCay96OzsVJs7 eFlWbN+t/gYJ7KftzJ8+0BRI= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id a11a52cd755f1f77; Mon, 14 Sep 2026 12:30:11 +0000 X-Mizu-Trace-ID: a11a52cd755f1f77 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio() Date: Mon, 14 Sep 2026 05:27:51 -0700 Message-ID: <20260914122950.3283997-2-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" pmd_to_softleaf_folio() reads as if it converted a PMD into a folio. What it does is decode the softleaf entry stored in the PMD and return the folio that entry references - the direction softleaf_to_folio() already spells out. No functional change intended. Suggested-by: Dev Jain Signed-off-by: Usama Arif Acked-by: David Hildenbrand (Arm) Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Zi Yan Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Baoquan He Reviewed-by: Barry Song Reviewed-by: Qi Zheng --- include/linux/leafops.h | 4 ++-- mm/huge_memory.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/leafops.h b/include/linux/leafops.h index 4c1476ae32343..7c13c58a5e218 100644 --- a/include/linux/leafops.h +++ b/include/linux/leafops.h @@ -657,7 +657,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd) } =20 /** - * pmd_to_softleaf_folio() - Convert the PMD entry to a folio. + * pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio. * @pmd: PMD entry. * * The PMD entry is expected to be a valid PMD softleaf entry. @@ -665,7 +665,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd) * Returns: the folio the softleaf entry references if this is a valid sof= tleaf * entry, otherwise NULL. */ -static inline struct folio *pmd_to_softleaf_folio(pmd_t pmd) +static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd) { const softleaf_t entry =3D softleaf_from_pmd(pmd); =20 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7140a1031fb2e..ee8d46827ffdc 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2518,7 +2518,7 @@ static struct folio *normal_or_softleaf_folio_pmd(str= uct vm_area_struct *vma, =20 if (!thp_migration_supported()) WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); - return pmd_to_softleaf_folio(pmdval); + return pmd_softleaf_to_folio(pmdval); } =20 static bool has_deposited_pgtable(struct vm_area_struct *vma, pmd_t pmdval, --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-101.mta0.migadu.com [91.218.175.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1035144A3F1 for ; Mon, 14 Sep 2026 12:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.101 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389056; cv=none; b=FBO8VoG6kcaSrU5eHpJqiCqsUq9A12XxHJYH/f13eyrSU4ITGtOXbomgkGecK0mZkhy+FKNSsmEITV1daS5GUL5U2X7wgZqfa89pIO+NlzsS+TmDtoSAYuO60oRsZHpcZllcJzB4HQWReHeLFlTRWa6U7NLQFRjuCZmoLfj4dKA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389056; c=relaxed/simple; bh=owEO/GVV8kpSqjtX6z75OJu1GLQvZh0Qa08KVOaicH4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h09CHJXAHDR8SyL/jGFPj1BMNp6h976tDCJshjKAFYzYYdRktGeMqWnDTcTyfrnWh4DP6z/ildJf4+nUZYBpNa/J3XtOTsO1xju8U7dVwmOBRioQ4Y1IicGndlcddquUbhPeO79q2R6t9Yr8FdDjwZo2NTroOi+KmOV1xhMedrE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=isnHcp9w; arc=none smtp.client-ip=91.218.175.101 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="isnHcp9w" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=owEO/GVV8kpSqjtX6z75OJu1GLQvZh0Qa08KVOaicH4=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389046; v=1; x=1789993846; b=isnHcp9w4G5qacNcvS5Mi2mZmyThOkjHOVouC8pU6qeH1NSF/4NNT1AjSPqvT2kBoal/Bl36 PpumCHIC/3aZbzZR8durkmpjgBuFwOyZtHY8FpN7SL6Dim1DUqq+wrUtM9tO+K/Z5xtgbj5daHa M8TnSIR9IYEB/oZefNVYqUZ4= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id 64381a4a761f26a4; Mon, 14 Sep 2026 12:30:46 +0000 X-Mizu-Trace-ID: 64381a4a761f26a4 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif , Catalin Marinas , Will Deacon Subject: [RESEND v7 02/29] arm64: mm: add PMD swap-exclusive helpers Date: Mon, 14 Sep 2026 05:27:52 -0700 Message-ID: <20260914122950.3283997-3-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and not just in a swap PTE. arm64 encodes a swap PMD exactly like a swap PTE, so the new helpers wrap the PTE ones and reuse PTE_SWP_EXCLUSIVE. Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Usama Arif --- arch/arm64/include/asm/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index e89ec5f4787b4..d3f53a601aed3 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -599,6 +599,12 @@ static inline int pmd_protnone(pmd_t pmd) #define pmd_swp_clear_uffd(pmd) \ pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd))) #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +#define pmd_swp_exclusive(pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_mkexclusive(pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_clear_exclusive(pmd) \ + pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) +#endif =20 #define pmd_write(pmd) pte_write(pmd_pte(pmd)) =20 --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-118.mta0.migadu.com [91.218.175.118]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 292FE45A2A1 for ; Mon, 14 Sep 2026 12:30:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.118 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389056; cv=none; b=Zuqsmk+XX5UqINVURTZKTn8htkxq1VAnL+oUl3q8nCEoUhxknf3rW2buxfnpRhpy0sjAl+RnGXEVcfUjTfVXeo/dHOa8+Y0RrwxfjFX3Ie5jy1Zl8yBOlK7mdNOUWy4j9z0E6gJ8QKGn2uqAfLRGgHvo+V1Vm/sDSmYqfBS6oEY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389056; c=relaxed/simple; bh=QiAy/7wN1ydZlsQx7DI1Vjrn3FeWZdoFpSxlDD4vX6A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dIn5D0LkXihLZgC+BAY9ZC7JH7Cot72ASPWsLU336zn6NveAidCuMqaf6XTQZgtK+yvqGCEbroxfLEqNejeW/nsdnrmXbW8iUl4N7/+USD2nezLLTVLBZqvYFhPM9W5RtgLZYHMVOaSYr3D9BaqoU+w0SLFpnH3g9o/Gz4Vm570= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=OAinCtuu; arc=none smtp.client-ip=91.218.175.118 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="OAinCtuu" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=QiAy/7wN1ydZlsQx7DI1Vjrn3FeWZdoFpSxlDD4vX6A=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389048; v=1; x=1789993848; b=OAinCtuupAMpJPULjtCJAIGtBGG71/6/dhf8cIE0WOAxUZHlxw4motUaVppxuVDpasZS3YDx exGS+XGpG9xGjy5X0fn49XOnJvqbkIFZiisEuC1wUMK2z70Bd2B3r/165VonXwZ8EFmAcYk7T/S QJnoZSshB0OnbVgxCSr79Tzw= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 208b78021ed50abd; Mon, 14 Sep 2026 12:30:48 +0000 X-Mizu-Trace-ID: 208b78021ed50abd X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif , Huacai Chen Subject: [RESEND v7 03/29] loongarch: mm: add PMD swap-exclusive helpers Date: Mon, 14 Sep 2026 05:27:53 -0700 Message-ID: <20260914122950.3283997-4-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and not just in a swap PTE. A LoongArch swap PMD is the swap PTE value plus _PAGE_HUGE, and _PAGE_SWP_EXCLUSIVE sits outside both the type and the offset field, so the PMD helpers can use the same bit. Cc: Huacai Chen Signed-off-by: Usama Arif Reviewed-by: David Hildenbrand (Arm) --- arch/loongarch/include/asm/pgtable.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/= asm/pgtable.h index cf29a4c8ac593..87fecc3a51001 100644 --- a/arch/loongarch/include/asm/pgtable.h +++ b/arch/loongarch/include/asm/pgtable.h @@ -351,6 +351,25 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return pte; } =20 +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + pmd_val(pmd) |=3D _PAGE_SWP_EXCLUSIVE; + return pmd; +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_val(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + pmd_val(pmd) &=3D ~_PAGE_SWP_EXCLUSIVE; + return pmd; +} +#endif + #define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL)) #define pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE)) #define pte_no_exec(pte) (pte_val(pte) & _PAGE_NO_EXEC) --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-137.mta0.migadu.com [91.218.175.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0700B46D090 for ; Mon, 14 Sep 2026 12:30:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.137 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389058; cv=none; b=Qwv13a1iggO7Huw1lOaBX1Wo0qT9ntOQRfmyKKatG/bNFrDJL9zqBLLYoxX6dVCXeGdNSTw4P3p00OlnDcR07Un9ut22DAlr97CsHqbfrzJcTpTXedIHWTcB3NhUK5iQoewsZ7FvznzEmjz7rgNzq1o3QWSAPdQDpHfkI/2Mq24= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389058; c=relaxed/simple; bh=BtQeHE2SZzfUtIrg0bS6PVLMNQTg1JxKJHlqS6nP29w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p+UHvN46Cxt9RuDyeVnTA8TQbj7ls0OBLBDmUaMoKRd+9h+aybzPnKXIycbFN/ZIxAGXhSiR9tU+B5PYnr3z8KpmDLPOjNh/QkqzmpkXUgybFY5VV3kr3M0ewNFCo8XjwRQnkZCdTePizsnSgZiL5KSKhr9QuOylRDb5PBvYm6c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=O26aFq95; arc=none smtp.client-ip=91.218.175.137 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="O26aFq95" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=BtQeHE2SZzfUtIrg0bS6PVLMNQTg1JxKJHlqS6nP29w=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389051; v=1; x=1789993851; b=O26aFq95QqbDPrrBZZAJN+/i5U6vyG752FUsZzaW5Iryt3TedNZY+Z49E+ceOAAKWNmiThEE 4xuXBjMLRG0W1o7P7M4qV3aUdHIlCHJmMztB2anIptP/bFVK/zpY98AhGcOR9b+k/FKqlBEHhLx W4ICbweYO7LQYxDCPSnNtC0I= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id 6a02ba748b326016; Mon, 14 Sep 2026 12:30:51 +0000 X-Mizu-Trace-ID: 6a02ba748b326016 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif , Madhavan Srinivasan Subject: [RESEND v7 04/29] powerpc: mm: add PMD swap-exclusive helpers Date: Mon, 14 Sep 2026 05:27:54 -0700 Message-ID: <20260914122950.3283997-5-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and not just in a swap PTE. book3s64 builds a swap PMD by running the PTE encoding over pmd_pte(), so the PMD helpers use the same _PAGE_SWP_EXCLUSIVE bit. It is also the only powerpc variant that selects ARCH_HAS_PMD_SOFTLEAVES, via PPC_THP. Cc: Madhavan Srinivasan Signed-off-by: Usama Arif --- arch/powerpc/include/asm/book3s/64/pgtable.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/in= clude/asm/book3s/64/pgtable.h index dff8790a047db..28943ef3c1c80 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -699,6 +699,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return __pte_raw(pte_raw(pte) & cpu_to_be64(~_PAGE_SWP_EXCLUSIVE)); } =20 +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return __pmd_raw(pmd_raw(pmd) | cpu_to_be64(_PAGE_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return !!(pmd_raw(pmd) & cpu_to_be64(_PAGE_SWP_EXCLUSIVE)); +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return __pmd_raw(pmd_raw(pmd) & cpu_to_be64(~_PAGE_SWP_EXCLUSIVE)); +} +#endif + static inline bool check_pte_access(unsigned long access, unsigned long pt= ev) { /* --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-156.mta0.migadu.com [91.218.175.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8591246D2C4 for ; Mon, 14 Sep 2026 12:30:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389064; cv=none; b=IgicLNqTYLiAjgkr0sUV+rMT5Wi4w8OLRLWnrQ/co+nTcU2PQCq41/nmtPiYm7yi1YZBJsc2RF6KQVLZkTI3fxM8ITz26/dGYOWIwFbZQQrxma2SR22hiBOKCjrnY3BkuRLneSUkWt7HeQ/N43/Mys/7K3wJnDKmu6ZfIiicD0E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389064; c=relaxed/simple; bh=JTBS0CwwRKFBvKNpf3wbmiqAAg6MPp22qwqUFuq1B2o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gIka9igDfsCDruuv0AvAAwogNkZQp+No0O+bNJpYx6O5w72XhFswZHjrqWbWWMmgBW6q9jG6Wk4H6rRH1nuvcJDgSzJraX+/tTuDhhJtPw0R/dSBAKS5GRmIjveULrDYFmmJ542H1HdDuSOYeZxjgBKfr1NQaXMgYlM83rENowg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=GVJTDJL2; arc=none smtp.client-ip=91.218.175.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="GVJTDJL2" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=JTBS0CwwRKFBvKNpf3wbmiqAAg6MPp22qwqUFuq1B2o=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389054; v=1; x=1789993854; b=GVJTDJL2915Flj6wf/6Lv7y0b7wdhpheBMW7zSIRfz+RR4pZ7mRf2y6yKUM++BlUeefTpgG2 su2M/c/JmtPjTqi8CS8jbAjB098jzXgkTUOIRRodhDER6D/qouHY6HJ0j4kGCNNQV7tkDcoR9yy Z8VZtq48Qu4sSi6FMzuqtleM= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id e87265f35c0b6f38; Mon, 14 Sep 2026 12:30:54 +0000 X-Mizu-Trace-ID: e87265f35c0b6f38 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif , Paul Walmsley , Palmer Dabbelt , Albert Ou Subject: [RESEND v7 05/29] riscv: mm: add PMD swap-exclusive helpers Date: Mon, 14 Sep 2026 05:27:55 -0700 Message-ID: <20260914122950.3283997-6-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and not just in a swap PTE. riscv encodes a swap PMD exactly like a swap PTE, so the new helpers wrap the PTE ones and reuse _PAGE_SWP_EXCLUSIVE. Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Signed-off-by: Usama Arif Reviewed-by: David Hildenbrand (Arm) --- arch/riscv/include/asm/pgtable.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index d48f90140841e..b644db16bda94 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -1219,6 +1219,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pt= e) } =20 #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pte_swp_exclusive(pmd_pte(pmd)); +} + +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))); +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))); +} + #define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) }) #define __swp_entry_to_pmd(swp) __pmd((swp).val) #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-178.mta0.migadu.com [91.218.175.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 848DA46D559 for ; Mon, 14 Sep 2026 12:31:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389069; cv=none; b=exLbaQlSVDrRxi/zskalh/Noy/tZltLP1+Mh2xTRQHWgaVTNHwteXwR9tVJv00Qe0NvQQy2oJ0rwofPZqFeSkP9KeMUQ6/W7zIxSWPNVW1AQxcZDb+2ZraXXAJUYkVH/foN7PpS2LMxg+xi7cZxwsxLgypB0j3fCng/UbdQXaDw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389069; c=relaxed/simple; bh=N+dRbJAiKiCxpTGr27ho63g/6orT5YaFsO8QYoCGDoE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L3rIurcvRpJnyR5vqYdaWzqDkYO6uriOauzfrhlN9tiGTIuN8nkGWzkMbaBd6Xh2PsJLVlwuDHwC7MOrZ5Cwoq21/zwVR+5BOKx7uJvjok7zQc0TUiM1F5guQr81VHtL7IrrF8mV/vMrNZfhXDXFf/Sf+5B8AcqStnPUE0LfMig= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=xsshYLyb; arc=none smtp.client-ip=91.218.175.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="xsshYLyb" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=N+dRbJAiKiCxpTGr27ho63g/6orT5YaFsO8QYoCGDoE=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389060; v=1; x=1789993860; b=xsshYLyb7yw87oCEK62zbNZcv//TVBLxEepVmEoW0YB7QmLIypzANdkvMPRjVIXWWWjxiYNK UivxatneTWdGrNfxO8AspBkSS6a3BqCVv9s+3yIXSja40/jjFR0hT91pUGe8Gok4Cu+3o6/wwNl VbAA7xil+5NUO23i9i8RP2Ak= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id ffd5ef34c1221106; Mon, 14 Sep 2026 12:31:00 +0000 X-Mizu-Trace-ID: ffd5ef34c1221106 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik Subject: [RESEND v7 06/29] s390: mm: add PMD swap-exclusive helpers Date: Mon, 14 Sep 2026 05:27:56 -0700 Message-ID: <20260914122950.3283997-7-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and not just in a swap PTE. s390 is the one architecture where a swap PMD is not a swap PTE in disguise: it is an RSTE with its own layout, converted to a fake PTE swap entry for the common code. Give it its own exclusive bit rather than borrowing the PTE-format macro. The two happen to have the same value, but that is a coincidence. Bit 52 was documented as unused; document what it is now. Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Signed-off-by: Usama Arif --- arch/s390/include/asm/pgtable.h | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtabl= e.h index 2d5c2ab06de98..0790a0884cfab 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -333,6 +333,7 @@ void setup_protection_map(void); /* Common bits in region and segment table entries, for swap entries */ #define _RST_ENTRY_COMM 0x0010 /* Common-Region/Segment, marks swap entry= */ #define _RST_ENTRY_INVALID 0x0020 /* invalid region/segment table entry */ +#define _RST_ENTRY_SWP_EXCLUSIVE 0x0800 /* SW exclusive swap bit, see mk_s= wap_rste() */ =20 #define _CRST_ENTRIES 2048 /* number of region/segment table entries */ #define _PAGE_ENTRIES 256 /* number of page table entries */ @@ -859,6 +860,28 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return clear_pte_bit(pte, __pgprot(_PAGE_SWP_EXCLUSIVE)); } =20 +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +/* + * A PMD swap entry is an RSTE, not a PTE, so it needs its own exclusive b= it + * rather than the PTE-format _PAGE_SWP_EXCLUSIVE. The two happen to have= the + * same value; see the RSTE swap layout above mk_swap_rste(). + */ +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_val(pmd) & _RST_ENTRY_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return clear_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE)); +} +#endif + static inline int pte_soft_dirty(pte_t pte) { return pte_val(pte) & _PAGE_SOFT_DIRTY; @@ -1900,15 +1923,16 @@ static inline swp_entry_t __swp_entry(unsigned long= type, unsigned long offset) * Bits 59 and 63 are used to indicate the swap entry. Bit 58 marks the rs= te * as invalid. * A swap entry is indicated by bit pattern (rste & 0x011) =3D=3D 0x010 - * | offset |Xtype |11TT|S0| + * | offset |Etype |11TT|S0| * |0000000000111111111122222222223333333333444444444455|555555|5566|66| * |0123456789012345678901234567890123456789012345678901|234567|8901|23| * * Bits 0-51 store the offset. + * Bit 52 (E) is used to remember PG_anon_exclusive + * (_RST_ENTRY_SWP_EXCLUSIVE), mirroring bit 52 of a swap pte. * Bits 53-57 store the type. * Bit 62 (S) is used for softdirty tracking. * Bits 60-61 (TT) indicate the table type: 0x01 for REGION3 and 0x00 for = SEGMENT. - * Bit 52 (X) is unused. */ =20 #define __SWP_OFFSET_MASK_RSTE ((1UL << 52) - 1) --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-206.mta0.migadu.com [91.218.175.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D55D46D2CF for ; Mon, 14 Sep 2026 12:31:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.206 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389078; cv=none; b=SH+HgmnRBGoK49Tf8myIwgt2r/7Sufw2HUYiB9Qgq5d7W3B8KTjFu9ayw1FCFeeRIkZ6W/TBL34zGNIu3mAEsU+Be0qcA0qnKAK5FJhYJndzM5FlX6H27zlbh2XqdpZHE4bhJnVjmBqRFfHj2sLW4yUqurfWdjlc9SzUP1FbUxU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389078; c=relaxed/simple; bh=m3zofWqXNwwubO6EsHPEBwbDlkXPNebnA/BZT3Zr5yk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=prkl/rQLV02PzIXoNwnGxnxHQxII/Ca55zqR0j3Feg23Mrhr3neTrkWhN5vuMcNMcUOVx9G3fe6PXUSDsRHq6iEOIxW3Dvih0QOO0DHiHv2PGpTgBGRpYkJfuhizBoA1Brk4Mtu/ourhu9DqV6ennHqCJc8+Ikfcpk02AO/eLMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=kNgl2V05; arc=none smtp.client-ip=91.218.175.206 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="kNgl2V05" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=m3zofWqXNwwubO6EsHPEBwbDlkXPNebnA/BZT3Zr5yk=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389067; v=1; x=1789993867; b=kNgl2V05Uht0hkwX4LIWWYrhCsbxBc5pvGo7DYJnj8pZrMHrFjI9PeUoIQkSB8W6cKqfdq9K Y6MVI4mOtKwmcxOtaaGk30lpJDKer0dX1L7eu9NVHllYdvUhJoJ4D8WZdO8mZf8kk+VwPvhRUsG aE7fYrQXpOfrVmiuQFabyLs0= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 10864e0dc6d3f851; Mon, 14 Sep 2026 12:31:06 +0000 X-Mizu-Trace-ID: 10864e0dc6d3f851 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org Subject: [RESEND v7 07/29] x86: mm: add PMD swap-exclusive helpers Date: Mon, 14 Sep 2026 05:27:57 -0700 Message-ID: <20260914122950.3283997-8-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and not just in a swap PTE. x86-64 encodes a swap PMD exactly like a swap PTE, so the new helpers reuse _PAGE_SWP_EXCLUSIVE, bit 3, which the swap-entry layout already reserves for PG_anon_exclusive. 32-bit x86 aliases that bit to _PAGE_PSE and does not select ARCH_HAS_PMD_SOFTLEAVES. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Usama Arif Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau (Meta) --- arch/x86/include/asm/pgtable.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index d551120a7c889..a2d1cd03cba23 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1525,6 +1525,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pt= e) return pte_clear_flags(pte, _PAGE_SWP_EXCLUSIVE); } =20 +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pmd_clear_flags(pmd, _PAGE_SWP_EXCLUSIVE); +} +#endif + #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty(pte_t pte) { --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-230.mta0.migadu.com [91.218.175.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B0A82C11DF for ; Mon, 14 Sep 2026 12:31:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.230 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389078; cv=none; b=QULVU76AoqRoabMTRT5LI3Mukdd2ShCr48fL8DpygFvaLWLbswC6IKGXwq9W9Zf+XykIpWpJSP/+qlOIE8c/ugyPEZOjeK7J5Zl9ngUzU+tSSt0UxXQv5y/NS9ZQL0uR/W6B9KbzDsVaYXmCoFwJZL56vI+ziLXHeAWo3ugwF94= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389078; c=relaxed/simple; bh=GA4gtvHV/+JkGd0QoK3boOJtdQKcFFleMRDfJWwQDhk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cuW3AAXxBn2eVmwZ5xlyVPTkAAAaIJOTkjevExUMS84/u7lFqAmkS3RhB4c8C8RlWsbH7z7u2oc/+glBmE90u3o8s8MdKc50HMoZ8apfO93G64KDs3xAKrw5gxImfkC0OM0vvgnsnwM80gJylE1IhH4dDwmav5ngR+Yigma454o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=sTuHQxrh; arc=none smtp.client-ip=91.218.175.230 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="sTuHQxrh" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=GA4gtvHV/+JkGd0QoK3boOJtdQKcFFleMRDfJWwQDhk=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389071; v=1; x=1789993871; b=sTuHQxrh2KqD5NSzX0l8OHiI2gEPht1xHOLsPeIABQ1em2uqi1hMKpo4WV9eLp6o3kd5++w2 dlFZfIGT5r6RfbeScqKhA4qM4BZ84CEP3Q6ToyxVXtARXGZsalJkneVXn3tIbaStPHpHLqABBrO VpTqi2wj17zqHb6Q/iCicQ+8= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id 8d6cb4f927396b23; Mon, 14 Sep 2026 12:31:10 +0000 X-Mizu-Trace-ID: 8d6cb4f927396b23 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 08/29] mm: recognize PMD swap entries in the softleaf layer Date: Mon, 14 Sep 2026 05:27:58 -0700 Message-ID: <20260914122950.3283997-9-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Reclaim splits a PMD-mapped anonymous THP into PTE-level swap entries before unmapping it, so an ordinary swap entry has never had to appear in a PMD. Later patches install one there instead, and the softleaf layer is where every consumer decodes non-present PMDs. Accept swap entries as valid PMD softleaves and add pmd_is_swap_entry(). A swap entry carries no PFN, so make pmd_softleaf_to_folio() warn and return NULL rather than interpret a swap offset as a page frame number. Unlike migration and device-private entries, a PMD swap entry can also carry the swap-exclusive marker, which softleaf_from_pmd() has to strip before decoding. Strip all three overlays unconditionally while we are here: each clear is a plain bit clear, so testing first only buys a branch. Signed-off-by: Usama Arif Acked-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau (Meta) --- include/linux/leafops.h | 40 ++++++++++++++++++++++++++++------------ include/linux/pgtable.h | 17 +++++++++++++++++ 2 files changed, 45 insertions(+), 12 deletions(-) diff --git a/include/linux/leafops.h b/include/linux/leafops.h index 7c13c58a5e218..ce176c78cefd4 100644 --- a/include/linux/leafops.h +++ b/include/linux/leafops.h @@ -98,10 +98,9 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd) if (pmd_present(pmd) || pmd_none(pmd)) return softleaf_mk_none(); =20 - if (pmd_swp_soft_dirty(pmd)) - pmd =3D pmd_swp_clear_soft_dirty(pmd); - if (pmd_swp_uffd(pmd)) - pmd =3D pmd_swp_clear_uffd(pmd); + pmd =3D pmd_swp_clear_soft_dirty(pmd); + pmd =3D pmd_swp_clear_uffd(pmd); + pmd =3D pmd_swp_clear_exclusive(pmd); arch_entry =3D __pmd_to_swp_entry(pmd); =20 /* Temporary until swp_entry_t eliminated. */ @@ -634,18 +633,29 @@ static inline bool pmd_is_migration_entry(pmd_t pmd) */ static inline bool softleaf_is_valid_pmd_entry(softleaf_t entry) { - /* Only device private, migration entries valid for PMD. */ return softleaf_is_device_private(entry) || - softleaf_is_migration(entry); + softleaf_is_migration(entry) || + softleaf_is_swap(entry); +} + +/** + * pmd_is_swap_entry() - Does this PMD entry encode an actual swap entry? + * @pmd: PMD entry. + * + * Returns: true if the PMD encodes a swap entry, otherwise false. + */ +static inline bool pmd_is_swap_entry(pmd_t pmd) +{ + return softleaf_is_swap(softleaf_from_pmd(pmd)); } =20 /** * pmd_is_valid_softleaf() - Is this PMD entry a valid softleaf entry? * @pmd: PMD entry. * - * PMD leaf entries are valid only if they are device private or migration - * entries. This function asserts that a PMD leaf entry is valid in this - * respect. + * PMD leaf entries are valid only if they are device private, migration, + * or swap entries. This function asserts that a PMD leaf entry is valid + * in this respect. * * Returns: true if the PMD entry is a valid leaf entry, otherwise false. */ @@ -660,10 +670,12 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd) * pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio. * @pmd: PMD entry. * - * The PMD entry is expected to be a valid PMD softleaf entry. + * The PMD entry is expected to be a valid PMD softleaf entry that referen= ces a + * PFN, that is a migration or device private entry. A PMD swap entry is a= valid + * softleaf entry but encodes swap slots rather than a PFN, so it has no f= olio. * - * Returns: the folio the softleaf entry references if this is a valid sof= tleaf - * entry, otherwise NULL. + * Returns: the folio the softleaf entry references, or NULL if the entry = is not + * a valid PMD softleaf entry or does not reference a PFN. */ static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd) { @@ -673,6 +685,10 @@ static inline struct folio *pmd_softleaf_to_folio(pmd_= t pmd) VM_WARN_ON_ONCE(true); return NULL; } + if (!softleaf_has_pfn(entry)) { + VM_WARN_ON_ONCE(true); + return NULL; + } return softleaf_to_folio(entry); } =20 diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e3c8ab96941c5..3f955f836abfe 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1917,6 +1917,23 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t p= md) } #endif =20 +#ifndef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pmd; +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pmd; +} +#endif + #ifndef __HAVE_PFNMAP_TRACKING /* * Interfaces that can be used by architecture code to keep track of --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-251.mta0.migadu.com [91.218.175.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DA05470457 for ; Mon, 14 Sep 2026 12:31:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.251 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389087; cv=none; b=ApDVkt2LqvDaRgWB/4dnRmwb8Wtf2ao60v4A79QNSN+WZCgjvW5r1WniG+mJlIyMbWkg+xHo25Otysx/dtuIimLz4dXnrR295COPGc3PdaW8gTXOeMLXvinuzgSW+N8A5RdZKo0+AYdJ0C7h7IVNFZyEs6MG92a5Qtr5KZcKOWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389087; c=relaxed/simple; bh=9WXF5SfkPGIzLBw1Y4CC+Epd6KLA6VeaEGUz3MVaYfc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kf3oFw+FQOW5P/mmSZga1iRSZ102W9wif9cHesPGylsO8DLQVMu+m4zXQ+DrtMNQISOivurtoVr6kOwOUPvseEtQgNyiY6KXiTL25Nh/b2y1KnB9b5uLPN4F6vFYK+1ZcOYyIJO8s/C2H1vH8LdwiybHZmeaNdUbEO/epzJWVBs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=I+bc+CII; arc=none smtp.client-ip=91.218.175.251 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="I+bc+CII" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=9WXF5SfkPGIzLBw1Y4CC+Epd6KLA6VeaEGUz3MVaYfc=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389078; v=1; x=1789993878; b=I+bc+CIIOmoLhHSDqxqajR21J1BpzQ1rEwFoAsC6RfZhPblFk9rXgPkOKWNs5naVzcDLjEa7 fsZHcGW8XVYAa9umoE197PRNT/sXuSJByFW/4XCfW/l2pNshpRs+Vlgq6T275i4kUHtUkU3LQqx guUK7gr5EcjQhk2dIs4zF3Go= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 60a161468f8d634e; Mon, 14 Sep 2026 12:31:17 +0000 X-Mizu-Trace-ID: 60a161468f8d634e X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 09/29] mm/debug_vm_pgtable: test PMD swap-exclusive helpers Date: Mon, 14 Sep 2026 05:27:59 -0700 Message-ID: <20260914122950.3283997-10-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" An architecture that picked a PMD exclusive bit overlapping the swap type or offset field would otherwise only be caught by data corruption at runtime. Mirror pte_swap_exclusive_tests() at PMD level. Suggested-by: David Hildenbrand (Arm) Signed-off-by: Usama Arif Acked-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau (Meta) --- mm/debug_vm_pgtable.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index 2875fd22d7bb0..863111c6d4eb3 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -802,6 +802,45 @@ static void __init pte_swap_exclusive_tests(struct pgt= able_debug_args *args) WARN_ON(memcmp(&entry, &softleaf, sizeof(entry))); } =20 +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static void __init pmd_swap_exclusive_tests(struct pgtable_debug_args *arg= s) +{ + swp_entry_t entry; + softleaf_t softleaf; + pmd_t pmd; + + if (!has_transparent_hugepage()) + return; + + pr_debug("Validating PMD swap exclusive\n"); + entry =3D args->swp_entry; + + pmd =3D softleaf_to_pmd(entry); + softleaf =3D softleaf_from_pmd(pmd); + + WARN_ON(pmd_swp_exclusive(pmd)); + WARN_ON(!softleaf_is_swap(softleaf)); + WARN_ON(memcmp(&entry, &softleaf, sizeof(entry))); + + pmd =3D pmd_swp_mkexclusive(pmd); + softleaf =3D softleaf_from_pmd(pmd); + + WARN_ON(!pmd_swp_exclusive(pmd)); + WARN_ON(!softleaf_is_swap(softleaf)); + WARN_ON(pmd_swp_soft_dirty(pmd)); + WARN_ON(memcmp(&entry, &softleaf, sizeof(entry))); + + pmd =3D pmd_swp_clear_exclusive(pmd); + softleaf =3D softleaf_from_pmd(pmd); + + WARN_ON(pmd_swp_exclusive(pmd)); + WARN_ON(!softleaf_is_swap(softleaf)); + WARN_ON(memcmp(&entry, &softleaf, sizeof(entry))); +} +#else /* !CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ +static void __init pmd_swap_exclusive_tests(struct pgtable_debug_args *arg= s) { } +#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ + static void __init pte_swap_tests(struct pgtable_debug_args *args) { swp_entry_t arch_entry; @@ -1322,6 +1361,7 @@ static int __init debug_vm_pgtable(void) pmd_leaf_soft_dirty_tests(&args); =20 pte_swap_exclusive_tests(&args); + pmd_swap_exclusive_tests(&args); =20 pte_swap_tests(&args); pmd_softleaf_tests(&args); --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-19.mta0.migadu.com [91.218.175.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DE853FA5F0 for ; Mon, 14 Sep 2026 12:31:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389093; cv=none; b=CXoFTnlkpEZ5EtqGPlgbLLOqMD97mt+hviyC9Iuq+q9KbuLl4Eks0aENENlLs7R9AAlHzhoqihSd1wwLya3NfgBlnrN9QHPQXrXSIID4c/xVGA5tEKMbyhPi7TvhlQ1+QNmouWeFFGYwl0RcGQ7/Kry98LH9jsANmqrAbZuc6WE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389093; c=relaxed/simple; bh=3XNEzeVU1O8SWrgui6NV2H1NKonCjPLxXmf/sJyhnNo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ct2mDG7MCqX0g8IgCvn/Y9SCtFtLlMFEleD52IhVMoV/0zT29XB2QOWb8hSFsMeFmu/YrCC65TBAeHdnIya1gI6E8HTFbCPy9bhi4bFlqoY8j4hBKbHtfRUr191Z9DLvo4SAmesrkNy5QLLBYkl21bgNf9e3gd/GIySANn6KKKM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=On/2qC4F; arc=none smtp.client-ip=91.218.175.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="On/2qC4F" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=3XNEzeVU1O8SWrgui6NV2H1NKonCjPLxXmf/sJyhnNo=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389084; v=1; x=1789993884; b=On/2qC4FRxiKg8BKss9FvrqxQfRXdEZVS+/Yd2Y8ReSiHpF5mbnzY43jFaDf7VlbtDspCMoq MJkZgU23BsVbPL+LgwpfcwJifc2DwirapYwEUf0UbdjcV21p7XBhrp/plSXIwjx3F+fyvGYRQ4y O/HXT3+7tXl0dcK2Fwl71B18= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id 178b94f0694f8731; Mon, 14 Sep 2026 12:31:23 +0000 X-Mizu-Trace-ID: 178b94f0694f8731 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 10/29] mm: make PMD migration-entry splitting explicit Date: Mon, 14 Sep 2026 05:28:00 -0700 Message-ID: <20260914122950.3283997-11-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __split_huge_pmd() and friends take a "freeze" boolean that every caller has to pass and almost every caller passes as false. The name says nothing about what it selects, and the one thing it does select - PTE migration entries instead of PTE mappings - is only ever wanted by the rmap migration path. Rename it to use_migration_entries, keep it private to mm/huge_memory.c, and add split_pmd_to_migration_entries() for try_to_migrate_one(), the only caller that wants it. migrate_vma_split_unmapped_folio() also passed freeze=3Dtrue, but only ever runs on a PMD that is already a migration entry, which the generic helper expands into PTE migration entries either way. Its folio_get() only existed to balance the put_page() that freeze=3Dtrue performs, so both go. No functional change intended. Suggested-by: David Hildenbrand (Arm) Signed-off-by: Usama Arif Reviewed-by: Kiryl Shutsemau (Meta) --- include/linux/huge_mm.h | 22 ++++++++------- mm/huge_memory.c | 60 ++++++++++++++++++++++++----------------- mm/memory.c | 4 +-- mm/migrate_device.c | 7 +---- mm/mprotect.c | 2 +- mm/rmap.c | 7 +++-- 6 files changed, 55 insertions(+), 47 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 8ca0fa3be2acb..64b6a2eea899d 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -430,7 +430,7 @@ int folio_memcg_alloc_deferred(struct folio *folio); void deferred_split_folio(struct folio *folio, bool partially_mapped); =20 void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, bool freeze); + unsigned long address); =20 /** * pmd_is_huge() - Is this PMD either a huge PMD entry or a software leaf = entry? @@ -462,12 +462,10 @@ static inline bool pmd_is_huge(pmd_t pmd) do { \ pmd_t *____pmd =3D (__pmd); \ if (pmd_is_huge(*____pmd)) \ - __split_huge_pmd(__vma, __pmd, __address, \ - false); \ + __split_huge_pmd(__vma, __pmd, __address); \ } while (0) =20 -void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long addr= ess, - bool freeze); +void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long addr= ess); =20 void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); @@ -590,7 +588,9 @@ static inline bool thp_migration_supported(void) } =20 void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addre= ss, - pmd_t *pmd, bool freeze); + pmd_t *pmd); +void split_pmd_to_migration_entries(struct vm_area_struct *vma, + unsigned long address, pmd_t *pmd); bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, struct folio *folio); void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd, @@ -690,12 +690,14 @@ static inline void deferred_split_folio(struct folio = *folio, bool partially_mapp do { } while (0) =20 static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, bool freeze) {} + unsigned long address) {} static inline void split_huge_pmd_address(struct vm_area_struct *vma, - unsigned long address, bool freeze) {} + unsigned long address) {} static inline void split_huge_pmd_locked(struct vm_area_struct *vma, - unsigned long address, pmd_t *pmd, - bool freeze) {} + unsigned long address, pmd_t *pmd) {} +static inline void +split_pmd_to_migration_entries(struct vm_area_struct *vma, + unsigned long address, pmd_t *pmd) {} =20 static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ee8d46827ffdc..873887aed0bc2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2033,7 +2033,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm= _struct *src_mm, pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); - __split_huge_pmd(src_vma, src_pmd, addr, false); + __split_huge_pmd(src_vma, src_pmd, addr); return -EAGAIN; } add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); @@ -2257,7 +2257,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) folio_unlock(folio); spin_unlock(vmf->ptl); fallback: - __split_huge_pmd(vma, vmf->pmd, vmf->address, false); + __split_huge_pmd(vma, vmf->pmd, vmf->address); return VM_FAULT_FALLBACK; } =20 @@ -3190,7 +3190,7 @@ static void __split_huge_zero_page_pmd(struct vm_area= _struct *vma, } =20 static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long haddr, bool freeze) + unsigned long haddr, bool use_migration_entries) { struct mm_struct *mm =3D vma->vm_mm; struct folio *folio; @@ -3291,10 +3291,10 @@ static void __split_huge_pmd_locked(struct vm_area_= struct *vma, pmd_t *pmd, * folios w.r.t anon exclusive handling. See the comments for * folio handling and anon_exclusive below. */ - if (freeze && anon_exclusive && + if (use_migration_entries && anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) - freeze =3D false; - if (!freeze) { + use_migration_entries =3D false; + if (!use_migration_entries) { rmap_t rmap_flags =3D RMAP_NONE; =20 folio_ref_add(folio, HPAGE_PMD_NR - 1); @@ -3344,11 +3344,11 @@ static void __split_huge_pmd_locked(struct vm_area_= struct *vma, pmd_t *pmd, VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); =20 /* - * Without "freeze", we'll simply split the PMD, propagating the - * PageAnonExclusive() flag for each PTE by setting it for + * Without migration entries, we'll simply split the PMD and + * propagate the PageAnonExclusive() flag for each PTE by setting it for * each subpage -- no need to (temporarily) clear. * - * With "freeze" we want to replace mapped pages by + * With migration entries we want to replace mapped pages by * migration entries right away. This is only possible if we * managed to clear PageAnonExclusive() -- see * set_pmd_migration_entry(). @@ -3359,10 +3359,10 @@ static void __split_huge_pmd_locked(struct vm_area_= struct *vma, pmd_t *pmd, * See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive =3D PageAnonExclusive(page); - if (freeze && anon_exclusive && + if (use_migration_entries && anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) - freeze =3D false; - if (!freeze) { + use_migration_entries =3D false; + if (!use_migration_entries) { rmap_t rmap_flags =3D RMAP_NONE; =20 folio_ref_add(folio, HPAGE_PMD_NR - 1); @@ -3387,7 +3387,7 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, * Note that NUMA hinting access restrictions are not transferred to * avoid any possibility of altering permissions across VMAs. */ - if (freeze || pmd_is_migration_entry(old_pmd)) { + if (use_migration_entries || pmd_is_migration_entry(old_pmd)) { pte_t entry; swp_entry_t swp_entry; =20 @@ -3420,8 +3420,8 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, for (i =3D 0, addr =3D haddr; i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE= ) { /* * anon_exclusive was already propagated to the relevant - * pages corresponding to the pte entries when freeze - * is false. + * pages corresponding to the pte entries when + * use_migration_entries is false. */ if (write) swp_entry =3D make_writable_device_private_entry( @@ -3469,7 +3469,7 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, =20 if (!pmd_is_migration_entry(*pmd)) folio_remove_rmap_pmd(folio, page, vma); - if (freeze) + if (use_migration_entries) put_page(page); =20 smp_wmb(); /* make pte visible before pmd */ @@ -3477,15 +3477,28 @@ static void __split_huge_pmd_locked(struct vm_area_= struct *vma, pmd_t *pmd, } =20 void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addre= ss, - pmd_t *pmd, bool freeze) + pmd_t *pmd) { VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); if (pmd_trans_huge(*pmd) || pmd_is_valid_softleaf(*pmd)) - __split_huge_pmd_locked(vma, pmd, address, freeze); + __split_huge_pmd_locked(vma, pmd, address, false); +} + +/* + * Split a present PMD into PTE migration entries, for the rmap migration + * walker. Like split_huge_pmd_locked(), the caller must hold the PMD loc= k and + * must already be inside an mmu_notifier invalidate range. + */ +void split_pmd_to_migration_entries(struct vm_area_struct *vma, + unsigned long address, pmd_t *pmd) +{ + VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); + if (pmd_trans_huge(*pmd) || pmd_is_valid_softleaf(*pmd)) + __split_huge_pmd_locked(vma, pmd, address, true); } =20 void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, bool freeze) + unsigned long address) { spinlock_t *ptl; struct mmu_notifier_range range; @@ -3495,20 +3508,19 @@ void __split_huge_pmd(struct vm_area_struct *vma, p= md_t *pmd, (address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); ptl =3D pmd_lock(vma->vm_mm, pmd); - split_huge_pmd_locked(vma, range.start, pmd, freeze); + split_huge_pmd_locked(vma, range.start, pmd); spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } =20 -void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long addr= ess, - bool freeze) +void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long addr= ess) { pmd_t *pmd =3D mm_find_pmd(vma->vm_mm, address); =20 if (!pmd) return; =20 - __split_huge_pmd(vma, pmd, address, freeze); + __split_huge_pmd(vma, pmd, address); } =20 static inline void split_huge_pmd_if_needed(struct vm_area_struct *vma, un= signed long address) @@ -3520,7 +3532,7 @@ static inline void split_huge_pmd_if_needed(struct vm= _area_struct *vma, unsigned if (!IS_ALIGNED(address, HPAGE_PMD_SIZE) && range_in_vma(vma, ALIGN_DOWN(address, HPAGE_PMD_SIZE), ALIGN(address, HPAGE_PMD_SIZE))) - split_huge_pmd_address(vma, address, false); + split_huge_pmd_address(vma, address); } =20 void vma_adjust_trans_huge(struct vm_area_struct *vma, diff --git a/mm/memory.c b/mm/memory.c index 926276d419202..477d7e359b447 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2096,7 +2096,7 @@ static inline unsigned long zap_pmd_range(struct mmu_= gather *tlb, next =3D pmd_addr_end(addr, end); if (pmd_is_huge(*pmd)) { if (next - addr !=3D HPAGE_PMD_SIZE) - __split_huge_pmd(vma, pmd, addr, false); + __split_huge_pmd(vma, pmd, addr); else if (zap_huge_pmd(tlb, vma, pmd, addr)) { addr =3D next; continue; @@ -6382,7 +6382,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault = *vmf) =20 split: /* COW or write-notify handled on pte level: split pmd. */ - __split_huge_pmd(vma, vmf->pmd, vmf->address, false); + __split_huge_pmd(vma, vmf->pmd, vmf->address); =20 return VM_FAULT_FALLBACK; } diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 0c437004329d9..4a0b61d50d222 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -918,12 +918,7 @@ static int migrate_vma_split_unmapped_folio(struct mig= rate_vma *migrate, unsigned long flags; int ret =3D 0; =20 - /* - * take a reference, since split_huge_pmd_address() with freeze =3D true - * drops a reference at the end. - */ - folio_get(folio); - split_huge_pmd_address(migrate->vma, addr, true); + split_huge_pmd_address(migrate->vma, addr); ret =3D folio_split_unmapped(folio, 0); if (ret) return ret; diff --git a/mm/mprotect.c b/mm/mprotect.c index 2888ee638d872..ee33bbb421008 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -530,7 +530,7 @@ static inline long change_pmd_range(struct mmu_gather *= tlb, if (pmd_is_huge(_pmd)) { if ((next - addr !=3D HPAGE_PMD_SIZE) || pgtable_split_needed(vma, cp_flags)) { - __split_huge_pmd(vma, pmd, addr, false); + __split_huge_pmd(vma, pmd, addr); /* * For file-backed, the pmd could have been * cleared; make sure pmd populated if diff --git a/mm/rmap.c b/mm/rmap.c index 5332c52909be1..feb751e29b992 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2290,7 +2290,7 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, * restart so we can process the PTE-mapped THP. */ split_huge_pmd_locked(vma, pvmw.address, - pvmw.pmd, false); + pvmw.pmd); flags &=3D ~TTU_SPLIT_HUGE_PMD; page_vma_mapped_walk_restart(&pvmw); continue; @@ -2515,13 +2515,12 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, =20 if (flags & TTU_SPLIT_HUGE_PMD) { /* - * split_huge_pmd_locked() might leave the + * split_pmd_to_migration_entries() might leave the * folio mapped through PTEs. Retry the walk * so we can detect this scenario and properly * abort the walk. */ - split_huge_pmd_locked(vma, pvmw.address, - pvmw.pmd, true); + split_pmd_to_migration_entries(vma, pvmw.address, pvmw.pmd); flags &=3D ~TTU_SPLIT_HUGE_PMD; page_vma_mapped_walk_restart(&pvmw); continue; --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-41.mta0.migadu.com [91.218.175.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EECFE45C71C for ; Mon, 14 Sep 2026 12:31:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389097; cv=none; b=POZprCsTT/EKAiNSPPp5A0FfiHAvaJzQQB3PY3ts21ZSjlx1HxDxBsQTVAM0w9bU2wc6QXEJcTui9c8tcE40HRmkIhkT3WM3YibSVJRwI1S7IKzL57gw+NJcri0FK5z9+OnXUG5a8OPd5dKfUvU3dsDSbYEjYR5ng1V+rmwID7E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389097; c=relaxed/simple; bh=Q7Vo+oMZWSnoi9kpct14wrtvWt95QszpUVVNkFyr/6c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IKYbpV87KPAX0huE5VDBc38V/ozFi3htz95/d3h5utonimi64eUocV7YHoOcXSSkAuAmx6TXb8C/7FywGvxBUSCE/K8nW8LpIdyc3Aup1sAWsEnW99wvB8mXE5yyUaol1nWTMxjhn91t/gqu+Eu2vFyPsDB3uUUUU40QFX6z3nc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Z5GhJrBL; arc=none smtp.client-ip=91.218.175.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Z5GhJrBL" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=Q7Vo+oMZWSnoi9kpct14wrtvWt95QszpUVVNkFyr/6c=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389086; v=1; x=1789993886; b=Z5GhJrBLiU2mFJd2D90uchgd529Y/fai1OvFUV9kqOBTIdFvKQ0Q6ePS/l9kPiszDgm6pNsb 8DvDCL7/SaRToSr+wmIf4qNysI19355EWN4HBhqZiJyPz/YplWoUDECQmwCxOWMporw7/UMe3cI d/6uRIU5A+MsstqisGYQ1IjY= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 56156af8bd9f1fa3; Mon, 14 Sep 2026 12:31:26 +0000 X-Mizu-Trace-ID: 56156af8bd9f1fa3 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 11/29] mm: split PMD swap entries into PTE swap entries Date: Mon, 14 Sep 2026 05:28:01 -0700 Message-ID: <20260914122950.3283997-12-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Once a PMD can hold a swap entry, everything that splits a PMD - mprotect() or munmap() over part of the range, MADV_FREE, a pagewalk with no PMD handler - has to be able to split that entry too, or the callers that rely on split_huge_pmd() to hand them a PTE table would find the PMD unchanged. No reference counting is needed: a swap entry pins no folio, and swap_map is already one per slot, so the PTEs simply take over what the PMD held. The migration-only entry point cannot reach the new branch, because page_vma_mapped_walk() never hands back a swap PMD for the folio being migrated. Warn if that ever changes, and force the regular split anyway, since the branch leaves folio and page uninitialised. Test the pre-split old_pmd rather than re-reading *pmd in the trailing folio_remove_rmap_pmd() gate, so every entry-type test in the function interrogates the same snapshot. That part is cosmetic: pmdp_invalidate() leaves the PMD present as far as software is concerned. Signed-off-by: Usama Arif --- mm/huge_memory.c | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 873887aed0bc2..0e347a545588c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3304,6 +3304,21 @@ static void __split_huge_pmd_locked(struct vm_area_s= truct *vma, pmd_t *pmd, folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, vma, haddr, rmap_flags); } + } else if (pmd_is_swap_entry(*pmd)) { + /* + * A PMD swap entry has no page, so it cannot be turned into + * PTE migration entries. page_vma_mapped_walk() never hands + * one back for the folio being migrated, so this should not + * happen; warn, but also force the regular split so that a + * broken invariant cannot make the code below dereference the + * uninitialised folio and page. + */ + VM_WARN_ON_ONCE(use_migration_entries); + use_migration_entries =3D false; + old_pmd =3D *pmd; + soft_dirty =3D pmd_swp_soft_dirty(old_pmd); + uffd_wp =3D pmd_swp_uffd(old_pmd); + anon_exclusive =3D pmd_swp_exclusive(old_pmd); } else { /* * Up to this point the pmd is present and huge and userland has @@ -3440,6 +3455,25 @@ static void __split_huge_pmd_locked(struct vm_area_s= truct *vma, pmd_t *pmd, VM_WARN_ON(!pte_none(ptep_get(pte + i))); set_pte_at(mm, addr, pte + i, entry); } + } else if (pmd_is_swap_entry(old_pmd)) { + const softleaf_t old_entry =3D softleaf_from_pmd(old_pmd); + pte_t pte_swp_entry; + swp_entry_t entry; + + for (i =3D 0, addr =3D haddr; i < HPAGE_PMD_NR; + i++, addr +=3D PAGE_SIZE) { + entry =3D swp_entry(swp_type(old_entry), + swp_offset(old_entry) + i); + pte_swp_entry =3D swp_entry_to_pte(entry); + if (soft_dirty) + pte_swp_entry =3D pte_swp_mksoft_dirty(pte_swp_entry); + if (uffd_wp) + pte_swp_entry =3D pte_swp_mkuffd(pte_swp_entry); + if (anon_exclusive) + pte_swp_entry =3D pte_swp_mkexclusive(pte_swp_entry); + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + set_pte_at(mm, addr, pte + i, pte_swp_entry); + } } else { pte_t entry; =20 @@ -3467,7 +3501,7 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, } pte_unmap(pte); =20 - if (!pmd_is_migration_entry(*pmd)) + if (!pmd_is_migration_entry(old_pmd) && !pmd_is_swap_entry(old_pmd)) folio_remove_rmap_pmd(folio, page, vma); if (use_migration_entries) put_page(page); --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-57.mta0.migadu.com [91.218.175.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 607934718CE for ; Mon, 14 Sep 2026 12:31:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.57 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389099; cv=none; b=tiu3M9Etgr4SUJPWWmvv8/l7pSrzKNE1acL8eyF7UqGJer9MfkTrpC5YCSopjqosCn38N8Fufc+rKMpULm4l8LsdrDjeGJVQuPvedhO0QgEV9aZuZcW7r62Gjx379KXxjJ10q04VvzmxILRFSG7IwWVBsTeSn8lNAeOZbCIFMVY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389099; c=relaxed/simple; bh=yJ5rM0Ieb2+vVwyVMyaZBbRDWPGmbZFWmsl3JUm02Do=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O+CamxoL7TOsVZeN6zWov5L6QQALDcdHAugiUGqx6AvPy7ZGti+cqE/LEuPMTW7m/LFjYqMV89kuOOFQ7DXv1uHTuAazS5MsoOIUCvFPUvzJdHIpjIJYL353uUZrtXb6yCIP6J8YhOGEtBpbmcJlgjjf4kofIhRo+DOvvL/AjKs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Xxiozo3b; arc=none smtp.client-ip=91.218.175.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Xxiozo3b" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=yJ5rM0Ieb2+vVwyVMyaZBbRDWPGmbZFWmsl3JUm02Do=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389088; v=1; x=1789993888; b=Xxiozo3bVuBO5D8EXJEc7p4eHJE6tbf/RUN4MvMfV0wGDdnOBop90KknMog8EzV3dN9vfQXf co4sw0d/prByyUzE0DQjRYF1lwnmNUOG+zUwhxesyHqECQh3apGN03W0JI6Yjj1HY5fitlOQPaF ydw+Yd4vYayJfNUoO3JC2HHY= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id acc2a20a9373363e; Mon, 14 Sep 2026 12:31:28 +0000 X-Mizu-Trace-ID: acc2a20a9373363e X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 12/29] mm: handle PMD swap entries in fork path Date: Mon, 14 Sep 2026 05:28:02 -0700 Message-ID: <20260914122950.3283997-13-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" copy_huge_pmd() only knows about migration and device-private PMDs, so a PMD swap entry would fall through to the present-PMD path and fork() would duplicate it without taking a reference on the slots it points at. Copy it the way copy_nonpresent_pte() copies a PTE swap entry: duplicate the swap references, clear the exclusive marker on the source, put the destination mm on mmlist, and account the child's slots to MM_SWAPENTS. Duplicating HPAGE_PMD_NR slots one at a time would be wasteful, so give swap_dup_entry_direct() an nr argument and rename it accordingly. Unlike the put side it hands nr straight to the per-cluster helper, so the range has to sit inside one cluster - which it does, since SWAPFILE_CLUSTER =3D= =3D HPAGE_PMD_NR under CONFIG_THP_SWAP and a PMD-order folio's slots are only ever allocated at a cluster head. Reject a crossing range with -EINVAL so a future caller cannot walk off the end of the swap table. The GFP_ATOMIC extend-table allocation inside the dup can fail; copy_huge_pmd() then drops both PMD locks and retries once with GFP_KERNEL. Bound it to one retry, because swap_retry_table_alloc() also returns 0 when it decides the table is not needed. Normalise any remaining failure to -ENOMEM: copy_pmd_range() treats every other error as "not a huge PMD" and would then reach pmd_none_or_clear_bad(), clearing the source PMD and leaking its swap slots. Signed-off-by: Usama Arif --- include/linux/swap.h | 4 +-- mm/huge_memory.c | 65 +++++++++++++++++++++++++++++++++++++++----- mm/memory.c | 4 +-- mm/swap.h | 5 ++-- mm/swapfile.c | 58 +++++++++++++++++++++++++++++---------- 5 files changed, 109 insertions(+), 27 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 43155e122b5c3..0c3006ece1ca5 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -414,7 +414,7 @@ sector_t swap_folio_sector(struct folio *folio); * All entries must be allocated by folio_alloc_swap(). And they must have * a swap count > 1. See comments of folio_*_swap helpers for more info. */ -int swap_dup_entry_direct(swp_entry_t entry); +int swap_dup_entries_direct(swp_entry_t entry, int nr); void swap_put_entries_direct(swp_entry_t entry, int nr); =20 /* @@ -458,7 +458,7 @@ static inline void free_swap_cache(struct folio *folio) { } =20 -static inline int swap_dup_entry_direct(swp_entry_t ent) +static inline int swap_dup_entries_direct(swp_entry_t ent, int nr) { return 0; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0e347a545588c..6dfe8ef6dd371 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1894,7 +1894,7 @@ bool touch_pmd(struct vm_area_struct *vma, unsigned l= ong addr, return false; } =20 -static void copy_huge_non_present_pmd( +static int copy_huge_non_present_pmd( struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, @@ -1940,14 +1940,40 @@ static void copy_huge_non_present_pmd( */ folio_try_dup_anon_rmap_pmd(src_folio, &src_folio->page, dst_vma, src_vma); + } else if (softleaf_is_swap(entry)) { + int err; + + /* + * PMD swap entry: duplicate swap references and clear + * exclusive on source, matching copy_nonpresent_pte(). + * + * A PMD swap entry only exists under CONFIG_THP_SWAP, where + * SWAPFILE_CLUSTER =3D=3D HPAGE_PMD_NR, and it is cluster aligned, + * so these HPAGE_PMD_NR slots are exactly one cluster - which + * is what swap_dup_entries_direct() requires. + */ + err =3D swap_dup_entries_direct(entry, HPAGE_PMD_NR); + if (err < 0) + return err; + + mm_prepare_for_swap_entries(dst_mm); + + if (pmd_swp_exclusive(pmd)) { + pmd =3D pmd_swp_clear_exclusive(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } } =20 - add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + if (softleaf_is_swap(entry)) + add_mm_counter(dst_mm, MM_SWAPENTS, HPAGE_PMD_NR); + else + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); if (!userfaultfd_protected(dst_vma)) pmd =3D pmd_swp_clear_uffd(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); + return 0; } =20 int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, @@ -1957,6 +1983,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm= _struct *src_mm, spinlock_t *dst_ptl, *src_ptl; struct page *src_page; struct folio *src_folio; + bool retried =3D false; pmd_t pmd; pgtable_t pgtable =3D NULL; int ret =3D -ENOMEM; @@ -1988,6 +2015,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm= _struct *src_mm, if (unlikely(!pgtable)) goto out; =20 +retry: dst_ptl =3D pmd_lock(dst_mm, dst_pmd); src_ptl =3D pmd_lockptr(src_mm, src_pmd); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -1995,11 +2023,34 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct = mm_struct *src_mm, ret =3D -EAGAIN; pmd =3D *src_pmd; =20 - if (unlikely(thp_migration_supported() && - pmd_is_valid_softleaf(pmd))) { - copy_huge_non_present_pmd(dst_mm, src_mm, dst_pmd, src_pmd, addr, - dst_vma, src_vma, pmd, pgtable); - ret =3D 0; + if (unlikely(pmd_is_valid_softleaf(pmd))) { + ret =3D copy_huge_non_present_pmd(dst_mm, src_mm, dst_pmd, src_pmd, + addr, dst_vma, src_vma, pmd, + pgtable); + if (ret) { + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + /* + * For PMD swap entries -ENOMEM means the per-cluster + * swap-extend table couldn't be GFP_ATOMIC-allocated. + * Try the GFP_KERNEL fallback once before giving up. + * swap_retry_table_alloc() also returns 0 when it + * decides the table is not needed after all, so bound + * this to a single retry rather than looping on it. + */ + if (ret =3D=3D -ENOMEM && !retried) { + softleaf_t entry =3D softleaf_from_pmd(pmd); + + retried =3D true; + if (softleaf_is_swap(entry) && + !swap_retry_table_alloc(entry, HPAGE_PMD_NR, + GFP_KERNEL)) + goto retry; + } + pte_free(dst_mm, pgtable); + ret =3D -ENOMEM; + goto out; + } goto out_unlock; } =20 diff --git a/mm/memory.c b/mm/memory.c index 477d7e359b447..84e1e1c22bffa 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -979,7 +979,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm= _struct *src_mm, struct page *page; =20 if (likely(softleaf_is_swap(entry))) { - if (swap_dup_entry_direct(entry) < 0) + if (swap_dup_entries_direct(entry, 1) < 0) return -EIO; =20 mm_prepare_for_swap_entries(dst_mm); @@ -1394,7 +1394,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, =20 if (ret =3D=3D -EIO) { VM_WARN_ON_ONCE(!entry.val); - if (swap_retry_table_alloc(entry, GFP_KERNEL) < 0) { + if (swap_retry_table_alloc(entry, 1, GFP_KERNEL) < 0) { ret =3D -ENOMEM; goto out; } diff --git a/mm/swap.h b/mm/swap.h index b3b54c28929a1..2321c9a2c2c58 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -222,7 +222,7 @@ static inline void swap_cluster_unlock_irq(struct swap_= cluster_info *ci) spin_unlock_irq(&ci->lock); } =20 -extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp); +int swap_retry_table_alloc(swp_entry_t entry, unsigned int nr, gfp_t gfp); =20 /* * Below are the core routines for doing swap for a folio. @@ -428,7 +428,8 @@ static inline int swap_writeout(struct swap_io_ctx *ctx= , struct folio *folio) return 0; } =20 -static inline int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp) +static inline int swap_retry_table_alloc(swp_entry_t entry, unsigned int n= r, + gfp_t gfp) { return -EINVAL; } diff --git a/mm/swapfile.c b/mm/swapfile.c index 280dd906eb187..27ae3964a158e 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1468,11 +1468,16 @@ static bool swap_sync_discard(void) =20 static int swap_extend_table_alloc(struct swap_info_struct *si, struct swap_cluster_info *ci, - unsigned int ci_off, gfp_t gfp) + unsigned int ci_off, unsigned int nr, + gfp_t gfp) { int count; + unsigned int i; void *table; =20 + /* The range must not run past the end of @ci's swap table. */ + VM_WARN_ON_ONCE(ci_off + nr > SWAPFILE_CLUSTER); + table =3D kzalloc(sizeof(ci->extend_table[0]) * SWAPFILE_CLUSTER, gfp); if (!table) return -ENOMEM; @@ -1486,15 +1491,21 @@ static int swap_extend_table_alloc(struct swap_info= _struct *si, */ if (!cluster_table_is_alloced(ci)) goto out_free; - count =3D swp_tb_get_count(__swap_table_get(ci, ci_off)); - if (count < (SWP_TB_COUNT_MAX - 1)) - goto out_free; if (ci->extend_table) goto out_free; - - ci->extend_table =3D table; - spin_unlock(&ci->lock); - return 0; + /* + * The caller may not know which slot in [ci_off, ci_off + nr) hit + * SWP_TB_COUNT_MAX - 1. Confirm at least one slot in the range still + * needs the extend table before committing the allocation. + */ + for (i =3D 0; i < nr; i++) { + count =3D swp_tb_get_count(__swap_table_get(ci, ci_off + i)); + if (count >=3D (SWP_TB_COUNT_MAX - 1)) { + ci->extend_table =3D table; + spin_unlock(&ci->lock); + return 0; + } + } =20 out_free: spin_unlock(&ci->lock); @@ -1502,7 +1513,7 @@ static int swap_extend_table_alloc(struct swap_info_s= truct *si, return 0; } =20 -int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp) +int swap_retry_table_alloc(swp_entry_t entry, unsigned int nr, gfp_t gfp) { int ret; struct swap_info_struct *si; @@ -1514,7 +1525,8 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t g= fp) return 0; =20 ci =3D __swap_offset_to_cluster(si, offset); - ret =3D swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp); + ret =3D swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), nr, + gfp); =20 put_swap_device(si); return ret; @@ -1690,6 +1702,9 @@ static int __swap_cluster_dup_entry(struct swap_clust= er_info *ci, * @offset: start offset of slots. * @nr: number of slots. * + * The range [offset, offset + nr) must not cross a cluster boundary; the + * caller is responsible for splitting a range that can. + * * Context: The specified slots must be pinned by existing swap count or s= wap * cache reference, so they won't be released until this helper returns. * Return: 0 on success. -ENOMEM if the swap count maxed out (SWP_TB_COUNT= _MAX) @@ -1704,6 +1719,7 @@ static int swap_dup_entries_cluster(struct swap_info_= struct *si, =20 ci_start =3D offset % SWAPFILE_CLUSTER; ci_end =3D ci_start + nr; + VM_WARN_ON_ONCE(ci_end > SWAPFILE_CLUSTER); ci_off =3D ci_start; ci =3D swap_cluster_lock(si, offset); restart: @@ -1712,7 +1728,8 @@ static int swap_dup_entries_cluster(struct swap_info_= struct *si, if (unlikely(err)) { if (err =3D=3D -ENOMEM) { spin_unlock(&ci->lock); - err =3D swap_extend_table_alloc(si, ci, ci_off, GFP_ATOMIC); + err =3D swap_extend_table_alloc(si, ci, ci_off, 1, + GFP_ATOMIC); spin_lock(&ci->lock); if (!err) goto restart; @@ -1723,6 +1740,7 @@ static int swap_dup_entries_cluster(struct swap_info_= struct *si, swap_cluster_unlock(ci); return 0; failed: + /* The caller's page-table or swap-cache reference pins every slot. */ while (ci_off-- > ci_start) __swap_cluster_put_entry(ci, ci_off); swap_cluster_unlock(ci); @@ -3966,8 +3984,9 @@ void si_swapinfo(struct sysinfo *val) } =20 /* - * swap_dup_entry_direct() - Increase reference count of a swap entry by o= ne. + * swap_dup_entries_direct() - Increase reference count of swap entries by= one. * @entry: first swap entry from which we want to increase the refcount. + * @nr: number of contiguous swap entries to duplicate. * * Returns 0 for success, or -ENOMEM if the extend table is required * but could not be atomically allocated. Returns -EINVAL if the swap @@ -3978,8 +3997,16 @@ void si_swapinfo(struct sysinfo *val) * owner. e.g., locking the PTL of a PTE containing the entry being increa= sed. * Also the swap entry must have a count >=3D 1. Otherwise folio_dup_swap = should * be used. + * + * Unlike swap_put_entries_direct(), the whole range [entry, entry + nr) m= ust + * lie within one swap cluster; a range that crosses a cluster boundary is + * rejected with -EINVAL. The only caller passing nr > 1 is the PMD swap e= ntry + * fork path: a PMD swap entry can only exist with CONFIG_THP_SWAP, where + * SWAPFILE_CLUSTER =3D=3D HPAGE_PMD_NR, and a PMD-order folio's slots are= only ever + * allocated at a cluster head (see alloc_swap_scan_cluster()), so such a = range + * is exactly one cluster. */ -int swap_dup_entry_direct(swp_entry_t entry) +int swap_dup_entries_direct(swp_entry_t entry, int nr) { struct swap_info_struct *si; =20 @@ -3989,6 +4016,9 @@ int swap_dup_entry_direct(swp_entry_t entry) return -EINVAL; } =20 + if (WARN_ON_ONCE(swp_cluster_offset(entry) + nr > SWAPFILE_CLUSTER)) + return -EINVAL; + /* * The caller must be increasing the swap count from a direct * reference of the swap slot (e.g. a swap entry in page table). @@ -3996,7 +4026,7 @@ int swap_dup_entry_direct(swp_entry_t entry) */ VM_WARN_ON_ONCE(!swap_entry_swapped(si, entry)); =20 - return swap_dup_entries_cluster(si, swp_offset(entry), 1); + return swap_dup_entries_cluster(si, swp_offset(entry), nr); } =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-74.mta0.migadu.com [91.218.175.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3EDFD4718E5 for ; Mon, 14 Sep 2026 12:31:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389100; cv=none; b=h81x4kkA5uSUzE41nHKhx31nWaMos+0HHKW65zqE1Z3Te4DgIta1KDrtl7WinNEa51QMHwqKqHxp16VxPIt8/i9jgtjBgta9cChrryeLVnydGhfnsz4sYCAD8mXZcXiIqwWLBSJGe9DssPbXYPZ0edrRmugW1vVY+vzF6gnqzQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389100; c=relaxed/simple; bh=gQpqzqvH+WoJJUAO2d/0PD0oFz7Cyx7fYSRyoCuO5m8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E+wJk9lD4VoUl+JKXgWlGmM+4BzLm+MK183QbIOmNg55wvZO0R+BOcCgWrBfpWUx6P20S3MXqWae3zatDoyLQt22fX/1TTi//eRc4FklPzVjSPEPpxwMj/YfcvnvXT424YdQHTfLjOFvpGrC8DhqsqdFIVDM8ujsieB/qS5L6Jw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=F7pBAob/; arc=none smtp.client-ip=91.218.175.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="F7pBAob/" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=gQpqzqvH+WoJJUAO2d/0PD0oFz7Cyx7fYSRyoCuO5m8=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389091; v=1; x=1789993891; b=F7pBAob/bwheiLNE8XpRUnDs3xIwQokA5+aYb/8RxFVlmMJAMhbtsCkENmXn11jKYtH2Ygzs PC3dEixu0cLhpq8rqMj2QnlT5imQ0cinwPwaTOK39eya0gH34Cx1dHNZ4F6fDaPn3psFefFKKmH Kt6dniiFl8PNaAfRbOiRF/38= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id c310bfb2d63ccb74; Mon, 14 Sep 2026 12:31:30 +0000 X-Mizu-Trace-ID: c310bfb2d63ccb74 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 13/29] mm: zswap: reject high-order swap cache allocations backed by zswap Date: Mon, 14 Sep 2026 05:28:03 -0700 Message-ID: <20260914122950.3283997-14-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zswap stores swapped-out pages one at a time, so a range of swap slots can be split between zswap entries and the backing device. A high-order swap-cache folio over such a range cannot be read back in one go: the disk read would return stale or zero data for every slot zswap owns. Reject that in __swap_cache_alloc(). Do the check after the folio is in the swap cache: from that point zswap can neither start storing nor start writing back any slot in the range, so the answer is stable. Callers already fall back to order-0, which zswap_load() handles per page. Export zswap_is_present() for the PMD-order swap-in users added later. Co-developed-by: Alexandre Ghiti Signed-off-by: Alexandre Ghiti Signed-off-by: Usama Arif --- include/linux/zswap.h | 6 ++++++ mm/swap_state.c | 39 +++++++++++++++++++++++++-------------- mm/zswap.c | 12 +++++++++--- 3 files changed, 40 insertions(+), 17 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index df6cafbe95dc0..94746fb71bb65 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -35,6 +35,7 @@ void zswap_lruvec_state_init(struct lruvec *lruvec); void zswap_folio_swapin(struct folio *folio); bool zswap_is_enabled(void); bool zswap_never_enabled(void); +bool zswap_is_present(swp_entry_t entry, unsigned int nr); #else =20 struct zswap_lruvec_state {}; @@ -73,6 +74,11 @@ static inline bool zswap_never_enabled(void) return true; } =20 +static inline bool zswap_is_present(swp_entry_t entry, unsigned int nr) +{ + return false; +} + #endif =20 #endif /* _LINUX_ZSWAP_H */ diff --git a/mm/swap_state.c b/mm/swap_state.c index 625c185a1ca4d..251b48b2c60e1 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -466,26 +467,27 @@ static struct folio *__swap_cache_alloc(struct swap_c= luster_info *ci, __swap_cache_do_add_folio(ci, folio, entry); spin_unlock(&ci->lock); =20 + /* + * Now that the folio is in the swap cache, zswap can no longer start + * storing or writing back any slot in the range, so this is a stable + * answer. Reject a high-order allocation over a range that already + * has per-page zswap entries. + */ + if (order && zswap_is_present(entry, nr_pages)) { + err =3D -EBUSY; + goto delete_folio; + } + if (mem_cgroup_swapin_charge_folio(folio, memcg_id, vmf ? vmf->vma->vm_mm : NULL, gfp)) { - spin_lock(&ci->lock); - __swap_cache_do_del_folio(ci, folio, entry, shadow); - spin_unlock(&ci->lock); - folio_unlock(folio); - /* nr_pages refs from swap cache, 1 from allocation */ - folio_put_refs(folio, nr_pages + 1); + err =3D -ENOMEM; count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE); - return ERR_PTR(-ENOMEM); + goto delete_folio; } =20 if (order > 1 && folio_memcg_alloc_deferred(folio)) { - spin_lock(&ci->lock); - __swap_cache_do_del_folio(ci, folio, entry, shadow); - spin_unlock(&ci->lock); - folio_unlock(folio); - /* nr_pages refs from swap cache, 1 from allocation */ - folio_put_refs(folio, nr_pages + 1); - return ERR_PTR(-ENOMEM); + err =3D -ENOMEM; + goto delete_folio; } =20 /* memsw uncharges swap when folio is added to swap cache */ @@ -499,6 +501,15 @@ static struct folio *__swap_cache_alloc(struct swap_cl= uster_info *ci, /* Caller will initiate read into locked new_folio */ folio_add_lru(folio); return folio; + +delete_folio: + spin_lock(&ci->lock); + __swap_cache_do_del_folio(ci, folio, entry, shadow); + spin_unlock(&ci->lock); + folio_unlock(folio); + /* nr_pages refs from swap cache, 1 from allocation */ + folio_put_refs(folio, nr_pages + 1); + return ERR_PTR(err); } =20 /** diff --git a/mm/zswap.c b/mm/zswap.c index 507f2d19fd2a4..f5b5827d30084 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1587,11 +1587,14 @@ bool zswap_store(struct folio *folio) * @entry: base swap entry of the range * @nr: number of contiguous slots to check * - * Context: The caller must keep the range pinned, otherwise the answer can - * change under it. + * Context: Unless the caller keeps the range pinned, the answer is only a + * hint: zswap can store or write back a slot right after this returns. A + * caller that needs a stable answer must pin the range first, as + * __swap_cache_alloc() does by inserting the folio into the swap cache be= fore + * asking. * Return: true if at least one slot in the range is in zswap. */ -static bool zswap_is_present(swp_entry_t entry, unsigned int nr) +bool zswap_is_present(swp_entry_t entry, unsigned int nr) { pgoff_t offset =3D swp_offset(entry); struct xarray *tree =3D swap_zswap_tree(entry); @@ -1605,6 +1608,9 @@ static bool zswap_is_present(swp_entry_t entry, unsig= ned int nr) */ BUILD_BUG_ON(SWAPFILE_CLUSTER > ZSWAP_ADDRESS_SPACE_PAGES); =20 + if (zswap_never_enabled()) + return false; + return xa_find(tree, &index, offset + nr - 1, XA_PRESENT); } =20 --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-99.mta0.migadu.com [91.218.175.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 127484718FE for ; Mon, 14 Sep 2026 12:31:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.99 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389105; cv=none; b=eDUWzloxOJ6vhZm4WhcHXB1xC45W69BTJu2BEDEbbBZ7ZwCavci/v+AUiuSB+RjaXFyjZqvq4qavRCJ4egM48N90RaQcIVA+1SrEAwKApO2mqJa6WUyfjdXH3CmLBa9TQG7pmk2S/eJ+wUTrGmdDNLsNBlIf8skNe7WF6uO+WJ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389105; c=relaxed/simple; bh=UU6mVmZlzcFG6qUy1vyDg8aJu6LdyzlIdDsmwZMkMj8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oAHVao2EReVU/tfsjkC/ufFWLJUDkXowmyW9xarY6LAtgeTAWNCWlDjQlynx/iv7JKt9VXxlqdJOXSGb2xdXaJ6j7+9qpSxmQkjRVSQN0bxwNT4+3jYb8J8Zz6Qtf46c0ru4AgvwUxwSWqyv2PyAoLKzBAjqth/Ho4jcD8AyYGM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=w+ylnq7e; arc=none smtp.client-ip=91.218.175.99 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="w+ylnq7e" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=UU6mVmZlzcFG6qUy1vyDg8aJu6LdyzlIdDsmwZMkMj8=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389097; v=1; x=1789993897; b=w+ylnq7eccpdCy8faNHjQfyD5aFsqeay/CkrFE7u+8UdKtWUywn1Uyhi0JoWPqat3gYilAK3 IB/8/ipub5YCDRBR3zJ+E4Va5Tlwg1qWXYsvT7Nlzo9MviULHIA7hGYQlkYFAqCFXqhGAEBNIkb P0ORBxIl68EdSx/UOgf+owr0= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 32f750c12cce9baf; Mon, 14 Sep 2026 12:31:36 +0000 X-Mizu-Trace-ID: 32f750c12cce9baf X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 14/29] mm: swap in PMD swap entries as whole THPs during swapoff Date: Mon, 14 Sep 2026 05:28:04 -0700 Message-ID: <20260914122950.3283997-15-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" swapoff walks every mm and faults every slot of the device back in. unuse_pte_range() only understands PTEs, so a PMD swap entry would never be found and swapoff would never complete. A PMD swap entry is a compact encoding for HPAGE_PMD_NR slots, not a promise that the swap cache holds one folio for them. Add swap_pmd_cache_lookup() to classify the covered range as empty, one PMD-sized folio, or already split, and unuse_pmd() to map the first two cases back in as one THP, preserving soft-dirty, exclusive and UFFD state. Everything else falls back to PTEs: a split cache, per-page zswap state, a failed PMD-order allocation or read, or a poisoned subpage. Check PageHWPoison on every subpage rather than the folio-level flag, which memory_failure() only sets after taking the folio lock. All the fallback reasons are observed without the PMD lock and possibly after sleeping, so they share one exit that re-checks the PMD is still the entry we were called for before splitting it. That exit also drops a folio that is not uptodate, or that has never been mapped, from the swap cache: the PTE path cannot re-read the first, and would add a single-page rmap to the second. Signed-off-by: Usama Arif --- mm/internal.h | 16 ++++ mm/swap.h | 17 +++++ mm/swap_state.c | 44 +++++++++++ mm/swapfile.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 266 insertions(+) diff --git a/mm/internal.h b/mm/internal.h index 05179c4b2090e..ec7f007bc2c0d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -24,6 +24,22 @@ =20 struct folio_batch; =20 +/* + * Unlike folio_contain_hwpoisoned_page(), this does not rely on the folio= -level + * PG_has_hwpoisoned, which memory_failure() only sets after taking the fo= lio + * lock and so can lag a tail-page poison. + */ +static inline bool folio_has_hwpoisoned_subpage(const struct folio *folio) +{ + long nr =3D folio_nr_pages(folio); + long i; + + for (i =3D 0; i < nr; i++) + if (PageHWPoison(folio_page(folio, i))) + return true; + return false; +} + /* mm/workingset.c */ bool workingset_test_recent(void *shadow, bool file, bool *workingset, bool flush); diff --git a/mm/swap.h b/mm/swap.h index 2321c9a2c2c58..0b54026f32d6f 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -311,6 +311,23 @@ static inline bool folio_matches_swap_entry(const stru= ct folio *folio, bool swap_cache_has_folio(swp_entry_t entry); struct folio *swap_cache_get_folio(swp_entry_t entry); void *swap_cache_get_shadow(swp_entry_t entry); +enum swap_pmd_cache { + SWAP_PMD_CACHE_EMPTY, + SWAP_PMD_CACHE_HUGE, + SWAP_PMD_CACHE_SPLIT, +}; + +#ifdef CONFIG_THP_SWAP +enum swap_pmd_cache swap_pmd_cache_lookup(swp_entry_t entry, + struct folio **foliop); +#else +static inline enum swap_pmd_cache swap_pmd_cache_lookup(swp_entry_t entry, + struct folio **foliop) +{ + *foliop =3D NULL; + return SWAP_PMD_CACHE_EMPTY; +} +#endif void swap_cache_del_folio(struct folio *folio); struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_m= ask, unsigned long orders, struct vm_fault *vmf, diff --git a/mm/swap_state.c b/mm/swap_state.c index 251b48b2c60e1..15f93b9a3eb2a 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -125,6 +125,50 @@ bool swap_cache_has_folio(swp_entry_t entry) return swp_tb_is_folio(swp_tb); } =20 +#ifdef CONFIG_THP_SWAP +/** + * swap_pmd_cache_lookup - classify the swap cache behind a PMD swap entry + * @entry: first swap slot encoded by the PMD swap entry + * @foliop: returned PMD-sized folio, with a reference, if present + * + * A PMD swap entry is a compact page-table encoding for HPAGE_PMD_NR + * consecutive swap slots. The swap cache behind those slots can be empty, + * one PMD-sized folio, or per-slot folios after the original folio was sp= lit. + * + * Context: Caller must keep @entry valid using the usual swap cache rules. + * Return: SWAP_PMD_CACHE_EMPTY if no slot in the PMD range has a cached f= olio, + * SWAP_PMD_CACHE_HUGE if one PMD-sized folio covers the range, or + * SWAP_PMD_CACHE_SPLIT if the range needs per-page handling. + */ +enum swap_pmd_cache swap_pmd_cache_lookup(swp_entry_t entry, + struct folio **foliop) +{ + unsigned int type =3D swp_type(entry); + pgoff_t offset =3D swp_offset(entry); + struct folio *folio; + int i; + + *foliop =3D NULL; + + folio =3D swap_cache_get_folio(entry); + if (folio) { + if (folio_nr_pages(folio) =3D=3D HPAGE_PMD_NR) { + *foliop =3D folio; + return SWAP_PMD_CACHE_HUGE; + } + folio_put(folio); + return SWAP_PMD_CACHE_SPLIT; + } + + for (i =3D 1; i < HPAGE_PMD_NR; i++) { + if (swap_cache_has_folio(swp_entry(type, offset + i))) + return SWAP_PMD_CACHE_SPLIT; + } + + return SWAP_PMD_CACHE_EMPTY; +} +#endif + /** * swap_cache_get_shadow - Looks up a shadow in the swap cache. * @entry: swap entry used for the lookup. diff --git a/mm/swapfile.c b/mm/swapfile.c index 27ae3964a158e..fe3cadfc3a017 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -42,6 +42,7 @@ #include #include #include +#include =20 #include #include @@ -2696,6 +2697,182 @@ static int unuse_pte_range(struct vm_area_struct *v= ma, pmd_t *pmd, return 0; } =20 +#ifdef CONFIG_THP_SWAP +/* + * unuse_pmd - Map a locked folio at PMD granularity during swapoff. + * + * The caller provides a locked, swapped-in folio. Returns 0 on success + * (PMD was mapped). Returns -EAGAIN if the swap cache folio no longer + * matches the entry or the PMD changed under the lock (try_to_unuse will + * rescan). Returns -EIO if the folio is not uptodate or contains a poison= ed + * subpage; in that case the PMD is split so unuse_pte_range() can handle + * individual pages. + */ +static int unuse_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, softleaf_t entry, + struct folio *folio) +{ + struct mm_struct *mm =3D vma->vm_mm; + struct page *page; + pmd_t new_pmd, old_pmd; + spinlock_t *ptl; + rmap_t rmap_flags =3D RMAP_NONE; + bool exclusive; + + if (unlikely(!folio_matches_swap_entry(folio, entry))) + return -EAGAIN; + + /* Let PTE fallback reread each slot, or isolate the poisoned ones. */ + if (unlikely(!folio_test_uptodate(folio))) + return -EIO; + if (unlikely(folio_has_hwpoisoned_subpage(folio))) + return -EIO; + + page =3D folio_page(folio, 0); + + ptl =3D pmd_lock(mm, pmd); + old_pmd =3D pmdp_get(pmd); + + if (!pmd_is_swap_entry(old_pmd) || + softleaf_from_pmd(old_pmd).val !=3D entry.val) { + spin_unlock(ptl); + return -EAGAIN; + } + + exclusive =3D pmd_swp_exclusive(old_pmd); + + /* + * Some architectures may have to restore extra metadata to the folio + * when reading from swap. This metadata may be indexed by swap entry + * so this must be called before folio_put_swap(). + */ + arch_swap_restore(folio_swap(entry, folio), folio); + + add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR); + + new_pmd =3D folio_mk_pmd(folio, vma->vm_page_prot); + new_pmd =3D pmd_mkold(new_pmd); + if (pmd_swp_soft_dirty(old_pmd)) + new_pmd =3D pmd_mksoft_dirty(new_pmd); + if (pmd_swp_uffd(old_pmd)) + new_pmd =3D pmd_mkuffd(new_pmd); + if (pmd_swp_uffd(old_pmd) && userfaultfd_rwp(vma)) + new_pmd =3D pmd_modify(new_pmd, PAGE_NONE); + + if (exclusive) + rmap_flags |=3D RMAP_EXCLUSIVE; + + folio_get(folio); + if (!folio_test_anon(folio)) + folio_add_new_anon_rmap(folio, vma, addr, rmap_flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, addr, rmap_flags); + + set_pmd_at(mm, addr, pmd, new_pmd); + folio_put_swap(folio, NULL); + + spin_unlock(ptl); + + folio_free_swap(folio); + return 0; +} + +/* + * Try to swap in a PMD swap entry as a whole THP. Returns 0 on success. + * If the swap cache no longer has one PMD-sized folio, zswap may require + * per-page loading, or a PMD-order allocation/read fails, split the PMD so + * the caller can fall back to unuse_pte_range(). Otherwise propagates the + * error from unuse_pmd(). + */ +static int unuse_pmd_entry(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, softleaf_t entry) +{ + enum swap_pmd_cache cache_state; + struct folio *folio; + pmd_t pmdval; + int ret; + + cache_state =3D swap_pmd_cache_lookup(entry, &folio); + if (cache_state =3D=3D SWAP_PMD_CACHE_SPLIT) { + ret =3D -EAGAIN; + goto split_fallback; + } + if (!folio) { + struct vm_fault vmf =3D { + .vma =3D vma, + .address =3D addr, + .real_address =3D addr, + .pmd =3D pmd, + }; + + if (zswap_is_present(entry, HPAGE_PMD_NR)) { + ret =3D -EAGAIN; + goto split_fallback; + } + + folio =3D swapin_sync(entry, GFP_HIGHUSER_MOVABLE, + BIT(HPAGE_PMD_ORDER), &vmf, NULL, 0); + if (IS_ERR_OR_NULL(folio)) { + ret =3D folio ? PTR_ERR(folio) : -ENOMEM; + goto split_fallback; + } + } + + folio_lock(folio); + folio_wait_writeback(folio); + /* + * If the cached folio is no longer PMD-sized (e.g. split in the + * swap cache by deferred_split_scan() or memory_failure() while + * the PMD swap entry was installed), the PMD swap entry no longer + * maps a single contiguous folio. Split the PMD swap entry so + * unuse_pte_range() can swap the per-slot folios in individually. + */ + if (folio_nr_pages(folio) !=3D HPAGE_PMD_NR) { + folio_unlock(folio); + folio_put(folio); + ret =3D -EAGAIN; + goto split_fallback; + } + ret =3D unuse_pmd(vma, pmd, addr, entry, folio); + /* + * PTE fallback cannot add a single-page rmap to a newly allocated, + * PMD-sized !anon folio, and it cannot do anything useful with a folio + * that failed to read. Remove either from the swap cache so each slot + * is read back into an order-0 folio. An uptodate anon swap-cache folio + * can be mapped one PTE at a time and must stay cached so poisoned + * subpages remain visible. + */ + if (ret && folio_matches_swap_entry(folio, entry) && + (!folio_test_uptodate(folio) || !folio_test_anon(folio))) + swap_cache_del_folio(folio); + folio_unlock(folio); + folio_put(folio); + if (ret =3D=3D -EIO) + goto split_fallback; + return ret; + +split_fallback: + /* + * Every reason we get here was observed without the PMD lock and + * possibly after sleeping, so re-check that the PMD is still the entry + * we were called for. A racing fault may already have swapped the range + * back in as a THP, and splitting that would demote it for nothing. + */ + pmdval =3D pmdp_get(pmd); + if (pmd_is_swap_entry(pmdval) && + softleaf_from_pmd(pmdval).val =3D=3D entry.val) + __split_huge_pmd(vma, pmd, addr); + return ret; +} +#else /* !CONFIG_THP_SWAP */ +static inline int unuse_pmd_entry(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, softleaf_t entry) +{ + return -EAGAIN; +} +#endif /* CONFIG_THP_SWAP */ + static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, unsigned long addr, unsigned long end, unsigned int type) @@ -2706,8 +2883,20 @@ static inline int unuse_pmd_range(struct vm_area_str= uct *vma, pud_t *pud, =20 pmd =3D pmd_offset(pud, addr); do { + pmd_t pmdval; + cond_resched(); next =3D pmd_addr_end(addr, end); + + pmdval =3D pmdp_get(pmd); + if (IS_ENABLED(CONFIG_THP_SWAP) && pmd_is_swap_entry(pmdval)) { + softleaf_t entry =3D softleaf_from_pmd(pmdval); + + if (swp_type(entry) =3D=3D type && + !unuse_pmd_entry(vma, pmd, addr, entry)) + continue; + } + ret =3D unuse_pte_range(vma, pmd, addr, next, type); if (ret) return ret; --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-120.mta0.migadu.com [91.218.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FC04471D0B for ; Mon, 14 Sep 2026 12:31:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.120 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389107; cv=none; b=ks7l1eYGOqTjl5kiYP8VpbhpfNMYjWiLbsGxB4vJA3nucrK0TbI3VEahzeqv5dHAldT1NaBTyBsH0a9wVTF9BVZUFwuLQRsGTVpAlrNLIyFCn5Q9Z7xbWw+SjyPjqPZY3Y+s1BAm0k/hLB4JrPuTzhyKdOwqaxfo3YTLWLNuz6s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389107; c=relaxed/simple; bh=ENO5TWLGQORFi4I6GbjA3CLf4xd6bsmKP3WV+1rfFXs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qUv8zbZDMhsXk80VQrsckGCjPyC0XdDC4dkvZu8bWmeiuTmMks7wdwo4XY9c7TyUa1Zxne8XXdrqjeXiO3GMfzBPYmM8+MoHipZ4Le78Cjp15yFrUigJzEJ4bKLIJYEh0SEebPJL7AEm+ePm88gXxTeHIlq4tXryVcneK3ov3bE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=h3QkhiwX; arc=none smtp.client-ip=91.218.175.120 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="h3QkhiwX" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=ENO5TWLGQORFi4I6GbjA3CLf4xd6bsmKP3WV+1rfFXs=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389099; v=1; x=1789993899; b=h3QkhiwXYcTZIuHPfBY/OApK3oAnJPSf4MiaDXPLEk9gGD9P6GFCYLAoIXZBL/jB3dUBs7VG FtFp5iyFFU+RiwZ8UliwaICS2TK2uZ70h4jBqnHRXIdK82c5JEM0SWi/byRUAi8R5q1edN5041a nF94ai5PgXcbzkDYzG9wWh48= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id da37515ffb07b15a; Mon, 14 Sep 2026 12:31:39 +0000 X-Mizu-Trace-ID: da37515ffb07b15a X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 15/29] fs/proc: account PMD swap entries in smaps Date: Mon, 14 Sep 2026 05:28:05 -0700 Message-ID: <20260914122950.3283997-16-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" smaps_pmd_entry() derives a page from a non-present PMD and gives up if it cannot. A PMD swap entry carries no PFN, so a swapped-out THP is accounted nowhere: its bytes show up in neither Swap nor SwapPss, and VmSwap silently loses them. The slots of a PMD swap entry can have different swap reference counts, so SwapPss has to be computed one slot at a time rather than divided once. Factor that loop out of smaps_pte_entry() and call it from both levels, then return early: a swapped-out THP is not resident and must not be added to AnonHugePages or Rss. pagemap needs no equivalent change - it already emits PM_SWAP and type | offset+idx for a PFN-less softleaf entry - but its thp_migration_supported() gate answers a different question than the decode requires, so use pmd_is_valid_softleaf() there instead. Signed-off-by: Usama Arif --- fs/proc/task_mmu.c | 42 ++++++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 24425e2308951..aa3f4d54a8580 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -957,6 +957,27 @@ static void smaps_pte_hole_lookup(unsigned long addr, = struct mm_walk *walk) #endif } =20 +static void smaps_account_swap(struct mem_size_stats *mss, + softleaf_t entry, unsigned long size) +{ + unsigned long nr_pages =3D size >> PAGE_SHIFT; + + mss->swap +=3D size; + do { + int mapcount =3D swp_swapcount(entry); + + if (mapcount >=3D 2) { + u64 pss_delta =3D (u64)PAGE_SIZE << PSS_SHIFT; + + do_div(pss_delta, mapcount); + mss->swap_pss +=3D pss_delta; + } else { + mss->swap_pss +=3D (u64)PAGE_SIZE << PSS_SHIFT; + } + entry.val++; + } while (--nr_pages); +} + static void smaps_pte_entry(pte_t *pte, unsigned long addr, struct mm_walk *walk) { @@ -978,18 +999,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long = addr, const softleaf_t entry =3D softleaf_from_pte(ptent); =20 if (softleaf_is_swap(entry)) { - int mapcount; - - mss->swap +=3D PAGE_SIZE; - mapcount =3D swp_swapcount(entry); - if (mapcount >=3D 2) { - u64 pss_delta =3D (u64)PAGE_SIZE << PSS_SHIFT; - - do_div(pss_delta, mapcount); - mss->swap_pss +=3D pss_delta; - } else { - mss->swap_pss +=3D (u64)PAGE_SIZE << PSS_SHIFT; - } + smaps_account_swap(mss, entry, PAGE_SIZE); } else if (softleaf_has_pfn(entry)) { if (softleaf_is_device_private(entry)) present =3D true; @@ -1019,9 +1029,13 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned lon= g addr, if (pmd_present(*pmd)) { page =3D vm_normal_page_pmd(vma, addr, *pmd); present =3D true; - } else if (unlikely(thp_migration_supported())) { + } else { const softleaf_t entry =3D softleaf_from_pmd(*pmd); =20 + if (softleaf_is_swap(entry)) { + smaps_account_swap(mss, entry, HPAGE_PMD_SIZE); + return; + } if (softleaf_has_pfn(entry)) page =3D softleaf_to_page(entry); } @@ -2000,7 +2014,7 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigne= d long addr, flags |=3D PM_UFFD_WP; if (pm->show_pfn) frame =3D pmd_pfn(pmd) + idx; - } else if (thp_migration_supported()) { + } else if (pmd_is_valid_softleaf(pmd)) { const softleaf_t entry =3D softleaf_from_pmd(pmd); unsigned long offset; =20 --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-139.mta0.migadu.com [91.218.175.139]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99A3745D930 for ; Mon, 14 Sep 2026 12:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.139 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389109; cv=none; b=R/KCbzlcnUabLSRrMqGZiBcsstOp8BZAtj3iOfceBrETN/qus6AAzjYQtx8h3rEeFMUK0gsJ63mKVzCQV3LcAwHpdhNcqCQ99Mo6JvqnzmkDng93dTtynJ7pb6Z6mo43wLiBGQiUkK52OAIKrcNFEeniiCo1FRJYPYPwYeI97+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389109; c=relaxed/simple; bh=2eHW0y7y7uGyU1RGoY9G808x9f5l4A7c7aMBHiLQq+A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BmRRdjTGNkInmRC6EdwGWgxOqU9qcmTfEZ5GgHMbJ2egmTKASYKumOQg42U0WkSbEliW8Qyf30mlJwv0vheV8RwVLbakHplywnehqK5uZhvRsb8ACOb2ZlBIHTNm4TtNBL6Xrtc4F71BS//2fAZ3m5EUg6HBW+9jo6KxrlRUYNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=d9WjUnY2; arc=none smtp.client-ip=91.218.175.139 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="d9WjUnY2" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=2eHW0y7y7uGyU1RGoY9G808x9f5l4A7c7aMBHiLQq+A=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389101; v=1; x=1789993901; b=d9WjUnY2evR8QgKHUj2PVKmLVy6KFGm5/nWQiO0iGSkyD3UtI0fv63IVqfX7onGpMML8L+Yp 7SoPe0qwjpykcWqmwS2FAxk3gfajIMB+7e8/gKa0dSKCQk7ti3/Dukkh2Fj3sLJvpo4PYJKyChz 1U87qA7KSWfsxxkobuDzjx1Q= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id d75a3d950def4de7; Mon, 14 Sep 2026 12:31:41 +0000 X-Mizu-Trace-ID: d75a3d950def4de7 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 16/29] mm: handle soft-dirty and uffd-wp on PMD swap entries Date: Mon, 14 Sep 2026 05:28:06 -0700 Message-ID: <20260914122950.3283997-17-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Soft-dirty and uffd-wp live in the PMD, not in the swap entry, so every path that manipulates them for a PMD migration entry has to do the same for a PMD swap entry. Otherwise clearing soft-dirty skips a swapped-out THP, UFFDIO_WRITEPROTECT does not arm it, and mremap does not mark the destination. change_huge_pmd() needs nothing beyond dropping its thp_migration_supported() gate: a PMD swap entry encodes no write permission, so it falls into the default arm of change_non_present_huge_pmd() and picks up only the uffd-wp update. Signed-off-by: Usama Arif --- fs/proc/task_mmu.c | 4 ++-- mm/huge_memory.c | 11 +++++++++-- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index aa3f4d54a8580..ad0d4c7f0988b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1654,7 +1654,7 @@ static void clear_soft_dirty_pmd(struct vm_area_struc= t *vma, pmd =3D pmd_clear_soft_dirty(pmd); =20 set_pmd_at(vma->vm_mm, addr, pmdp, pmd); - } else if (pmd_is_migration_entry(pmd)) { + } else if (pmd_is_migration_entry(pmd) || pmd_is_swap_entry(pmd)) { pmd =3D pmd_swp_clear_soft_dirty(pmd); set_pmd_at(vma->vm_mm, addr, pmdp, pmd); } @@ -2480,7 +2480,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *v= ma, old =3D pmdp_invalidate_ad(vma, addr, pmdp); pmd =3D pmd_mkuffd(old); set_pmd_at(vma->vm_mm, addr, pmdp, pmd); - } else if (pmd_is_migration_entry(pmd)) { + } else if (pmd_is_migration_entry(pmd) || pmd_is_swap_entry(pmd)) { pmd =3D pmd_swp_mkuffd(pmd); set_pmd_at(vma->vm_mm, addr, pmdp, pmd); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6dfe8ef6dd371..d3d0fec9a5ff3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2660,7 +2660,8 @@ static inline int pmd_move_must_withdraw(spinlock_t *= new_pmd_ptl, static pmd_t move_soft_dirty_pmd(pmd_t pmd) { if (pgtable_supports_soft_dirty()) { - if (unlikely(pmd_is_migration_entry(pmd))) + if (unlikely(pmd_is_migration_entry(pmd) || + pmd_is_swap_entry(pmd))) pmd =3D pmd_swp_mksoft_dirty(pmd); else if (pmd_present(pmd)) pmd =3D pmd_mksoft_dirty(pmd); @@ -2752,6 +2753,12 @@ static void change_non_present_huge_pmd(struct mm_st= ruct *mm, pmd_t newpmd; =20 VM_WARN_ON(!pmd_is_valid_softleaf(*pmd)); + + /* + * Note that a PMD swap entry falls into the default branch below: it + * does not encode write permission in the entry type, so only the + * uffd_wp flag update at the end applies to it. + */ if (softleaf_is_migration_write(entry)) { const struct folio *folio =3D softleaf_to_folio(entry); =20 @@ -2812,7 +2819,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, if (!ptl) return 0; =20 - if (thp_migration_supported() && pmd_is_valid_softleaf(*pmd)) { + if (pmd_is_valid_softleaf(*pmd)) { change_non_present_huge_pmd(mm, addr, pmd, uffd_prot, uffd_prot_resolve); goto unlock; --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-162.mta0.migadu.com [91.218.175.162]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 892F745C6EB for ; Mon, 14 Sep 2026 12:31:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.162 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389117; cv=none; b=FCEb+fx+B8Kw7nomJLWUKyWKS/UOD8rrnn8RkIH+lPd2OS1n36SyQrSb9w8u03POWwFpm/F6sc3yPiIlqipxK63wO39x1cwn240o5P+TR2RHF0xlIGKuU6n8lt2RYPNB2TAET+XLfK1dZ0fxGz+eVVsVb8gkeQHrlBxdsYK6hLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389117; c=relaxed/simple; bh=EsV4QmuE2zIVMKtO/SOhRrFUuEtNtiAe7xebMuMrx54=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mHSo46rS5xaa7ePLaSj2itFDhFLq0uBv5+04eSdQhEWfWuxuiGwNJzjSVUglReGdr3ygCQ8aw5CyAlJ1fILHWfUtciWJrLZYPXI/CMrathILOIFUuNUHdXXzAI8aMgrPsaU4OVtcta8KqXTIyNZwywuULsOfMiJVf3fdogWx4Yc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=QhQaOh26; arc=none smtp.client-ip=91.218.175.162 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="QhQaOh26" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=EsV4QmuE2zIVMKtO/SOhRrFUuEtNtiAe7xebMuMrx54=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389103; v=1; x=1789993903; b=QhQaOh261LjKiwmhFo7BQr8VctZnuLzKLjEp6oDQVw5IplJ2Rkxh5gSunTOiHUYOG9iFB3OG gTKYbLaytZVkuyK2vP/g+Qh8FeBe4vvd5e4BtbIy9znwvjHVlUTMZGmok+tEFf3gfXu5bqAa+xp rKIA/i3g7E/Azb2/oRks0voA= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id e796bb344a1921fe; Mon, 14 Sep 2026 12:31:43 +0000 X-Mizu-Trace-ID: e796bb344a1921fe X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 17/29] mm/hmm: fault PMD swap entries on demand Date: Mon, 14 Sep 2026 05:28:07 -0700 Message-ID: <20260914122950.3283997-18-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" hmm_vma_handle_absent_pmd() reports anything non-present that is not device-private as HMM_PFN_ERROR, so a swapped-out THP would look like a permanent failure rather than memory that needs paging in. Treat it like a PTE swap entry: record the fault when one was asked for, and otherwise report the range as non-resident with no flags, which is what hmm_vma_handle_pte() already does in snapshot mode. Signed-off-by: Usama Arif --- mm/hmm.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index 2f1e98c6b6440..95575ac378888 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -377,12 +377,21 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *= walk, unsigned long start, required_fault =3D hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0); if (required_fault) { - if (softleaf_is_device_private(entry)) + if (softleaf_is_device_private(entry) || + softleaf_is_swap(entry)) return hmm_record_fault(addr, end, required_fault, walk); else return -EFAULT; } =20 + /* + * A swapped-out THP is not resident. Report it as not-valid, + * matching what hmm_vma_handle_pte() does for a PTE swap entry when + * no fault was requested. + */ + if (softleaf_is_swap(entry)) + return hmm_pfns_fill(start, end, range, 0); + return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR); } #else --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-187.mta0.migadu.com [91.218.175.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B252E45D1A8 for ; Mon, 14 Sep 2026 12:31:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389117; cv=none; b=I9bdFK+5b0DE7KsHL4VTnW0PtRgHrm2dzBnIF3GTxBkDPFfsNbWfMsMkOOXoxLZOKKWAxWKBkcgUTPMBXxkljadBDBtG/N0oEny1HVll7YyQqPxcLfjkVyWs48op4J/libVUqUcECj4Axe2Iizx5O0GIdMgVEUVUT1FJC0rWc4Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389117; c=relaxed/simple; bh=0pxvz2tA8d/MqnhA+ARBMVfrRyK0IVtO3kcKq5jVvyM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FmZTSM1OeUHElDfzrs5HiU/QbkiJyB3z3ui4CWdCqsIsxj5hAxnX0xoIo2jMyxD4E9GP8IGjRIYaowXiCBQ/8Kq0UC3rPNUPh0pnAxXD326VbY5Ia0OFxiG/rvUdsKXb9MHS5CfpMmVZxLR8FPMPCOnvtGAG1KFC1DTIuyrgAr4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=cUeMcohk; arc=none smtp.client-ip=91.218.175.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="cUeMcohk" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=0pxvz2tA8d/MqnhA+ARBMVfrRyK0IVtO3kcKq5jVvyM=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389105; v=1; x=1789993905; b=cUeMcohkP1jsBtxE6fis2Pimxgeg45wvpSINk23Rh9u406i7JrWBmjRGOU3FLpGJil5tQs34 Y27hM6JDd30TjiXdkGpgQR2wCV/TAZjVMnElrtxK+mKnmfvZQax78mkl+4MNh4kAKDzHPp8pMWQ LVFNKoDG/PvexTPaz3c7SExU= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id a8f2f31ac8e52b34; Mon, 14 Sep 2026 12:31:45 +0000 X-Mizu-Trace-ID: a8f2f31ac8e52b34 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 18/29] mm: free PMD swap entries in zap_huge_pmd() Date: Mon, 14 Sep 2026 05:28:08 -0700 Message-ID: <20260914122950.3283997-19-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zap_huge_pmd() assumes any non-present PMD it clears references a folio, so a PMD swap entry would reach normal_or_softleaf_folio_pmd() and warn. Worse, nothing would release the slots it points at: unmapping a swapped-out THP would leak the whole PMD's worth of swap. Handle it before the folio lookup, as zap_pte_range() already does for a PTE swap entry. Signed-off-by: Usama Arif --- mm/huge_memory.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d3d0fec9a5ff3..689a1f4286471 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2628,6 +2628,16 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_= area_struct *vma, arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); =20 + if (pmd_is_swap_entry(orig_pmd)) { + softleaf_t entry =3D softleaf_from_pmd(orig_pmd); + + zap_deposited_table(mm, pmd); + spin_unlock(ptl); + swap_put_entries_direct(entry, HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR); + return true; + } + is_present =3D pmd_present(orig_pmd); folio =3D normal_or_softleaf_folio_pmd(vma, addr, orig_pmd, is_present); has_deposit =3D has_deposited_pgtable(vma, orig_pmd, folio); --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-208.mta0.migadu.com [91.218.175.208]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CC6F463B68 for ; Mon, 14 Sep 2026 12:31:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.208 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389118; cv=none; b=Y1WzIhP0rz+M18qfGsw1uvlo+X9iH8f1XQStr1T3g2ek0nBix82bOluRSzCc2QknUjAWnke8PPtMDPQetxenFZSapxjwQ4ViQTeSJEkfc4OcH3LOjx+AK6nBZKBdh2wk4HiNC3Wps3Qp0dHjBEj5Q8GH5r6UkPSCIN5S6SIiz+8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389118; c=relaxed/simple; bh=/ejeggqsmQIRp7dTYKNmiDe/IHY+KtxEWhGqQu4hgx0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tjk3nqh95EVpsIzhIO6dwvkZRjZbnKgu4uOYm+caBF4sZ0RLh1ax8Dvn3AxU0Bl57mSw6pgS8MQZT1hCtpTTQFWe2mWFoCG5DAVH/tqTUHBgiFyQTW//WwlxzopYL9R0UQbd6yOsIeDWI7to3LEsveOdyNoJenjsElvlPtHqDrU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=dI59SaSo; arc=none smtp.client-ip=91.218.175.208 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="dI59SaSo" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=/ejeggqsmQIRp7dTYKNmiDe/IHY+KtxEWhGqQu4hgx0=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389112; v=1; x=1789993912; b=dI59SaSo3o3og/j4mLbGKRwOkjr4VqVmEocuL56p277x9h3WBHVhKnC7giJk4KtBdo8e1k3Q +RYv3Y6KWkfu+e2FxN9ZOxSYjPaoMsNOzZGNM5hsqPN6M9UfBKxex3hNE48FwZ052p0EURAPZXe EHEFA7wNszjB3U0eGAQwCc7g= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 00067274d6bc2178; Mon, 14 Sep 2026 12:31:51 +0000 X-Mizu-Trace-ID: 00067274d6bc2178 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 19/29] mm/madvise: free PMD swap entries with MADV_FREE Date: Mon, 14 Sep 2026 05:28:09 -0700 Message-ID: <20260914122950.3283997-20-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" madvise_free_pte_range() only calls madvise_free_huge_pmd() for a present THP, and its PTE loop cannot descend into a non-present PMD, so MADV_FREE over a swapped-out THP silently does nothing. Over the same range as PTE swap entries it frees every slot, which is what userspace gets today. Free it at PMD granularity when the advice covers the whole PMD, and split and let the PTE path handle the subrange when it does not. zap_deposited_table() moves above its new caller; it is otherwise unchanged. Signed-off-by: Usama Arif --- mm/huge_memory.c | 34 +++++++++++++++++++++++++--------- mm/madvise.c | 2 +- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 689a1f4286471..5f3d620c64a94 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2447,6 +2447,15 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vm= f) return 0; } =20 +static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd) +{ + pgtable_t pgtable; + + pgtable =3D pgtable_trans_huge_withdraw(mm, pmd); + pte_free_defer(mm, pgtable); + mm_dec_nr_ptes(mm); +} + /* * Return true if we do MADV_FREE successfully on entire pmd page. * Otherwise, return false. @@ -2471,6 +2480,22 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, s= truct vm_area_struct *vma, goto out; =20 if (unlikely(!pmd_present(orig_pmd))) { + if (pmd_is_swap_entry(orig_pmd)) { + softleaf_t entry =3D softleaf_from_pmd(orig_pmd); + + if (next - addr !=3D HPAGE_PMD_SIZE) { + spin_unlock(ptl); + __split_huge_pmd(vma, pmd, addr); + goto out_unlocked; + } + + pmdp_huge_get_and_clear(mm, addr, pmd); + zap_deposited_table(mm, pmd); + spin_unlock(ptl); + swap_put_entries_direct(entry, HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR); + return true; + } VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) && !pmd_is_device_private_entry(orig_pmd)); goto out; @@ -2525,15 +2550,6 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, s= truct vm_area_struct *vma, return ret; } =20 -static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd) -{ - pgtable_t pgtable; - - pgtable =3D pgtable_trans_huge_withdraw(mm, pmd); - pte_free_defer(mm, pgtable); - mm_dec_nr_ptes(mm); -} - static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct= *vma, pmd_t pmdval, struct folio *folio, bool is_present) { diff --git a/mm/madvise.c b/mm/madvise.c index fbb72ab49aa64..70acb39b8b3dd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -678,7 +678,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, int nr, max_nr; =20 next =3D pmd_addr_end(addr, end); - if (pmd_trans_huge(*pmd)) + if (pmd_trans_huge(*pmd) || pmd_is_swap_entry(*pmd)) if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next)) return 0; =20 --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-227.mta0.migadu.com [91.218.175.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2417245D939 for ; Mon, 14 Sep 2026 12:31:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.227 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389120; cv=none; b=eMtEgUtOMy6OLLm5SFJkqm+3Vjbg3PpDgyCtRHYyd5ipYIJaRJZ1O4kr8I6QcblVDz4T2B8t7fZBWFZULJdIIuAOcbMz4XzbN9ZZinRDl+jC0VnsKMQlhSoNArW4cRgyHoSn7Ajx3CeCOyKOaKShoUBV4xpnNJvzxrlTuNPibsU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389120; c=relaxed/simple; bh=O3Mn4KnTxiYaTlhEiHGB+/12dl1tDr6TCxd6G0YbQtw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RFu6TXzdcFQjdCe6JjLHS0AojXOVnYZ3ppD81Dk4AYu63veRiKrJ76Sk0UMR2kcruoh9GeIH/pO/ANTlGNbmwkfuzTSQlhy4RbB6HFdlLWn2a6FaRVobhB7BmEVNgDVWsil3P/G8KN+3S/w2rB+HbaamHt/5vDzdjiRoRjabmKE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=V+pRhF7b; arc=none smtp.client-ip=91.218.175.227 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="V+pRhF7b" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=O3Mn4KnTxiYaTlhEiHGB+/12dl1tDr6TCxd6G0YbQtw=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389114; v=1; x=1789993914; b=V+pRhF7bePafelNnSWtdASntDuFZFJAL1WKkfdOKocMP4eA5yguWiFjSAONjB/RL+TFOvuxA 1pUjuwHkNd53hSoL8gVGXKQiJZbxim33Be/UwgM/Yj/3zwvx/got9n4cFIQ6OFDPr0LXaNw6Jcr q5z6cBm8Jm0bnf1lUyAKLmIA= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id 849d2883d7f4742b; Mon, 14 Sep 2026 12:31:53 +0000 X-Mizu-Trace-ID: 849d2883d7f4742b X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 20/29] mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT Date: Mon, 14 Sep 2026 05:28:10 -0700 Message-ID: <20260914122950.3283997-21-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A swapped-out THP has no resident folio to deactivate or reclaim, which is also the conclusion the PTE path reaches when it walks the swap entries and skips every one. madvise_cold_or_pageout_pte_range() gets there the long way round: pmd_trans_huge() is false for a PMD swap entry, so it falls out of the huge branch and tries to map a PTE table that a non-present PMD does not have. The entry can also appear after that first, lockless test, so re-check under the PMD lock; otherwise a PMD that turned into a swap entry while pmd_trans_huge_lock() was being taken trips the VM_WARN_ON_ONCE() that expects a migration or device-private entry. Signed-off-by: Usama Arif --- mm/madvise.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/madvise.c b/mm/madvise.c index 70acb39b8b3dd..d3b5ed7e3cffb 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -385,6 +385,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pm= d, !can_do_file_pageout(vma); =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* + * Swapped-out THPs have no resident folio to deactivate or reclaim. + * Avoid descending into or splitting a PMD swap entry. + */ + if (pmd_is_swap_entry(*pmd)) { + walk->action =3D ACTION_CONTINUE; + return 0; + } + if (pmd_trans_huge(*pmd)) { pmd_t orig_pmd; unsigned long next =3D pmd_addr_end(addr, end); @@ -395,6 +404,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, return 0; =20 orig_pmd =3D *pmd; + if (pmd_is_swap_entry(orig_pmd)) + goto huge_unlock; + if (unlikely(!pmd_present(orig_pmd))) { VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) && !pmd_is_device_private_entry(orig_pmd)); --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-245.mta0.migadu.com [91.218.175.245]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9677C45FFDE for ; Mon, 14 Sep 2026 12:32:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.245 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389131; cv=none; b=U8FDN20R4vjmFqJ1g6H5xphWtPHxWeMh3Ude9Y/9u2p946rWKK/2iedF/LieQeySN9COpJeA0M3xWlyIX6aFNOxfBeJk2vOwyfaCqPc496aT9hOPFkAqLp04xhdLtTAl+2ka9vw0IZevTwcNGX6KbEorOs9WU2JLUCKSNeaIS0w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389131; c=relaxed/simple; bh=PmpEBPMCGBePeR65tGhRvp39CU+cDN/bw2BxnAbshOU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Cz0ka5oY0uGk+NEHhfBBu8HfNpMhPGYdqjkOLkNbu7Sraa8+zbW18oElnITq30OLJjXqGIl9HMnL8DZnG6URqFkKDDj+S+8/DAq1leh3Twl8cAj83eny1gSYEzbEZ9U18qPzuuSZ/NfbWnjbeO6RVu8IcfmzIDKu72JMMqGkxlM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=a5EMW5Ki; arc=none smtp.client-ip=91.218.175.245 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="a5EMW5Ki" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=PmpEBPMCGBePeR65tGhRvp39CU+cDN/bw2BxnAbshOU=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389118; v=1; x=1789993918; b=a5EMW5KiIJFHiP6RUSv5DT9A/vgiXnUcIICfNI+cQRPfyIMq2ORlllHe6vsKWhdx5Qmf2Gq0 DIC7HekNqovFX23Oyc9X69qQgZ3KmS3t+tEqB8ACeDX+lFyVp0NQBURGhP+8jO3LOYB14rqr2At szJXH2ygEQg9uOEU+v3+34Es= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id abc9569ad6a47579; Mon, 14 Sep 2026 12:31:57 +0000 X-Mizu-Trace-ID: abc9569ad6a47579 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 21/29] mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE Date: Mon, 14 Sep 2026 05:28:11 -0700 Message-ID: <20260914122950.3283997-22-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Both callbacks test bare pmd_trans_huge(), which is false for a PMD swap entry, so they return without consuming the PMD and walk_pmd_range() splits it - only for the PTE callbacks to find the range populated and abort, or to find no guard markers and do nothing. Both already short-circuit at PMD level for a present THP. Do the same for a swapped-out one: it is just as populated, and just as free of guard markers. Signed-off-by: Usama Arif --- mm/madvise.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index d3b5ed7e3cffb..64019668d69fb 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1269,8 +1269,12 @@ static int guard_install_pmd_entry(pmd_t *pmd, unsig= ned long addr, { pmd_t pmdval =3D pmdp_get(pmd); =20 - /* If huge return >0 so we abort the operation + zap. */ - return pmd_trans_huge(pmdval); + /* + * If huge return >0 so we abort the operation + zap. A PMD swap entry + * is a swapped-out THP: also populated, and splitting it here would + * only demote it before the zap. + */ + return pmd_trans_huge(pmdval) || pmd_is_swap_entry(pmdval); } =20 static int guard_install_pte_entry(pte_t *pte, unsigned long addr, @@ -1409,8 +1413,12 @@ static int guard_remove_pmd_entry(pmd_t *pmd, unsign= ed long addr, { pmd_t pmdval =3D pmdp_get(pmd); =20 - /* If huge, cannot have guard pages present, so no-op - skip. */ - if (pmd_trans_huge(pmdval)) + /* + * If huge, cannot have guard pages present, so no-op - skip. The same + * is true of a PMD swap entry, which must not be split just to discover + * there is nothing to remove. + */ + if (pmd_trans_huge(pmdval) || pmd_is_swap_entry(pmdval)) walk->action =3D ACTION_CONTINUE; =20 return 0; --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-12.mta0.migadu.com [91.218.175.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B75E4611C9 for ; Mon, 14 Sep 2026 12:32:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389130; cv=none; b=OYbr2ExVjSmR402iPhMVZJ3JBbSPZ56zvoDW3wAHe2M0tSm2eQkxoPoc+h82doHWWO/7yqq7nRbNBI5wEKRP/CwxveeZRD2AeBG8f5OQXEboDGkWvZtFOyBp0i8dXH9N+TnpfTZfl5CUVOSWdERX6dllOH19+zTLBEa59WILH2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389130; c=relaxed/simple; bh=w0gztRkUfUOE/+G3kO+FMjzHwEfBAB20UzLM0QrOt9Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cJRHaBT6926vEEXuEyvOHv7H1KUac3I2kTzlRh12BzrQ2fNYH8ti+YMPasj/YsXrbPgLWv4Okry5rWW6HZ+GhN7fAOeFHSKGLfQR2RlMsEBuGOcVNqa33Pgm1IDVaJorREpKcg1+fe52lC5f5MaFTwJ/25hPo2z/ijAb7wYJL9g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=N9h+BqDN; arc=none smtp.client-ip=91.218.175.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="N9h+BqDN" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=w0gztRkUfUOE/+G3kO+FMjzHwEfBAB20UzLM0QrOt9Y=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389120; v=1; x=1789993920; b=N9h+BqDNu2H6KLMXdpMIXdLgPuFHyp95Bkge701twKjC4/Ib7B0+69PngqCU/zzAMUDuIkmE Foer+v+sMC0EmoxOhUWfpntEhbp8B+py4vPbufwTtv+ELhKFxV+OeLpTOEcaFNIpRTZbHjx1Axc 4T7TZ4Qjboq10gF+2hNEvJDc= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id 95316f54d43a6e01; Mon, 14 Sep 2026 12:32:00 +0000 X-Mizu-Trace-ID: 95316f54d43a6e01 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 22/29] mm/mincore: report PMD swap-cache residency Date: Mon, 14 Sep 2026 05:28:12 -0700 Message-ID: <20260914122950.3283997-23-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" mincore_pte_range() reports every page under a huge PMD as resident. That is wrong for a swapped-out one: mincore() would tell userspace that memory it will have to fault in is already in core. A PMD swap entry is only a compact encoding for HPAGE_PMD_NR slots, so answer from the swap cache instead. One PMD-sized folio covering the range makes every page resident exactly when that folio is uptodate; an empty cache makes none of them; and if the folio was split while the entry stayed in place, fall back to looking up each covered slot as mincore_swap() does for a PTE swap entry. Signed-off-by: Usama Arif --- mm/mincore.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/mm/mincore.c b/mm/mincore.c index c086836bc4bcc..1846c6f68ae63 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -85,6 +85,48 @@ static unsigned char mincore_swap(swp_entry_t entry, boo= l shmem) return present; } =20 +#ifdef CONFIG_THP_SWAP +static void mincore_pmd_swap(swp_entry_t entry, unsigned long addr, + unsigned long end, unsigned char *vec) +{ + unsigned long haddr =3D addr & HPAGE_PMD_MASK; + unsigned long start =3D (addr - haddr) >> PAGE_SHIFT; + unsigned long nr =3D (end - addr) >> PAGE_SHIFT; + struct folio *folio; + enum swap_pmd_cache state; + int i; + + state =3D swap_pmd_cache_lookup(entry, &folio); + if (state =3D=3D SWAP_PMD_CACHE_HUGE) { + memset(vec, folio_test_uptodate(folio), nr); + folio_put(folio); + return; + } + + if (state =3D=3D SWAP_PMD_CACHE_EMPTY) { + memset(vec, 0, nr); + return; + } + + /* + * The PMD swap entry is only a compact encoding for consecutive swap + * slots. If the PMD-sized swapcache folio was split, report residency + * from the individual slots covered by this mincore() range. + */ + for (i =3D 0; i < nr; i++) + vec[i] =3D mincore_swap(swp_entry(swp_type(entry), + swp_offset(entry) + start + i), + false); +} +#else /* !CONFIG_THP_SWAP */ +static void mincore_pmd_swap(swp_entry_t entry, unsigned long addr, + unsigned long end, unsigned char *vec) +{ + /* Nothing produces PMD swap entries without CONFIG_THP_SWAP. */ + memset(vec, 0, (end - addr) >> PAGE_SHIFT); +} +#endif /* CONFIG_THP_SWAP */ + /* * Later we can get more picky about what "in core" means precisely. * For now, simply check to see if the page is in the page cache, @@ -171,7 +213,10 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long= addr, unsigned long end, =20 ptl =3D pmd_trans_huge_lock(pmd, vma); if (ptl) { - memset(vec, 1, nr); + if (pmd_is_swap_entry(*pmd)) + mincore_pmd_swap(softleaf_from_pmd(*pmd), addr, end, vec); + else + memset(vec, 1, nr); spin_unlock(ptl); goto out; } --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-29.mta0.migadu.com [91.218.175.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8A0F463B74 for ; Mon, 14 Sep 2026 12:32:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389134; cv=none; b=H9KtG+cbpbdnvdcRf2GCaMBpA8MYg8Q4JTgs7atJwR5/8WXRvV2X4AixfBUnOrOZFSTikXcONxEDc6/UQTalPYvTJZae2zrDZhZ20xqIT7G2CnWBZzcynYIoiPCPwKACvgZq93gr+rFSZH11gMZS6bVkOOQVKAT346HTUSqt880= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389134; c=relaxed/simple; bh=m4azhIZnpKty8kdqDwfR1PkRU+4t+lf/1EC6ZF7XDMc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e3E5t1j8aXD902T7DAkafCK6KVnkjLI9zpfp0GF+/Sr7xDix91rXf793Qr9Csx9COB9sWBUDNZbuB/AUIUaNLvcJnGroAp+b+1J4bU+3xnwCHTmsrb0dMxoXGGKnvypY/++lobB/G5O4nUWHKms0TM9/BE0U7r287NSq6loBxrQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=SYhMEomT; arc=none smtp.client-ip=91.218.175.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="SYhMEomT" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=m4azhIZnpKty8kdqDwfR1PkRU+4t+lf/1EC6ZF7XDMc=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389123; v=1; x=1789993923; b=SYhMEomTiM1mkK5PkCE1gEHXoIWv4ccLvr3evYTTgNpXZgPsXtetQGlyNh3V+woWnFHO3mKR PAj3JsteRxOGIZ6X19X5oi4uQz6PQOdLERZ2HO1QOkbTL+PEsGXvxsB/FKMfgqq5dMG4P+p3cii yAZ5Oi5hTHhYkkCgqQu6S87A= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 25c92a897014d0e5; Mon, 14 Sep 2026 12:32:02 +0000 X-Mizu-Trace-ID: 25c92a897014d0e5 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 23/29] mm/khugepaged: treat PMD swap entries as mapped THPs Date: Mon, 14 Sep 2026 05:28:13 -0700 Message-ID: <20260914122950.3283997-24-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" check_pmd_state() would report a PMD swap entry as SCAN_NO_PTE_TABLE - it is not present and, on x86, is also pmd_bad(). The range is already a THP; it just happens to be on disk. Return SCAN_PMD_MAPPED next to the existing migration-entry case and before the pmd_present() and pmd_bad() tests. This does change MADV_COLLAPSE. Today a swapped-out THP is HPAGE_PMD_NR PTE swap entries and MADV_COLLAPSE, which ignores khugepaged_max_ptes_swap, swaps them all in. With a PMD swap entry it reports SCAN_PMD_MAPPED, which madvise_collapse() counts as success, so the call returns 0 without making the range resident. It still faults back in as a whole THP on first touch; only the point at which the I/O happens moves. Signed-off-by: Usama Arif --- mm/khugepaged.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 0d8cd44b14c57..bbbf96c616295 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1052,6 +1052,12 @@ static inline enum scan_result check_pmd_state(pmd_t= *pmd) */ if (pmd_is_migration_entry(pmde)) return SCAN_PMD_MAPPED; + /* + * A PMD-mapped THP that has been swapped out is still a THP from + * khugepaged's perspective; treat it like a present huge PMD. + */ + if (pmd_is_swap_entry(pmde)) + return SCAN_PMD_MAPPED; if (!pmd_present(pmde)) return SCAN_NO_PTE_TABLE; if (pmd_trans_huge(pmde)) --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-47.mta0.migadu.com [91.218.175.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFA25466AE7 for ; Mon, 14 Sep 2026 12:32:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389139; cv=none; b=R8BCLQ+f8RBx03kVwsUZ7p5sj0O0yj3Ne79xru13IweD2WfPs96pr7EP8JnZ01ZB+kcR02YoTCnwNtfya8AAe5iuDu2bSGjlApOSLVHBG8XFTwi4YtuzLmPgn4N0ccvKuLp8Gdp7MMon7MwdT6OfD8TX723ukzSgBWS76hb5yuE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389139; c=relaxed/simple; bh=EyUDdenA2PgGa7+xcdmhQ/lN4jJ+bH9xLwGVwaO34zE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AzdeKTPQhnQLN5oerG1mhJmlnP+8S9o9Z+PJYjKl0T2rlIl3FE78BJGTOh7e210tMwSSBvaot+IFy3B/WvXh0Bh2PKaHmyneucaA4sOSEm3S702NvRmiqz0Pde6CJubZUZ8Y1lUBVxGm7yHB0NdASHPTAgd9qbDii8JplPwI3d4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=o+PtNzf2; arc=none smtp.client-ip=91.218.175.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="o+PtNzf2" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=EyUDdenA2PgGa7+xcdmhQ/lN4jJ+bH9xLwGVwaO34zE=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389125; v=1; x=1789993925; b=o+PtNzf27GE/b+iZOhJuMYSGaw+SoU61D9gUGwQcvTVIZWrSjtJuBRiCJ2PAOJC+2VWCyTlv cNAL0LaLQGg/xTldtn+p2raeydbzDlhqaguUwgQhM7V4QlBHQu7cs8BlLnavWU1GBPfH8p23rAs UxVA9wNFUUVGZCM+AiIGH4iY= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id c218f079dab4b45a; Mon, 14 Sep 2026 12:32:05 +0000 X-Mizu-Trace-ID: c218f079dab4b45a X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 24/29] mm: handle PMD swap entries in MADV_WILLNEED Date: Mon, 14 Sep 2026 05:28:14 -0700 Message-ID: <20260914122950.3283997-25-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" swapin_walk_pmd_entry() walks PTEs and cannot descend into a non-present PMD, so MADV_WILLNEED is a no-op on a PMD swap entry. Reading the slots back one at a time would not be much better: order-0 readahead populates per-page swap-cache state, and the fault that follows would then have to split the entry the prefetch was supposed to help. Prefetch the whole range as one PMD-order folio instead, leaving the entry in place so the eventual fault still takes do_huge_pmd_swap_page(). Check the THP policy first: the range was PMD-mapped when it was swapped out, but MADV_NOHUGEPAGE or the sysfs knob may have changed since, and reading at PMD order would then hand do_swap_page() a PMD-sized folio to map with PTEs. A split cache, per-page zswap state, or a failed PMD-order read or allocation falls back to PTEs. Leaving the entry alone would prefetch nothing at all, while the PTE path can still read the slots at order 0. A folio that failed to read is dropped from the swap cache first, so the PTE retry re-reads each slot rather than every later fault returning SIGBUS. Only split if the PMD still looks like the entry that was observed - the decision is taken after the PMD lock has been dropped, so a racing fault may have swapped the range back in as a THP. The test is lockless, so it narrows that window rather than closing it. Signed-off-by: Usama Arif --- mm/madvise.c | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) diff --git a/mm/madvise.c b/mm/madvise.c index 64019668d69fb..ec1cb1b71070a 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -33,6 +33,7 @@ #include #include #include +#include =20 #include =20 @@ -194,6 +195,108 @@ static int madvise_update_vma(vm_flags_t new_flags, } =20 #ifdef CONFIG_SWAP +/* + * Prefetch a whole PMD swap entry as one PMD-order folio. + * + * Called with the PMD lock held; always drops it. Returns true when the + * caller should ask the walker to retry so the PTE path can handle the + * covered slots individually. + */ +static bool swapin_pmd_swap_entry(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, softleaf_t entry, + spinlock_t *ptl) +{ + struct vm_fault vmf =3D { + .vma =3D vma, + .address =3D addr, + .real_address =3D addr, + .pmd =3D pmd, + }; + enum swap_pmd_cache cache_state; + struct swap_info_struct *si; + struct folio *folio; + bool split =3D false; + + /* + * The range was PMD-mapped when it was swapped out, but the policy may + * have changed since: MADV_NOHUGEPAGE, or the sysfs knob. Reading it + * back at PMD order would then hand do_swap_page() a PMD-sized folio to + * map with PTEs, which is exactly what the policy forbids. Split now + * and let the PTE path prefetch at order 0 - the next fault would split + * the entry anyway, so nothing is lost that the VMA still permits. + */ + if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_PAGEFAULT, + HPAGE_PMD_ORDER)) { + spin_unlock(ptl); + return true; + } + + cache_state =3D swap_pmd_cache_lookup(entry, &folio); + if (cache_state =3D=3D SWAP_PMD_CACHE_HUGE) { + /* Already cached as one PMD-sized folio, nothing to do. */ + folio_put(folio); + spin_unlock(ptl); + return false; + } + if (cache_state =3D=3D SWAP_PMD_CACHE_SPLIT || + zswap_is_present(entry, HPAGE_PMD_NR)) { + spin_unlock(ptl); + return true; + } + + /* + * Pin the swap device under the PMD lock so the PMD-swap-entry + * observation keeps the entry valid for swapin_sync(). + */ + si =3D get_swap_device(entry); + spin_unlock(ptl); + if (IS_ERR_OR_NULL(si)) + return false; + + folio =3D swapin_sync(entry, GFP_HIGHUSER_MOVABLE, BIT(HPAGE_PMD_ORDER), + &vmf, NULL, 0); + + /* + * Fall back to PTE-order swapin: a PMD-order failure does not mean + * that individual slots cannot be read. + */ + if (IS_ERR_OR_NULL(folio)) { + split =3D true; + goto out; + } + + if (folio_nr_pages(folio) !=3D HPAGE_PMD_NR) { + split =3D true; + goto out_put; + } + + /* + * A trylock only succeeds once the read has completed, so this never + * blocks MADV_WILLNEED on in-flight I/O. A read that failed - a + * PMD-order zswap load that found per-page state, or an I/O error - + * leaves the folio clean and not uptodate. Drop it from the swap cache + * so the PTE retry reads each slot again; leaving it there would make + * the next fault return VM_FAULT_SIGBUS. Another thread may have + * removed it already, so revalidate the association first. + */ + if (!folio_trylock(folio)) + goto out_put; + + if (!folio_test_uptodate(folio)) { + if (folio_matches_swap_entry(folio, entry)) + swap_cache_del_folio(folio); + split =3D true; + } + folio_unlock(folio); + +out_put: + folio_put(folio); +out: + /* Keep the device pinned until the last use of @entry. */ + put_swap_device(si); + return split; +} + static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, unsigned long end, struct mm_walk *walk) { @@ -203,6 +306,41 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned = long start, spinlock_t *ptl; unsigned long addr; =20 + ptl =3D pmd_trans_huge_lock(pmd, vma); + if (ptl) { + pmd_t pmdval =3D *pmd; + + if (pmd_is_swap_entry(pmdval)) { + /* swapin_pmd_swap_entry() always drops the PMD lock. */ + if (!swapin_pmd_swap_entry(vma, pmd, start, + softleaf_from_pmd(pmdval), + ptl)) + goto ret; + /* + * Only split if this still looks like the entry we + * observed. The fallback was decided after the PMD lock + * was dropped, so a racing fault may have swapped the + * range back in as a THP, and splitting that would + * demote a perfectly good huge mapping for an advisory + * hint. The test is lockless, so it narrows that window + * rather than closing it. + */ + if (pmd_same(pmdval, pmdp_get_lockless(pmd))) { + __split_huge_pmd(vma, pmd, start); + walk->action =3D ACTION_AGAIN; + goto ret; + } + /* + * Somebody else changed the PMD. Leave it alone and let + * the PTE loop below deal with whatever is there now; + * it simply finds no page table if the range came back + * as a THP. + */ + } else { + spin_unlock(ptl); + } + } + for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { pte_t pte; softleaf_t entry; @@ -231,6 +369,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned l= ong start, if (ptep) pte_unmap_unlock(ptep, ptl); swap_read_submit(&ctx); +ret: cond_resched(); =20 return 0; --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-65.mta0.migadu.com [91.218.175.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72A143546D8 for ; Mon, 14 Sep 2026 12:32:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.65 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389138; cv=none; b=kZVGDeYFJ79qI8vVutx9Qjzpt53RsEr3zmVjdENcoDcZvzSQknfWsV92t36oywzb3NcxSRJc6MtAsveiGhHMhUPFQKPrtRlcS1OMa7ej6/bKa7aWZlyZfKtOfeCtt/ZTRIk51F4iM771R7WrGhQj+EAxDPouo3rOLExNaAGpbqE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389138; c=relaxed/simple; bh=l9rS6hCS40oLFaHf8DDPyFgRnX3di/CMgWWuQWOguRk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aANpivTL4vM7eLPqS0T2dSVGLigj5mkRpjXF11/l5KeFeLA1zIkWmKdKR/F5IjcuolpXE6RwUjfPjENvuvwNUQeI0g5jH6ip3d5QzvJDzASs7FZM+i8BW7dYHJ2aZrar8dRv5RYsjXPToBVhiLpflfQuYWK9nEZNRk4EFZC/o18= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=w3Vsb0hN; arc=none smtp.client-ip=91.218.175.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="w3Vsb0hN" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=l9rS6hCS40oLFaHf8DDPyFgRnX3di/CMgWWuQWOguRk=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389128; v=1; x=1789993928; b=w3Vsb0hNhcuTJVOFM1gCOUpSQravM71il3YZg6kCY59NTZkrirvExSnvUra2Aqsfiyau6eXW 9eC1Rvz/ww0JpAVNsTDT9jOSCaFDK20Z+DS+FwkjLwEs7aWv+/JqE67YvEzzdOE3wkFjIxTuFRh ptrwjleA8srVMWg06h52LqsY= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id def300bf9b7a4cdf; Mon, 14 Sep 2026 12:32:07 +0000 X-Mizu-Trace-ID: def300bf9b7a4cdf X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 25/29] mm: handle PMD swap entries in UFFDIO_MOVE Date: Mon, 14 Sep 2026 05:28:15 -0700 Message-ID: <20260914122950.3283997-26-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" move_pages_huge_pmd() returns -ENOENT for any PMD that is neither trans_huge nor a migration entry, so an aligned UFFDIO_MOVE over a swapped-out THP fails even though a PMD swap entry is a perfectly good mapping to move. Falling back to the PTE path is no help either: splitting yields PTE swap entries pointing at the same swap-cache folio, and move_pages_ptes() refuses any swap-cache folio that is still large. move_swap_pmd() is modelled on move_swap_pte(): it moves the entry under both PMD locks, propagates soft-dirty, arms the UFFD marker for an RWP-registered destination, carries the deposited page table across, and requires pmd_swp_exclusive() for the same single-owner semantics. The entry can only be moved whole while the covered swap cache is empty or holds one PMD-sized folio. A cached folio is locked and revalidated, then its anon rmap is re-anchored to the destination VMA; an empty cache is re-checked slot by slot under both PMD locks, because a per-slot folio that appeared meanwhile would need the PTE path to fix up its rmap metadata. A range that is already split is split and retried through PTEs. Revalidation failure just returns -EAGAIN: its usual cause is a racing fault that made src_pmd a healthy present THP, which must not be shattered. Finally, reject a PMD swap entry at the *destination* with -EEXIST. It is not a hole, and unlike a migration entry it does not resolve on its own: pte_alloc() skips a !pmd_none PMD, pte_offset_map_rw_nolock() then fails, and the resulting -EAGAIN would be retried forever. Signed-off-by: Usama Arif --- mm/huge_memory.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++- mm/userfaultfd.c | 14 +++++ 2 files changed, 171 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5f3d620c64a94..497f677a3ef71 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2972,6 +2972,78 @@ int change_huge_pud(struct mmu_gather *tlb, struct v= m_area_struct *vma, #endif =20 #ifdef CONFIG_USERFAULTFD +#ifdef CONFIG_THP_SWAP +/* + * Move a PMD-level swap entry from src_pmd to dst_pmd. Both PMD locks are + * acquired here; src_folio (if present) must already be locked. The depos= ited + * page table backing the source THP is moved across with the entry. + */ +static int move_swap_pmd(struct mm_struct *mm, struct vm_area_struct *dst_= vma, + unsigned long dst_addr, unsigned long src_addr, + pmd_t *dst_pmd, pmd_t *src_pmd, + pmd_t orig_dst_pmd, pmd_t orig_src_pmd, + spinlock_t *dst_ptl, spinlock_t *src_ptl, + struct folio *src_folio, swp_entry_t entry) +{ + pgtable_t src_pgtable; + pmd_t moved_pmd; + + /* + * The folio may have been freed and reused for a different swap entry + * while it was unlocked. Re-verify the association. + */ + if (src_folio && unlikely(!folio_matches_swap_entry(src_folio, entry) || + folio_nr_pages(src_folio) !=3D HPAGE_PMD_NR)) + return -EAGAIN; + + double_pt_lock(dst_ptl, src_ptl); + + if (!pmd_same(*src_pmd, orig_src_pmd) || + !pmd_same(*dst_pmd, orig_dst_pmd)) { + double_pt_unlock(dst_ptl, src_ptl); + return -EAGAIN; + } + + /* + * If the folio is in the swap cache, re-anchor its anon rmap to the + * destination VMA so a future swap-in fault at dst_addr finds it. + * Otherwise, re-check the whole PMD swap range: a PMD swap entry is + * only a compact encoding for HPAGE_PMD_NR swap slots, and any per-slot + * cached folio would need the PTE move path to update its rmap + * metadata. + */ + if (src_folio) { + folio_move_anon_rmap(src_folio, dst_vma); + src_folio->index =3D linear_anon_page_index(dst_vma, dst_addr); + } else { + unsigned int type =3D swp_type(entry); + pgoff_t offset =3D swp_offset(entry); + int i; + + for (i =3D 0; i < HPAGE_PMD_NR; i++) { + if (swap_cache_has_folio(swp_entry(type, offset + i))) { + double_pt_unlock(dst_ptl, src_ptl); + return -EAGAIN; + } + } + } + + moved_pmd =3D pmdp_huge_get_and_clear(mm, src_addr, src_pmd); + if (pgtable_supports_soft_dirty()) + moved_pmd =3D pmd_swp_mksoft_dirty(moved_pmd); + /* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */ + if (userfaultfd_rwp(dst_vma)) + moved_pmd =3D pmd_swp_mkuffd(moved_pmd); + set_pmd_at(mm, dst_addr, dst_pmd, moved_pmd); + + src_pgtable =3D pgtable_trans_huge_withdraw(mm, src_pmd); + pgtable_trans_huge_deposit(mm, dst_pmd, src_pgtable); + + double_pt_unlock(dst_ptl, src_ptl); + return 0; +} +#endif /* CONFIG_THP_SWAP */ + /* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by * the caller, but it must return after releasing the page_table_lock. @@ -3006,11 +3078,95 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, pm } =20 if (!pmd_trans_huge(src_pmdval)) { - spin_unlock(src_ptl); if (pmd_is_migration_entry(src_pmdval)) { + spin_unlock(src_ptl); pmd_migration_entry_wait(mm, src_pmd); return -EAGAIN; } +#ifdef CONFIG_THP_SWAP + if (pmd_is_swap_entry(src_pmdval)) { + swp_entry_t entry; + struct swap_info_struct *si; + enum swap_pmd_cache cache_state; + + /* + * UFFDIO_MOVE on anon mappings requires single-owner + * semantics; refuse to move a shared swap entry. + */ + if (!pmd_swp_exclusive(src_pmdval)) { + spin_unlock(src_ptl); + return -EBUSY; + } + + entry =3D softleaf_from_pmd(src_pmdval); + spin_unlock(src_ptl); + + /* + * Pin the swap device against a racing swapoff. NULL + * means swapoff is in progress, which resolves on its + * own, so ask the caller to retry. An error pointer + * means the entry names no swap device at all: that + * never resolves, so report it instead of spinning in + * the caller's -EAGAIN loop. + */ + si =3D get_swap_device(entry); + if (!si) + return -EAGAIN; + if (IS_ERR(si)) + return PTR_ERR(si); + + src_folio =3D NULL; + cache_state =3D swap_pmd_cache_lookup(entry, &src_folio); + if (cache_state =3D=3D SWAP_PMD_CACHE_SPLIT) { + put_swap_device(si); + __split_huge_pmd(src_vma, src_pmd, src_addr); + return -EAGAIN; + } + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, + mm, src_addr, + src_addr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + if (src_folio) { + folio_lock(src_folio); + /* + * Do not split on failure here. The usual cause + * is that a racing fault swapped the range back + * in and dropped the folio from the swap cache, + * so src_pmd is now a healthy present THP; + * splitting it would destroy the very mapping + * UFFDIO_MOVE is trying to move whole. The + * caller's -EAGAIN retry re-reads src_pmd and + * picks the right path, exactly as + * move_swap_pte() relies on for the PTE case. + */ + if (!folio_matches_swap_entry(src_folio, entry) || + folio_nr_pages(src_folio) !=3D HPAGE_PMD_NR) { + folio_unlock(src_folio); + folio_put(src_folio); + mmu_notifier_invalidate_range_end(&range); + put_swap_device(si); + return -EAGAIN; + } + } + + dst_ptl =3D pmd_lockptr(mm, dst_pmd); + err =3D move_swap_pmd(mm, dst_vma, dst_addr, src_addr, + dst_pmd, src_pmd, dst_pmdval, + src_pmdval, dst_ptl, src_ptl, + src_folio, entry); + + mmu_notifier_invalidate_range_end(&range); + if (src_folio) { + folio_unlock(src_folio); + folio_put(src_folio); + } + put_swap_device(si); + return err; + } +#endif /* CONFIG_THP_SWAP */ + spin_unlock(src_ptl); return -ENOENT; } =20 diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 79cc7b546f130..e9e1df254fd72 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -2053,6 +2053,20 @@ static ssize_t move_pages(struct userfaultfd_ctx *ct= x, unsigned long dst_start, break; } =20 + /* + * A PMD swap entry at dst is a swapped-out THP, not a hole, + * and unlike a PMD migration entry it will not resolve on its + * own. Nothing below faults it back in: pte_alloc() skips a + * !pmd_none PMD, pte_offset_map_rw_nolock() then fails on the + * non-present PMD, and the -EAGAIN that produces would be + * retried forever by the loop below. Be strict, exactly as for + * a present THP. + */ + if (unlikely(pmd_is_swap_entry(dst_pmdval))) { + err =3D -EEXIST; + break; + } + ptl =3D pmd_trans_huge_lock(src_pmd, src_vma); if (ptl) { /* Check if we can move the pmd without splitting it. */ --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-89.mta0.migadu.com [91.218.175.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6AE4472555 for ; Mon, 14 Sep 2026 12:32:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.89 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389140; cv=none; b=BYyjLJg2IGzh2EcT532Mh2ts6L6+9CAzLUFbmX6CRCrjFE5c5v/MOCsGHohS7SPqwDo+VLtjXdmO0cNBqspEqtFLY3vJThXpKzJK/ejjIvrixy9yM2h06Rfsp746rxlNHpWmXl5JizJ+wYlbiF54AG2iB7SOYp5vYlir0X89sks= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389140; c=relaxed/simple; bh=qHCwemBvEHnl6RrkodLgJVOAA4pYdgV5Aeyu0ypk9I0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YaMzLvP4RW7pl+zKAX9lZfbJHXI8hnBwi8ttzJ3QB13saVz5gFvElNsbgkEWO779Iiz5lpfhLWpfiz1e2qss12eGuCZXtAoTmFjD7J+ker1h0TMHIQTpLUCKPRnIGP55BHzy/NHlOO+aWYc/hVpYq6FW2r9yOuk0DduIxMZAZGk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=khIxBPYu; arc=none smtp.client-ip=91.218.175.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="khIxBPYu" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=qHCwemBvEHnl6RrkodLgJVOAA4pYdgV5Aeyu0ypk9I0=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389130; v=1; x=1789993930; b=khIxBPYuAg5FmSdTIET+iSz+77DhfXdHs772s08vMSFKGBywDQZueERFl7DI7CK0If9efImv ZPyIUvWxC3N4VWON4a0G6fU74Zo7yq5wwEhIb49oMvcRbfDpYnQ7CQPzYcTGKLaeJq2Ee+1xr9Y 7YdPKV8nQQ+GGW7CdnRoxiTk= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id e967df5ce3b87961; Mon, 14 Sep 2026 12:32:10 +0000 X-Mizu-Trace-ID: e967df5ce3b87961 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 26/29] mm: don't PTE-batch a swap-in over a hardware-poisoned subpage Date: Mon, 14 Sep 2026 05:28:16 -0700 Message-ID: <20260914122950.3283997-27-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" do_swap_page() checks PageHWPoison() on the faulting page, but the large-folio batching path then maps the rest of the folio without looking at the other subpages. A poisoned subpage that is not the one being faulted gets mapped anyway, and the fault that would have reported it never happens. Skip batching for such a folio and let each page take its own fault, where the existing check catches it. Test every subpage rather than the folio-level PG_has_hwpoisoned: memory_failure() sets PageHWPoison() on the subpage before it acquires the folio lock, which we hold here. Signed-off-by: Usama Arif --- mm/memory.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 84e1e1c22bffa..aa1f67b378587 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5020,7 +5020,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_idx =3D 0; address =3D vmf->address; ptep =3D vmf->pte; - if (folio_test_large(folio) && folio_test_swapcache(folio)) { + /* + * Scan every subpage rather than testing the folio-level + * PG_has_hwpoisoned: memory_failure() sets PageHWPoison on the subpage + * before it takes the folio lock, and we hold that lock, so the + * folio-level flag can still be clear here. + */ + if (folio_test_large(folio) && folio_test_swapcache(folio) && + !folio_has_hwpoisoned_subpage(folio)) { int nr =3D folio_nr_pages(folio); unsigned long idx =3D folio_page_idx(folio, page); unsigned long folio_start =3D address - idx * PAGE_SIZE; --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-112.mta0.migadu.com [91.218.175.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84E67477992 for ; Mon, 14 Sep 2026 12:32:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389144; cv=none; b=tCIjTbQMzGpsbI/5cc6JCBl5p3Ktg7AC30oZM6cB3wSHUI9fY2fAqa+JKeTLHcMoFccQizVqVB/8eKrIroNAgPthbNw64ydkEr1HXU9NvDpOWPYjA8bjqt4n1rx36k2M7uA+bQWr5LyGmHIoPnHAN5SDuNhbGisJL5XROCX7wNM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389144; c=relaxed/simple; bh=i8lsaF6ASDY8IVJmy1+rFL4wXtW1f9Bd1clcvnPDbM0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kuwrav7dsie2z+ut/BEbKM0zFj0KutL8DMtvG8s5CXLKkFZHiVIM15hYiDwiBFPlxf7TA5L56o6kxs7HNaEsUmwau2No1PaLublRScc1xAou2u/Ko4SgOYGeNZmOkHdqFl6Nb2YEiyF4lUoeefYXpZc8rVH65ImWnazFRRKY98k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=WhozPU8k; arc=none smtp.client-ip=91.218.175.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="WhozPU8k" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=i8lsaF6ASDY8IVJmy1+rFL4wXtW1f9Bd1clcvnPDbM0=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389133; v=1; x=1789993933; b=WhozPU8kVxFHlfhi5ZWXIkUXVKdopkuQRPCgeMvsEDrp3n2qSrnunZbVeI7uAXFGBV6KTY3L rGyXfqHBOvmTdAfmVx/QGxEXYXktuIdefy3YwoNaKNIFSqZE36rsS5eqSKyHS5JThgJqRbHBVKX 03REshFIjnixCbHdDcsFU72c= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta10.migadu.com with ESMTPS id 8157c0f3f9b29dfd; Mon, 14 Sep 2026 12:32:12 +0000 X-Mizu-Trace-ID: 8157c0f3f9b29dfd X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 27/29] mm: handle PMD swap entry faults on swap-in Date: Mon, 14 Sep 2026 05:28:17 -0700 Message-ID: <20260914122950.3283997-28-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Nothing faults a PMD swap entry back in. __handle_mm_fault() recognises a non-present PMD as device-private or a migration entry and returns 0 for anything else, so a swapped-out THP would refault forever. do_huge_pmd_swap_page() resolves the whole mapping in one go, mirroring do_swap_page() at PMD granularity, and restores soft-dirty, uffd-wp and write permission the same way. It deliberately skips the order-0 readahead paths: the fault already asks for the whole range, and readahead would populate per-page swap-cache state and force the entry to split before the fault could finish. A PMD swap entry only promises HPAGE_PMD_NR consecutive slots, not that the cache still holds one folio for them. When it does not - the VMA no longer permits PMD-order THPs, the cache has been split, a slot is in zswap, the allocation or read fails, the memcg charge fails, or the cache insertion loses a race - split the entry and return 0 so the retry lands in do_swap_page(). That keeps a transient PMD-order allocation failure from becoming VM_FAULT_OOM. Only split if the PMD is still the entry we were called for, since every reason to fall back was observed without the PMD lock. Before falling back with a locked folio, drop it from the swap cache if it has never been mapped or is not uptodate: do_swap_page() would hand an unmapped PMD-sized folio to folio_add_new_anon_rmap() as a whole while installing one PTE, and a folio that failed to read would make every later fault return VM_FAULT_SIGBUS instead of re-reading the slots. An uptodate anon folio stays cached so a poisoned subpage stays visible. This mirrors unuse_pmd_entry(). Refuse to map a folio with a poisoned subpage so do_swap_page() can report VM_FAULT_HWPOISON for the offending page, and drop the exclusive marker when the folio is under writeback to an SWP_STABLE_WRITES backend such as zram, so a later write COWs rather than corrupting the writeback. When the PMD ends up read-only but the fault was a write, call wp_huge_pmd() from the same handler so the COW does not need a second fault. Mask VM_FAULT_FALLBACK out of what it returns: splitting to PTE level is a normal outcome, but the bit is part of VM_FAULT_ERROR and arch fault handlers BUG() on it without an accompanying signal. Signed-off-by: Usama Arif --- include/linux/huge_mm.h | 14 +++ mm/huge_memory.c | 257 ++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 42 +++++++ mm/memory.c | 40 +------ 4 files changed, 319 insertions(+), 34 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 64b6a2eea899d..b44a228dfe20c 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -548,6 +548,15 @@ vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf); =20 vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf); =20 +#ifdef CONFIG_THP_SWAP +vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf); +#else +static inline vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) +{ + return 0; +} +#endif + extern struct folio *huge_zero_folio; extern unsigned long huge_zero_pfn; =20 @@ -747,6 +756,11 @@ static inline vm_fault_t do_huge_pmd_device_private(st= ruct vm_fault *vmf) return 0; } =20 +static inline vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) +{ + return 0; +} + static inline bool is_huge_zero_folio(const struct folio *folio) { return false; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 497f677a3ef71..bd9cc24c2b011 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -42,6 +42,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -2447,6 +2448,262 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *v= mf) return 0; } =20 +#ifdef CONFIG_THP_SWAP +/** + * do_huge_pmd_swap_page() - Handle a fault on a PMD-level swap entry. + * @vmf: Fault context. vmf->orig_pmd contains the swap PMD. + * + * A PMD swap entry is a compact encoding for HPAGE_PMD_NR consecutive swap + * slots. If the swap cache still has one PMD-sized folio covering the ran= ge, + * map it directly at PMD level. If the range has been split into per-page + * cache state, or zswap may have per-page state for it, split the PMD swap + * entry and retry at PTE granularity. + * + * Return: VM_FAULT_* flags. + */ +vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) +{ + struct vm_area_struct *vma =3D vmf->vma; + struct mm_struct *mm =3D vma->vm_mm; + struct folio *folio; + struct page *page; + struct swap_info_struct *si; + unsigned long haddr =3D vmf->address & HPAGE_PMD_MASK; + softleaf_t entry; + swp_entry_t swp_entry; + pmd_t pmd; + vm_fault_t ret =3D 0; + bool exclusive, stable_writes, rwp_restore =3D false; + bool write =3D vmf->flags & FAULT_FLAG_WRITE; + rmap_t rmap_flags =3D RMAP_NONE; + enum swap_pmd_cache cache_state; + + entry =3D softleaf_from_pmd(vmf->orig_pmd); + if (unlikely(!softleaf_is_swap(entry))) + return 0; + + if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_PAGEFAULT, + HPAGE_PMD_ORDER)) { + __split_huge_pmd(vma, vmf->pmd, haddr); + return 0; + } + + swp_entry =3D entry; + + /* Prevent swapoff from happening to us. */ + si =3D get_swap_device(swp_entry); + if (IS_ERR_OR_NULL(si)) { + if (IS_ERR(si)) + return VM_FAULT_SIGBUS; + return 0; + } + + cache_state =3D swap_pmd_cache_lookup(swp_entry, &folio); + if (cache_state =3D=3D SWAP_PMD_CACHE_SPLIT) + goto split_fallback; + if (!folio) { + /* + * PMD swap entries encode ordinary per-page swap slots. If any + * slot is in zswap, split and let the PTE swap path load the + * range per page. Otherwise the range is all on disk and can be + * read back as one PMD-sized folio. + */ + if (zswap_is_present(swp_entry, HPAGE_PMD_NR)) + goto split_fallback; + + folio =3D swapin_sync(swp_entry, GFP_HIGHUSER_MOVABLE, + BIT(HPAGE_PMD_ORDER), vmf, NULL, 0); + if (IS_ERR_OR_NULL(folio)) + goto split_fallback; + + /* Had to read from swap area: Major fault */ + ret =3D VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + } + + ret |=3D folio_lock_or_retry(folio, vmf); + if (ret & VM_FAULT_RETRY) + goto out_release; + + /* Verify the folio is still in swap cache and matches our entry */ + if (unlikely(!folio_matches_swap_entry(folio, swp_entry))) + goto out_page; + + /* + * Folio should be PMD-sized; if not (e.g. split in swap cache), + * split the PMD swap entry and retry at PTE level. + */ + if (folio_nr_pages(folio) !=3D HPAGE_PMD_NR) + goto unlock_split_fallback; + + /* + * A read that failed - a PMD-order zswap load that found per-page + * state, or an I/O error - leaves the folio clean and not uptodate. + * Fall back so the PTE retry reads each slot again rather than + * returning SIGBUS for the whole range. + */ + if (unlikely(!folio_test_uptodate(folio))) + goto unlock_split_fallback; + + /* + * If any subpage is hardware-poisoned, split the PMD swap entry and + * let the PTE swap-in path handle each page individually so + * do_swap_page() can return VM_FAULT_HWPOISON for the poisoned + * subpage rather than mapping the corrupted memory as one THP. + */ + if (unlikely(folio_has_hwpoisoned_subpage(folio))) + goto unlock_split_fallback; + + page =3D folio_page(folio, 0); + arch_swap_restore(folio_swap(swp_entry, folio), folio); + + folio_throttle_swaprate(folio, GFP_KERNEL); + + /* Lock the PMD and verify it hasn't changed */ + vmf->ptl =3D pmd_lock(mm, vmf->pmd); + if (unlikely(!pmd_same(vmf->orig_pmd, pmdp_get(vmf->pmd)))) { + spin_unlock(vmf->ptl); + goto out_page; + } + + exclusive =3D pmd_swp_exclusive(vmf->orig_pmd); + + /* + * Some swap backends (e.g. zram) don't support concurrent page + * modifications while under writeback. If we map exclusive on such + * a backend while the folio is still under writeback, the writeback + * may see partial modifications and corrupt the swap slot. Drop the + * exclusive marker and only map R/O for that case; further GUP + * references can't appear once the page is fully unmapped, so this + * is safe. + */ + /* Lockless like do_swap_page(): SWP_STABLE_WRITES never changes. */ + stable_writes =3D data_race(si->flags & SWP_STABLE_WRITES); + if (exclusive && folio_test_writeback(folio) && stable_writes) + exclusive =3D false; + + /* + * Set up the PMD mapping. Similar to do_swap_page() but at PMD level. + */ + add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR); + + pmd =3D folio_mk_pmd(folio, vma->vm_page_prot); + pmd =3D pmd_mkyoung(pmd); + + if (pmd_swp_soft_dirty(vmf->orig_pmd)) + pmd =3D pmd_mksoft_dirty(pmd); + if (pmd_swp_uffd(vmf->orig_pmd)) + pmd =3D pmd_mkuffd(pmd); + if (pmd_swp_uffd(vmf->orig_pmd) && userfaultfd_rwp(vma)) { + pmd =3D pmd_modify(pmd, PAGE_NONE); + rwp_restore =3D true; + } + + /* + * Check exclusivity to determine if we can map writable. + */ + if (exclusive) { + if (!rwp_restore && (vma->vm_flags & VM_WRITE) && + !userfaultfd_huge_pmd_wp(vma, pmd) && + !pmd_needs_soft_dirty_wp(vma, pmd)) { + pmd =3D pmd_mkwrite(pmd, vma); + if (write) + pmd =3D pmd_mkdirty(pmd); + } + rmap_flags |=3D RMAP_EXCLUSIVE; + } + + flush_icache_pages(vma, page, HPAGE_PMD_NR); + + if (!folio_test_anon(folio)) + folio_add_new_anon_rmap(folio, vma, haddr, rmap_flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, haddr, rmap_flags); + + folio_put_swap(folio, NULL); + + set_pmd_at(mm, haddr, vmf->pmd, pmd); + update_mmu_cache_pmd(vma, haddr, vmf->pmd); + + /* Update orig_pmd for any follow-up wp_huge_pmd() below. */ + vmf->orig_pmd =3D pmd; + + /* + * Conditionally try to free up the swap cache. Do it after mapping, + * so raced page faults will likely see the folio in swap cache and + * wait on the folio lock. + */ + if (should_try_to_free_swap(si, folio, vma, exclusive, vmf->flags)) + folio_free_swap(folio); + + spin_unlock(vmf->ptl); + + folio_unlock(folio); + put_swap_device(si); + + /* + * If the write fault wasn't satisfied above (folio is shared without + * exclusivity), call wp_huge_pmd() to handle COW or + * userfaultfd-wp without forcing a second fault. + * + * wp_huge_pmd() may return VM_FAULT_FALLBACK if it had to split the + * PMD; that's a normal outcome, and the natural PTE-level refault will + * complete the COW. Mask it so callers (and the arch fault handler) + * don't see VM_FAULT_FALLBACK as a fatal VM_FAULT_ERROR. + */ + if (write && !pmd_write(pmd) && !rwp_restore) { + vm_fault_t wp_ret =3D wp_huge_pmd(vmf); + + wp_ret &=3D ~VM_FAULT_FALLBACK; + ret |=3D wp_ret; + if (ret & VM_FAULT_ERROR) + ret &=3D VM_FAULT_ERROR; + } + + return ret; + +out_page: + folio_unlock(folio); +out_release: + folio_put(folio); + put_swap_device(si); + return ret; + +unlock_split_fallback: + /* + * PTE fallback cannot add a single-page rmap to a PMD-sized folio that + * has never been mapped: do_swap_page() would hand the whole folio to + * folio_add_new_anon_rmap() while installing one PTE. Nor can it do + * anything useful with a folio that failed to read. Remove either from + * the swap cache so each slot is read back into its own order-0 folio. + * An uptodate anon swap-cache folio can be mapped one PTE at a time and + * must stay cached, so that any poisoned subpage stays visible to + * do_swap_page(). This mirrors unuse_pmd_entry(). + */ + if (folio_matches_swap_entry(folio, swp_entry) && + (!folio_test_uptodate(folio) || !folio_test_anon(folio))) + swap_cache_del_folio(folio); + folio_unlock(folio); + folio_put(folio); + +split_fallback: + /* + * Only split if the PMD is still the swap entry we were called for. + * All the reasons we get here (allocation failure, zswap state, a + * split or poisoned cached folio) were observed without the PMD lock, + * so a racing thread may already have swapped the range back in as a + * THP -- splitting that would silently demote a perfectly good huge + * mapping. + */ + if (pmd_same(vmf->orig_pmd, pmdp_get_lockless(vmf->pmd))) + __split_huge_pmd(vma, vmf->pmd, haddr); + put_swap_device(si); + return 0; +} +#endif /* CONFIG_THP_SWAP */ + static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd) { pgtable_t pgtable; diff --git a/mm/internal.h b/mm/internal.h index ec7f007bc2c0d..1a5480e4b5071 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -577,6 +577,48 @@ static inline vm_fault_t vmf_anon_prepare(struct vm_fa= ult *vmf) } =20 vm_fault_t do_swap_page(struct vm_fault *vmf); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +vm_fault_t wp_huge_pmd(struct vm_fault *vmf); +#else +static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) +{ + return VM_FAULT_FALLBACK; +} +#endif + +/* + * Check if we should call folio_free_swap to free the swap cache. + * folio_free_swap only frees the swap cache to release the slot if swap + * count is zero, so we don't need to check the swap count here. + */ +static inline bool should_try_to_free_swap(struct swap_info_struct *si, + struct folio *folio, + struct vm_area_struct *vma, + bool exclusive, + unsigned int fault_flags) +{ + if (!folio_test_swapcache(folio)) + return false; + /* + * Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap + * cache can help save some IO or memory overhead, but these devices + * are fast, and meanwhile, swap cache pinning the slot deferring the + * release of metadata or fragmentation is a more critical issue. + */ + if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) + return true; + if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || + folio_test_mlocked(folio)) + return true; + + /* + * Free the swapcache only if we are the exclusive user and + * this is a write fault. + */ + return (fault_flags & FAULT_FLAG_WRITE) && exclusive; +} + void folio_rotate_reclaimable(struct folio *folio); bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); diff --git a/mm/memory.c b/mm/memory.c index aa1f67b378587..63b51ba46b0b5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4603,38 +4603,6 @@ static vm_fault_t remove_device_exclusive_entry(stru= ct vm_fault *vmf) return 0; } =20 -/* - * Check if we should call folio_free_swap to free the swap cache. - * folio_free_swap only frees the swap cache to release the slot if swap - * count is zero, so we don't need to check the swap count here. - */ -static inline bool should_try_to_free_swap(struct swap_info_struct *si, - struct folio *folio, - struct vm_area_struct *vma, - bool exclusive, - unsigned int fault_flags) -{ - if (!folio_test_swapcache(folio)) - return false; - /* - * Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap - * cache can help save some IO or memory overhead, but these devices - * are fast, and meanwhile, swap cache pinning the slot deferring the - * release of metadata or fragmentation is a more critical issue. - */ - if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) - return true; - if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || - folio_test_mlocked(folio)) - return true; - - /* - * Free the swapcache only if we are the exclusive user and - * this is a write fault. - */ - return (fault_flags & FAULT_FLAG_WRITE) && exclusive; -} - static vm_fault_t pte_marker_clear(struct vm_fault *vmf) { vmf->pte =3D pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, @@ -6362,8 +6330,8 @@ static inline vm_fault_t create_huge_pmd(struct vm_fa= ult *vmf) return VM_FAULT_FALLBACK; } =20 -/* `inline' is required to avoid gcc 4.1.2 build error */ -static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +vm_fault_t wp_huge_pmd(struct vm_fault *vmf) { struct vm_area_struct *vma =3D vmf->vma; const bool unshare =3D vmf->flags & FAULT_FLAG_UNSHARE; @@ -6393,6 +6361,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault = *vmf) =20 return VM_FAULT_FALLBACK; } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ =20 static vm_fault_t create_huge_pud(struct vm_fault *vmf) { @@ -6656,6 +6625,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_st= ruct *vma, =20 if (pmd_is_migration_entry(vmf.orig_pmd)) pmd_migration_entry_wait(mm, vmf.pmd); + else if (IS_ENABLED(CONFIG_THP_SWAP) && + pmd_is_swap_entry(vmf.orig_pmd)) + return do_huge_pmd_swap_page(&vmf); return 0; } if (pmd_trans_huge(vmf.orig_pmd)) { --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-118.mta0.migadu.com [91.218.175.118]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72A27463B84 for ; Mon, 14 Sep 2026 12:32:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.118 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389144; cv=none; b=k9aVAh3/5RPVqsXeRSe7sDLQ4cwYv+ELfIwHsBfgcyu3d5gVhv4oqa8Axk5VXR//T9sovuq9KoKzntEEm7I3jqwAUbyZoZSJ9IbR++nX2n2w6Na5rYoq+7LcaZzzwMC/SHNBzIX8aUrgpQncinDqum5S42vzs6CV38jmKCBG0UU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389144; c=relaxed/simple; bh=DPe1I53E+kRlfcEsvDmSLP4EjuaTV/Jtl8i3KROoEu4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nkhN4qZVg+OFoFVrpU+D1KYLVzrryttjflcN7P7ouVvR/eI7gt4wwajXlAlDDeYWfTG/6lkd+jRsUpxeEzEn00r7aFWnnfTTW0l33Q5lk/PI8EPtbq0IlyxSOe183DpkjbjyXqphBOOHw/ozW49NAdSvbXriO/SWJuYXBwDtbaE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=cWHd3aIb; arc=none smtp.client-ip=91.218.175.118 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="cWHd3aIb" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=DPe1I53E+kRlfcEsvDmSLP4EjuaTV/Jtl8i3KROoEu4=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389134; v=1; x=1789993934; b=cWHd3aIbAE0S4Gf3wLWy2lcbND2U0Gtf2X92xQsL+kp1icn5x4TtyI7/kjNmTch0G5/Sljpa 1oSfY9kiYacqfXYv1Um1K6+oS12w6khouByqCjHHOPPlnVJStAAforDABS+7bj4lhMvqTG2ihUM tUfdhFhNPYsjpFbPQ1GkJy80= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta11.migadu.com with ESMTPS id 374a9e1bbe50b91d; Mon, 14 Sep 2026 12:32:14 +0000 X-Mizu-Trace-ID: 374a9e1bbe50b91d X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 28/29] mm: install PMD swap entries on swap-out Date: Mon, 14 Sep 2026 05:28:18 -0700 Message-ID: <20260914122950.3283997-29-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When reclaim swaps out a PMD-mapped anonymous THP it first splits the PMD via TTU_SPLIT_HUGE_PMD. The huge mapping is lost for the whole swap round-trip: swapping the range back in takes HPAGE_PMD_NR faults and leaves as many small mappings, and the process waits for khugepaged to collapse them again. The PMD does not have to be split. A contiguous run of slots was already secured when the folio was added to the swap cache - a non-contiguous allocation would have split the folio first - so the whole mapping can be replaced by one PMD-level swap entry encoding the first slot. shrink_folio_list() therefore stops asking for TTU_SPLIT_HUGE_PMD for a PMD-mappable folio already in the swap cache, and try_to_unmap_one() grows a PMD branch. TTU_SPLIT_HUGE_PMD remains the fallback for everything else. set_pmd_swap_entry() is deliberately close in shape to set_pmd_migration_entry(): invalidate the mapping while keeping the original for rollback, take a swap reference on every slot, transfer the exclusive state, propagate the dirty bit to the folio so writeback is not lost, add the mm to mmlist before the entry becomes visible, and carry over soft-dirty and uffd-wp. Any step that can fail restores the mapping first. The entry encodes exactly what the PTE entries would, so swap_map accounting is unchanged: each slot carries a count of one, released individually on a later split or together on swap-in. zswap needs no handling here. It stores the folio as order-0 entries, and the PMD-order swap-in users split and fall back to PTEs if any covered slot turns out to have a zswap entry. thp_swpout_pmd counts PMD mappings replaced this way. Unlike thp_swpout it counts mappings rather than folios, so a fork-shared THP can increment it once per mapping. Signed-off-by: Usama Arif --- Documentation/admin-guide/mm/transhuge.rst | 5 ++ include/linux/huge_mm.h | 2 + include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 84 ++++++++++++++++++++++ mm/rmap.c | 19 +++++ mm/vmscan.c | 9 ++- mm/vmstat.c | 1 + 7 files changed, 120 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/adm= in-guide/mm/transhuge.rst index b187d618452f4..64d413d9fd83e 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -632,6 +632,11 @@ thp_swpout is incremented every time a huge page is swapout in one piece without splitting. =20 +thp_swpout_pmd + is incremented every time a PMD mapping is replaced by a PMD-level + swap entry. A fork-shared THP can increment this counter once for each + PMD mapping that is swapped out. + thp_swpout_fallback is incremented if a huge page has to be split before swapout. Usually because failed to allocate some continuous swap space diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b44a228dfe20c..86c65e3a6c5d1 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -550,6 +550,8 @@ vm_fault_t do_huge_pmd_device_private(struct vm_fault *= vmf); =20 #ifdef CONFIG_THP_SWAP vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf); +int set_pmd_swap_entry(struct page_vma_mapped_walk *pvmw, + struct folio *folio); #else static inline vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 2628ccda076a0..f8fd4e13698c3 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -108,6 +108,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, THP_ZERO_PAGE_ALLOC_FAILED, THP_SWPOUT, THP_SWPOUT_FALLBACK, + THP_SWPOUT_PMD, #endif #ifdef CONFIG_BALLOON BALLOON_INFLATE, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bd9cc24c2b011..317d2108105de 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -5736,3 +5736,87 @@ void remove_migration_pmd(struct page_vma_mapped_wal= k *pvmw, struct folio *folio trace_remove_migration_pmd(address, pmd_val(pmde)); } #endif + +#ifdef CONFIG_THP_SWAP +/** + * set_pmd_swap_entry() - Replace a PMD mapping with a PMD-level swap entr= y. + * @pvmw: Page vma mapped walk context, must have pvmw->pmd set and + * pvmw->pte NULL (i.e. PMD-mapped). + * @folio: The folio being swapped out. Must be in the swap cache. + * + * This installs a PMD-level swap entry in place of a present PMD mapping, + * avoiding the need to split the PMD into PTE-level swap entries. + * + * Return: 0 on success, negative error code on failure. + */ +int set_pmd_swap_entry(struct page_vma_mapped_walk *pvmw, + struct folio *folio) +{ + struct vm_area_struct *vma =3D pvmw->vma; + struct mm_struct *mm =3D vma->vm_mm; + unsigned long address =3D pvmw->address; + unsigned long haddr =3D address & HPAGE_PMD_MASK; + struct page *page =3D folio_page(folio, 0); + bool anon_exclusive; + pmd_t pmdval; + swp_entry_t entry; + pmd_t pmdswp; + + /* + * try_to_unmap_one() only gets here for a PMD-mapped, anonymous, + * PMD-sized folio that is already in the swap cache, and a swapcache + * folio is always swapbacked. Refuse instead of crashing should that + * ever stop being true: the caller aborts the rmap walk and the folio + * simply stays mapped. + */ + if (unlikely(!pvmw->pmd || pvmw->pte || + !folio_test_anon(folio) || + !folio_test_swapcache(folio) || + !folio_test_swapbacked(folio) || + folio_nr_pages(folio) !=3D HPAGE_PMD_NR)) { + VM_WARN_ON_ONCE_FOLIO(true, folio); + return -EBUSY; + } + + flush_cache_range(vma, haddr, haddr + HPAGE_PMD_SIZE); + + pmdval =3D pmdp_invalidate(vma, haddr, pvmw->pmd); + + /* Update high watermark before we lower rss */ + update_hiwater_rss(mm); + + if (folio_dup_swap(folio, NULL) < 0) { + set_pmd_at(mm, haddr, pvmw->pmd, pmdval); + return -ENOMEM; + } + + /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ + anon_exclusive =3D PageAnonExclusive(page); + if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) { + folio_put_swap(folio, NULL); + set_pmd_at(mm, haddr, pvmw->pmd, pmdval); + return -EBUSY; + } + + mm_prepare_for_swap_entries(mm); + + if (pmd_dirty(pmdval)) + folio_mark_dirty(folio); + + entry =3D folio->swap; + pmdswp =3D softleaf_to_pmd(entry); + if (pmd_soft_dirty(pmdval)) + pmdswp =3D pmd_swp_mksoft_dirty(pmdswp); + if (pmd_uffd(pmdval)) + pmdswp =3D pmd_swp_mkuffd(pmdswp); + if (anon_exclusive) + pmdswp =3D pmd_swp_mkexclusive(pmdswp); + set_pmd_at(mm, haddr, pvmw->pmd, pmdswp); + + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); + + count_vm_event(THP_SWPOUT_PMD); + return 0; +} +#endif /* CONFIG_THP_SWAP */ diff --git a/mm/rmap.c b/mm/rmap.c index feb751e29b992..3745f16cf5716 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2284,6 +2284,25 @@ static bool try_to_unmap_one(struct folio *folio, st= ruct vm_area_struct *vma, goto walk_abort; } =20 +#ifdef CONFIG_THP_SWAP + /* + * If the folio is in the swap cache and we're not + * asked to split, install a PMD-level swap entry. + */ + if (!(flags & TTU_SPLIT_HUGE_PMD) && + folio_test_anon(folio) && + folio_test_swapcache(folio)) { + if (set_pmd_swap_entry(&pvmw, folio)) + goto walk_abort; + + add_mm_counter(mm, MM_ANONPAGES, + -HPAGE_PMD_NR); + add_mm_counter(mm, MM_SWAPENTS, + HPAGE_PMD_NR); + goto walk_done; + } +#endif + if (flags & TTU_SPLIT_HUGE_PMD) { /* * We temporarily have to drop the PTL and diff --git a/mm/vmscan.c b/mm/vmscan.c index c2eb8fa9d5e50..7648a2a0d0813 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1408,7 +1408,14 @@ static unsigned int shrink_folio_list(struct list_he= ad *folio_list, enum ttu_flags flags =3D TTU_BATCH_FLUSH; bool was_swapbacked =3D folio_test_swapbacked(folio); =20 - if (folio_test_pmd_mappable(folio)) + /* + * With THP_SWAP, PMD-mappable folios already in the + * swap cache can be unmapped with a PMD-level swap + * entry, avoiding the cost of splitting the PMD. + */ + if (folio_test_pmd_mappable(folio) && + !(IS_ENABLED(CONFIG_THP_SWAP) && + folio_test_swapcache(folio))) flags |=3D TTU_SPLIT_HUGE_PMD; /* * Without TTU_SYNC, try_to_unmap will only begin to diff --git a/mm/vmstat.c b/mm/vmstat.c index a3e809c57f295..5badcce8ff0ad 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1435,6 +1435,7 @@ const char * const vmstat_text[] =3D { [I(THP_ZERO_PAGE_ALLOC_FAILED)] =3D "thp_zero_page_alloc_failed", [I(THP_SWPOUT)] =3D "thp_swpout", [I(THP_SWPOUT_FALLBACK)] =3D "thp_swpout_fallback", + [I(THP_SWPOUT_PMD)] =3D "thp_swpout_pmd", #endif #ifdef CONFIG_BALLOON [I(BALLOON_INFLATE)] =3D "balloon_inflate", --=20 2.53.0-Meta From nobody Fri Sep 25 10:03:23 2026 Received: from mta0.migadu.com (out-149.mta0.migadu.com [91.218.175.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEA70478E2B for ; Mon, 14 Sep 2026 12:32:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.149 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389151; cv=none; b=kKxaHtLIVMvuI7UGTMZH1Pk7AFoph4bfHbqLMvRl90jVt1G1+tQa7dmL6/tbzHvelCAhNmyVL6tcsLAZmIzbIQzJOpvkgqOwThCeqxshR+UKBN+RYV61fh+cJ+wP6Ar037yuKCr4+NYOTIAwGRZZoLgJ8nOHW4WKTe0Y7Sb9g5Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789389151; c=relaxed/simple; bh=Kq5WPEUBHPtzpUrc0VTAW4N+/tG3GLb7GwrSVhcziMg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oBRVk5vcR7Y/VR9VcEzZ2LoVg5dYvrX/wx0xF+Uz1b6ZZ9n6RV/hmd+eA9dY0hvHiw5/PYn9pvJNhAlZLPeVirljNAq9fLldZGzTjvmCyYLKslUKHOCCNOVX0hvXaXSGw9yWYotfAtqvYMIkjP+IYeHcZxAfg9jV5tdaL57qwg0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=o/Lngni0; arc=none smtp.client-ip=91.218.175.149 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="o/Lngni0" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=Kq5WPEUBHPtzpUrc0VTAW4N+/tG3GLb7GwrSVhcziMg=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789389141; v=1; x=1789993941; b=o/Lngni0FBmsIAaacbCnKLldniKqjR1XAw1g5Wd2lMFqUwPnNToC9wA7v1XDrw9AKjqlAAH5 /4UeDf7WjL4dEFOpTE4rnI7A8ayR/7++b9i9tykyUzJhOxYX66vOPlIobPK15LUow7ZtFOklElG tpi/pPDKNbil4VdlNl4CDOQE= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id f59a0d854ceee8d5; Mon, 14 Sep 2026 12:32:20 +0000 X-Mizu-Trace-ID: f59a0d854ceee8d5 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, qi.zheng@linux.dev, luizcap@redhat.com, kernel-team@meta.com, Usama Arif Subject: [RESEND v7 29/29] selftests/mm: add PMD swap entry tests Date: Mon, 14 Sep 2026 05:28:19 -0700 Message-ID: <20260914122950.3283997-30-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914122950.3283997-1-usama.arif@linux.dev> References: <20260914122950.3283997-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Each test gets a fresh PMD-mapped THP from fixture setup, fills it with a pattern that differs between base pages so a reordered split is detectable, swaps it out with MADV_PAGEOUT, and verifies thp_swpout_pmd increased before the test body runs. The tests are basic, swapin_sync, fork, fork_cow, write, rwp_swapin, munmap, mprotect, split_mprotect, split_munmap, uffdio_move, mremap, pagemap, mincore, madvise_free, madvise_willneed and swapoff. MADV_PAGEOUT leaves the folio in the swap cache on an asynchronous device, so a fault would only remap it and never reach swapin_sync(). Fixture setup therefore asks the current cgroup to reclaim afterwards, and uses mincore() - which reports a PMD swap entry as resident exactly while the cache holds it - to tell whether that worked. Tests run either way; swapin_sync skips if the cache survived. The swapoff test runs only when PMD_SWAP_DEVICE is the sole active swap device, and restores it at its original priority. Skip rather than fail where the kernel never promised anything. thp_swpout_pmd and thp_swpout_fallback are system-wide, so test the fallback counter first: if reclaim split any THP we cannot be sure it was not ours. That also covers a kernel without CONFIG_THP_SWAP. A kernel with no thp_swpout_pmd counter does not implement the feature at all. With zswap enabled the range may legitimately come back through the PTE fallback, so those runs skip the PMD-restoration assertions and say so. Add a /proc/vmstat field reader to vm_util.c so other tests can use it too, and register pmd_swap with run_vmtests.sh and the default runner. Assisted-by: LLM Signed-off-by: Usama Arif --- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 + tools/testing/selftests/mm/pmd_swap.c | 989 ++++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + tools/testing/selftests/mm/vm_util.c | 24 + tools/testing/selftests/mm/vm_util.h | 2 + 6 files changed, 1025 insertions(+) create mode 100755 tools/testing/selftests/mm/ksft_pmd_swap.sh create mode 100644 tools/testing/selftests/mm/pmd_swap.c diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/= mm/Makefile index d3e9bd67904aa..a685988a3d569 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -104,6 +104,7 @@ TEST_GEN_FILES +=3D guard-regions TEST_GEN_FILES +=3D merge TEST_GEN_FILES +=3D rmap TEST_GEN_FILES +=3D folio_split_race_test +TEST_GEN_FILES +=3D pmd_swap TEST_GEN_FILES +=3D soft-dirty =20 ifeq ($(ARCH),x86_64) @@ -162,6 +163,7 @@ TEST_PROGS +=3D ksft_mremap.sh TEST_PROGS +=3D ksft_pagemap.sh TEST_PROGS +=3D ksft_pfnmap.sh TEST_PROGS +=3D ksft_pkey.sh +TEST_PROGS +=3D ksft_pmd_swap.sh TEST_PROGS +=3D ksft_process_madv.sh TEST_PROGS +=3D ksft_process_mrelease.sh TEST_PROGS +=3D ksft_rmap.sh diff --git a/tools/testing/selftests/mm/ksft_pmd_swap.sh b/tools/testing/se= lftests/mm/ksft_pmd_swap.sh new file mode 100755 index 0000000000000..0f070b4729a89 --- /dev/null +++ b/tools/testing/selftests/mm/ksft_pmd_swap.sh @@ -0,0 +1,4 @@ +#!/bin/sh -e +# SPDX-License-Identifier: GPL-2.0 + +./run_vmtests.sh -t pmd_swap diff --git a/tools/testing/selftests/mm/pmd_swap.c b/tools/testing/selftest= s/mm/pmd_swap.c new file mode 100644 index 0000000000000..91bac9ae5560e --- /dev/null +++ b/tools/testing/selftests/mm/pmd_swap.c @@ -0,0 +1,989 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Test PMD-level swap entries and their users. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "kselftest_harness.h" +#include "vm_util.h" + +#define ZSWAP_ENABLED_PATH "/sys/module/zswap/parameters/enabled" + +/* pagemap: bits 0-54 hold the PFN, or type|offset for a swap entry. */ +#define PM_PFRAME_MASK ((1ULL << 55) - 1) +/* Must match MAX_SWAPFILES_SHIFT in include/linux/swap.h. */ +#define MAX_SWAPFILES_SHIFT 5 + +static bool check_swapped(int pagemap_fd, char *addr, unsigned long size) +{ + unsigned long off; + + for (off =3D 0; off < size; off +=3D getpagesize()) + if (!pagemap_is_swapped(pagemap_fd, addr + off)) + return false; + return true; +} + +static bool zswap_enabled(void) +{ + char enabled =3D 0; + FILE *f; + + f =3D fopen(ZSWAP_ENABLED_PATH, "r"); + if (!f) + return false; + + if (fscanf(f, " %c", &enabled) !=3D 1) + enabled =3D 0; + fclose(f); + + return enabled =3D=3D 'Y' || enabled =3D=3D 'y' || enabled =3D=3D '1'; +} + +static bool swap_available(unsigned long required_bytes) +{ + unsigned long required_kb =3D (required_bytes + 1023) / 1024; + unsigned long size_kb, used_kb; + char line[256]; + bool ret =3D false; + FILE *f; + + f =3D fopen("/proc/swaps", "r"); + if (!f) + return false; + + /* Skip the header. */ + if (!fgets(line, sizeof(line), f)) + goto out; + + while (fgets(line, sizeof(line), f)) { + if (sscanf(line, "%*s %*s %lu %lu", &size_kb, &used_kb) =3D=3D 2 && + size_kb >=3D used_kb && size_kb - used_kb >=3D required_kb) { + ret =3D true; + break; + } + } + +out: + fclose(f); + return ret; +} + +static bool same_swap_device(const struct stat *a, const struct stat *b) +{ + if (S_ISBLK(a->st_mode) && S_ISBLK(b->st_mode)) + return a->st_rdev =3D=3D b->st_rdev; + return a->st_dev =3D=3D b->st_dev && a->st_ino =3D=3D b->st_ino; +} + +/* + * Returns true if @swap_dev is the one and only active swap device, and s= tores + * its /proc/swaps priority in *prio so the caller can put it back the way= it + * found it. + */ +static bool swap_device_is_only_active(const char *swap_dev, int *prio) +{ + struct stat expected, active; + char path[256], line[512]; + unsigned int nr_active =3D 0; + bool matches =3D false; + FILE *f; + + *prio =3D -1; + + if (stat(swap_dev, &expected)) + return false; + + f =3D fopen("/proc/swaps", "r"); + if (!f) + return false; + if (!fgets(line, sizeof(line), f)) + goto out; + + while (fgets(line, sizeof(line), f)) { + int line_prio; + + if (sscanf(line, "%255s %*s %*s %*s %d", path, &line_prio) !=3D 2) + continue; + nr_active++; + if (nr_active > 1) + goto out; + matches =3D !stat(path, &active) && + same_swap_device(&expected, &active); + if (matches) + *prio =3D line_prio; + } + +out: + fclose(f); + return nr_active =3D=3D 1 && matches; +} + +/* Re-enable a device swapoff()ed by this test, at its original priority. = */ +static int swapon_restore(const char *swap_dev, int prio) +{ + int flags =3D 0; + + if (prio >=3D 0) + flags =3D SWAP_FLAG_PREFER | + ((prio << SWAP_FLAG_PRIO_SHIFT) & SWAP_FLAG_PRIO_MASK); + return swapon(swap_dev, flags); +} + +/* Locate this task's cgroup-v2 directory. An empty relative path is the r= oot. */ +static bool cgroup2_self_dir(char *buf, size_t len) +{ + char mnt[PATH_MAX] =3D "", type[64], rel[PATH_MAX] =3D ""; + char line[2 * PATH_MAX]; + size_t rel_len; + FILE *f; + + f =3D fopen("/proc/self/mounts", "r"); + if (!f) + return false; + while (fgets(line, sizeof(line), f)) { + if (sscanf(line, "%*s %4095s %63s", mnt, type) =3D=3D 2 && + !strcmp(type, "cgroup2")) + goto found_mnt; + } + mnt[0] =3D '\0'; +found_mnt: + fclose(f); + if (!mnt[0]) + return false; + + f =3D fopen("/proc/self/cgroup", "r"); + if (!f) + return false; + while (fgets(line, sizeof(line), f)) { + if (strncmp(line, "0::", 3)) + continue; + rel_len =3D strcspn(line + 3, "\n"); + if (rel_len && rel_len < sizeof(rel)) { + memcpy(rel, line + 3, rel_len); + rel[rel_len] =3D '\0'; + } + break; + } + fclose(f); + if (!rel[0]) + return false; + if (!strcmp(rel, "/")) + rel[0] =3D '\0'; + return snprintf(buf, len, "%s%s/memory.reclaim", mnt, rel) < (int)len; +} + +static bool cgroup_reclaim(unsigned long bytes) +{ + char path[PATH_MAX], val[32]; + ssize_t written; + int fd, len, err; + + if (!cgroup2_self_dir(path, sizeof(path))) + return false; + + fd =3D open(path, O_WRONLY); + if (fd < 0) + return false; + len =3D snprintf(val, sizeof(val), "%lu", bytes); + written =3D write(fd, val, len); + err =3D errno; + close(fd); + /* -EAGAIN means it reclaimed something but fell short, which is fine. */ + return written =3D=3D len || err =3D=3D EAGAIN; +} + +/* + * mincore() reports 1 over a PMD swap entry iff the swap cache still hold= s it. + * This samples the first slot only: it is a hint used to decide whether t= he + * swap cache was evicted, not an assertion about the whole range. + */ +static bool swapcache_resident(char *mem) +{ + unsigned char vec[1]; + + if (mincore(mem, getpagesize(), vec)) + return true; + return vec[0] & 1; +} + +/* + * MADV_PAGEOUT leaves the folio in the swap cache on asynchronous swap + * devices, so faulting would just remap the cached folio and never reach + * swapin_sync(). Push it out so the PMD-order swap-in path is exercised. + */ +static bool drop_swapcache(char *mem, unsigned long pmd_size) +{ + unsigned long want =3D pmd_size; + int i; + + for (i =3D 0; i < 3 && swapcache_resident(mem); i++, want *=3D 4) + if (!cgroup_reclaim(want)) + break; + return !swapcache_resident(mem); +} + +static unsigned int random_seed(void) +{ + unsigned int seed; + + if (getrandom(&seed, sizeof(seed), 0) !=3D sizeof(seed)) + seed =3D (unsigned int)time(NULL); + return seed; +} + +static unsigned long test_page_size(void) +{ + static unsigned long page_size; + + if (!page_size) + page_size =3D getpagesize(); + return page_size; +} + +/* + * Two base pages of the same PMD must never hold identical bytes, or the = split + * tests cannot tell that the slots came back in the wrong order. A single= byte + * cannot encode a page index on its own - HPAGE_PMD_NR is 8192 on arm64 w= ith + * 64K pages - so spell the index out in the first two bytes of every page. + */ +static unsigned char pattern_byte(unsigned int seed, unsigned long off) +{ + unsigned long page_size =3D test_page_size(); + unsigned long idx =3D off & (page_size - 1); + + if (idx < 2) + return (unsigned char)(seed + ((off / page_size) >> (idx * 8))); + + return (unsigned char)(seed + off + (off >> 8) + (off >> 16)); +} + +static void fill_pattern(char *buf, unsigned long size, unsigned int seed) +{ + unsigned long i; + + for (i =3D 0; i < size; i++) + buf[i] =3D (char)pattern_byte(seed, i); +} + +static bool verify_pattern_range(char *buf, unsigned long size, + unsigned int seed, unsigned long offset) +{ + unsigned long i; + + for (i =3D 0; i < size; i++) + if ((unsigned char)buf[i] !=3D pattern_byte(seed, offset + i)) + return false; + return true; +} + +static bool verify_pattern(char *buf, unsigned long size, unsigned int see= d) +{ + return verify_pattern_range(buf, size, seed, 0); +} + +static bool verify_zero(char *buf, unsigned long size) +{ + unsigned long i; + + for (i =3D 0; i < size; i++) + if (buf[i]) + return false; + return true; +} + +/* + * mmap an anonymous PMD-aligned region of pmd_size bytes. Over-allocates + * by one PMD and trims the unaligned head/tail so the returned address is + * PMD-aligned (required for whole-PMD UFFDIO_MOVE). + */ +static char *mmap_pmd_aligned(unsigned long pmd_size) +{ + unsigned long pad =3D pmd_size; + char *raw, *aligned; + + raw =3D mmap(NULL, pmd_size + pad, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (raw =3D=3D MAP_FAILED) + return MAP_FAILED; + + aligned =3D (char *)(((uintptr_t)raw + pmd_size - 1) & ~(pmd_size - 1)); + if (aligned !=3D raw) + munmap(raw, aligned - raw); + if (aligned + pmd_size !=3D raw + pmd_size + pad) + munmap(aligned + pmd_size, + (raw + pmd_size + pad) - (aligned + pmd_size)); + return aligned; +} + +/* Per-process swapped size in bytes, from /proc/self/status VmSwap. */ +static unsigned long read_vmswap(void) +{ + char line[256]; + unsigned long kb =3D 0; + FILE *f; + + f =3D fopen("/proc/self/status", "r"); + if (!f) + return 0; + while (fgets(line, sizeof(line), f)) { + if (!strncmp(line, "VmSwap:", 7)) { + kb =3D strtoul(line + 7, NULL, 10); + break; + } + } + fclose(f); + return kb * 1024; +} + +/* + * Swap the PMD range out. Returns true if a PMD swap entry was installed. + * On failure *swap_failed is set unless this environment simply cannot sw= ap + * at PMD granularity, in which case the caller should skip rather than fa= il. + */ +static bool swap_out_pmd(char *mem, unsigned long pmd_size, int pagemap_fd, + bool *swap_failed) +{ + long pmd_before =3D read_vmstat("thp_swpout_pmd"); + long fallback_before =3D read_vmstat("thp_swpout_fallback"); + long pmd_after, fallback_after; + bool swapped; + + /* + * A kernel without PMD swap entry support has no thp_swpout_pmd at + * all. Skip rather than report a failure the kernel never promised. + */ + if (pmd_before < 0 || fallback_before < 0) { + ksft_print_msg("no thp_swpout_pmd counter; PMD swap entries unsupported\= n"); + return false; + } + + if (madvise(mem, pmd_size, MADV_PAGEOUT)) { + ksft_print_msg("MADV_PAGEOUT failed: %s\n", strerror(errno)); + *swap_failed =3D true; + return false; + } + + swapped =3D check_swapped(pagemap_fd, mem, pmd_size); + pmd_after =3D read_vmstat("thp_swpout_pmd"); + fallback_after =3D read_vmstat("thp_swpout_fallback"); + ksft_print_msg("thp_swpout_pmd: %ld -> %ld, fallback: %ld -> %ld\n", + pmd_before, pmd_after, fallback_before, fallback_after); + + if (!swapped) { + ksft_print_msg("MADV_PAGEOUT did not swap the whole PMD range\n"); + *swap_failed =3D true; + return false; + } + /* + * Both counters are system-wide, so another task swapping a THP during + * the window above can move either of them. Test the fallback counter + * first: if reclaim split *any* THP we cannot be sure ours was not the + * one, and skipping is the safe direction. That also covers a kernel + * built without CONFIG_THP_SWAP, where folio_alloc_swap() returns + * -E2BIG for every PMD-order folio. + */ + if (fallback_after > fallback_before) { + ksft_print_msg("PMD swap unavailable; reclaim used PTE fallback\n"); + return false; + } + if (pmd_after > pmd_before) + return true; + + *swap_failed =3D true; + return false; +} + +static char *alloc_fill_swap_thp(unsigned long pmd_size, int pagemap_fd, + unsigned int seed, bool *swap_failed) +{ + char *mem; + + *swap_failed =3D false; + + mem =3D mmap_pmd_aligned(pmd_size); + if (mem =3D=3D MAP_FAILED) + return MAP_FAILED; + + if (madvise(mem, pmd_size, MADV_HUGEPAGE)) { + ksft_print_msg("MADV_HUGEPAGE failed: %s\n", strerror(errno)); + munmap(mem, pmd_size); + return MAP_FAILED; + } + fill_pattern(mem, pmd_size, seed); + + if (!check_huge_anon(mem, pmd_size, 1, pmd_size)) { + munmap(mem, pmd_size); + return MAP_FAILED; + } + if (!swap_out_pmd(mem, pmd_size, pagemap_fd, swap_failed)) { + munmap(mem, pmd_size); + return MAP_FAILED; + } + + return mem; +} + +struct rwp_access_args { + unsigned char *addr; + unsigned char expected; + bool write; + bool ok; +}; + +static void *rwp_access_thread(void *data) +{ + struct rwp_access_args *args =3D data; + + if (args->write) + *args->addr =3D args->expected; + args->ok =3D *args->addr =3D=3D args->expected; + return NULL; +} + +static int register_rwp(char *addr, unsigned long size, bool protect) +{ + struct uffdio_register reg =3D {}; + struct uffdio_rwprotect rwp =3D {}; + struct uffdio_api api =3D {}; + int uffd; + + uffd =3D syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + if (uffd < 0) + return -1; + + api.api =3D UFFD_API; + api.features =3D UFFD_FEATURE_RWP; + if (ioctl(uffd, UFFDIO_API, &api) || + !(api.features & UFFD_FEATURE_RWP)) + goto error; + + reg.range.start =3D (unsigned long)addr; + reg.range.len =3D size; + reg.mode =3D UFFDIO_REGISTER_MODE_RWP; + if (ioctl(uffd, UFFDIO_REGISTER, ®)) + goto error; + + if (!protect) + return uffd; + + rwp.range.start =3D (unsigned long)addr; + rwp.range.len =3D size; + rwp.mode =3D UFFDIO_RWPROTECT_MODE_RWP; + if (!ioctl(uffd, UFFDIO_RWPROTECT, &rwp)) + return uffd; + +error: + close(uffd); + return -1; +} + +static bool expect_rwp_fault(int uffd, char *addr, unsigned long size, + unsigned char expected, bool write) +{ + struct rwp_access_args args =3D { + .addr =3D (unsigned char *)addr, + .expected =3D expected, + .write =3D write, + }; + struct uffdio_rwprotect rwp =3D { + .range =3D { + .start =3D (unsigned long)addr, + .len =3D size, + }, + }; + struct pollfd pollfd =3D { + .fd =3D uffd, + .events =3D POLLIN, + }; + struct uffd_msg msg =3D {}; + pthread_t thread; + bool saw_rwp =3D false; + int ret; + + if (pthread_create(&thread, NULL, rwp_access_thread, &args)) + return false; + + ret =3D poll(&pollfd, 1, 5000); + if (ret =3D=3D 1 && (pollfd.revents & POLLIN) && + read(uffd, &msg, sizeof(msg)) =3D=3D (ssize_t)sizeof(msg)) { + saw_rwp =3D msg.event =3D=3D UFFD_EVENT_PAGEFAULT && + (msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_RWP); + } + + /* Resolve the access even on failure so the worker cannot remain blocked= . */ + ioctl(uffd, UFFDIO_RWPROTECT, &rwp); + if (pthread_join(thread, NULL)) + return false; + return saw_rwp && args.ok; +} + +FIXTURE(pmd_swap) +{ + unsigned long pmd_size; + unsigned long mem_len; + int pagemap_fd; + int uffd; + unsigned int seed; + bool zswap_enabled; + bool swap_disabled; + int swap_prio; + bool swapcache_dropped; + const char *swap_dev; + char *mem; + char *aux; +}; + +FIXTURE_SETUP(pmd_swap) +{ + bool swap_failed; + + self->pagemap_fd =3D -1; + self->uffd =3D -1; + self->mem =3D MAP_FAILED; + self->aux =3D MAP_FAILED; + self->mem_len =3D 0; + self->swap_disabled =3D false; + self->swap_prio =3D -1; + self->swapcache_dropped =3D false; + self->swap_dev =3D NULL; + + self->pmd_size =3D read_pmd_pagesize(); + if (!self->pmd_size) + SKIP(return, "Cannot determine PMD size\n"); + + self->pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (self->pagemap_fd < 0) + SKIP(return, "Cannot open /proc/self/pagemap\n"); + + if (!swap_available(self->pmd_size)) + SKIP(return, "No active swap device has enough free space\n"); + + self->seed =3D random_seed(); + self->zswap_enabled =3D zswap_enabled(); + self->mem =3D alloc_fill_swap_thp(self->pmd_size, self->pagemap_fd, + self->seed, &swap_failed); + if (self->mem =3D=3D MAP_FAILED) { + ASSERT_FALSE(swap_failed); + SKIP(return, "Could not create swapped THP\n"); + } + self->mem_len =3D self->pmd_size; + self->swapcache_dropped =3D drop_swapcache(self->mem, self->pmd_size); + ksft_print_msg("swap cache %s the swapped-out THP\n", + self->swapcache_dropped ? "no longer holds" + : "still holds"); +} + +FIXTURE_TEARDOWN(pmd_swap) +{ + int swap_err =3D 0; + int swap_ret =3D 0; + + if (self->swap_disabled) { + swap_ret =3D swapon_restore(self->swap_dev, self->swap_prio); + swap_err =3D errno; + } + if (self->uffd >=3D 0) + close(self->uffd); + if (self->aux !=3D MAP_FAILED) + munmap(self->aux, self->pmd_size); + if (self->mem !=3D MAP_FAILED) + munmap(self->mem, self->mem_len); + if (self->pagemap_fd >=3D 0) + close(self->pagemap_fd); + + EXPECT_EQ(swap_ret, 0) { + TH_LOG("swapon(%s) failed: %s", self->swap_dev, + strerror(swap_err)); + } +} + +TEST_F(pmd_swap, basic) +{ + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); +} + +/* + * With the swap cache evicted, the fault cannot be served by remapping a + * cached folio, so this covers the PMD-order swapin_sync() read. + */ +TEST_F(pmd_swap, swapin_sync) +{ + if (!self->swapcache_dropped) + SKIP(return, "Could not evict the folio from the swap cache\n"); + + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); + if (self->zswap_enabled) + ksft_print_msg("zswap enabled: PMD restoration not checked\n"); + else + ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1, + self->pmd_size)); +} + +TEST_F(pmd_swap, fork) +{ + pid_t pid; + int status; + + pid =3D fork(); + ASSERT_GE(pid, 0); + + if (pid =3D=3D 0) + _exit(verify_pattern(self->mem, self->pmd_size, + self->seed) ? 0 : 1); + + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); + + ASSERT_EQ(waitpid(pid, &status, 0), pid); + ASSERT_TRUE(WIFEXITED(status)); + ASSERT_EQ(WEXITSTATUS(status), 0); +} + +TEST_F(pmd_swap, fork_cow) +{ + unsigned int parent_seed =3D self->seed; + unsigned int child_seed =3D ~self->seed; + unsigned int new_seed =3D self->seed ^ 0xa5a5a5a5; + int release_child[2]; + bool parent_ok; + char c =3D 0; + pid_t pid; + int status, ret; + + ASSERT_EQ(pipe(release_child), 0); + + pid =3D fork(); + ASSERT_GE(pid, 0); + + if (pid =3D=3D 0) { + close(release_child[1]); + if (read(release_child[0], &c, 1) !=3D 1) + _exit(1); + if (!verify_pattern(self->mem, self->pmd_size, parent_seed)) + _exit(2); + fill_pattern(self->mem, self->pmd_size, child_seed); + if (!verify_pattern(self->mem, self->pmd_size, child_seed)) + _exit(3); + _exit(0); + } + + close(release_child[0]); + fill_pattern(self->mem, self->pmd_size, new_seed); + parent_ok =3D verify_pattern(self->mem, self->pmd_size, new_seed); + ret =3D write(release_child[1], &c, 1); + close(release_child[1]); + ASSERT_EQ(waitpid(pid, &status, 0), pid); + ASSERT_EQ(ret, 1); + ASSERT_TRUE(parent_ok); + ASSERT_TRUE(WIFEXITED(status)); + ASSERT_EQ(WEXITSTATUS(status), 0); + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, new_seed)); +} + +TEST_F(pmd_swap, write) +{ + self->mem[0] =3D 0xbb; + ASSERT_EQ(self->mem[0], (char)0xbb); + ASSERT_TRUE(verify_pattern_range(self->mem + 1, self->pmd_size - 1, + self->seed, 1)); + if (self->zswap_enabled) + ksft_print_msg("zswap enabled: PMD restoration not checked\n"); + else + ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1, + self->pmd_size)); +} + +TEST_F(pmd_swap, rwp_swapin) +{ + self->uffd =3D register_rwp(self->mem, self->pmd_size, true); + if (self->uffd < 0) + SKIP(return, "Userfaultfd RWP unsupported\n"); + + ASSERT_TRUE(expect_rwp_fault(self->uffd, self->mem, self->pmd_size, + pattern_byte(self->seed, 0), false)); + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, munmap) +{ + unsigned long swap_before, swap_after; + int ret; + + swap_before =3D read_vmswap(); + ASSERT_GE(swap_before, self->pmd_size); + + ret =3D munmap(self->mem, self->pmd_size); + if (!ret) { + self->mem =3D MAP_FAILED; + self->mem_len =3D 0; + } + ASSERT_EQ(ret, 0); + + swap_after =3D read_vmswap(); + ASSERT_LE(swap_after, swap_before - self->pmd_size); +} + +TEST_F(pmd_swap, mprotect) +{ + ASSERT_EQ(mprotect(self->mem, self->pmd_size, PROT_READ), 0); + ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem, + self->pmd_size)); + ASSERT_EQ(mprotect(self->mem, self->pmd_size, + PROT_READ | PROT_WRITE), 0); + ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem, + self->pmd_size)); + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, split_mprotect) +{ + unsigned long half =3D self->pmd_size / 2; + + ASSERT_EQ(mprotect(self->mem, half, PROT_READ), 0); + ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem, + self->pmd_size)); + ASSERT_EQ(mprotect(self->mem, half, PROT_READ | PROT_WRITE), 0); + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, split_munmap) +{ + unsigned long half =3D self->pmd_size / 2; + unsigned long swap_before =3D read_vmswap(); + unsigned long i; + char *base =3D self->mem; + int ret; + + ASSERT_GE(swap_before, half); + ret =3D munmap(base, half); + if (!ret) { + self->mem =3D base + half; + self->mem_len =3D half; + } + ASSERT_EQ(ret, 0); + ASSERT_LE(read_vmswap(), swap_before - half); + + for (i =3D 0; i < half; i +=3D getpagesize()) + ASSERT_TRUE(pagemap_is_swapped(self->pagemap_fd, + self->mem + i)); + ASSERT_TRUE(verify_pattern_range(self->mem, half, self->seed, half)); +} + +TEST_F(pmd_swap, uffdio_move) +{ + struct uffdio_register reg =3D {}; + struct uffdio_move move =3D {}; + struct uffdio_api api =3D {}; + bool rwp; + + self->aux =3D mmap_pmd_aligned(self->pmd_size); + if (self->aux =3D=3D MAP_FAILED) + SKIP(return, "Could not mmap aligned dst\n"); + ASSERT_EQ(madvise(self->aux, self->pmd_size, MADV_HUGEPAGE), 0); + + self->uffd =3D syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + if (self->uffd < 0) + SKIP(return, "userfaultfd unavailable\n"); + + api.api =3D UFFD_API; + api.features =3D UFFD_FEATURE_MOVE | UFFD_FEATURE_RWP; + if (ioctl(self->uffd, UFFDIO_API, &api) || + !(api.features & UFFD_FEATURE_MOVE)) + SKIP(return, "UFFD_FEATURE_MOVE unsupported\n"); + rwp =3D api.features & UFFD_FEATURE_RWP; + + reg.range.start =3D (unsigned long)self->aux; + reg.range.len =3D self->pmd_size; + reg.mode =3D UFFDIO_REGISTER_MODE_MISSING | + (rwp ? UFFDIO_REGISTER_MODE_RWP : 0); + ASSERT_EQ(ioctl(self->uffd, UFFDIO_REGISTER, ®), 0); + + move.dst =3D (unsigned long)self->aux; + move.src =3D (unsigned long)self->mem; + move.len =3D self->pmd_size; + ASSERT_EQ(ioctl(self->uffd, UFFDIO_MOVE, &move), 0); + ASSERT_EQ(move.move, self->pmd_size); + + ASSERT_TRUE(check_swapped(self->pagemap_fd, self->aux, + self->pmd_size)); + if (rwp) + ASSERT_TRUE(expect_rwp_fault(self->uffd, self->aux, + self->pmd_size, + pattern_byte(self->seed, 0), false)); + ASSERT_TRUE(verify_pattern(self->aux, self->pmd_size, self->seed)); + if (self->zswap_enabled) + ksft_print_msg("zswap enabled: PMD restoration not checked\n"); + else + ASSERT_TRUE(check_huge_anon(self->aux, self->pmd_size, 1, + self->pmd_size)); +} + +TEST_F(pmd_swap, mremap) +{ + char *new_mem, *dst; + + self->aux =3D mmap_pmd_aligned(self->pmd_size); + if (self->aux =3D=3D MAP_FAILED) + SKIP(return, "Could not mmap aligned dst\n"); + dst =3D self->aux; + + new_mem =3D mremap(self->mem, self->pmd_size, self->pmd_size, + MREMAP_MAYMOVE | MREMAP_FIXED, dst); + if (new_mem !=3D MAP_FAILED) { + self->mem =3D new_mem; + self->aux =3D MAP_FAILED; + } + ASSERT_NE(new_mem, MAP_FAILED); + ASSERT_EQ(new_mem, dst); + + ASSERT_TRUE(check_swapped(self->pagemap_fd, new_mem, self->pmd_size)); + ASSERT_TRUE(verify_pattern(new_mem, self->pmd_size, self->seed)); +} + +TEST_F(pmd_swap, pagemap) +{ + uint64_t entry, first; + unsigned long off; + + entry =3D pagemap_get_entry(self->pagemap_fd, self->mem); + ASSERT_TRUE(entry & (1ULL << 62)); + ASSERT_FALSE(entry & (1ULL << 63)); + first =3D entry & PM_PFRAME_MASK; + + /* + * The kernel zeroes the swap type/offset payload for readers without + * CAP_SYS_ADMIN, so the slot-ordering check below would silently test + * nothing. Skip instead of passing vacuously. + */ + if (!first) + SKIP(return, "pagemap swap offsets need CAP_SYS_ADMIN\n"); + + for (off =3D getpagesize(); off < self->pmd_size; off +=3D getpagesize())= { + uint64_t idx =3D off / getpagesize(); + + entry =3D pagemap_get_entry(self->pagemap_fd, self->mem + off); + ASSERT_TRUE(entry & (1ULL << 62)); + ASSERT_FALSE(entry & (1ULL << 63)); + ASSERT_EQ(entry & PM_PFRAME_MASK, + first + (idx << MAX_SWAPFILES_SHIFT)); + } +} + +TEST_F(pmd_swap, mincore) +{ + unsigned long pages =3D self->pmd_size / getpagesize(); + unsigned char vec[pages]; + unsigned long i; + + ASSERT_EQ(mincore(self->mem, self->pmd_size, vec), 0); + /* + * Nothing in this test faults the range, and the fixture already + * evicted the swap cache, so every slot must report not-resident. The + * other direction is not stable - reclaim can drop the cached folio + * between fixture setup and here, and a split cache reports per slot - + * so only check that mincore() answered. + */ + if (self->swapcache_dropped) { + for (i =3D 0; i < pages; i++) + ASSERT_EQ(vec[i] & 1, 0); + } + ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem, + self->pmd_size)); +} + +TEST_F(pmd_swap, madvise_free) +{ + unsigned long swap_before =3D read_vmswap(); + unsigned long i; + + ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem, + self->pmd_size)); + ASSERT_GE(swap_before, self->pmd_size); + ASSERT_EQ(madvise(self->mem, self->pmd_size, MADV_FREE), 0); + for (i =3D 0; i < self->pmd_size; i +=3D getpagesize()) + ASSERT_FALSE(pagemap_is_swapped(self->pagemap_fd, + self->mem + i)); + ASSERT_LE(read_vmswap(), swap_before - self->pmd_size); + ASSERT_TRUE(verify_zero(self->mem, self->pmd_size)); +} + +TEST_F(pmd_swap, madvise_willneed) +{ + ASSERT_EQ(madvise(self->mem, self->pmd_size, MADV_WILLNEED), 0); + ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem, + self->pmd_size)); + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); + if (self->zswap_enabled) + ksft_print_msg("zswap enabled: PMD restoration not checked\n"); + else + ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1, + self->pmd_size)); +} + +TEST_F(pmd_swap, swapoff) +{ + int ret, err; + + self->swap_dev =3D getenv("PMD_SWAP_DEVICE"); + if (!self->swap_dev) + SKIP(return, "PMD_SWAP_DEVICE env var not set\n"); + /* + * Otherwise a higher-priority device may have taken the PMD swap + * entry and swapoff() would operate on the wrong backend. + */ + if (!swap_device_is_only_active(self->swap_dev, &self->swap_prio)) + SKIP(return, "PMD_SWAP_DEVICE must be the only active swap device\n"); + + self->uffd =3D register_rwp(self->mem, self->pmd_size, true); + + ret =3D swapoff(self->swap_dev); + err =3D errno; + if (!ret) + self->swap_disabled =3D true; + ASSERT_EQ(ret, 0) { + TH_LOG("swapoff(%s) failed: %s", self->swap_dev, strerror(err)); + } + + /* + * Check residency before touching the memory. If we read + * first, a bug that left a PMD swap entry in place after swapoff + * would silently trigger do_huge_pmd_swap_page() and reinstall a + * PMD mapping, masking the regression. + */ + if (self->zswap_enabled) + ksft_print_msg("zswap enabled: PMD restoration not checked\n"); + else + ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1, + self->pmd_size)); + if (self->uffd >=3D 0) + ASSERT_TRUE(expect_rwp_fault(self->uffd, self->mem, + self->pmd_size, + pattern_byte(self->seed, 0), false)); + ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed)); + + ret =3D swapon_restore(self->swap_dev, self->swap_prio); + err =3D errno; + if (!ret) + self->swap_disabled =3D false; + ASSERT_EQ(ret, 0) { + TH_LOG("swapon(%s) failed: %s", self->swap_dev, strerror(err)); + } +} + +TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/self= tests/mm/run_vmtests.sh index 19755fb4edb94..720ba5cbacd62 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -69,6 +69,8 @@ separated by spaces: test pagemap_scan IOCTL - pfnmap tests for VM_PFNMAP handling +- pmd_swap + tests for PMD-level swap entries - process_madv test for process_madv - cow @@ -414,6 +416,8 @@ CATEGORY=3D"pagemap" run_test ./pagemap_ioctl =20 CATEGORY=3D"pfnmap" run_test ./pfnmap =20 +CATEGORY=3D"pmd_swap" run_test ./pmd_swap + # COW tests CATEGORY=3D"cow" run_test ./cow =20 diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests= /mm/vm_util.c index 80bc9f597b521..e6133784abad2 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -180,6 +180,30 @@ uint64_t read_pmd_pagesize(void) return strtoul(buf, NULL, 10); } =20 +long read_vmstat(const char *name) +{ + char line[MAX_LINE_LENGTH]; + size_t name_len =3D strlen(name); + long val =3D -1; + FILE *f; + + f =3D fopen("/proc/vmstat", "r"); + if (!f) + return -1; + /* + * Match the whole field name: "thp_swpout" is a prefix of + * "thp_swpout_fallback" and "thp_swpout_pmd". + */ + while (fgets(line, sizeof(line), f)) { + if (!strncmp(line, name, name_len) && line[name_len] =3D=3D ' ') { + val =3D strtol(line + name_len + 1, NULL, 10); + break; + } + } + fclose(f); + return val; +} + unsigned long rss_anon(void) { unsigned long rss_anon =3D 0; diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests= /mm/vm_util.h index 9a49af88702e4..6355d830cd4a8 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -89,6 +89,8 @@ unsigned long pagemap_get_pfn(int fd, char *start); void clear_softdirty(void); bool check_for_pattern(FILE *fp, const char *pattern, char *buf, size_t le= n); uint64_t read_pmd_pagesize(void); +/* Returns the counter value, or -1 if /proc/vmstat has no such field. */ +long read_vmstat(const char *name); unsigned long rss_anon(void); bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage= _size); bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage= _size); --=20 2.53.0-Meta