From nobody Sun Dec 14 02:06:06 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2A691756D for ; Mon, 4 Mar 2024 08:14:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540070; cv=none; b=OoUAYIB7Uxe/8p4ZFoL02cl5u5gfNQfTLscxzGIs7LTgpW62U/sAPkuuc32ylvdMi8RryZN0PjLDYXJoiXWt+ggYU4Mvw4/rNbhjtqj4ZDPDMjrBjg821He+irYjH7D7vtQD+6xCUlH9X6+UBW7szmWwnvfloE9Fj8tsODXx3cg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540070; c=relaxed/simple; bh=TthfiCTSHniBg07/g9kff9ziRLL3IKn2kCZUYvskwmg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NeLUf9HSCi2hdOKoLW9W0BQdfBfpBK1YbZhHcJP9ooSDemXI7Zmy+2CbOkq8quK00fqkKEXWC2G+335HCMfrMtetyZBCXD738ZNnPRMfqUwQ94smXWvLEFJiMI8RBozOuExJYwUlS5tqKo9T0mDFlrqCGcqNVP3aywrxyNxFyJM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Nt5iBoBh; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Nt5iBoBh" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6e558a67f70so3659650b3a.0 for ; Mon, 04 Mar 2024 00:14:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709540068; x=1710144868; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y59EUmsn936TktKB/aH9UYKKlrxQ0A+jm6Fw0p9O+fA=; b=Nt5iBoBhVkZOGXpTIU7AKV1U0/ax2pulTqFUxQn5c1/Gc59xGjn/bfVcq0ZK9RRF06 jhbesD3fT2PnqSWm62/U6Sjqr3GXhgDSvu6d0BKpJsi1L/UxpGB5NtLYZqriR3KsWscx SafRcHqx4aBs9wW4f3V580g+ybc7XNdPYaJY3kXkYLYs3iKQp9un26TwKDgqaUjQrB49 nluAgQ2kMbph4fVyO4ARD3OeObN7RfkTgTb9SMBKCbFt7daJfSSN75VrZwOnhHjJT41L UEWDcvdfN5vV+sHw6C0SdWhuf8/T+bEtzPufl6niC1dVVbul76Rv4HCwX9R2r2a9VAfA De+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709540068; x=1710144868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y59EUmsn936TktKB/aH9UYKKlrxQ0A+jm6Fw0p9O+fA=; b=hLKURDW81yhHmcErRvEv+vq5uL59OP8VMJvUmHKMYl1Dx9RoWuS+WvWTdAq7o2AHhB MqDBw7fwIqwrZpKiwHPn6CeXc6IRHq6VBUKPx7NIkZ6Y7uZSAv22BdlrT5Gwzkl055x2 7qGTfM/1kw7ZMv6py7qGsNkxA/0Ftc91Cv20Ken3MHpdqMqZxqGqdtLDcRnMrrf0Y1R2 3nVFw3Hb2+onja5nf55pBzLSN0us93iPKa++KAkpWvDUddCVnFEFkuAy6HoZErMyGOtt 2eTOG9f8ReirkCFnvKZqUCiS8yZW2WszHGLrgriIdlWvbanDbgaI6cideXxQybWb6WOr kXSg== X-Forwarded-Encrypted: i=1; AJvYcCU1IEuKTKH2kdxysfSodM6di+rZfrnninIkGto6vLaqqXc6MESiD6Q0JLiTx4uLeoGNVOOLGktfEp/4D6318lFZpe48c2P+mR3YUe51 X-Gm-Message-State: AOJu0Yz2ZQ7rFXSoB9qo3ECBmmbHn4JtD4GNFXm/EHKF+V/PwyxXJwwj RDsKkT4BXaBWIFsQUcjF72H1DvRvjWA6W7PC562uFSBbALwJx15J X-Google-Smtp-Source: AGHT+IGA7fWsmwmgOxEFHP05B2QC6S7nXvlivm8Kflb15xIyMMY/ZdBjS77UiDjrNdXg/50SGdBWOA== X-Received: by 2002:a05:6a00:b8f:b0:6e6:13ec:7178 with SMTP id g15-20020a056a000b8f00b006e613ec7178mr2688020pfj.32.1709540067899; Mon, 04 Mar 2024 00:14:27 -0800 (PST) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id ka42-20020a056a0093aa00b006e558a67374sm6686387pfb.0.2024.03.04.00.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 00:14:27 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, ryan.roberts@arm.com Cc: chengming.zhou@linux.dev, chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mhocko@suse.com, nphamcs@gmail.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, Barry Song , Catalin Marinas , Will Deacon , Mark Rutland , Kemeng Shi , Anshuman Khandual , Peter Collingbourne , Peter Xu , Lorenzo Stoakes , "Mike Rapoport (IBM)" , Hugh Dickins , "Aneesh Kumar K.V" , Rick Edgecombe Subject: [RFC PATCH v3 1/5] arm64: mm: swap: support THP_SWAP on hardware with MTE Date: Mon, 4 Mar 2024 21:13:44 +1300 Message-Id: <20240304081348.197341-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240304081348.197341-1-21cnbao@gmail.com> References: <20240304081348.197341-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Commit d0637c505f8a1 ("arm64: enable THP_SWAP for arm64") brings up THP_SWAP on ARM64, but it doesn't enable THP_SWP on hardware with MTE as the MTE code works with the assumption tags save/restore is always handling a folio with only one page. The limitation should be removed as more and more ARM64 SoCs have this feature. Co-existence of MTE and THP_SWAP becomes more and more important. This patch makes MTE tags saving support large folios, then we don't need to split large folios into base pages for swapping out on ARM64 SoCs with MTE any more. arch_prepare_to_swap() should take folio rather than page as parameter because we support THP swap-out as a whole. It saves tags for all pages in a large folio. As now we are restoring tags based-on folio, in arch_swap_restore(), we may increase some extra loops and early-exitings while refaulting a large folio which is still in swapcache in do_swap_page(). In case a large folio has nr pages, do_swap_page() will only set the PTE of the particular page which is causing the page fault. Thus do_swap_page() runs nr times, and each time, arch_swap_restore() will loop nr times for those subpages in the folio. So right now the algorithmic complexity becomes O(nr^2). Once we support mapping large folios in do_swap_page(), extra loops and early-exitings will decrease while not being completely removed as a large folio might get partially tagged in corner cases such as, 1. a large folio in swapcache can be partially unmapped, thus, MTE tags for the unmapped pages will be invalidated; 2. users might use mprotect() to set MTEs on a part of a large folio. arch_thp_swp_supported() is dropped since ARM64 MTE was the only one who needed it. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: David Hildenbrand Cc: Kemeng Shi Cc: "Matthew Wilcox (Oracle)" Cc: Anshuman Khandual Cc: Peter Collingbourne Cc: Steven Price Cc: Yosry Ahmed Cc: Peter Xu Cc: Lorenzo Stoakes Cc: "Mike Rapoport (IBM)" Cc: Hugh Dickins CC: "Aneesh Kumar K.V" Cc: Rick Edgecombe Signed-off-by: Barry Song Reviewed-by: Steven Price Acked-by: Chris Li --- arch/arm64/include/asm/pgtable.h | 19 ++------------ arch/arm64/mm/mteswap.c | 43 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 48 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 401087e8a43d..7a54750770b8 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1) #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ =20 -static inline bool arch_thp_swp_supported(void) -{ - return !system_supports_mte(); -} -#define arch_thp_swp_supported arch_thp_swp_supported - /* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1095,12 +1089,7 @@ static inline pmd_t pmdp_establish(struct vm_area_st= ruct *vma, #ifdef CONFIG_ARM64_MTE =20 #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap(struct page *page) -{ - if (system_supports_mte()) - return mte_save_tags(page); - return 0; -} +extern int arch_prepare_to_swap(struct folio *folio); =20 #define __HAVE_ARCH_SWAP_INVALIDATE static inline void arch_swap_invalidate_page(int type, pgoff_t offset) @@ -1116,11 +1105,7 @@ static inline void arch_swap_invalidate_area(int typ= e) } =20 #define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore(swp_entry_t entry, struct folio *foli= o) -{ - if (system_supports_mte()) - mte_restore_tags(entry, &folio->page); -} +extern void arch_swap_restore(swp_entry_t entry, struct folio *folio); =20 #endif /* CONFIG_ARM64_MTE */ =20 diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc5..295836fef620 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -68,6 +68,13 @@ void mte_invalidate_tags(int type, pgoff_t offset) mte_free_tag_storage(tags); } =20 +static inline void __mte_invalidate_tags(struct page *page) +{ + swp_entry_t entry =3D page_swap_entry(page); + + mte_invalidate_tags(swp_type(entry), swp_offset(entry)); +} + void mte_invalidate_tags_area(int type) { swp_entry_t entry =3D swp_entry(type, 0); @@ -83,3 +90,39 @@ void mte_invalidate_tags_area(int type) } xa_unlock(&mte_pages); } + +int arch_prepare_to_swap(struct folio *folio) +{ + long i, nr; + int err; + + if (!system_supports_mte()) + return 0; + + nr =3D folio_nr_pages(folio); + + for (i =3D 0; i < nr; i++) { + err =3D mte_save_tags(folio_page(folio, i)); + if (err) + goto out; + } + return 0; + +out: + while (i--) + __mte_invalidate_tags(folio_page(folio, i)); + return err; +} + +void arch_swap_restore(swp_entry_t entry, struct folio *folio) +{ + if (system_supports_mte()) { + long i, nr =3D folio_nr_pages(folio); + + entry.val -=3D swp_offset(entry) & (nr - 1); + for (i =3D 0; i < nr; i++) { + mte_restore_tags(entry, folio_page(folio, i)); + entry.val++; + } + } +} diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index de0c89105076..e04b93c43965 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -535,16 +535,4 @@ static inline int split_folio_to_order(struct folio *f= olio, int new_order) #define split_folio_to_list(f, l) split_folio_to_list_to_order(f, l, 0) #define split_folio(f) split_folio_to_order(f, 0) =20 -/* - * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to - * limitations in the implementation like arm64 MTE can override this to - * false - */ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported(void) -{ - return true; -} -#endif - #endif /* _LINUX_HUGE_MM_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e1b22903f709..bfcfe3386934 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1106,7 +1106,7 @@ static inline int arch_unmap_one(struct mm_struct *mm, * prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap(struct page *page) +static inline int arch_prepare_to_swap(struct folio *folio) { return 0; } diff --git a/mm/page_io.c b/mm/page_io.c index ae2b49055e43..a9a7c236aecc 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_= control *wbc) * Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =3D arch_prepare_to_swap(&folio->page); + ret =3D arch_prepare_to_swap(folio); if (ret) { folio_mark_dirty(folio); folio_unlock(folio); diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 90973ce7881d..53abeaf1371d 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -310,7 +310,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =3D 0; =20 if (folio_test_large(folio)) { - if (IS_ENABLED(CONFIG_THP_SWAP) && arch_thp_swp_supported()) + if (IS_ENABLED(CONFIG_THP_SWAP)) get_swap_pages(1, &entry, folio_nr_pages(folio)); goto out; } --=20 2.34.1 From nobody Sun Dec 14 02:06:06 2025 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B2F3171CE for ; Mon, 4 Mar 2024 08:14:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540077; cv=none; b=si3+7K3dRAbji60nU4EdEwJc1VWV/+OAioNAzJdKFZxFG0PwX2+Go9nvC554r6+oI5pd72MuBo6NlJi28f4tHpUJ4mKGS5fKcAURlkMDerDzMYX5Ca1x7MsBctty0GCeN+0DtaRnYr8o5VopX6evHOhXsPV/X+B+RkQd7ibhCoE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540077; c=relaxed/simple; bh=PHC2OmMBrBHeHzSR537jJ2GXTFYmQehyNnf3GsevdDE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Uiq+5tt6PyohlMdXWPMgWyuiAzPmBi6NcqCFFsTteKsul4CMYN2csm0ChxzV88mj1GcVwAj3+jf5xmW0EpdM1X5cUxjI/YdQ3zODdK6K39v9f9upZ5t0BSVGUshnKhk+xfYQLzrEKeqlBdoFCMbeRzjMwTFxcvhpzKPhu145LJ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=D9d3OlLT; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="D9d3OlLT" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6da202aa138so3020867b3a.2 for ; Mon, 04 Mar 2024 00:14:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709540076; x=1710144876; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AOQhq+D2hctSN3bqux7faSvf1ZbbOD2cN4aBjG+5ulk=; b=D9d3OlLTK1TC2jOvPi5Z2pb9hrOzomJa8LOHJ1pOnCCRV2Ed7pJBudmN3kYk5vvRnE UuknfVLeyC1D8M51y3+1rjX94ysmP/g4lOij6qgYb5otD3QEds50EzaMhJ5Z1IyvW2Nr 9rwEN1/Cu+weVpibnDuqd+eqEnutI5estrPx3cKCp9gV5/ffJbHn8LPC+6RF6ql7NlOu gWrB4a2kqE7ArAd3k9fxSi+JzEgGoYkash5/7B6id8M8y51awA7bvNfqxxXBbX8qv/Sx TC2NhrQhDy0yqEEQercxGTUEb3ch6BnDW66eopNBRGx1CA/V+pqI6Mi55WoeSoxG8BYm Kw+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709540076; x=1710144876; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AOQhq+D2hctSN3bqux7faSvf1ZbbOD2cN4aBjG+5ulk=; b=h1Qpv1clVrcb2SNTYbiXr/hji/WvqIqEQaVOz/PdDkB4nNbWgto0FYESn/l5MSluh/ b5eNKKbx08CW/q+XN/5YJKnTi+vN3N9Ev9GUGBnV+SsoAOX0nOX0MBe9XGcH580eGLYO tTTrPQnmOEN8tYAfgwZD1mlSHi9QboK4YXctbpWA+T71fHPLB7Uwyg8XH9pcA8HNb6ZA 9KnCjjqreVjn7EU8JZvGIqPHlSCHntrt848CHk3+33VyKsboLPD//7CZhcCb2lcBSNnu LdUzPSenKK2ncg1Oln5avATKaHxMkidW4aoy7h0CHnglcuc8/1/tBnKT628+agM/hkbW M0XQ== X-Forwarded-Encrypted: i=1; AJvYcCWYgyxOVjpNM1daiSzF0X31qvOHd5I8TqoYSPXOSCHEyfwI4j6tC3kU0la/mDspKqiBJqQqw8cRWtP9L5u8+EXYPrTPpXmaDHHO+moc X-Gm-Message-State: AOJu0YwPtiFb5GUtuDKfBMlEfI6++ZopdsRdc0vF+QKwMmENQRCE00R5 umi+fwlIMK8D4sxkwZzVA+Na9n6F+IYQ5gSTW96yzl+D1nDoF0I9 X-Google-Smtp-Source: AGHT+IGEsGkarNyt9Jp8whgPA0poRrIOWMKKVHKbcHA3ADNFk6EzEEaBlL35RANkuwLw0QLGLyniCQ== X-Received: by 2002:a05:6a20:979b:b0:1a1:3000:ab67 with SMTP id hx27-20020a056a20979b00b001a13000ab67mr9356185pzc.46.1709540075740; Mon, 04 Mar 2024 00:14:35 -0800 (PST) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id ka42-20020a056a0093aa00b006e558a67374sm6686387pfb.0.2024.03.04.00.14.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 00:14:35 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, ryan.roberts@arm.com Cc: chengming.zhou@linux.dev, chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mhocko@suse.com, nphamcs@gmail.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, Chuanhua Han , Barry Song Subject: [RFC PATCH v3 2/5] mm: swap: introduce swap_nr_free() for batched swap_free() Date: Mon, 4 Mar 2024 21:13:45 +1300 Message-Id: <20240304081348.197341-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240304081348.197341-1-21cnbao@gmail.com> References: <20240304081348.197341-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- include/linux/swap.h | 6 ++++++ mm/swapfile.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 2955f7a78d8d..d6ab27929458 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -481,6 +481,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_nr_free(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern int free_swap_and_cache(swp_entry_t); int swap_type_of(dev_t device, sector_t offset); @@ -561,6 +562,11 @@ static inline void swap_free(swp_entry_t swp) { } =20 +void swap_nr_free(swp_entry_t entry, int nr_pages) +{ + +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index 3f594be83b58..244106998a69 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1341,6 +1341,41 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } =20 +/* + * Called after swapping in a large folio, batched free swap entries + * for this large folio, entry should be for the first subpage and + * its offset is aligned with nr_pages + */ +void swap_nr_free(swp_entry_t entry, int nr_pages) +{ + int i; + struct swap_cluster_info *ci; + struct swap_info_struct *p; + unsigned type =3D swp_type(entry); + unsigned long offset =3D swp_offset(entry); + DECLARE_BITMAP(usage, SWAPFILE_CLUSTER) =3D { 0 }; + + /* all swap entries are within a cluster for mTHP */ + VM_BUG_ON(offset % SWAPFILE_CLUSTER + nr_pages > SWAPFILE_CLUSTER); + + if (nr_pages =3D=3D 1) { + swap_free(entry); + return; + } + + p =3D _swap_info_get(entry); + + ci =3D lock_cluster(p, offset); + for (i =3D 0; i < nr_pages; i++) { + if (__swap_entry_free_locked(p, offset + i, 1)) + __bitmap_set(usage, i, 1); + } + unlock_cluster(ci); + + for_each_clear_bit(i, usage, nr_pages) + free_swap_slot(swp_entry(type, offset + i)); +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ --=20 2.34.1 From nobody Sun Dec 14 02:06:06 2025 Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FBC617BA9 for ; Mon, 4 Mar 2024 08:14:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540085; cv=none; b=A0cfFhy2sHYMMhrmWxmIj0KOuAhTqQfP7/6Fs+3yI1AEBc/e7ArnQlvS8MVUVW1QyrekqV7vQoCWdCLxCnKYxabV+lo6OXJYPuZvevbzkDAjW/slBz0L3tkyZrrrb0PDcinIcroBvV0/2jldytAxt2RJsc0mJA7svUjEhmdHa/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540085; c=relaxed/simple; bh=97XECp6ImGl0bxcVjD9vOm2RDIYctvChEeuK0x7tRrg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jjXKh7u9f6mE3BKorRqm2wB8s1/702yXd7n19NGJ44qIu2rN/BP1cVE5qpxI9opx9YRUjLUXGS8H2hJtcfd18TI8hpSlGOBEGhm9t3fCQmIR6ZqSBJrUQNNEKgO0xP1Vo5a2RgmBl7yD9Vqup7477vecYC3bObBFH5jXzjfZI2Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=H6e/S3c4; arc=none smtp.client-ip=209.85.166.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="H6e/S3c4" Received: by mail-io1-f43.google.com with SMTP id ca18e2360f4ac-7bed9fb159fso259233639f.1 for ; Mon, 04 Mar 2024 00:14:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709540083; x=1710144883; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nPNnhD7u3Mm6wgXrNGhON5/bGI4wy7Z421ED7ypUMCI=; b=H6e/S3c4QDzprosK/gcKxHAuMi17BoaNAUBWA6yvutonMR1hjZ5quIM9q4ydHdLUPK mYYbkKOxbGEFGRTYqRxYbtfbBZL/qEwNff4sjuBQNIUP1yRI3WbKTegaRgzMLwLjJv2Y 4sSItnarMVwRUXPEaiqugCursDRdZxQ5ahWFyO+UaU5pTyqpPKe7ruadjtpADL/kVsyi tsUCduR+85/UgU2pMH1nTdXEgWWgr6ovWgues7skBZ5GmhLxuzv1OCgXHUqVzKrx9Jvf JVvLKCS43ny1UPy4xp40fe77dG/bh9EtjkjSBMy4IE2upl2CMU8Bq/JtGaDtIdWZCbl/ IxoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709540083; x=1710144883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nPNnhD7u3Mm6wgXrNGhON5/bGI4wy7Z421ED7ypUMCI=; b=W34g+PakdrWOj+2qyCDgX8wGJbZ34QopM9PVrGA536bR0WnMrDZ9zIiGEi3VUx8oly ck8dodIdviikGtywI78uLcIrbCfSoBCP+Ozgl1L+99s3aSKfBmXeXXDvKWIBa8rW6yY3 VZznoOLTdR78k/YoiflUrathnNFcENJtMnfPErPeAUzUzdpTolG0GLQ0AT7Z+vG9+YDT 4JVBJTgsDlcdahEVeI1f1xgXuG7dvnKzWm2bhqBnZee/vn4W7i1EGrw4821YiYX2zXBp wAKG8tQ7NAqrMHzprLG3A6YzdgUfGjBuUo3d29diYrZRtkkscaOxTu5ae7a2R6cOwRDz aqQQ== X-Forwarded-Encrypted: i=1; AJvYcCWMluaJzmg6eulZth0Fq3eX1YgGR6TMQzG+PkQIM9/T2c7dkEXLyMwr1nMurTZ0x24xAKuNbGMWe8SmeSml2RU0+DeUc2PZ408XK1C2 X-Gm-Message-State: AOJu0YxWi2WNjsV9KCUiRJmqQspwSKVIEn4jB4RIhm3qoduoyMQkb6UP AfKiHptdCycclF1p6dPo8KLOGIOnHqyODNEZW08YrtRn5NmOD01V X-Google-Smtp-Source: AGHT+IGUq+bnojeKaoe2OXAEqxwKRkmXLSREbs2hTuKe5Xt3pOS0+qxiK9Mn4Apyji0BmRbt8d2UYg== X-Received: by 2002:a05:6e02:1567:b0:365:f8d:50c3 with SMTP id k7-20020a056e02156700b003650f8d50c3mr11150779ilu.21.1709540083664; Mon, 04 Mar 2024 00:14:43 -0800 (PST) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id ka42-20020a056a0093aa00b006e558a67374sm6686387pfb.0.2024.03.04.00.14.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 00:14:43 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, ryan.roberts@arm.com Cc: chengming.zhou@linux.dev, chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mhocko@suse.com, nphamcs@gmail.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, Chuanhua Han , Barry Song Subject: [RFC PATCH v3 3/5] mm: swap: make should_try_to_free_swap() support large-folio Date: Mon, 4 Mar 2024 21:13:46 +1300 Message-Id: <20240304081348.197341-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240304081348.197341-1-21cnbao@gmail.com> References: <20240304081348.197341-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han should_try_to_free_swap() works with an assumption that swap-in is always d= one at normal page granularity, aka, folio_nr_pages =3D 1. To support large fol= io swap-in, this patch removes the assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index abd4f33d62c9..e0d34d705e07 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3837,7 +3837,7 @@ static inline bool should_try_to_free_swap(struct fol= io *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) =3D=3D 2; + folio_ref_count(folio) =3D=3D (1 + folio_nr_pages(folio)); } =20 static vm_fault_t pte_marker_clear(struct vm_fault *vmf) --=20 2.34.1 From nobody Sun Dec 14 02:06:06 2025 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9C1F17BAC for ; Mon, 4 Mar 2024 08:14:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540094; cv=none; b=s/1hhjbadgRKxkYyAtx9gTQHcakpE7WdmaTQUoVbRfsb0SziHHrMcosKYVD23CdWVnymnmLP8OpwZF1Immk9tC22YoVJo9AfdTyMOXOQXc/1oRWYZASkc3PzDKoNbCGmm7AfrpKJyaxM8UzjTjUcrjX2endOTSrgw10wjATxlhc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540094; c=relaxed/simple; bh=9NiAmaNMcQqQy5qDH5R7xvBLevxdcjq0TuHgInEcdWc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HQjAc3mwY1WUoi4F64x6yNXjhB8eVvglSLLgDnwM+z+K0sBlHskJacsI/4ycUJRNubQJqbAhpUTAtU4wndX3VAZ/1fL3cJoy+1diV/U0SivemxkYKyVu4mUHbFpM8NH6eOVmNQoG+0GetTWTVF8TMmxp1zOH3MxBZ09lNT2S6p0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=keCarv5z; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="keCarv5z" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6e4560664b5so3374452b3a.1 for ; Mon, 04 Mar 2024 00:14:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709540092; x=1710144892; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VEHcUaBOURO/yC0z8mfKLfYRHHxSSbfVWo8QOD6O6SM=; b=keCarv5zcLnNsdKKqHVC8mIPIVGxGPvxnkCXOXfnozFXtGWOpwdc4lzlto5K58x6Kx WhYfZrlepmdtytTTX2JJdEf4ZsCaIQoVEZXkmIw6Ii/b4es0AiUbc27jXdLb6vDCOcOp QGLK0VtcXOg14hGujft5QgPSh1nqzxAyXMxvfDKZ3W9tSZrz6KVCD8LOnQsnAuLIAVU8 N0Uc4gKhy3/WVWcxfEHNKYD+aXWDFUuCPcHRrds9H6sSqfTqXN1QXTevT1x0cnrOuh/C X2MCiRbLgb86duy5kD8vOoxufaDWepOpwcuqrEbY4AvzX8CikSIdxuNdekISqi1P49KZ 2O3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709540092; x=1710144892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VEHcUaBOURO/yC0z8mfKLfYRHHxSSbfVWo8QOD6O6SM=; b=dY0zX8irtsi8M9DN8vePNPIpv/VwEs1LqiVY8KgSZ57CIKUvru0A+0RAC+7klLuOQs N+bAt6pnRK8U3jUgBCwhhUcyCt+cdlpGaKq2s4YJhd+i61tnGwnK73Z3QXcDqxCwc2ES IrKT2w/OHuEsa0yS3B1CcOjLeCHSFphXsS5IwibLhBzZc5oQRsMO5CtXPDHALcHP3Cxp g7OxgsYWhonw8DRnFmfcw7wdEXb+EbY2xbcrZQ95zCSQWV9p1QMFdRErROXArADFFyBN na9T5HS9u2XuscoS2Dg06upQjZo8hhZ8r/C+0aBNldgRd1V+9G0TDrN58jlOBgelwhg6 3L7g== X-Forwarded-Encrypted: i=1; AJvYcCWrPyPq9kEm4rw6Nw3YCcA8cmDWgZZV0hkyHdol2gDdvBSkzAa/JtuRBAaWDk+prD9NsnDq+Pocdu1ggIQlUZ49OHtNWSdhOd7r9RVY X-Gm-Message-State: AOJu0YwOZzpreSwp0s70I9X/uI2wD4KKQuDyYwUczyEK0W2HMhxkhRqv U3iH9lzFHoVcnBkKs07U1Jrkqr3lrAG/qoJPvt+AIdtR1mZ5qjVT X-Google-Smtp-Source: AGHT+IGE7Q24s92a3q9O7BOh15+H0RGSvY6KsWDz94mjR2aHEmrzB8E5OFPS9feP8EbqMCx8xVkVsg== X-Received: by 2002:a05:6a00:2d99:b0:6e6:2499:d189 with SMTP id fb25-20020a056a002d9900b006e62499d189mr2330890pfb.2.1709540091964; Mon, 04 Mar 2024 00:14:51 -0800 (PST) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id ka42-20020a056a0093aa00b006e558a67374sm6686387pfb.0.2024.03.04.00.14.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 00:14:51 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, ryan.roberts@arm.com Cc: chengming.zhou@linux.dev, chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mhocko@suse.com, nphamcs@gmail.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, Barry Song , Hugh Dickins , Minchan Kim , SeongJae Park Subject: [RFC PATCH v3 4/5] mm: swap: introduce swapcache_prepare_nr and swapcache_clear_nr for large folios swap-in Date: Mon, 4 Mar 2024 21:13:47 +1300 Message-Id: <20240304081348.197341-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240304081348.197341-1-21cnbao@gmail.com> References: <20240304081348.197341-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Commit 13ddaf26be32 ("mm/swap: fix race when skipping swapcache") supports one entry only, to support large folio swap-in, we need to handle multiple swap entries. Cc: Kairui Song Cc: "Huang, Ying" Cc: David Hildenbrand Cc: Chris Li Cc: Hugh Dickins Cc: Johannes Weiner Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Minchan Kim Cc: Yosry Ahmed Cc: Yu Zhao Cc: SeongJae Park Signed-off-by: Barry Song --- include/linux/swap.h | 1 + mm/swap.h | 1 + mm/swapfile.c | 118 ++++++++++++++++++++++++++----------------- 3 files changed, 74 insertions(+), 46 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index d6ab27929458..22105f0fe2d4 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -480,6 +480,7 @@ extern int add_swap_count_continuation(swp_entry_t, gfp= _t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); +extern int swapcache_prepare_nr(swp_entry_t entry, int nr); extern void swap_free(swp_entry_t); extern void swap_nr_free(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); diff --git a/mm/swap.h b/mm/swap.h index fc2f6ade7f80..1cec991efcda 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -42,6 +42,7 @@ void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry); +void swapcache_clear_nr(struct swap_info_struct *si, swp_entry_t entry, in= t nr); struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); struct folio *filemap_get_incore_folio(struct address_space *mapping, diff --git a/mm/swapfile.c b/mm/swapfile.c index 244106998a69..bae1b8165b11 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3309,7 +3309,7 @@ void si_swapinfo(struct sysinfo *val) } =20 /* - * Verify that a swap entry is valid and increment its swap map count. + * Verify that nr swap entries are valid and increment their swap map coun= t. * * Returns error code in following case. * - success -> 0 @@ -3319,66 +3319,76 @@ void si_swapinfo(struct sysinfo *val) * - swap-cache reference is requested but the entry is not used. -> ENOENT * - swap-mapped reference requested but needs continued swap count. -> EN= OMEM */ -static int __swap_duplicate(swp_entry_t entry, unsigned char usage) +static int __swap_duplicate_nr(swp_entry_t entry, int nr, unsigned char us= age) { struct swap_info_struct *p; struct swap_cluster_info *ci; unsigned long offset; - unsigned char count; - unsigned char has_cache; - int err; + unsigned char count[SWAPFILE_CLUSTER]; + unsigned char has_cache[SWAPFILE_CLUSTER]; + int err, i; =20 p =3D swp_swap_info(entry); =20 offset =3D swp_offset(entry); ci =3D lock_cluster_or_swap_info(p, offset); =20 - count =3D p->swap_map[offset]; - - /* - * swapin_readahead() doesn't check if a swap entry is valid, so the - * swap entry could be SWAP_MAP_BAD. Check here with lock held. - */ - if (unlikely(swap_count(count) =3D=3D SWAP_MAP_BAD)) { - err =3D -ENOENT; - goto unlock_out; - } - - has_cache =3D count & SWAP_HAS_CACHE; - count &=3D ~SWAP_HAS_CACHE; - err =3D 0; - - if (usage =3D=3D SWAP_HAS_CACHE) { + for (i =3D 0; i < nr; i++) { + count[i] =3D p->swap_map[offset + i]; =20 - /* set SWAP_HAS_CACHE if there is no cache and entry is used */ - if (!has_cache && count) - has_cache =3D SWAP_HAS_CACHE; - else if (has_cache) /* someone else added cache */ - err =3D -EEXIST; - else /* no users remaining */ + /* + * swapin_readahead() doesn't check if a swap entry is valid, so the + * swap entry could be SWAP_MAP_BAD. Check here with lock held. + */ + if (unlikely(swap_count(count[i]) =3D=3D SWAP_MAP_BAD)) { err =3D -ENOENT; + goto unlock_out; + } =20 - } else if (count || has_cache) { - - if ((count & ~COUNT_CONTINUED) < SWAP_MAP_MAX) - count +=3D usage; - else if ((count & ~COUNT_CONTINUED) > SWAP_MAP_MAX) - err =3D -EINVAL; - else if (swap_count_continued(p, offset, count)) - count =3D COUNT_CONTINUED; - else - err =3D -ENOMEM; - } else - err =3D -ENOENT; /* unused swap entry */ + has_cache[i] =3D count[i] & SWAP_HAS_CACHE; + count[i] &=3D ~SWAP_HAS_CACHE; + err =3D 0; + + if (usage =3D=3D SWAP_HAS_CACHE) { + + /* set SWAP_HAS_CACHE if there is no cache and entry is used */ + if (!has_cache[i] && count[i]) + has_cache[i] =3D SWAP_HAS_CACHE; + else if (has_cache[i]) /* someone else added cache */ + err =3D -EEXIST; + else /* no users remaining */ + err =3D -ENOENT; + } else if (count[i] || has_cache[i]) { + + if ((count[i] & ~COUNT_CONTINUED) < SWAP_MAP_MAX) + count[i] +=3D usage; + else if ((count[i] & ~COUNT_CONTINUED) > SWAP_MAP_MAX) + err =3D -EINVAL; + else if (swap_count_continued(p, offset + i, count[i])) + count[i] =3D COUNT_CONTINUED; + else + err =3D -ENOMEM; + } else + err =3D -ENOENT; /* unused swap entry */ =20 - if (!err) - WRITE_ONCE(p->swap_map[offset], count | has_cache); + if (err) + break; + } =20 + if (!err) { + for (i =3D 0; i < nr; i++) + WRITE_ONCE(p->swap_map[offset + i], count[i] | has_cache[i]); + } unlock_out: unlock_cluster_or_swap_info(p, ci); return err; } =20 +static int __swap_duplicate(swp_entry_t entry, unsigned char usage) +{ + return __swap_duplicate_nr(entry, 1, usage); +} + /* * Help swapoff by noting that swap entry belongs to shmem/tmpfs * (in which case its reference count is never incremented). @@ -3417,17 +3427,33 @@ int swapcache_prepare(swp_entry_t entry) return __swap_duplicate(entry, SWAP_HAS_CACHE); } =20 -void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) +int swapcache_prepare_nr(swp_entry_t entry, int nr) +{ + return __swap_duplicate_nr(entry, nr, SWAP_HAS_CACHE); +} + +void swapcache_clear_nr(struct swap_info_struct *si, swp_entry_t entry, in= t nr) { struct swap_cluster_info *ci; unsigned long offset =3D swp_offset(entry); - unsigned char usage; + unsigned char usage[SWAPFILE_CLUSTER]; + int i; =20 ci =3D lock_cluster_or_swap_info(si, offset); - usage =3D __swap_entry_free_locked(si, offset, SWAP_HAS_CACHE); + for (i =3D 0; i < nr; i++) + usage[i] =3D __swap_entry_free_locked(si, offset + i, SWAP_HAS_CACHE); unlock_cluster_or_swap_info(si, ci); - if (!usage) - free_swap_slot(entry); + for (i =3D 0; i < nr; i++) { + if (!usage[i]) { + free_swap_slot(entry); + entry.val++; + } + } +} + +void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) +{ + swapcache_clear_nr(si, entry, 1); } =20 struct swap_info_struct *swp_swap_info(swp_entry_t entry) --=20 2.34.1 From nobody Sun Dec 14 02:06:06 2025 Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2E3A17731 for ; Mon, 4 Mar 2024 08:15:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540102; cv=none; b=tvMlfWdEgqa5F+eUiHewZFBHpFRV6Lbc5LJZajLQZkVQI95/70IlOM7uoiSWl4SFkWe9U6tTbKMC9mXPX4JwvT7gRA9WqNUcwZBo7hhLRpBKZM4ANUxCazNO/HyZrTVme3YmYbs6y4Zpl7G0AkkOk2arokb2U7EFt4jdy+OF1gI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709540102; c=relaxed/simple; bh=qJPCusJ04ma2IPDts7aPIAhRV53b4FPmjk2hD0OS2RQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CuHNjREKLh0txdgDvnW2Snif7qQ+HLV/wlZW7JGexjgAEE+77ikxE5ov17/6uO2tkAJGotVsem95weC0Bn6NFelNRrbK1ZVqrGl2lFXXmoSw8Q64NIbGLSLlm0OaK8mrSviHMrrDYJL0trtWQ8CTJM1Up+EBWKJJfIWNkheGRBc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Oka7TYIe; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Oka7TYIe" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3c1adc90830so2570183b6e.0 for ; Mon, 04 Mar 2024 00:15:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709540100; x=1710144900; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CAJqxwL1m9o2obT3SwA5MTdLBPpoaZY5B/QAJ6xv5j8=; b=Oka7TYIeZTjtj0Jjn8q/UYoM9aL+z2hJJNIU73PtM0LNRISK4dw2e+4xQugwJVRjFg hOs5QUBhB2v7XQxSJAD6vP5ArzS8D8BmK5eP5XbVNyW0YB6M86Ob/VJs8w2NlKS2QxIc 5Pg4LtUXMWGIbTvRK1PP056QxB2cMNzq2PMmZcBwms9FHpVyMVqMfe5/xm89/wonTUsC MVv2xIZEl9YSoOluU+kbml1f/OklpN5GTrr35Amlhbe0gKOpgO+frG2RgnbJ8I/2vMod /8fLY53WS4saOvRL65DmE4QrN08qSvgDVUEd1T4dVH/zIZDpQ3hKObOsKsVjGeVNlaZn tZXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709540100; x=1710144900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CAJqxwL1m9o2obT3SwA5MTdLBPpoaZY5B/QAJ6xv5j8=; b=t81bCGQakg/oBPj4SNxB/paZKuXT5qJAye8dPXenSU1qQlvOe/C7XRA+o3c+xb8DU6 2I+X5bP5O7D3FcOrZVQtw5TrZR/kkys9EXAyLf8PrnxtSiqCPCBGSDGpB5drCIwEykRX 5THWJwkj8NXPfoUKZAeB41cRxJg2YHMChaxuc08EP+2BDZ/n5g0oD/9VtbxdstJpZsPY G69NOmFvJ1Cm9l9qkc4Uda8+hXsmjASyZ6ka5/AGv44VOlLk6f0WZ9yYN04vf0nKOBpc mNYX1hNtiJuD6MwQGMHKl8gvMNk1OeQ9fSpIHcU9SihnSIDVEs0NSfV5X70M/dWdmvep Cdyg== X-Forwarded-Encrypted: i=1; AJvYcCUKjkR5k5RtHAJAyI9TvPsuhcteNgFuNnqwk7m72EOgKNp7AdygVr+YQPplPpj4EdXSdnKPc28jlHD07WG2RxIFdlk7QqgYsjFD2PKj X-Gm-Message-State: AOJu0YySPxHca66lXica159pyaxkg8RliS1dwAixQl1hG2zfNGfBblo3 JCbIRvUYAUZMvwhGyA3x3+n0Zpn2JmQFuHdcTVlsGB3cQ/mu5fEI X-Google-Smtp-Source: AGHT+IH6AIE3OmLeVTX7JovqJlGUGKUv2cWCft66hk/1+/3mqzDatNzNe/9A2nspzbNZTbVsGn1pPA== X-Received: by 2002:a05:6870:e91:b0:21e:d80d:3f13 with SMTP id mm17-20020a0568700e9100b0021ed80d3f13mr11743190oab.58.1709540099802; Mon, 04 Mar 2024 00:14:59 -0800 (PST) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id ka42-20020a056a0093aa00b006e558a67374sm6686387pfb.0.2024.03.04.00.14.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 00:14:59 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, ryan.roberts@arm.com Cc: chengming.zhou@linux.dev, chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mhocko@suse.com, nphamcs@gmail.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, Chuanhua Han , Barry Song Subject: [RFC PATCH v3 5/5] mm: support large folios swapin as a whole Date: Mon, 4 Mar 2024 21:13:48 +1300 Message-Id: <20240304081348.197341-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240304081348.197341-1-21cnbao@gmail.com> References: <20240304081348.197341-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han On an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to background, its most memory might be swapped-out. Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE. This patch brings up mTHP swap-in support. Right now, we limit mTHP swap-in to those contiguous swaps which were likely swapped out from mTHP as a whole. Meanwhile, the current implementation only covers the SWAP_SYCHRONOUS case. It doesn't support swapin_readahead as large folios yet since this kind of shared memory is much less than memory mapped by single process. Right now, we are re-faulting large folios which are still in swapcache as a whole, this can effectively decrease extra loops and early-exitings which we have increased in arch_swap_restore() while supporting MTE restore for foli= os rather than page. On the other hand, it can also decrease do_swap_page as PTEs used to be set one by one even we hit a large folio in swapcache. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 250 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 212 insertions(+), 38 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index e0d34d705e07..501ede745ef3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3907,6 +3907,136 @@ static vm_fault_t handle_pte_marker(struct vm_fault= *vmf) return VM_FAULT_SIGBUS; } =20 +/* + * check a range of PTEs are completely swap entries with + * contiguous swap offsets and the same SWAP_HAS_CACHE. + * pte must be first one in the range + */ +static bool is_pte_range_contig_swap(pte_t *pte, int nr_pages) +{ + int i; + struct swap_info_struct *si; + swp_entry_t entry; + unsigned type; + pgoff_t start_offset; + char has_cache; + + entry =3D pte_to_swp_entry(ptep_get_lockless(pte)); + if (non_swap_entry(entry)) + return false; + start_offset =3D swp_offset(entry); + if (start_offset % nr_pages) + return false; + + si =3D swp_swap_info(entry); + type =3D swp_type(entry); + has_cache =3D si->swap_map[start_offset] & SWAP_HAS_CACHE; + for (i =3D 1; i < nr_pages; i++) { + entry =3D pte_to_swp_entry(ptep_get_lockless(pte + i)); + if (non_swap_entry(entry)) + return false; + if (swp_offset(entry) !=3D start_offset + i) + return false; + if (swp_type(entry) !=3D type) + return false; + /* + * while allocating a large folio and doing swap_read_folio for the + * SWP_SYNCHRONOUS_IO path, which is the case the being faulted pte + * doesn't have swapcache. We need to ensure all PTEs have no cache + * as well, otherwise, we might go to swap devices while the content + * is in swapcache + */ + if ((si->swap_map[start_offset + i] & SWAP_HAS_CACHE) !=3D has_cache) + return false; + } + + return true; +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +/* + * Get a list of all the (large) orders below PMD_ORDER that are enabled + * for this vma. Then filter out the orders that can't be allocated over + * the faulting address and still be fully contained in the vma. + */ +static inline unsigned long get_alloc_folio_orders(struct vm_fault *vmf) +{ + struct vm_area_struct *vma =3D vmf->vma; + unsigned long orders; + + orders =3D thp_vma_allowable_orders(vma, vma->vm_flags, false, true, true, + BIT(PMD_ORDER) - 1); + orders =3D thp_vma_suitable_orders(vma, vmf->address, orders); + return orders; +} +#endif + +static struct folio *alloc_swap_folio(struct vm_fault *vmf) +{ + struct vm_area_struct *vma =3D vmf->vma; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + unsigned long orders; + struct folio *folio; + unsigned long addr; + pte_t *pte; + gfp_t gfp; + int order; + + /* + * If uffd is active for the vma we need per-page fault fidelity to + * maintain the uffd semantics. + */ + if (unlikely(userfaultfd_armed(vma))) + goto fallback; + + /* + * a large folio being swapped-in could be partially in + * zswap and partially in swap devices, zswap doesn't + * support large folios yet, we might get corrupted + * zero-filled data by reading all subpages from swap + * devices while some of them are actually in zswap + */ + if (is_zswap_enabled()) + goto fallback; + + orders =3D get_alloc_folio_orders(vmf); + if (!orders) + goto fallback; + + pte =3D pte_offset_map(vmf->pmd, vmf->address & PMD_MASK); + if (unlikely(!pte)) + goto fallback; + + /* + * For do_swap_page, find the highest order where the aligned range is + * completely swap entries with contiguous swap offsets. + */ + order =3D highest_order(orders); + while (orders) { + addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order); + if (is_pte_range_contig_swap(pte + pte_index(addr), 1 << order)) + break; + order =3D next_order(&orders, order); + } + + pte_unmap(pte); + + /* Try allocating the highest of the remaining orders. */ + gfp =3D vma_thp_gfp_mask(vma); + while (orders) { + addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order); + folio =3D vma_alloc_folio(gfp, order, vma, addr, true); + if (folio) + return folio; + order =3D next_order(&orders, order); + } + +fallback: +#endif + return vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); +} + + /* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -3928,6 +4058,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret =3D 0; void *shadow =3D NULL; + int nr_pages =3D 1; + unsigned long start_address; + pte_t *start_pte; =20 if (!pte_unmap_same(vmf)) goto out; @@ -3991,35 +4124,41 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) =3D=3D 1) { - /* - * Prevent parallel swapin from proceeding with - * the cache flag. Otherwise, another thread may - * finish swapin first, free the entry, and swapout - * reusing the same entry. It's undetectable as - * pte_same() returns true due to entry reuse. - */ - if (swapcache_prepare(entry)) { - /* Relax a bit to prevent rapid repeated page faults */ - schedule_timeout_uninterruptible(1); - goto out; - } - need_clear_cache =3D true; - /* skip swapcache */ - folio =3D vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); + folio =3D alloc_swap_folio(vmf); page =3D &folio->page; if (folio) { __folio_set_locked(folio); __folio_set_swapbacked(folio); =20 + if (folio_test_large(folio)) { + nr_pages =3D folio_nr_pages(folio); + entry.val =3D ALIGN_DOWN(entry.val, nr_pages); + } + + /* + * Prevent parallel swapin from proceeding with + * the cache flag. Otherwise, another thread may + * finish swapin first, free the entry, and swapout + * reusing the same entry. It's undetectable as + * pte_same() returns true due to entry reuse. + */ + if (swapcache_prepare_nr(entry, nr_pages)) { + /* Relax a bit to prevent rapid repeated page faults */ + schedule_timeout_uninterruptible(1); + goto out; + } + need_clear_cache =3D true; + if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, GFP_KERNEL, entry)) { ret =3D VM_FAULT_OOM; goto out_page; } - mem_cgroup_swapin_uncharge_swap(entry); + + for (swp_entry_t e =3D entry; e.val < entry.val + nr_pages; e.val++) + mem_cgroup_swapin_uncharge_swap(e); =20 shadow =3D get_shadow_from_swap_cache(entry); if (shadow) @@ -4118,6 +4257,42 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + start_address =3D vmf->address; + start_pte =3D vmf->pte; + if (start_pte && folio_test_large(folio)) { + unsigned long nr =3D folio_nr_pages(folio); + unsigned long addr =3D ALIGN_DOWN(vmf->address, nr * PAGE_SIZE); + pte_t *aligned_pte =3D vmf->pte - (vmf->address - addr) / PAGE_SIZE; + + /* + * case 1: we are allocating large_folio, try to map it as a whole + * iff the swap entries are still entirely mapped; + * case 2: we hit a large folio in swapcache, and all swap entries + * are still entirely mapped, try to map a large folio as a whole. + * otherwise, map only the faulting page within the large folio + * which is swapcache + */ + if (!is_pte_range_contig_swap(aligned_pte, nr)) { + if (nr_pages > 1) /* ptes have changed for case 1 */ + goto out_nomap; + goto check_pte; + } + + start_address =3D addr; + start_pte =3D aligned_pte; + /* + * the below has been done before swap_read_folio() + * for case 1 + */ + if (unlikely(folio =3D=3D swapcache)) { + nr_pages =3D nr; + entry.val =3D ALIGN_DOWN(entry.val, nr_pages); + page =3D &folio->page; + } + } + +check_pte: if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; =20 @@ -4185,12 +4360,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_nr_free(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); =20 - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); + pte =3D mk_pte(page, vma->vm_page_prot); =20 /* @@ -4200,14 +4377,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) =3D=3D 1)) { + (exclusive || folio_ref_count(folio) =3D=3D nr_pages)) { if (vmf->flags & FAULT_FLAG_WRITE) { pte =3D maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &=3D ~FAULT_FLAG_WRITE; } rmap_flags |=3D RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte =3D pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) @@ -4216,17 +4393,19 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) =20 /* ksm created a completely new copy */ if (unlikely(folio !=3D swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, start_address); folio_add_lru_vma(folio, vma); + } else if (!folio_test_anon(folio)) { + folio_add_new_anon_rmap(folio, vma, start_address); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, rmap_flags); } =20 VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, vmf->orig_pte); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { @@ -4243,6 +4422,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } =20 if (vmf->flags & FAULT_FLAG_WRITE) { + if (nr_pages > 1) + vmf->orig_pte =3D ptep_get(vmf->pte); + ret |=3D do_wp_page(vmf); if (ret & VM_FAULT_ERROR) ret &=3D VM_FAULT_ERROR; @@ -4250,14 +4432,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); out: /* Clear the swap cache pin for direct swapin after PTL unlock */ if (need_clear_cache) - swapcache_clear(si, entry); + swapcache_clear_nr(si, entry, nr_pages); if (si) put_swap_device(si); return ret; @@ -4273,7 +4455,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_put(swapcache); } if (need_clear_cache) - swapcache_clear(si, entry); + swapcache_clear_nr(si, entry, nr_pages); if (si) put_swap_device(si); return ret; @@ -4309,15 +4491,7 @@ static struct folio *alloc_anon_folio(struct vm_faul= t *vmf) if (unlikely(userfaultfd_armed(vma))) goto fallback; =20 - /* - * Get a list of all the (large) orders below PMD_ORDER that are enabled - * for this vma. Then filter out the orders that can't be allocated over - * the faulting address and still be fully contained in the vma. - */ - orders =3D thp_vma_allowable_orders(vma, vma->vm_flags, false, true, true, - BIT(PMD_ORDER) - 1); - orders =3D thp_vma_suitable_orders(vma, vmf->address, orders); - + orders =3D get_alloc_folio_orders(vmf); if (!orders) goto fallback; =20 --=20 2.34.1