From nobody Wed Jun 17 02:50:08 2026 Received: from mail-dl1-f42.google.com (mail-dl1-f42.google.com [74.125.82.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5AB637AA81 for ; Tue, 21 Apr 2026 23:02:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776812570; cv=none; b=gju9+UlJot5FJqYP6GXxsWWyni4LplVlo3wDoNYcFejbjgLgYUraG1CYv0pa8Lj7HADh8sdJm8SM4JrZWg9sBUAjEJeydhKGTYAmLmgDrlAga8YbFIhuiYTfHp3m76pwfrTVRGLnb8x3bmJ8QUtEZfExnTSZpDEqH6Xqkn8mn1w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776812570; c=relaxed/simple; bh=Mwv33OFn1P/G3Y2GF6fMvwNlS7lhrWXSX04G+ayxp10=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fIzlZ6Pa002LuKiPripvnwsS3LP70NGIh1xanyB2G/IuFSupnbrDn2iUIh2rO9bpjR/DseU0MMMElowJHKLNNGUgzX0Gy72MX6m5ggoil916M3Nx1EI2lh80zPJ2eqFsZqSlVc5JKZ5uRXIqQLxaEdU4Egoytiko1GPIVm1UZoQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aNFa+W2V; arc=none smtp.client-ip=74.125.82.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aNFa+W2V" Received: by mail-dl1-f42.google.com with SMTP id a92af1059eb24-12c726f46baso5904043c88.1 for ; Tue, 21 Apr 2026 16:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1776812568; x=1777417368; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=mdOzQUx1kuF/ItixYZyEN7MAejmBx2S9M5lxrAx/c7w=; b=aNFa+W2VnQz+F24fJVGirJAxv8qNhlU667QtvmiNEkGYG4GStY5IstlNX8OcZ7OZ6X r7ecNpL+0E4zcE1GnJaj/0QYP1HC75YveutpkjUedT1X4o9iJ3d4K3hU7rBiE09xUVfw CHJpzCKuM9pP9VuzRNpABXMsVnuQveEmr00L9EyISwqcU6dONj7sY+T1Y4taRXYmYtZ1 r9gBPiwrb9eFPLN+oesXWdJELTJQ6Ek1aGs4P9jy7V3BBv9eGbZenLGViY+r39kZi4gH Q3GV9ZQKbfLqwl+WyRMBc0rHlhGNBhwjpaqeuxPq/M1o1ZQPP88jzUkMadzMRYNnaZOM YBiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776812568; x=1777417368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mdOzQUx1kuF/ItixYZyEN7MAejmBx2S9M5lxrAx/c7w=; b=tLuXpGyE77fKHYEQFIx8XG6gMdD/fc6FEUX9h44fGRF5V3yDlQpmu1qJLOA7tWbXWB uwc32Ampap91/JevfexZIIdoq5ImbNYAW/odvgryK6rX/StjONM3uknaAJwrr5d0b2gn JqEFJKesP23Bdd668sPFPyAct/TQKZ/UrENdmgGT9H65CcSMTljBQSgnDY5ma9FkrdnJ PGuiXzzQuGSvOwncdwqMWUkbHXsstPJO6a5rMGsgI03Lf6BSI5M6mnBCGmvlHaTlOJ0B pkc/Zy/Js/fdxlNnH/gIXmlFpHxK4maSckovGki209+tU60aoBnoUWqtzkSKRpQW6uye 4sjQ== X-Forwarded-Encrypted: i=1; AFNElJ90YBk1tbfufmlHSIf7fVhfWQzl33QEtBmh6b8SSFXvdY4YBPDEBjD6LVqyfIL1ltQguMS57S2JSzFqBf4=@vger.kernel.org X-Gm-Message-State: AOJu0YxDtt7fjU2sOlJfnrF95o5kaqyevfTW+PN8LIRKcdsjlr3IAQHZ U2ix+FZqEw2/i2woW+/OLDVnPyXYMHlkMSi0JFE7BS1h/4xSJxg0P7ln X-Gm-Gg: AeBDievqpjbgImAowDCeynAshQnPCHKpLgN/DXCtOIEQ4m6KJXA0GEM4Qh5GAU7CZqE Yiy95jbzaAM6Ayp7y116ealCpAW4pQK8gt7p9uccLE2N8uRJxiZqLBgeDCi9FEwU2GENg2N7qyr ngBEphT/eTyxxOdduSaKC5xOr6SwDRfh7AnRO3KBJ3yY7A6vn3PBFERjxbxSBbh3NJmXNto5M2h 2rtgxTvSXYHyYxoATbLT/hmyITzSLRGKwElGdvccHEY58uRtT2fBrYqcyHt5m45Ms9ktpmQkBdy yGwLMxSLi+v/40vZguyxqaHTCg5LdVMj9CPtfHMopkiqncsFVoEGDwH/ZpnZbVVsST7/b5z0oWQ 1zDCjxIB22+UXCHteMxF2OiCyunzfGLWOdpQQcBxgRd9s7oA1OFGhrpx5AeOz3Tt9s3B3/vpegG SQKhvZgQimxn/yq3bM54yl06XL3XPME7COT3PmMbRDW3x/ILjK8sU5BOBMT6s8LrmtGXQRLX7G7 +ptpO1N7vTGiw== X-Received: by 2002:a05:7022:5f14:b0:12c:87f1:f40a with SMTP id a92af1059eb24-12c87f1f7dbmr3847202c88.18.1776812567826; Tue, 21 Apr 2026 16:02:47 -0700 (PDT) Received: from bbox-1.mtv.corp.google.com ([2a00:79e0:2e7c:8:4678:d28b:b946:bcc]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-12c74a20eb5sm26453546c88.14.2026.04.21.16.02.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2026 16:02:47 -0700 (PDT) Sender: Minchan Kim From: Minchan Kim To: akpm@linux-foundation.org Cc: hca@linux.ibm.com, linux-s390@vger.kernel.org, david@kernel.org, mhocko@suse.com, brauner@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, surenb@google.com, timmurray@google.com, Minchan Kim , Minchan Kim Subject: [PATCH v1 1/3] mm: process_mrelease: expedite clean file folio reclaim via mmu_gather Date: Tue, 21 Apr 2026 16:02:37 -0700 Message-ID: <20260421230239.172582-2-minchan@kernel.org> X-Mailer: git-send-email 2.54.0.rc1.555.g9c883467ad-goog In-Reply-To: <20260421230239.172582-1-minchan@kernel.org> References: <20260421230239.172582-1-minchan@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, process_mrelease() unmaps pages but file-backed pages are not evicted and stay in the pagecache, relying on standard memory reclaim (kswapd or direct reclaim) to eventually free them. This delays the immediate recovery of system memory under Android's LMKD scenarios, leading to redundant background apps kills. This patch implements an expedited eviction mechanism for clean pagecache folios in the mmu_gather code, similar to how swapcache folios are handled. It drops them from the pagecache (i.e., evicting them) if they are complete= ly unmapped during reaping. Within this single unified loop, anonymous pages are released via free_swap_cache(), and file-backed folios are symmetrically released via free_file_cache(). Signed-off-by: Minchan Kim --- arch/s390/include/asm/tlb.h | 2 +- include/linux/swap.h | 5 ++--- mm/mmu_gather.c | 7 ++++--- mm/swap.c | 42 +++++++++++++++++++++++++++++++++++++ mm/swap_state.c | 26 ----------------------- 5 files changed, 49 insertions(+), 33 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 619fd41e710e..2736dbb571a8 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -62,7 +62,7 @@ static inline bool __tlb_remove_folio_pages(struct mmu_ga= ther *tlb, VM_WARN_ON_ONCE(delay_rmap); VM_WARN_ON_ONCE(page_folio(page) !=3D page_folio(page + nr_pages - 1)); =20 - free_pages_and_swap_cache(encoded_pages, ARRAY_SIZE(encoded_pages)); + free_pages_and_caches(tlb->mm, encoded_pages, ARRAY_SIZE(encoded_pages)); return false; } =20 diff --git a/include/linux/swap.h b/include/linux/swap.h index 62fc7499b408..bdb784966343 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -414,7 +414,9 @@ extern int sysctl_min_unmapped_ratio; extern int sysctl_min_slab_ratio; #endif =20 +struct mm_struct; void check_move_unevictable_folios(struct folio_batch *fbatch); +void free_pages_and_caches(struct mm_struct *mm, struct encoded_page **pag= es, int nr); =20 extern void __meminit kswapd_run(int nid); extern void __meminit kswapd_stop(int nid); @@ -433,7 +435,6 @@ static inline unsigned long total_swapcache_pages(void) =20 void free_swap_cache(struct folio *folio); void free_folio_and_swap_cache(struct folio *folio); -void free_pages_and_swap_cache(struct encoded_page **, int); /* linux/mm/swapfile.c */ extern atomic_long_t nr_swap_pages; extern long total_swap_pages; @@ -510,8 +511,6 @@ static inline void put_swap_device(struct swap_info_str= uct *si) do { (val)->freeswap =3D (val)->totalswap =3D 0; } while (0) #define free_folio_and_swap_cache(folio) \ folio_put(folio) -#define free_pages_and_swap_cache(pages, nr) \ - release_pages((pages), (nr)); =20 static inline void free_swap_cache(struct folio *folio) { diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index fe5b6a031717..3c6c315d3c48 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -100,7 +100,8 @@ void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_= area_struct *vma) */ #define MAX_NR_FOLIOS_PER_FREE 512 =20 -static void __tlb_batch_free_encoded_pages(struct mmu_gather_batch *batch) +static void __tlb_batch_free_encoded_pages(struct mm_struct *mm, + struct mmu_gather_batch *batch) { struct encoded_page **pages =3D batch->encoded_pages; unsigned int nr, nr_pages; @@ -135,7 +136,7 @@ static void __tlb_batch_free_encoded_pages(struct mmu_g= ather_batch *batch) } } =20 - free_pages_and_swap_cache(pages, nr); + free_pages_and_caches(mm, pages, nr); pages +=3D nr; batch->nr -=3D nr; =20 @@ -148,7 +149,7 @@ static void tlb_batch_pages_flush(struct mmu_gather *tl= b) struct mmu_gather_batch *batch; =20 for (batch =3D &tlb->local; batch && batch->nr; batch =3D batch->next) - __tlb_batch_free_encoded_pages(batch); + __tlb_batch_free_encoded_pages(tlb->mm, batch); tlb->active =3D &tlb->local; } =20 diff --git a/mm/swap.c b/mm/swap.c index bb19ccbece46..e44bc8cefceb 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -1043,6 +1043,48 @@ void release_pages(release_pages_arg arg, int nr) } EXPORT_SYMBOL(release_pages); =20 +static inline void free_file_cache(struct folio *folio) +{ + if (folio_trylock(folio)) { + mapping_evict_folio(folio_mapping(folio), folio); + folio_unlock(folio); + } +} + +/* + * Passed an array of pages, drop them all from swapcache and then release + * them. They are removed from the LRU and freed if this is their last us= e. + * + * If @try_evict_file_folios is true, this function will proactively evict= clean + * file-backed folios if they are no longer mapped. + */ +void free_pages_and_caches(struct mm_struct *mm, struct encoded_page **pag= es, int nr) +{ + bool try_evict_file_folios =3D mm_flags_test(MMF_UNSTABLE, mm); + struct folio_batch folios; + unsigned int refs[PAGEVEC_SIZE]; + + folio_batch_init(&folios); + for (int i =3D 0; i < nr; i++) { + struct folio *folio =3D page_folio(encoded_page_ptr(pages[i])); + + if (folio_test_anon(folio)) + free_swap_cache(folio); + else if (unlikely(try_evict_file_folios)) + free_file_cache(folio); + + refs[folios.nr] =3D 1; + if (unlikely(encoded_page_flags(pages[i]) & + ENCODED_PAGE_BIT_NR_PAGES_NEXT)) + refs[folios.nr] =3D encoded_nr_pages(pages[++i]); + + if (folio_batch_add(&folios, folio) =3D=3D 0) + folios_put_refs(&folios, refs); + } + if (folios.nr) + folios_put_refs(&folios, refs); +} + /* * The folios which we're about to release may be in the deferred lru-addi= tion * queues. That would prevent them from really being freed right now. Th= at's diff --git a/mm/swap_state.c b/mm/swap_state.c index 6d0eef7470be..7576bf36d920 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -400,32 +400,6 @@ void free_folio_and_swap_cache(struct folio *folio) folio_put(folio); } =20 -/* - * Passed an array of pages, drop them all from swapcache and then release - * them. They are removed from the LRU and freed if this is their last us= e. - */ -void free_pages_and_swap_cache(struct encoded_page **pages, int nr) -{ - struct folio_batch folios; - unsigned int refs[PAGEVEC_SIZE]; - - folio_batch_init(&folios); - for (int i =3D 0; i < nr; i++) { - struct folio *folio =3D page_folio(encoded_page_ptr(pages[i])); - - free_swap_cache(folio); - refs[folios.nr] =3D 1; - if (unlikely(encoded_page_flags(pages[i]) & - ENCODED_PAGE_BIT_NR_PAGES_NEXT)) - refs[folios.nr] =3D encoded_nr_pages(pages[++i]); - - if (folio_batch_add(&folios, folio) =3D=3D 0) - folios_put_refs(&folios, refs); - } - if (folios.nr) - folios_put_refs(&folios, refs); -} - static inline bool swap_use_vma_readahead(void) { return READ_ONCE(enable_vma_readahead) && !atomic_read(&nr_rotate_swap); --=20 2.54.0.rc1.555.g9c883467ad-goog From nobody Wed Jun 17 02:50:08 2026 Received: from mail-dl1-f41.google.com (mail-dl1-f41.google.com [74.125.82.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6C6E37BE6B for ; Tue, 21 Apr 2026 23:02:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776812572; cv=none; b=DTf3QpJWwh41fDIr5xK/sPUKv9YdpKYwdgSDJvT1ubZBVL5V/D4o7J6xhrXwcPf7rYggVYr3J0/rtRUwHMLvkw30nxkQjOgY+AcK3n3KQq4on8e2wpxhCJoKoMO4SzIWmRhVeSlPV+NNzZjfhg183VM1/dArcvoPR6+8cC86JiU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776812572; c=relaxed/simple; bh=9NM8vR01ylWdR69tTI/CB/XMMKN6dhRFED8qVzMJFYg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y2dv4GgI8Y6sFdiImuzrMkZnko4vVqDy1RIKFSW1CpQo9GT8SmeVGo1/s0BJoW4ugIQqb6uHMbtFgghV9y9rwI7PSVpIjGfw61xvCpVSgsjrUD2Gsj/9P7W4Q0je+U59T9TTULqRyHrO47Q8dC9vMwUEpmzmczDH2KF35khnYQM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=qxZpJS/4; arc=none smtp.client-ip=74.125.82.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qxZpJS/4" Received: by mail-dl1-f41.google.com with SMTP id a92af1059eb24-12c8cc7a77eso1820712c88.1 for ; Tue, 21 Apr 2026 16:02:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1776812570; x=1777417370; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=VDzLT8xllq1iBGA+qtx0OEDuRYSfNoyHs8R3lM9bdik=; b=qxZpJS/4UUR1J37RA5G5fIuwrVmskObSTSwd2ynP7IcSUvOHLFrwAG2fU+vC8o4UFA LRSje7v5h/Vm1UUEJvRUOd9XbsEAP3g82juFNX3WceyMxx9b9g48jVua+eQHlxRD6ue1 GPyO56jxQ7zTuPSDRRPHxtR01BM+JfJnY1xAFpfwE6XaN5yL+kypn9en6AukTUcj/HxQ PJKEYP8q27XlN+UPdPDolT9c82p5TMWUTt+xDSztWCQ2MrGakt6K/qOVPZ9zIcDURQWc PEVcTFd6aw9gPyRQjTtinFb962RAEg03bbSlEscE7iua4lnDCM+kNxNn0T8yqSTgcTYn S4QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776812570; x=1777417370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VDzLT8xllq1iBGA+qtx0OEDuRYSfNoyHs8R3lM9bdik=; b=gz4Wzp8s3vTobQE/PHRZ1WJNhFN+/AXArfRWTjdsnHJBECWAdjO2QB3kKrHiVeB2mB cv8r2bgBJiAWljZ7LC06Hck9LN0JAx3abgtxh8buqBuGuBMf1fLBLOR4BYqebxaC9E5K ZJAFrVOMekaXYpGW3ETE2XzcpY46VapS2A81l6egFywQimNizWG2h2qNGcCYed3Z2Iyz QXL9lyuCeiodmhlQU8OaDrs39X5uI2UK1LUloVcar5FXlyNH6G8VLkdSzCOat3voR7CD MAlaH6wJ080XU/aE3Wh7hvtEZ4X6ClWu9rZ5R+P107UMrGVpC6MXtI9GOiPxABZwfSpW NAtQ== X-Forwarded-Encrypted: i=1; AFNElJ8G7S8vL8MwcTzlUJcbdgniBVC6LmNq7JcTUMpRYrXmDYs/fuxRn5dl3tzzo+vwbje+OexhY93GOWbWT4Y=@vger.kernel.org X-Gm-Message-State: AOJu0YzPbJbHUsW/9tggTPoWDv8AOtEQJ1dGImnOzJW+1Ecnum5BPv7M X/ZvQ5+Ew1ORaSK4FJ2psmtM6PzskDlYfCj+2jb2DJBU4pgGZ9aPLs/n X-Gm-Gg: AeBDietuKgPj08KliDnwGMfx1L6KkqzLJTUdCqXs3BChzM2wmXF3uU5ZdMKgQsZExAj FDUbBRdiY1bEFosBnM5gr+9mT3RFAlUHqPwTwxyCWg+/0iIxM8wknGsMaQdWgu7ucXB0ExBWC6V mFlvj8SeWbdOlPV3/Mwdv32ik2J2c4z6b0pv/StZeZfczoM7pr/ywJ6RKHRHzwSgpGewB9tKxWd F8BJPMhrD0wdOXqis1R/VOkrquD8vmQY7hdo+6DdevLi0ModnAuCf1ZAdEKmHKZz66zjVATAkf8 GWNMNU79ckWRvNzcUTDMRKZiG7Bp/+ffoqMD7CXvON3fVxt66wTgfhL4YFvRPjBIS3OjUTzhn5S uuEZKq3gVR5UQc5w3HrwhbbhRf+x5ixfezQHQK3O/2n1/9a2/lodbQA7NKQkb12f0CifsYcZFrE D2dXD0BQ5ZS/JtcNuzTuzj8Cd8BoV4z3EZZohqx+8Z0YHMoI6aGcD/fqDzHgSIqo/2ksdGwaGcd JVOexEwti7orA== X-Received: by 2002:a05:7022:e985:b0:12d:b2ca:a9e9 with SMTP id a92af1059eb24-12db2caab80mr2271908c88.8.1776812569428; Tue, 21 Apr 2026 16:02:49 -0700 (PDT) Received: from bbox-1.mtv.corp.google.com ([2a00:79e0:2e7c:8:4678:d28b:b946:bcc]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-12c74a20eb5sm26453546c88.14.2026.04.21.16.02.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2026 16:02:48 -0700 (PDT) Sender: Minchan Kim From: Minchan Kim To: akpm@linux-foundation.org Cc: hca@linux.ibm.com, linux-s390@vger.kernel.org, david@kernel.org, mhocko@suse.com, brauner@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, surenb@google.com, timmurray@google.com, Minchan Kim , Minchan Kim Subject: [PATCH v1 2/3] mm: process_mrelease: skip LRU movement for exclusive file folios Date: Tue, 21 Apr 2026 16:02:38 -0700 Message-ID: <20260421230239.172582-3-minchan@kernel.org> X-Mailer: git-send-email 2.54.0.rc1.555.g9c883467ad-goog In-Reply-To: <20260421230239.172582-1-minchan@kernel.org> References: <20260421230239.172582-1-minchan@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For the process_mrelease reclaim, skip LRU handling for exclusive file-backed folios since they will be freed soon so pointless to move around in the LRU. This avoids costly LRU movement which accounts for a significant portion of the time during unmap_page_range. - 91.31% 0.00% mmap_exit_test [kernel.kallsyms] [.] exit_mm exit_mm __mmput exit_mmap unmap_vmas - unmap_page_range - 55.75% folio_mark_accessed + 48.79% __folio_batch_add_and_move 4.23% workingset_activation + 12.94% folio_remove_rmap_ptes + 9.86% page_table_check_clear + 3.34% tlb_flush_mmu 1.06% __page_table_check_pte_clear Signed-off-by: Minchan Kim --- mm/memory.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 2f815a34d924..fcb57630bb8d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1640,6 +1640,8 @@ static __always_inline void zap_present_folio_ptes(st= ruct mmu_gather *tlb, bool delay_rmap =3D false; =20 if (!folio_test_anon(folio)) { + bool skip_mark_accessed; + ptent =3D get_and_clear_full_ptes(mm, addr, pte, nr, tlb->fullmm); if (pte_dirty(ptent)) { folio_mark_dirty(folio); @@ -1648,7 +1650,16 @@ static __always_inline void zap_present_folio_ptes(s= truct mmu_gather *tlb, *force_flush =3D true; } } - if (pte_young(ptent) && likely(vma_has_recency(vma))) + + /* + * For the process_mrelease reclaim, skip LRU handling for exclusive + * file-backed folios since they will be freed soon so pointless + * to move around in the LRU. + */ + skip_mark_accessed =3D mm_flags_test(MMF_UNSTABLE, mm) && + !folio_maybe_mapped_shared(folio); + if (likely(!skip_mark_accessed) && pte_young(ptent) && + likely(vma_has_recency(vma))) folio_mark_accessed(folio); rss[mm_counter(folio)] -=3D nr; } else { --=20 2.54.0.rc1.555.g9c883467ad-goog From nobody Wed Jun 17 02:50:08 2026 Received: from mail-dl1-f51.google.com (mail-dl1-f51.google.com [74.125.82.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E43DF38655B for ; Tue, 21 Apr 2026 23:02:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776812573; cv=none; b=u9gQ77xAyqaf9eSEzWkTY15KKQ2XqAzLCj1Nk2T53pJQbdNJR28xZUEnCh1bkqkqS2nNe0rbmvydck8EkGLjHw+WI+HM8WQex+TVSSx6xPAfB7Tlv69duty7dRZGTMCoTslbY0tLdIR81q7/5FmewN1KdaQZzlwzNQwphq0bSdw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776812573; c=relaxed/simple; bh=uCqTMx2Vrw9oUnN/NUn/zWaQzWq7nFi9S8FCWPuGSDE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hhyT0NilLRaFujtjPz8OnME8L8HrsSgNR2bA6mcjrt/2zqwAHi0a+JeNyBQhc8n9kRXPOsUOJf3nPZWnXb9wbZou1UgBVDGTY28Q8MyX+pjMjpL7hnMTxr6aiSoEkcgiKfI7X1XALPTOrZ8vMcQbSKosO4vsqw44QvXKJFdI9R4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XlcNzM/I; arc=none smtp.client-ip=74.125.82.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XlcNzM/I" Received: by mail-dl1-f51.google.com with SMTP id a92af1059eb24-12c7212836bso12630257c88.0 for ; Tue, 21 Apr 2026 16:02:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1776812571; x=1777417371; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=lwpK6htKdPchlWbZaSeEAHqzDpDsOuggPLlaoXdeCJA=; b=XlcNzM/IyJTeUf5regBMn924oOIzWAhakBeum3kCBO7S42JL3e7yNlI3ERVgtNWrlr wmP7luqzFVpWHlffLkRaLQJbBIVSlPCbuolRR17NyoUYl14EfhwyOxEUgXqb/YKqnq+j 6gGqbKVqLgOLIcyh3l+Lt6/1pzpVKcpfJaHQshErNChm580Rlyopc9saATwJgJ7TWV/k OttQGhe2MRPT0TeQBrOPruleBlgUScKtbLIks2xmaC9iVonYjuYIEunw9FLZsJpVExv8 C3BuA/sRPuEoNzqEhqrJd2+h3JyD3z2f4HChAXFV6emm7GbptaYCc76gg2cXD/kVvFLX n37Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776812571; x=1777417371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lwpK6htKdPchlWbZaSeEAHqzDpDsOuggPLlaoXdeCJA=; b=Z/uZcMNVbLNB0ZmuRnjpUdjYHC7kC0Gg4RgoL7WXBB5/XUgKwuyB2hxaEkMdTnQLJP ZAN9Mlh+Ht/NfwV2zmaTJhWyV9fD3e2b5IXtlEZ67GtnHxHarGZetJbpuT4mLLl9PDVW wOlRIXwCrlFU8MxVx6OD461asS/lSEsCU8j4RxIaRK+UZNJMSL4iqa4Z5mYlakr5pRY+ 2Jl8uzZZ24gs7J9ZsS8U2CEuB/zTMCKHPqihgmYc0/eClI/CeWRoLRcEGroz9Jxxe/6C BvGyZ/2l69L4fDNE0r4j4x5eAqvmiX1T9eF1o90qm6D0WgSKB9cyd8YvrDjpqw0DhG1z 8vXQ== X-Forwarded-Encrypted: i=1; AFNElJ8kvJ6IsWu1XZd/Lw8amWZRqfZjQHjsQ2nMY4DN8RlyLH5qi1NirmfmegMW9mrzNPRIRsen/FD3WhJLNiI=@vger.kernel.org X-Gm-Message-State: AOJu0Yw8+IHZPri2Irb92u4Ssf4jsF9emTHh1vsVeGM3Q2V4Zy3ebX/R hEGdUqfBGqa8HUF7/hTzd6tgn+Q7/dOScrIEQlVvz3ATWbuD6voXUdlZ X-Gm-Gg: AeBDietWm8dPkinNX2j8Z9IsUBD7YbLD8SYVIHfgY8T92ZV0svj7Z3Unb5JMO7ywx4S N2gSer/WTvO76LsSEQHMWVd6dBJP+qObw2wSweOV1/0neEcm1y+zRE6vZw2n5IpDl1X7zDNw/2M OneDyyLRMWwM0n+JD1ijywRTOZiM1JpWX/nnoWBKsJDZMb6C2tH6ILoD4hnw6UAHTytrSwv579d o6N19LTiMAslaMamAup9AV9KGnTCSmeDAb2tS8kHzi0PGf3BKC6rik5jGHlQ5YIimBRQ1xfGZkc E/Tr0EBZx1f0t9D1BID9B6+gSPS5rbG1Q+zICmv8Awwypnk7F0yuKkcwGHY9KYJmp3D95pNzflf ELKulO+7IkZoUz3iCQ0kF9kcLubc4uwwinCIccuRbShOZNgjCI1ldIbf/o5CMNzYtbDvuTKBgd4 X3NSptDvG6XEMdy/54/BkCTb8PqqjLYQo0PZEW71jHqHPNFTH0ug6gq4ismvGQ+sZgVXD07jVx/ v8KeWCwfFZvmQ== X-Received: by 2002:a05:701b:2415:b0:12d:b26f:cafd with SMTP id a92af1059eb24-12db26fcc16mr2120556c88.5.1776812570960; Tue, 21 Apr 2026 16:02:50 -0700 (PDT) Received: from bbox-1.mtv.corp.google.com ([2a00:79e0:2e7c:8:4678:d28b:b946:bcc]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-12c74a20eb5sm26453546c88.14.2026.04.21.16.02.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2026 16:02:50 -0700 (PDT) Sender: Minchan Kim From: Minchan Kim To: akpm@linux-foundation.org Cc: hca@linux.ibm.com, linux-s390@vger.kernel.org, david@kernel.org, mhocko@suse.com, brauner@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, surenb@google.com, timmurray@google.com, Minchan Kim , Minchan Kim Subject: [PATCH v1 3/3] mm: process_mrelease: introduce PROCESS_MRELEASE_REAP_KILL flag Date: Tue, 21 Apr 2026 16:02:39 -0700 Message-ID: <20260421230239.172582-4-minchan@kernel.org> X-Mailer: git-send-email 2.54.0.rc1.555.g9c883467ad-goog In-Reply-To: <20260421230239.172582-1-minchan@kernel.org> References: <20260421230239.172582-1-minchan@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, process_mrelease() requires userspace to send a SIGKILL signal prior to the call. This separation introduces a scheduling race window where the victim task may receive the signal and enter the exit path before the reaper can invoke process_mrelease(). When the victim enters the exit path (do_exit -> exit_mm), it clears its task->mm immediately. This causes process_mrelease() to fail with -ESRCH, leaving the actual address space teardown (exit_mmap) to be deferred until the mm's reference count drops to zero. In Android, arbitrary reference cou= nts (e.g., async I/O, reading /proc//cmdline, or various other remote VM accesses) frequently delay this teardown indefinitely, defeating the purpose of expedited reclamation. This delay keeps memory pressure high, forcing the system to unnecessarily kill additional innocent background apps before the memory from the first victim is recovered. This patch introduces the PROCESS_MRELEASE_REAP_KILL UAPI flag to support an integrated auto-kill mode. When specified, process_mrelease() directly injects a SIGKILL into the target task. To solve the race condition deterministically, we grab the mm reference via mmget() and set the MMF_UNSTABLE flag *before* sending the SIGKILL. Using mmget() instead of mmgrab() keeps mm_users > 0, preventing the victim from calling exit_mmap() in its own exit path. This ensures that the memory is reclaimed synchronously and deterministically by the reaper in the context of process_mrelease(), avoiding delays caused by non-deterministic scheduling of the victim task. Signed-off-by: Minchan Kim Reviewed-by: Suren Baghdasaryan --- include/uapi/linux/mman.h | 4 +++ mm/oom_kill.c | 56 +++++++++++++++++++++++++++------------ 2 files changed, 43 insertions(+), 17 deletions(-) diff --git a/include/uapi/linux/mman.h b/include/uapi/linux/mman.h index e89d00528f2f..4266976b45ad 100644 --- a/include/uapi/linux/mman.h +++ b/include/uapi/linux/mman.h @@ -56,4 +56,8 @@ struct cachestat { __u64 nr_recently_evicted; }; =20 +/* Flags for process_mrelease */ +#define PROCESS_MRELEASE_REAP_KILL (1 << 0) +#define PROCESS_MRELEASE_VALID_FLAGS (PROCESS_MRELEASE_REAP_KILL) + #endif /* _UAPI_LINUX_MMAN_H */ diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 5c6c95c169ee..730ba0d19b53 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -20,6 +20,7 @@ =20 #include #include +#include #include #include #include @@ -850,7 +851,7 @@ bool oom_killer_disable(signed long timeout) return true; } =20 -static inline bool __task_will_free_mem(struct task_struct *task) +static inline bool __task_will_free_mem(struct task_struct *task, bool ign= ore_exit) { struct signal_struct *sig =3D task->signal; =20 @@ -862,6 +863,9 @@ static inline bool __task_will_free_mem(struct task_str= uct *task) if (sig->core_state) return false; =20 + if (ignore_exit) + return true; + if (sig->flags & SIGNAL_GROUP_EXIT) return true; =20 @@ -878,7 +882,7 @@ static inline bool __task_will_free_mem(struct task_str= uct *task) * Caller has to make sure that task->mm is stable (hold task_lock or * it operates on the current). */ -static bool task_will_free_mem(struct task_struct *task) +static bool task_will_free_mem(struct task_struct *task, bool ignore_exit) { struct mm_struct *mm =3D task->mm; struct task_struct *p; @@ -892,7 +896,7 @@ static bool task_will_free_mem(struct task_struct *task) if (!mm) return false; =20 - if (!__task_will_free_mem(task)) + if (!__task_will_free_mem(task, ignore_exit)) return false; =20 /* @@ -916,7 +920,7 @@ static bool task_will_free_mem(struct task_struct *task) continue; if (same_thread_group(task, p)) continue; - ret =3D __task_will_free_mem(p); + ret =3D __task_will_free_mem(p, false); if (!ret) break; } @@ -1034,7 +1038,7 @@ static void oom_kill_process(struct oom_control *oc, = const char *message) * so it can die quickly */ task_lock(victim); - if (task_will_free_mem(victim)) { + if (task_will_free_mem(victim, false)) { mark_oom_victim(victim); queue_oom_reaper(victim); task_unlock(victim); @@ -1135,7 +1139,7 @@ bool out_of_memory(struct oom_control *oc) * select it. The goal is to allow it to allocate so that it may * quickly exit and free its memory. */ - if (task_will_free_mem(current)) { + if (task_will_free_mem(current, false)) { mark_oom_victim(current); queue_oom_reaper(current); return true; @@ -1217,8 +1221,9 @@ SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigne= d int, flags) unsigned int f_flags; bool reap =3D false; long ret =3D 0; + bool reap_kill; =20 - if (flags) + if (flags & ~PROCESS_MRELEASE_VALID_FLAGS) return -EINVAL; =20 task =3D pidfd_get_task(pidfd, &f_flags); @@ -1236,19 +1241,33 @@ SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsig= ned int, flags) } =20 mm =3D p->mm; - mmgrab(mm); =20 - if (task_will_free_mem(p)) - reap =3D true; - else { - /* Error only if the work has not been done already */ - if (!mm_flags_test(MMF_OOM_SKIP, mm)) + reap_kill =3D !!(flags & PROCESS_MRELEASE_REAP_KILL); + reap =3D task_will_free_mem(p, reap_kill); + if (!reap) { + if (reap_kill || !mm_flags_test(MMF_OOM_SKIP, mm)) ret =3D -EINVAL; + + task_unlock(p); + goto put_task; } - task_unlock(p); =20 - if (!reap) - goto drop_mm; + if (reap_kill) { + /* + * We use mmget() instead of mmgrab() to keep mm_users > 0, + * preventing the victim from calling exit_mmap() in its + * own exit path. This ensures that the memory is reclaimed + * synchronously and deterministically by the reaper. + */ + mmget(mm); + task_unlock(p); + ret =3D kill_pid(task_tgid(task), SIGKILL, 0); + if (ret) + goto drop_mm; + } else { + mmgrab(mm); + task_unlock(p); + } =20 if (mmap_read_lock_killable(mm)) { ret =3D -EINTR; @@ -1263,7 +1282,10 @@ SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsign= ed int, flags) mmap_read_unlock(mm); =20 drop_mm: - mmdrop(mm); + if (reap_kill) + mmput(mm); + else + mmdrop(mm); put_task: put_task_struct(task); return ret; --=20 2.54.0.rc1.555.g9c883467ad-goog