From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f50.google.com (mail-yx1-f50.google.com [74.125.224.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5BB55389473 for ; Wed, 9 Sep 2026 09:42:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788946938; cv=none; b=VbvvYjWFsqvHD3cMB07Gh65UewMa4XTQMJvhPE+f0Zdd3Na8o4jsjp1C0WLOH1PodKIeNVdRh/xVVoKVozTWq3UepSZgDxj0bKHldw4Ky7NHihgQ4umWtD5+ctkcs16l4xiPr6UU1A0hY3SuL4TsDW8ioTBaMAgUtNxX+yz3jnY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788946938; c=relaxed/simple; bh=N/QTtPy3Eq36G6jCKFgYYujyz3PwIl5VjjdzH4vnlfY=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=YW2Qn+raOWWfJRj/9VWwYOmCTF4jdeujwaHxWK4zVf738IiwPuheu8dWNVZOKQnQggksSlP3SFfCQRZSUGQC6GHcJIJuyIiOFec0FFymFfK5XWC4PdogB9tINTXAWt6Al0JaxRrub4hVYciW3ReTJ72a61+SKr+50Pn+qW1wN60= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HcRhWs0M; arc=none smtp.client-ip=74.125.224.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HcRhWs0M" Received: by mail-yx1-f50.google.com with SMTP id 956f58d0204a3-66f7f62e915so6259435d50.3 for ; Wed, 09 Sep 2026 02:42:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788946936; x=1789551736; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+ZhIjaKSKu0jnDwRfSlp/FTaG5xLy46pxEQXC6FqYeA=; b=HcRhWs0MZClW1Z66abZaVnOFx91ik7apWsDxqQBmAPPsCGETn8eTjhzQ1UpbjEFJHo /GvxoAxhjCsF9BMxMoh+veQRo61WlM7guQss/g7xDLj0ujkyhZlr8s/BPOkGZjs/STb7 A8P51g1RhmJ0UsC0XcO8WfO5Z8bWoHzK72srU1kr6t6W+14Usb6GguWw++CIdljQBi9a G9k24gGyLZbwbO1Oj8EmOB4ySbmdzpmNqdc4jrL17lOo9p4Eo7OGD67ETqjJegfIB/fG WxniyFu5Z1bXuCg/eGi6KEaZuHehM35z2N8V+KJr1JRYSPs9CBSRF3Mait4j4rH+lpLS 3bWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788946936; x=1789551736; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+ZhIjaKSKu0jnDwRfSlp/FTaG5xLy46pxEQXC6FqYeA=; b=s2J9v/ifKhOYXkpcDJ+XOHuE7yTG8ZHjyyy9A9Cl/87Qytxdyxvmmf7LbVUUaRoEIP aMe5N0zKP5+edKSeYO8Bfl08mMfmtHQjRmJnD+3TD3WCY5nqlQgI+O8q3DXMvmuWB+qy K0cyYM13Y35DBBABqhP9ZzqcNVY3v0qXNpV9dSpzZMnr0Yvv7caD3St8RoYRiotS9iVY 9g4601aey75EIpxAEuhSoest0IlvzLaJ6v9Jyj0JaXz7UaPHp74e0i2H5OAMb0Evb8Gl ipG00m5GjiNBXy3NXu+8HYaiUZhwCYyBuKoaC1BOPoZI7W20kTYIVzsXbNZUi2f0hwBE tF+g== X-Forwarded-Encrypted: i=1; AKwUvByNCrGU+s+Uzms0tGn3mL0UPb+t6io+QZFpATxBMGotrETahwJZo+q8YSCR5ZY3W8ePIuvUa8/hfz351J0=@vger.kernel.org X-Gm-Message-State: AFuF++lkm48KHd9yDEmL9ZIwzqomWzME9XZdEMhBSLSStqLsUbuUmhY/ yfJ41w6OHQ3sUF0DOyfGWP3OQtiUTBQzts9cNd8iEywQKCEveHERt49vXM219NDb5Q== X-Gm-Gg: AYBFou3cTmzbE9ax12sSnAKNLIlSjAY79u4jpnsPcFQh9uFq30jYiV3adfqGdTi7fxk j4rzDg5nQ29mHNB2JZMQ/WmMmC32hDx0ss4gMTThEFUkE4mh8tSIqpslhZK3BM5MILFwhN2nnph 8RqV+8kK2JIFxPf1k5T+dEO582Vgv9lycRhKlXXuxgIgW8agHG7m3IT1N2Z2x1E+1KAMt7xS37B 1TURRzGzYsT916QWWyZlfyHXpNrqtO2odni3aoZUVRctKyntgXqS8OwiEDxIDqY+Baf/AysM+RO QvosrNfEWRZJK7eQPVw+rXzOOTeRezcHQ9ect3DVoCOBu+AKBVYfZQqy9B/IkorSY7D7qNHyTs3 +fgdUsPBKpUEa2GwI1eI0jLSEuXgJBzOfsWYN3lbmkAKWDUjA1XiHd/lxp4luCypD2poQlk6C0q qhdmgPgtRoGoXkToj6pYSNmL9kwxwV9y4Ft5ZgTvqJWU8H0MAG311p+axmtCQpazwSRfIqz9dke 2NQf3pJXyVqMery7LtYADB3d4uMUWLynD0F7TH/A2OVhuo6a763msrdBcmu X-Received: by 2002:a53:a043:0:b0:66f:c1be:3187 with SMTP id 956f58d0204a3-66fc1be358emr5989382d50.94.1788946935368; Wed, 09 Sep 2026 02:42:15 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb47a8b8fsm12141411d50.2.2026.09.09.02.42.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:42:14 -0700 (PDT) Date: Wed, 9 Sep 2026 02:42:10 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 01/26] mm/fbatch: remove !CONFIG_SMP special case of folio_activate() In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" 3.0 commit eb709b0d062e ("mm: batch activate_page() to reduce lock contention") brought in an ifdef CONFIG_SMP around activate batching: https://lore.kernel.org/linux-mm/20100805140755.501af8a7.akpm@linux-foundat= ion.org/ shows a sensitivity to bloat that day, not any incompatibility with UP. No other batching here has a UP alternative, and it's a bit confusing: simplify mm/folio.c a little by removing it now. Certainly we can reduce UP bloat (and/or 32-bit bloat) by, say, lowering FOLIO_BATCH_SIZE from 31: traditionally 16, 14, 15, then raised to 31 by 6.9 commit 9cecde80aae0 ("mm: increase folio batch size"); or by giving just the static per-cpu folio batches a type of their own with a smaller array size on UP (1? or a little batching worthwhile even on UP?). But not right now, it's orthogonal to this series. And I suspect that the old ifdef led to lru_activate being placed last, whereas it's usually the second most popular fbatch: move it there, to match cpu_needs_drain() comment "Check these in order of likelihood that they're not zero". Signed-off-by: Hugh Dickins Acked-by: David Hildenbrand (Arm) Reviewed-By: Vlastimil Babka (SUSE) --- mm/folio.c | 40 ++++++---------------------------------- 1 file changed, 6 insertions(+), 34 deletions(-) diff --git a/mm/folio.c b/mm/folio.c index 50a6dbe55998..c093ca900a3e 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -48,12 +48,10 @@ struct cpu_fbatches { */ local_lock_t lock; struct folio_batch lru_add; + struct folio_batch lru_activate; struct folio_batch lru_deactivate_file; struct folio_batch lru_deactivate; struct folio_batch lru_lazyfree; -#ifdef CONFIG_SMP - struct folio_batch lru_activate; -#endif /* Protecting the following batches which require disabling interrupts */ local_lock_t lock_irq; struct folio_batch lru_move_tail; @@ -283,15 +281,6 @@ static void lru_activate(struct lruvec *lruvec, struct= folio *folio) count_memcg_events(lruvec_memcg(lruvec), PGACTIVATE, nr_pages); } =20 -#ifdef CONFIG_SMP -static void folio_activate_drain(int cpu) -{ - struct folio_batch *fbatch =3D &per_cpu(cpu_fbatches.lru_activate, cpu); - - if (folio_batch_count(fbatch)) - folio_batch_move_lru(fbatch, lru_activate); -} - void folio_activate(struct folio *folio) { if (folio_test_active(folio) || folio_test_unevictable(folio) || @@ -301,25 +290,6 @@ void folio_activate(struct folio *folio) folio_batch_add_and_move(folio, lru_activate); } =20 -#else -static inline void folio_activate_drain(int cpu) -{ -} - -void folio_activate(struct folio *folio) -{ - struct lruvec *lruvec; - - if (!folio_test_clear_lru(folio)) - return; - - lruvec =3D folio_lruvec_lock_irq(folio); - lru_activate(lruvec, folio); - lruvec_unlock_irq(lruvec); - folio_set_lru(folio); -} -#endif - static void __lru_cache_activate_folio(struct folio *folio) { struct folio_batch *fbatch; @@ -628,6 +598,10 @@ void lru_add_drain_cpu(int cpu) trace_mm_lru_add_drain_tp(cpu, nr_folios); } =20 + fbatch =3D &fbatches->lru_activate; + if (folio_batch_count(fbatch)) + folio_batch_move_lru(fbatch, lru_activate); + fbatch =3D &fbatches->lru_move_tail; /* Disabling interrupts below acts as a compiler barrier. */ if (data_race(folio_batch_count(fbatch))) { @@ -650,8 +624,6 @@ void lru_add_drain_cpu(int cpu) fbatch =3D &fbatches->lru_lazyfree; if (folio_batch_count(fbatch)) folio_batch_move_lru(fbatch, lru_lazyfree); - - folio_activate_drain(cpu); } =20 /** @@ -759,11 +731,11 @@ static bool cpu_needs_drain(unsigned int cpu) =20 /* Check these in order of likelihood that they're not zero */ return data_race(folio_batch_count(&fbatches->lru_add) || + folio_batch_count(&fbatches->lru_activate) || folio_batch_count(&fbatches->lru_move_tail) || folio_batch_count(&fbatches->lru_deactivate_file) || folio_batch_count(&fbatches->lru_deactivate) || folio_batch_count(&fbatches->lru_lazyfree) || - folio_batch_count(&fbatches->lru_activate) || need_mlock_drain(cpu)) || has_bh_in_lru(cpu, NULL); } --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6379485CE1 for ; Wed, 9 Sep 2026 09:44:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947083; cv=none; b=dwCRY/rqbVnKJCW+pRoCYwYF1+dBNIgh3STC2umIDjgLva4pwNzaWN6r16FZgVjCcOt6FYZh7duWmUuQOct6Fb1WlVUX8ZRCpqyY7xtJiztQCUho5yQS9tu//SxsDDTsvicLzRFh5htzxsVbel/jFBldnXZQ0nov3N/f5bkxw5Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947083; c=relaxed/simple; bh=bTUnCSRZY4G+HM39v0Xkm1aeH5odwpKTGFZaGF+zFkU=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=gHLHILadtFoPs4aqQ7SYd6JhkCVuZ3+0t7fIcrRcLZPX/1vWp0SdEQnzyWVdAs0zYzTXjJWsnf7jwr6uB8+umbqqMyBnQQdS5LNDaS9jwiCYXTPOXMe7W2gmW+r4UKrtXIBJaMqvQtGqMvUpPz4NTURoWZzuGxaYOkPmV6B3XRE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tdkF8xNK; arc=none smtp.client-ip=209.85.128.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tdkF8xNK" Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-86162c086f8so53598687b3.1 for ; Wed, 09 Sep 2026 02:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947081; x=1789551881; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=l/77qcoGASc/bcotpYpu7s8Toxgxo4GzXR5d/4rRuxI=; b=tdkF8xNKlSCsIfsWgL91//5JVWgbkFK8LI9ElBpjlcX2oh0npUSYMyYmAKaCdOFusp lzLiD0XQai4k68ZG54JivS5/UV9clbwroSxZjzDWKC/XRqHNG+PpAPOvrp9xtVCCpVLi w5PywqOGLYlOtNtQ7Mgj4iDroomAxlO8O9QpcHgCKmwr74pp6TDdr/r1CoWVwt9FuiVK O23DIzXJzukdhqm3IXUwBBV984B1jGtcg6hWtagQoz3HgCPFL8v2/vSCaWqekEVJbo8L jh2bXikkE0jWIqw4pouWidbBZb8Kik95aoNPF1D5Ai6yygBz4gICB8vPsBmPSaBfoc8H ve2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947081; x=1789551881; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=l/77qcoGASc/bcotpYpu7s8Toxgxo4GzXR5d/4rRuxI=; b=JN6+KF5bk2DOoMXVQDXh9LACaUHgGfYaFDGYMYHhLf9tiIHjS1rkMdknLTK5O+FcSd nC/ryFOhC7y2rVW9uYUE8VtGk4JunNleCzGgKKZi6hglXK7eGWnRNxfWpqTt11kf4ASh lkn1p7gW73JNfzfJNXrchp4wYFuvt14Bff/2pbVHLhomheMkcFswmv9Yt9g7epDRiEEC 0o9pNOC4I6J4TRBRIJeo1j5eGVN+7t0LC8ACKM5PE948r4geamJO1oaCGifOgdIxKJxS lobzLFbDkQFPJa7sMQub5iEpBBnP8J7Z33xLHkifLP3VBZstBWKOHlKmcRl0LfJEEOP3 XmSg== X-Forwarded-Encrypted: i=1; AKwUvBx3a+9+R0Gdui5+HgfIAg2Jk1t47XioSItQ2Vg6azAC57FUjMnDJjsbgXN72D6Jz5IrK8ckv3Uh/5GOCqc=@vger.kernel.org X-Gm-Message-State: AFuF++m64UXN8lYUfyOFtXlMjnoZK2BQ5u5Og10VYg5PHyCQsnJT5BtP eWGQO4QHMmQiYZtQBBO5V1OpHgdSyCph1i7tVOV6YRDz9gQammKyO75YqLC5TjNF6A== X-Gm-Gg: AYBFou3je6of3CSiy83Mx4vLvw0rse7K2LrcJDevizTJJR5cdtEZM0EPYUlqJBhP/Lk wmQkKreZQIHi7HIFc0X5V9H1aHXKkJJJjefQ+44D/SGCRaqKDmfAARtu8CNUjkPdA8mfLEYukHW Zh3VDZE78jBMinmSuxWU7rT/4kgvxp3fSR+SfeCvKxrB7UoJmkCfY5fNW4Vy37KUJB5nNoqnUSc KO/7FUU2b1WI5fd2RFlaUqk1ybEmm3kEgmI6hkfz/QJJU8FgI3sRmMnKzssyEFuhH8yV00Z/+ai JW4LARllpAmtKguTryIVJvEGCQKmkm3OrhQ1bmRzxLMJUjrn9Q5ULj/DPa1/hnJAJ2jJb4x2I5N 0Bgsl0dJpgpmCQriSSJGaxQepIckpU/PvlCkxlALfjgTSxyHRjMpBJDLJgp8qwhaPd5aMFeWtBL FU9NM0AU5IZvH9ByN06VEKvGt2+zd3s3a8iBOPYMuuv4XzkYrpqe4Tmh9f259X3/rHo7c6Q5Sgd uzSXdlzf9Y6JTW8mRQV0V+Xdib7WSCfcbLWLyyNB47oDigJzQXRkJawXoKy X-Received: by 2002:a05:690c:a84:b0:7f0:38f7:6ca6 with SMTP id 00721157ae682-86e6d7905ccmr129247067b3.5.1788947079991; Wed, 09 Sep 2026 02:44:39 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-873194f76e3sm92930227b3.5.2026.09.09.02.44.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:44:39 -0700 (PDT) Date: Wed, 9 Sep 2026 02:44:34 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 02/26] mm/fbatch: allow folios_put_refs() to skip xa_is_value() entries In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let folios_put_refs() (hence folio_batch_release()) skip xa_is_value() entries, and therefore remove unneeded folio_batch_remove_exceptionals(). It made some sense when introduced in 3.1 for shmem swap entries only, but workingset shadows popularized exceptional entries in 3.15, and it's silly for so many sites to be squashing exceptionals out of the fbatch, merely to suit an inadequacy in folios_put_refs(). But remove exceptionals on leaving truncate_folio_batch_exceptionals(), one of whose callers then passes the fbatch on to others less tolerant. No longer essential to this series, since 7.2 commit 9669b87065a6 ("mm/lruvec: preemptively free dead folios during lru_add drain") allowed folios_put_refs() to skip NULLs; but still an improvement. Signed-off-by: Hugh Dickins Acked-by: David Hildenbrand (Arm) Reviewed-By: Vlastimil Babka (SUSE) --- include/linux/folio_batch.h | 5 +---- mm/folio.c | 25 ++++--------------------- mm/shmem.c | 2 -- mm/truncate.c | 16 ++++++++++------ 4 files changed, 15 insertions(+), 33 deletions(-) diff --git a/include/linux/folio_batch.h b/include/linux/folio_batch.h index b45946adc50b..e1cc8ae023f1 100644 --- a/include/linux/folio_batch.h +++ b/include/linux/folio_batch.h @@ -22,8 +22,7 @@ struct folio; * The folio_batch is used to amortise the cost of retrieving and * operating on a set of folios. The order of folios in the batch may be * significant (eg delete_from_page_cache_batch()). Some users of the - * folio_batch store "exceptional" entries in it which can be removed - * by calling folio_batch_remove_exceptionals(). + * folio_batch store "exceptional" (xa_is_value) entries in it too. */ struct folio_batch { unsigned char nr; @@ -100,6 +99,4 @@ static inline void folio_batch_release(struct folio_batc= h *fbatch) if (folio_batch_count(fbatch)) __folio_batch_release(fbatch); } - -void folio_batch_remove_exceptionals(struct folio_batch *fbatch); #endif /* _LINUX_FOLIO_BATCH_H */ diff --git a/mm/folio.c b/mm/folio.c index c093ca900a3e..ad6c64a4c22c 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -964,6 +964,10 @@ void folios_put_refs(struct folio_batch *folios, unsig= ned int *refs) if (!folio) continue; =20 + /* Skip any "exceptional" (workingset or shmem swap) entry. */ + if (xa_is_value(folio)) + continue; + if (is_huge_zero_folio(folio)) continue; =20 @@ -1069,27 +1073,6 @@ void __folio_batch_release(struct folio_batch *fbatc= h) } EXPORT_SYMBOL(__folio_batch_release); =20 -/** - * folio_batch_remove_exceptionals() - Prune non-folios from a batch. - * @fbatch: The batch to prune - * - * find_get_entries() fills a batch with both folios and shadow/swap/DAX - * entries. This function prunes all the non-folio entries from @fbatch - * without leaving holes, so that it can be passed on to folio-only batch - * operations. - */ -void folio_batch_remove_exceptionals(struct folio_batch *fbatch) -{ - unsigned int i, j; - - for (i =3D 0, j =3D 0; i < folio_batch_count(fbatch); i++) { - struct folio *folio =3D fbatch->folios[i]; - if (!xa_is_value(folio)) - fbatch->folios[j++] =3D folio; - } - fbatch->nr =3D j; -} - #ifdef CONFIG_MEMCG static void lruvec_reparent_lru(struct lruvec *child_lruvec, struct lruvec *parent_lruvec, diff --git a/mm/shmem.c b/mm/shmem.c index 897fa2b61346..1f2809adec54 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1157,7 +1157,6 @@ static void shmem_undo_range(struct inode *inode, lof= f_t lstart, uoff_t lend, truncate_inode_folio(mapping, folio); folio_unlock(folio); } - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched(); } @@ -1277,7 +1276,6 @@ static void shmem_undo_range(struct inode *inode, lof= f_t lstart, uoff_t lend, } folio_unlock(folio); } - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); } =20 diff --git a/mm/truncate.c b/mm/truncate.c index b58ba940be47..4151f7a167e3 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -53,7 +53,7 @@ static void clear_shadow_entries(struct address_space *ma= pping, /* * Unconditionally remove exceptional entries. Usually called from truncate * path. Note that the folio_batch may be altered by this function by remo= ving - * exceptional entries similar to what folio_batch_remove_exceptionals() d= oes. + * exceptional entries. * Please note that indices[] has entries in ascending order as guaranteed= by * either find_get_entries() or find_lock_entries(). */ @@ -95,7 +95,7 @@ static void truncate_folio_batch_exceptionals(struct addr= ess_space *mapping, dax_delete_mapping_entry(mapping, indices[i]); } } - goto out; + goto squash; } =20 xas_set(&xas, indices[j]); @@ -113,8 +113,14 @@ static void truncate_folio_batch_exceptionals(struct a= ddress_space *mapping, if (mapping_shrinkable(mapping)) inode_lru_list_add(mapping->host); spin_unlock(&mapping->host->i_lock); -out: - folio_batch_remove_exceptionals(fbatch); + +squash: + for (i =3D j + 1; i < nr; i++) { + folio =3D fbatch->folios[i]; + if (!xa_is_value(folio)) + fbatch->folios[j++] =3D folio; + } + fbatch->nr =3D j; } =20 /** @@ -575,7 +581,6 @@ unsigned long mapping_try_invalidate(struct address_spa= ce *mapping, if (xa_has_values) clear_shadow_entries(mapping, indices[0], indices[nr-1]); =20 - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched(); } @@ -732,7 +737,6 @@ int invalidate_inode_pages2_range(struct address_space = *mapping, if (xa_has_values) clear_shadow_entries(mapping, indices[0], indices[nr-1]); =20 - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched(); } --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 88BFD47ECF7 for ; Wed, 9 Sep 2026 09:46:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947214; cv=none; b=R3Bk7IT2y5x8SMERHlivQi0Z4+qFhUdDFu4NVZU2wgLghwUw0G0KOTsYuV0s4BN4mHNHbk9R+W/mBVPcyMQb88nKxgTGaJYyBLBdEej6i/7NIi9jLO21Cy/+8y1uwqMQvM9waAFi1EIbqddbbY+0ZdtDWe0W2rXxVFSMiER+fo8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947214; c=relaxed/simple; bh=yz9phphkpKB30xOl4FfJtCYJLrw/kdVME667JYPkLrw=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=ZqVSX3m0qrEF1lM8RyW8RhvSXPhFfKNNf6XONWEN7dkCNkKLlHZukP48tn0M3cUdeZqlUSeE3BkgEOHUm4P8E8KUUR2+MqsDinNN+800w6iT5p7PClHN0goWRAek3Rdjx02aJKEGHeC/DAKRZhcB0prhSfnZzBPzcaoW2Q4JrR4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YJxIfVzB; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YJxIfVzB" Received: by mail-yx2-f12.google.com with SMTP id 00721157ae682-85d46e4cdccso8525167b3.1 for ; Wed, 09 Sep 2026 02:46:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947211; x=1789552011; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=5+7MnKyiB0fAnf+ZBPoUiSQUKpsvD64pam2O0QOh8zY=; b=YJxIfVzBzbyp1rCrC/UYAyN61dhGtbLDcYpDojgpKogDd5Hlbpr4ivQU+2cW8np1xZ nqUT3K59Ak+GAkYkO955repklb+APUlLa5tWbWOWrMJ1gVs+JTM6FQ7GnsdFQtEJQIf/ 1zOSBORaG59rTroCAbCCwJzErPzg84xeaI3gaL6GuGgUdx+YmaNJptqcxJ14TfXuQP3T QDp26JBdBE2LWOLqRs0TuIJ4rsaEJEy3irRO6aepdcsDgdY/FNymdCHPUc+kz5yRW0tm ixoyD4ohVIKA1geXHlxoXzkkNl2t2jtvoZ/wiKWTk13gog/Ru4VHwTLmJBWV/syA4XrT 4k/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947211; x=1789552011; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=5+7MnKyiB0fAnf+ZBPoUiSQUKpsvD64pam2O0QOh8zY=; b=LuOsEVFoUEXQMHIwgkNTp7XXW7Ui0YCnPumv4Gm8i5ZfzuzXEu4yLsQ3GccChjHXPY ZHpvv8tq0n6omnOTAzGl1odKvEffzX7P+Drps5ASNg0D/KhEVsBko1EWXjgp2wlI8vxM 8+8M+v8H8cF+EPU0rkLhwt3gHlITc5+7uRIlSnk4pioaIOiLUFwgM7s5uydg6twf04I/ fSaLGwOgoiX0yqByuQc+deYY6kXC+O3xZ5qBNhvDxNfOjjOFP7FmZozViMyVYECU5M9/ pdf5gP2r//p7GUxAlZPDp9EQRVDOhgU+PP6Z5LNQm4CXZg2NxGDUcca5M1dzDp5iBusi vOpw== X-Forwarded-Encrypted: i=1; AKwUvBzoYDIf6nTyGQxEhJ/c6LP9McyRpQivBjRJJ6Bq8lRebL5ib/sHHE+lwUbDyfm9z4+iGYdXEHUSPz7JjT8=@vger.kernel.org X-Gm-Message-State: AFuF++mNx8EVJpg7pdtaDoA0arukkO9CVVlCNZBWkqeH4mU50ItFG6bD pyaxXuhVaNbeHbNa52lwGKHfTVCHkTd3mQ7vX4oChLZNZJVN9X+oprglkX11bPmEDQ== X-Gm-Gg: AYBFou1Gup2wRn2XPlX7JJ8byBzEr6g7I9aiICQ+lfXg1Xk5yJ2hXmGgKblIU2kewrK mBcN7SKKRXo0IgUP65ZIR34uBePLLmKuRiBmbVxGbdEQ/dVi06SRznfAUuXBLztbwKUrqJHQ7QL z30WslEN1ydgRQ2wu39sioLKy+1FZmrDCMZLzTPxrE9XyJL8crCpttn3v9h2xdUPG0y8bl/cEKx xxlcgyX24whlqJtmqtCXNVo4IWg82FFoHhsq0b2OOJzdNOMzzn1RXJ16hZZbmHCAg+NfBBk0wlM z9qT0zwbojwSo9nrOj5pHLFjsxeCxe1WR3LT39aXIuQ2cE4dJnf6apPKQIiWWcWdeWKEgmyQ+Rw r0xrdFXq9MqdR0x3PBLoKcVTCBC5JxtFl3CwnzPlCuic8q9nS6wSUKeUj0TXZtvewKykPUSyZ94 f6LITSfwQh+AfhfZrADNq0muGurugTIvs0yXJJMUknSZBQPSXbyXqq7476SPLnQuQI6A2LaZ05E V9tL4VydoF2QmsH+XXbNzkSvAXcTL00I1FST4aoPikZruEqZhlWBH8lbOw= X-Received: by 2002:a05:690c:650c:b0:85b:35f6:bc5d with SMTP id 00721157ae682-87f23fe7a98mr24694497b3.13.1788947210602; Wed, 09 Sep 2026 02:46:50 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-87149316356sm106597417b3.9.2026.09.09.02.46.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:46:49 -0700 (PDT) Date: Wed, 9 Sep 2026 02:46:45 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 03/26] mm/fbatch: temporarily disable lazyfree and mlock+munlock batching In-Reply-To: Message-ID: <5953f6aa-ca44-c9a2-97e1-a5b3b92e9e50@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It will not matter if we occasionally get activation or deactivation wrong; but a mistaken lazyfree (MADV_FREE) would likely cause dataloss. So disable its batching while reworking the per-cpu fbatch handling, then re-enable it with more thought afterwards. Similarly disable mlock+munlock batching temporarily: they will need some redesign before re-enabling. Just insert one disabling line for now, leaving the rest of the code as it was, for consideration later. Signed-off-by: Hugh Dickins Reviewed-By: Vlastimil Babka (SUSE) --- mm/folio.c | 2 ++ mm/mlock.c | 3 +++ 2 files changed, 5 insertions(+) diff --git a/mm/folio.c b/mm/folio.c index ad6c64a4c22c..b9dc4f5e10a6 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -219,6 +219,8 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, local_lock(&cpu_fbatches.lock); =20 if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || + /* XXX Temporarily disable lazyfree batching */ + fbatch =3D=3D &cpu_fbatches.lru_lazyfree || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 diff --git a/mm/mlock.c b/mm/mlock.c index 39215a3eab1f..929abeac85d0 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -255,6 +255,7 @@ void mlock_folio(struct folio *folio) =20 folio_get(folio); if (!folio_batch_add(fbatch, mlock_lru(folio)) || + true || /* XXX Temporarily disable mlock batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); @@ -278,6 +279,7 @@ void mlock_new_folio(struct folio *folio) =20 folio_get(folio); if (!folio_batch_add(fbatch, mlock_new(folio)) || + true || /* XXX Temporarily disable mlock_new batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); @@ -299,6 +301,7 @@ void munlock_folio(struct folio *folio) */ folio_get(folio); if (!folio_batch_add(fbatch, folio) || + true || /* XXX Temporarily disable munlock batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 218B03BF69E for ; Wed, 9 Sep 2026 09:49:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947365; cv=none; b=GNEDSYeRqhq8htGEYMCH+85WzGNzKYiVJBXbbvCzgiaRtKvGKhnJLn54j5g5HH031DIx58EZcg8HbFQEm0Ul0Et4yUj6B0GxabmT5Bki86w8GMPWS10Sb4KC5skmxPVUl10Ie0AkjWPvGsgOUHZZBdz1Csbe5D71X4hQw2b8EEk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947365; c=relaxed/simple; bh=bmiYMnfxqxqXTndNn9OHugj8rDer4W9a5xt6AtmTM94=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=fnqDtnXUCglnaPj06vXPasjzrE7VP5ObSp1hE1dXbJHwvdxxULOfVxA2sV7XnS2QqvadGz/9JpKK9C7NSlLxjs3ZX5y+eidE5Xmxhiop0GUwzLKmAxIrT8eSkhykW9ePHn2ADECh1hZwTT/sAUaFdIDUYJNIakNHMLi+Gr55t/Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Xy1CfrsA; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Xy1CfrsA" Received: by mail-yx2-f12.google.com with SMTP id 00721157ae682-85d46e4cdcdso9307187b3.2 for ; Wed, 09 Sep 2026 02:49:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947362; x=1789552162; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=UufK0vMlC6Z+KADxCs5ZoL9LcMsl7R0DiwRuRItb70g=; b=Xy1CfrsA3ykgtEBzxr7he0QYrrmY3qjtfiMjhWVfHWvD82Lmk495w9h/A7UoEC7EzO eXtlFRoWGRO+0ipdKtnQhjzpxrzCObInhbqjlZX0+cuIUkWzJ+btylZsFb17ElAEFbik +ubFA4B6r3+ze2+ybMtgE1NZJdL7C6dlgkpDkXOfSWqSw/p0uJ4OeRFX7fPPlX0PUegI G2NVTzmc+E7kIyPHIw4GIAqIIEH524CGYYb0yO8ZP8ggJQKyznkO1d9hFGkUkYyULa3N w5CidnpQp/TmCwcvnk50txFpDrhOeYS49TZYOhF2wDUDLiwtFUl7lI1N6ez09IQRfpRu 7bFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947362; x=1789552162; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=UufK0vMlC6Z+KADxCs5ZoL9LcMsl7R0DiwRuRItb70g=; b=k0g2WcxOD1lY9Tk6qFEBsaL9ggDjBnk/KsRiF/cSz65q0qWTkxd26C3dkraxQHfToI a44JftTsGbpgYENyLdEb+gfDBDddjOfzdkZ7EAopv7qmj+4MAttztDxHb14OO+6u4UzY wvAYxEiQbsZIGL+iVvG2C+PrbhGeRIkq85j4ss4CZ8IXw9LwUxYpvPmWg6HgagmltW6G 3U0Uajftn5pxG+mH65yi0eH7gNyZPhUi+1L6K4DPHkM/WxUTBsAu/25sx7FRbLl9K93H CeYjOQtrcW10Yg6F9m9RMwoSJXnGZC7/QGq59EAA033wBNeLEkcRowQeawT2c/bAxOcU nA0A== X-Forwarded-Encrypted: i=1; AKwUvBxAs3uW0N029idfr1CRFKA8CD2GiQ1PFQYknJ+/NezQjCzYLEebjY9kJPvAVnX81vE2VGX4/iUwVtp7UuA=@vger.kernel.org X-Gm-Message-State: AFuF++mjnEiCyrTXHly2ROh5ipL9dAEjwCR0I7Y/dEK6ASDsEM+kHG3q E7P6pUaG0xAqDQzMS8jlcq/etUXcmm/e7dJ0ChSZoiSuGXfmCKDw6G0LsoKQKrMqNw== X-Gm-Gg: AYBFou3hTGuVpDqevk8VSJBGZsINj436QBdEN5i5kvzlsorzOWi8laYD3XKCU29J4pI Fbbkf+Aay2MpO+rj8wpbB4chTf5QtotYKBRqf6EyHIrDEEgWG05h9d0UoOsFEnpDGBLUJUe0AcI S995zJqniBtMYjCmqDi9kPBCfVhdazmOvcUAmpx49OxaVIB7mrykoI1hF23BYy/QRb/vX1sc22X c0Pulb7OVqD+8ozkR6/+7x+6pE1JvdqbwJ0FwCLaf4T504AHnBeAIqB1XP1yMUNrJHApZYTBQ0/ C1aedIwt9YsdCXHhUJ+LcDmo+wti+4Cf8bWiO4G3p3zsyv4g1hGYpnF0X+KwWcs6tv0AB9NRfuU 3UF8CoeBVPgrBRepH9JDHYtkjvc49ZYL6vBHaOVLIjSJLaz5IyjT66vDEQDD6LnC3UQNaTgeeQz G4nPFJUB2tB6VCDaRJSf65YpGKl7V0lw+8Pye8IQNnR5qLLoT6Uzpqx4qz2ul5hnSzlEP3+t47S WuAzdG6Df3PCwNtBClwiqVfuiivBFOFv4a7PAXnPilJO0FsbOo6bHdcEvM= X-Received: by 2002:a05:690c:e3f2:b0:87c:35a5:4aea with SMTP id 00721157ae682-87f22b7eaf5mr23815837b3.6.1788947361454; Wed, 09 Sep 2026 02:49:21 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-87149600f48sm107580907b3.13.2026.09.09.02.49.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:49:20 -0700 (PDT) Date: Wed, 9 Sep 2026 02:49:15 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 04/26] mm/fbatch: lru bit set, no extra ref, while folio on per-cpu fbatch In-Reply-To: Message-ID: <61e15506-940f-3532-5fc9-4086f7612c10@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Treat folios on a per-cpu fbatch as if they were already on the lruvec: with PG_lru set, without holding an extra reference. This will enable the removal of most lru_add_drain() and lru_add_drain_all() calls soon. Recognize such a folio by 0x02 set in the folio->lru.next pointer by folio_add_lru(). Then lruvec_del_folio() (aided by "lru_add_del_folio") can pretend to unlink it, and folio_batch_move_lru()'s lru_add case can check whether one of the others has already moved it to lruvec. (That bit is also used in a transient way by set_page_pfmemalloc(), to inform interested callers whether page_is_pfmemalloc(): but those callers are in networking, not putting folios on LRU; and accept that any use of the page->lru field already erases page_is_pfmemalloc() information.) Let folio->lru.next point to the lru_add fbatch entry, but this is now just for debugging: it seemed to be important for folio_batch_move_lru() to distinguish fresh from stale entries, but then it turned out that it has to processs them identically. Activate, deactivates and move_tail, holding no reference on the folio, might come to act on a stale folio when the fbatch is drained: but it's acquired by try_get and test_clear_lru, so safe even when suboptimal. Reclaim is not an exact science, and there have been no complaints of missed actions since 5.11 commit fc574c23558c ("mm/swap.c: serialize memcg changes in pagevec_lru_move_fn") introduced the TestClearPageLRU protocol: so don't expect complaints of a few surprisingly taken actions. Signed-off-by: Hugh Dickins --- include/linux/mm_inline.h | 25 ++++++++ include/linux/mm_types.h | 6 +- mm/folio.c | 119 +++++++++++++------------------------- mm/huge_memory.c | 6 +- 4 files changed, 74 insertions(+), 82 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 621c8653d8f7..8420b1276535 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -343,6 +343,29 @@ static inline void folio_migrate_refs(struct folio *ne= w, const struct folio *old } #endif /* CONFIG_LRU_GEN */ =20 +enum { + LRU_NEXT_NEVER_TAIL =3D 0, /* Used by a tail's compound_head */ + LRU_NEXT_BATCHED =3D 1, /* Not used by any aligned pointer */ + NR_LRU_NEXT_FLAGS +}; + +static __always_inline +bool lru_add_del_folio(struct folio *folio) +{ + unsigned long lru_next =3D READ_ONCE(folio->lru_next); + + /* BUG_ON(folio_test_lru(folio) && folio_ref_count(folio)); */ + if (!(lru_next & BIT(LRU_NEXT_BATCHED))) + return false; + + WRITE_ONCE(folio->lru.next, LIST_POISON1); + /* BUG_ON(folio->lru_next & BIT(LRU_NEXT_BATCHED)); */ + + /* Ensure folio->lru_next visible when folio_set_lru() called later */ + smp_mb__before_atomic(); + return true; +} + static __always_inline void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) { @@ -384,6 +407,8 @@ void lruvec_del_folio(struct lruvec *lruvec, struct fol= io *folio) =20 if (lru_gen_del_folio(lruvec, folio, false)) return; + if (lru_add_del_folio(folio)) + return; =20 if (lru !=3D LRU_UNEVICTABLE) list_del(&folio->lru); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6d815f6440c9..fe6220b97cf3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -85,6 +85,8 @@ struct page { * WARNING: bit 0 of the first word is used for PageTail(). That * means the other users of this union MUST NOT use the bit to * avoid collision and false-positive PageTail(). + * Bit 1 of the first word is used by page_is_pfmemalloc(). + * Bit 1 of the first word (lru_next) is also used by folio_add_lru(). */ union { struct { /* Page cache and anonymous pages */ @@ -410,10 +412,8 @@ struct folio { union { struct list_head lru; /* private: avoid cluttering the output */ - /* For the Unevictable "LRU list" slot */ struct { - /* Avoid compound_info */ - void *__filler; + unsigned long lru_next; /* public: */ unsigned int mlock_count; /* private: */ diff --git a/mm/folio.c b/mm/folio.c index b9dc4f5e10a6..e743cd539b9e 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -152,57 +152,34 @@ static void folio_batch_move_lru(struct folio_batch *= fbatch, move_fn_t move_fn) int i; struct lruvec *lruvec =3D NULL; unsigned long flags =3D 0; - struct folio_batch free_fbatch; - bool is_lru_add =3D (move_fn =3D=3D lru_add); - - /* - * If we're adding to the LRU, preemptively filter dead folios. Use - * this dedicated folio batch for temp storage and deferred cleanup. - */ - if (is_lru_add) - folio_batch_init(&free_fbatch); =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { struct folio *folio =3D fbatch->folios[i]; =20 - /* block memcg migration while the folio moves between lru */ - if (!is_lru_add && !folio_test_clear_lru(folio)) - continue; - - /* - * Filter dead folios by moving them from the add batch to the temp - * batch for freeing after this loop. - * - * We're bypassing normal cleanup. Clear flags that are not - * applicable to dead folios. - * - * Since the folio may be part of a huge page, unqueue from - * deferred split list to avoid a dangling list entry. - */ - if (is_lru_add && folio_ref_freeze(folio, 1)) { - __folio_clear_active(folio); - __folio_clear_unevictable(folio); - folio_unqueue_deferred_split(folio); + if (!folio_try_get(folio)) { fbatch->folios[i] =3D NULL; - folio_batch_add(&free_fbatch, folio); continue; } =20 + if (!folio_test_clear_lru(folio)) + continue; + + /* Do not add to LRU if it has already been added */ + if (move_fn =3D=3D lru_add && !lru_add_del_folio(folio)) + goto restore_lru; + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); =20 + /* Do add to LRU if not already there (move_fn skipped) */ + if (lru_add_del_folio(folio)) + lruvec_add_folio(lruvec, folio); +restore_lru: folio_set_lru(folio); } =20 if (lruvec) lruvec_unlock_irqrestore(lruvec, flags); - - /* Cleanup filtered dead folios. */ - if (is_lru_add) { - mem_cgroup_uncharge_folios(&free_fbatch); - free_unref_folios(&free_fbatch); - } - folios_put(fbatch); } =20 @@ -211,8 +188,6 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, { unsigned long flags; =20 - folio_get(folio); - if (disable_irq) local_lock_irqsave(&cpu_fbatches.lock_irq, flags); else @@ -273,7 +248,6 @@ static void lru_activate(struct lruvec *lruvec, struct = folio *folio) if (folio_test_active(folio) || folio_test_unevictable(folio)) return; =20 - lruvec_del_folio(lruvec, folio); folio_set_active(folio); lruvec_add_folio(lruvec, folio); @@ -289,37 +263,12 @@ void folio_activate(struct folio *folio) !folio_test_lru(folio)) return; =20 - folio_batch_add_and_move(folio, lru_activate); -} - -static void __lru_cache_activate_folio(struct folio *folio) -{ - struct folio_batch *fbatch; - int i; - - local_lock(&cpu_fbatches.lock); - fbatch =3D this_cpu_ptr(&cpu_fbatches.lru_add); - /* - * Search backwards on the optimistic assumption that the folio being - * activated has just been added to this batch. Note that only - * the local batch is examined as a !LRU folio could be in the - * process of being released, reclaimed, migrated or on a remote - * batch that is currently being drained. Furthermore, marking - * a remote batch's folio active potentially hits a race where - * a folio is marked active just after it is added to the inactive - * list causing accounting errors and BUG_ON checks to trigger. + * XXX: It is curiously difficult to recreate safely the old + * __lru_cache_activate_folio() optimization (folio_set_active() + * directly if it's on the local lru_add fbatch): revisit later. */ - for (i =3D folio_batch_count(fbatch) - 1; i >=3D 0; i--) { - struct folio *batch_folio =3D fbatch->folios[i]; - - if (batch_folio =3D=3D folio) { - folio_set_active(folio); - break; - } - } - - local_unlock(&cpu_fbatches.lock); + folio_batch_add_and_move(folio, lru_activate); } =20 #ifdef CONFIG_LRU_GEN @@ -410,16 +359,7 @@ void folio_mark_accessed(struct folio *folio) * unevictable page accessed has no effect. */ } else if (!folio_test_active(folio)) { - /* - * If the folio is on the LRU, queue it for activation via - * cpu_fbatches.lru_activate. Otherwise, assume the folio is in a - * folio_batch, mark it active and it'll be moved to the active - * LRU on the next drain. - */ - if (folio_test_lru(folio)) - folio_activate(folio); - else - __lru_cache_activate_folio(folio); + folio_activate(folio); folio_clear_referenced(folio); workingset_activation(folio); } @@ -439,6 +379,10 @@ EXPORT_SYMBOL(folio_mark_accessed); */ void folio_add_lru(struct folio *folio) { + struct folio_batch *fbatch; + unsigned long lru_next; + bool full; + VM_BUG_ON_FOLIO(folio_test_active(folio) && folio_test_unevictable(folio), folio); VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); @@ -458,7 +402,26 @@ void folio_add_lru(struct folio *folio) folio_mark_accessed(folio); } =20 - folio_batch_add_and_move(folio, lru_add); + local_lock(&cpu_fbatches.lock); + fbatch =3D this_cpu_ptr(&cpu_fbatches.lru_add); + + /* Storing this address is only for debugging */ + lru_next =3D (unsigned long)&fbatch->folios[fbatch->nr]; + /* This mask will do nothing on 64-bit */ + lru_next &=3D ~(BIT(NR_LRU_NEXT_FLAGS) - 1); + lru_next |=3D BIT(LRU_NEXT_BATCHED); + folio->lru_next =3D lru_next; + + full =3D !folio_batch_add(fbatch, folio); + + /* Ensure folio->lru_next visible to folio_test_clear_lru() callers */ + smp_mb__before_atomic(); + folio_set_lru(folio); + + if (full || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + folio_batch_move_lru(fbatch, lru_add); + + local_unlock(&cpu_fbatches.lock); } EXPORT_SYMBOL(folio_add_lru); =20 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index afbb5974bd22..c7510d875433 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3995,8 +3995,12 @@ static int __folio_freeze_and_split_unmapped(struct = folio *folio, unsigned int n } =20 /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - if (do_lru) + if (do_lru) { lruvec =3D folio_lruvec_lock(folio); + /* Move from fbatch to lruvec before lru_add_split_folio()s */ + if (lru_add_del_folio(folio)) + lruvec_add_folio(lruvec, folio); + } =20 ret =3D __split_unmapped_folio(folio, new_order, split_at, xas, mapping, split_type); --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f50.google.com (mail-yx1-f50.google.com [74.125.224.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F43D47DF85 for ; Wed, 9 Sep 2026 09:51:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947496; cv=none; b=lpzoi6m9TC01nWSjmwxKRIOLidWlazRMF3tT9DfFw1HJrz/G/pe6ldSLd47vq50LNQn5CaSDo3LDF0LzBvjrwtu+5M568SbdIKaaNPGRWpDABeSzvqj3/rFa7h8oPTsfNBojajYedHrt5ZLwNQopfa0GLVNVaWgteysRSWYuP18= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947496; c=relaxed/simple; bh=zEN3i+bht4wJK9nMYZ0X1V5lt+UAKAX+zLdvRlqVUhs=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=OYgvVnLVdXLGksBZRAifsZklMhPd0aSiu+ecF9YRBLqCdyKfMOaXsPO9zm77nYwoYXSHC9o3tSJ35sVH0ly/w6gytnuBdpzzGIofTi6CIj7dgD1BvPwm9IcJgvudalqrcrDSppOD3Rs2732epOfYMat4WTnJ7RIPg7jsI9fkRHs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kGnX2DKj; arc=none smtp.client-ip=74.125.224.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kGnX2DKj" Received: by mail-yx1-f50.google.com with SMTP id 956f58d0204a3-66fa10f35caso5178187d50.3 for ; Wed, 09 Sep 2026 02:51:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947494; x=1789552294; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=mn+TRjSVVIAZoTaeIYbM//CzX6IvKNmptuaikzRYa5A=; b=kGnX2DKj5lnH5TOHANwvHWEMVwQIEp3f4GgGRZXADv0eNY2m2fw8q33D3/MO9JfAwT 1ZcJZVgZQkSIDSzb6T5+E8sqTQg50xUC0+CCDRjDctfJAGKrlcSwy3UTnhB9Z4ENzo3x OPnvb4MBYE4UpXz6RCRFEQL/jktk64V0atEPf3rIwAcfpW9cTn0oThrLP9Y5xzRATvnx Mn0HTU8OPvLjn2OPvXDmGjOAJH7D//Mz7uj5WB9LCAYmppi0mer7j62U++HoqKCUsCgW SDECqbINvgOIRG6fubgzAGl3hGUrza5t3QDpB8YQpDTH1uuxa70krfzzYAD8aCv5vA4m 4W5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947494; x=1789552294; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=mn+TRjSVVIAZoTaeIYbM//CzX6IvKNmptuaikzRYa5A=; b=qnYH9T63+SIveztyB/UBZwKiTcNMnsz0pZwku74DlKwUbGsrFB5yUI1lKGQj9chRdQ aK0jT7N6e3iURY7ZkFxGoN94uggF5Lyc05dqypbA41QuiOUESVIEeBnnXqxPOLOgiEPw VklqShZOJU/7nrZjGjuIjQsdkiEOPIBq7UoHyjMOBwD9nw/PluJBMQLkleMTKr4LLfVU zdWPN+A7eaWPMFai7geBUVXBjl4111Qg4vZLdKw01q4ezIjDxHtW3Z3qgnzw9d8hiWhg HsQDYzxWNRsGrfgzPt+hp1WjL94t9Lwp+EuL8YokHNHGubqr501DJVRjUQ0e5c0Yz0zF iTiw== X-Forwarded-Encrypted: i=1; AKwUvBzwpEShonH78uHYZe2qMwbdgT/Ozq7h1mrwPLVZEUrIPoW3m12EpFeNcrzOTbsKe20GfmBfhyAE1ngu93Q=@vger.kernel.org X-Gm-Message-State: AFuF++n57yFfEnyY3RsvoA6BQKXYxKie7w9d+AQOGlxfFRfgS0uejLyq Xoz574iRtSwYDNu0bKXQbEwEApP1h9RAdwhKCeYPlX73TpNGKyt69IS373V3S4sGww== X-Gm-Gg: AYBFou3ECV87Qu9zKUJApTIL0gMtLk2N/G1z4jdR+v/A/0fEsjOgcJxXi4dpQbST0Z8 VQGFOCguzJm6iE9NeNorTiDGZivXQ62Ceh6zvpA0jvl9p/92tGBnp6DpXlrK0g7Gm/zJZOvOeBL gOWN/W//cs52HUxLUROC8bsZSg/ZMDmxcFbOC3aReUW4je/V5mZdw+v8xS4L83gc/h14gj7Doj0 m1v7y+ALuZv0HP+yc3kqm0PF9aeWh/q0ssEWewRP17eSMpSbGEkd4y2WXHajPZEMfwH1hlm4x35 7evXDDbpNmc7hivbbkySm9ZlVnri2vHvPh/NuFUYDW9xsd6nTDUOvugpWUoB7aI3Ye9SS0tr5er 0egfpYeHLKxlFd2Z1/HjpZ6yXMx0gkd55YYdEQcGoBl1+mzzEffvJp4iH4CjfL9U3ByGcMubPog lfmPhoAChuuGYsI04DCkO/WrU4f+Ja0q46Qz8juyNL7wYGDbO8HSRuBDLAXbmZU1pMQKB56Dr5O dXPXXxJy/IvGHkDf9ngSETpaLih9QculC4TvdG7ZwzuevwcdBxZBxlsUCU= X-Received: by 2002:a05:690e:b8d:b0:66f:c774:5b4 with SMTP id 956f58d0204a3-66fc77411b8mr7040210d50.78.1788947493066; Wed, 09 Sep 2026 02:51:33 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8749af79c56sm78820707b3.22.2026.09.09.02.51.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:51:32 -0700 (PDT) Date: Wed, 9 Sep 2026 02:51:27 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 05/26] mm/fbatch: lru_add_del_folio()+folio_add_lru() after clear_lru() In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Most callers of folio_test_clear_lru() then proceed to remove the folio from its lru, and add it back at the end when they're done (if still in use). But isolate_migratepages_block() and check_move_unevictable_pages() sometimes decide against, and release immediately with a folio_set_lru(). Which usually works fine: but there's now a small chance that while they held the folio with lru bit cleared, an lru_add fbatch drain came along, and had to skip that folio because its lru bit was transiently cleared (previously, the lru_add fbatch drain relied on finding lru bit never yet set). This risks leaving that folio off lru, unreclaimable until freed. Fix such cases by trying lru_add_del_folio() (which only takes action and returns true if the folio was on an lru_add fbatch), then folio_add_lru() if it succeeded: invalidating the old fbatch slot, appending in a new one. Signed-off-by: Hugh Dickins Reviewed-by: Vlastimil Babka (SUSE) --- mm/compaction.c | 13 +++++++++++-- mm/vmscan.c | 16 +++++++++------- 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index a049415512c6..9e045a90ba21 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1204,7 +1204,13 @@ isolate_migratepages_block(struct compact_control *c= c, unsigned long low_pfn, !cc->alloc_contig)) { low_pfn +=3D folio_nr_pages(folio) - 1; nr_scanned +=3D folio_nr_pages(folio) - 1; - folio_set_lru(folio); + if (lru_add_del_folio(folio)) { + lruvec_unlock_irqrestore(locked, flags); + folio_add_lru(folio); + locked =3D NULL; + } else { + folio_set_lru(folio); + } goto isolate_fail_put; } } @@ -1293,7 +1299,10 @@ isolate_migratepages_block(struct compact_control *c= c, unsigned long low_pfn, if (locked) lruvec_unlock_irqrestore(locked, flags); if (folio) { - folio_set_lru(folio); + if (lru_add_del_folio(folio)) + folio_add_lru(folio); + else + folio_set_lru(folio); folio_put(folio); } =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index f11491ee9ed5..4e8d5cc34f07 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -8093,17 +8093,19 @@ void check_move_unevictable_folios(struct folio_bat= ch *fbatch) folio_clear_unevictable(folio); lruvec_add_folio(lruvec, folio); pgrescued +=3D nr_pages; + } else if (lru_add_del_folio(folio)) { + lruvec_unlock_irq(lruvec); + folio_add_lru(folio); + lruvec =3D NULL; } - folio_set_lru(folio); + if (lruvec) + folio_set_lru(folio); } =20 - if (lruvec) { - __count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); - __count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); + if (lruvec) lruvec_unlock_irq(lruvec); - } else if (pgscanned) { - count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); - } + count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); + count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); } EXPORT_SYMBOL_GPL(check_move_unevictable_folios); =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2539A48B39D for ; Wed, 9 Sep 2026 09:53:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947625; cv=none; b=U8MQQ4m1RV+dpaiIOGkrDZArZWoQ7ZBJzySWyJl5TvJ53sQem6DRSoDxgTTKEFcTx60inMuW8R+n1iSKs19eDXZJkT5BTLS6lnimf8DdpIiNbjwKZ5U9heYgeV2sNYeq21gFysG884aUIYSE9LOD6RL1s1Jj0PBQEgZ5rs3su50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947625; c=relaxed/simple; bh=FCbXp+9OF7RbrCuYxplGLnxtBxnCcxr/co7K/0ctoAQ=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=Jww9BqVsjnqZTM0gqPWA/IW0/pOgkB+IhYfuvTG+En2SDejRtHprrUcv1AhawQQaxdpWOgHE561iV2j24I6oTI3tL5ZotmEVkivtkX1/fSKPh8KOD0laQ9E+zxSOttp+Nljo2AicBPK47xdNEbn9wUNBbggOsSGSRyr7bZTAYBE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qV9pWXKk; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qV9pWXKk" Received: by mail-yx2-f12.google.com with SMTP id 00721157ae682-85d46e4cdccso8573617b3.1 for ; Wed, 09 Sep 2026 02:53:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947621; x=1789552421; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=klQabvPBSdHuFjYsQkDymGQq+sX0bqmcbAmBz8NKUgY=; b=qV9pWXKkkeoTOXcXrYSq6yrRF8zv3vV2DI+LJbqHMJrJEGV85gkZLMnIq6H9adxO8y px9Hf+Dzfh4pWk3cz9od2dc62JaPQhNr4UVagmyeDxFQJWDAH4MksRDnUzjCS0MDr4jK p5adWM6NvBq2+t5pz/JoQYzU43xWc0QHoL40EJV9fmY1GiCoig/hEUvNpb/MPVffe1eM xnL3U6fylG9yDPb6wLum6luwnE4gN0InBn1BP8d97Y/1wvYAgPk4z7Ek1smcWUZhC2Pv dHZZqjmY50HqCB6ZtT+sf/uPsI5MEehdBg/PzNYAuGDVZHJy+hUUxHPtgXDVeMxCpRCx uM7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947621; x=1789552421; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=klQabvPBSdHuFjYsQkDymGQq+sX0bqmcbAmBz8NKUgY=; b=TgSnIPkbUokEM9awfiAM1MX87e6hphxsB5eHoEdfP+8nhH8P4/1+utWVuBjOy2BLpE Nw7BGupcixxk6zw3rb3CGLleVNGilEmtyvN7nbKCDjabLqhaS5MnKeurWGprAsD6BXGS X8lka5ZnrNyMDD8AF8EHm4TKxJ23sHhIwv3zroAmtMlvVXe6c2qZcT8ZILI0Ued8vWeF QrW11Q+oufYaz6Hq11raG4XCCEltWQJjEaIEMGW2duwdNYbr41p/fD3s5UwgjLGQhFbW qGi6b68Iu+rvx6ok119QXyqM/ogaV86rd2yk02CHlz/TvnKi/1ggER9bBex1qLtDLu6u jI8Q== X-Forwarded-Encrypted: i=1; AKwUvByG6qrwOxzLhnMdWYKsDfb8DT+1gAE3uqxatNaPg9tQ+MetJ+WGAVa3gj9yS/hoKyzMufYUmFvdus6eVfI=@vger.kernel.org X-Gm-Message-State: AFuF++nX0ZpnFW1cJW2KPM5fYS1usOez/hWUBIPP2gJRNZIXZ5mdch45 Pwh3cNEkBFaPZbfMeeK0LJrUsG46ONpXECcz5ecGfvahTb5Jfuor9ljwbaPGsxYBxg== X-Gm-Gg: AYBFou0IOAZKSmrqJQGlrDTwv/j6lxbnjwisj7BFMpey+aeMc8Tc4swlHugoUK07yVp u5N+L10dYxwpQsGRXm3y4aWOazv6mYR80/xoYl/SKKbRTuBBK7ddAfELvm/xKjgqNSL4UqEAKyu lZQYKPP0jqYjOHn7HKhPgfj9ebvOZkFob0Z6d9s+cxb9TJsCUM3dJvxWvDY1rC+IEWwn9n/ytnY Vfhij0UpC3TzWakrn4TVdT6/14/+rFo6M8Q8xPnNer2xxGCxewCjll9SkzcjtNSoQv7MI6k1oJc xWkc2enX5iwutM8frqWgiuEculgNt3xHLX8SqhGVGmX7x9sxg9qo63yOjD882NjfSlxKzynadxl 7/FsmsQBsqH5C0v4whaB8N+71/n/s94+v8wSZZGvADxTygdSjI3ti5M4gEnyWGjvIMJqLnhvdZD nwucouAbWnktwg7ousp77JQiW57AsURQ6/rEyP5V1ShrxqoQDQYJVRGMlv6uIrbVh1/mv2svViC bf23TnXb3hUIMb96FVL0y//qEFV4nxvh1+7lp5vgkidb1o7TWIxq6Q4a3U= X-Received: by 2002:a05:690c:c0b:b0:81e:1b3c:9e4 with SMTP id 00721157ae682-87f23fe8f10mr27261477b3.14.1788947620263; Wed, 09 Sep 2026 02:53:40 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714b62503dsm108292997b3.38.2026.09.09.02.53.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:53:39 -0700 (PDT) Date: Wed, 9 Sep 2026 02:53:35 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 06/26] mm/fbatch: fbatch_drain_lazyfree(onstack fbatch) before ptl unlock In-Reply-To: Message-ID: <17e1a6c3-525b-1cc3-0731-349f0850e3ea@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Re-enable lazyfree batching for MADV_FREE. But it's not safe now to leave potentially stale (then reused) folios in a per-cpu fbatch for lazyfree. Instead, madvise_free_pte_range() keep an fbatch on its stack, and drain it each time before dropping pagetable lock, while the folios are secure. Ignore folio_may_be_lru_cached() and lru_cache_disabled(): limitations irrelevant to this fbatch drained under spinlock (even if RT); though in practice madvise_free_huge_pmd() does have to drain every time. Signed-off-by: Hugh Dickins Reviewed-by: Vlastimil Babka (SUSE) --- include/linux/huge_mm.h | 6 ++++-- mm/folio.c | 34 +++++++++++++++++++++------------- mm/huge_memory.c | 6 ++++-- mm/internal.h | 3 ++- mm/madvise.c | 9 +++++++-- 5 files changed, 38 insertions(+), 20 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c745f7ad2298..d50906327d1d 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -24,9 +24,11 @@ static inline void huge_pud_set_accessed(struct vm_fault= *vmf, pud_t orig_pud) } #endif =20 -vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf); +struct folio_batch; bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *= vma, - pmd_t *pmd, unsigned long addr, unsigned long next); + pmd_t *pmd, unsigned long addr, unsigned long next, + struct folio_batch *fbatch); +vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf); bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_= t *pmd, unsigned long addr); int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t= *pud, diff --git a/mm/folio.c b/mm/folio.c index e743cd539b9e..a18d8ef6afd5 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -51,7 +51,6 @@ struct cpu_fbatches { struct folio_batch lru_activate; struct folio_batch lru_deactivate_file; struct folio_batch lru_deactivate; - struct folio_batch lru_lazyfree; /* Protecting the following batches which require disabling interrupts */ local_lock_t lock_irq; struct folio_batch lru_move_tail; @@ -194,8 +193,6 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, local_lock(&cpu_fbatches.lock); =20 if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || - /* XXX Temporarily disable lazyfree batching */ - fbatch =3D=3D &cpu_fbatches.lru_lazyfree || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 @@ -585,10 +582,6 @@ void lru_add_drain_cpu(int cpu) fbatch =3D &fbatches->lru_deactivate; if (folio_batch_count(fbatch)) folio_batch_move_lru(fbatch, lru_deactivate); - - fbatch =3D &fbatches->lru_lazyfree; - if (folio_batch_count(fbatch)) - folio_batch_move_lru(fbatch, lru_lazyfree); } =20 /** @@ -634,19 +627,35 @@ void folio_deactivate(struct folio *folio) =20 /** * folio_mark_lazyfree - make an anon folio lazyfree - * @folio: folio to deactivate + * @fbatch: batch to which folio will be added + * @folio: folio to be lazily freed * - * folio_mark_lazyfree() moves @folio to the inactive file list. - * This is done to accelerate the reclaim of @folio. + * folio_mark_lazyfree() moves @folio to the inactive file list + * via @fbatch. This is done to accelerate the reclaim of @folio. */ -void folio_mark_lazyfree(struct folio *folio) +void folio_mark_lazyfree(struct folio_batch *fbatch, struct folio *folio) { if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) || !folio_test_lru(folio) || folio_test_swapcache(folio) || folio_test_unevictable(folio)) return; =20 - folio_batch_add_and_move(folio, lru_lazyfree); + if (!folio_batch_add(fbatch, folio)) + folio_batch_move_lru(fbatch, lru_lazyfree); +} + +/** + * fbatch_drain_lazyfree - drain the caller's folio batch + * @fbatch: batch of folios to be lazily freed + * + * Must be called before caller drops the page table lock: that is, + * before dropping the last certain reference to the folios in @fbatch. + * It would be very bad to lazyfree a folio after it was freed and reused. + */ +void fbatch_drain_lazyfree(struct folio_batch *fbatch) +{ + if (folio_batch_count(fbatch)) + folio_batch_move_lru(fbatch, lru_lazyfree); } =20 void lru_add_drain(void) @@ -700,7 +709,6 @@ static bool cpu_needs_drain(unsigned int cpu) folio_batch_count(&fbatches->lru_move_tail) || folio_batch_count(&fbatches->lru_deactivate_file) || folio_batch_count(&fbatches->lru_deactivate) || - folio_batch_count(&fbatches->lru_lazyfree) || need_mlock_drain(cpu)) || has_bh_in_lru(cpu, NULL); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c7510d875433..abebd8a23e56 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2356,7 +2356,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * Otherwise, return false. */ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *= vma, - pmd_t *pmd, unsigned long addr, unsigned long next) + pmd_t *pmd, unsigned long addr, unsigned long next, + struct folio_batch *fbatch) { spinlock_t *ptl; pmd_t orig_pmd; @@ -2417,7 +2418,8 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, st= ruct vm_area_struct *vma, tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } =20 - folio_mark_lazyfree(folio); + folio_mark_lazyfree(fbatch, folio); + fbatch_drain_lazyfree(fbatch); ret =3D true; out: spin_unlock(ptl); diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c9..0d78406eb126 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -63,7 +63,8 @@ void lru_add_drain(void); void lru_add_drain_cpu(int cpu); void lru_add_drain_cpu_zone(struct zone *zone); void folio_deactivate(struct folio *folio); -void folio_mark_lazyfree(struct folio *folio); +void folio_mark_lazyfree(struct folio_batch *fbatch, struct folio *folio); +void fbatch_drain_lazyfree(struct folio_batch *fbatch); =20 /* mm/vmscan.c */ unsigned long zone_reclaimable_pages(struct zone *zone); diff --git a/mm/madvise.c b/mm/madvise.c index eeee82cf2b3f..b2eab519af19 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -666,6 +667,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, struct mmu_gather *tlb =3D walk->private; struct mm_struct *mm =3D tlb->mm; struct vm_area_struct *vma =3D walk->vma; + struct folio_batch fbatch; spinlock_t *ptl; pte_t *start_pte, *pte, ptent; struct folio *folio; @@ -673,9 +675,10 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned= long addr, unsigned long next; int nr, max_nr; =20 + folio_batch_init(&fbatch); next =3D pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) - if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next)) + if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next, &fbatch)) return 0; =20 tlb_change_page_size(tlb, PAGE_SIZE); @@ -733,6 +736,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, continue; folio_get(folio); lazy_mmu_mode_disable(); + fbatch_drain_lazyfree(&fbatch); pte_unmap_unlock(start_pte, ptl); start_pte =3D NULL; err =3D split_folio(folio); @@ -777,13 +781,14 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigne= d long addr, clear_young_dirty_ptes(vma, addr, pte, nr, cydp_flags); tlb_remove_tlb_entries(tlb, pte, nr, addr); } - folio_mark_lazyfree(folio); + folio_mark_lazyfree(&fbatch, folio); } =20 if (nr_swap) add_mm_counter(mm, MM_SWAPENTS, nr_swap); if (start_pte) { lazy_mmu_mode_disable(); + fbatch_drain_lazyfree(&fbatch); pte_unmap_unlock(start_pte, ptl); } cond_resched(); --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01217463B82 for ; Wed, 9 Sep 2026 09:55:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947751; cv=none; b=HZXrG1RqqYbbgH89J6PmYx67BwAHrdavpvTTW7CfevYEvLxfjzgw47mtUaJBu6Z5WcTVK1t4o+M43DuM63DN9DfsJMm2cm3IAGl2rfax4j11tgiB7NJwUPy1rNuH5VgYm9BElNUqosdsAg3wsYtzsuW6V8w3rs3IH9skW7npzIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947751; c=relaxed/simple; bh=kbaqt5hPDumKhXkheDk4P0vgQgXUA4ehrbmFU/MMECM=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=UpAx0jLdGG/mJ/oCijfgVzRTgqw9EDMAAE+/NGUBIkJIkN85iV57bW8XRtMXgW6/7fs5WvDavH1ZjffhWDtaa/NlsVNB0IqYd1/XbsCEItaqUZ+jHYDWOnHXXlrogKIJ6JJI8/ETdwuXuWVFai7/ZCOf2mFfN3vJmJFgyYOJrX8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UPq81nAg; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UPq81nAg" Received: by mail-yx2-f12.google.com with SMTP id 00721157ae682-85d46e4cdcaso7912857b3.3 for ; Wed, 09 Sep 2026 02:55:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947748; x=1789552548; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9OZ5HJWp6Sq+cHPfV8As3zDywlXal3eQY4p7OoCaDH4=; b=UPq81nAgi1NHwETvvlhae7DRR7pQB7Uux4kDjmUA4a8HfV1dnf/uEzAX+acThrkBOu lk9L8p9cEGe8MJGhGONrVADoSFcTg4UuLmL4nEfwaea00NQ+zO82DDIDsaAvh7AHAd/n ec82RprQTxXaWXZyamg7ECdIPzIH4LIyuMefDOIU8/IJGup5MxBRc5AF0MsXQ7duRVSC b8pk+xDTtOD/fDLbZBSKkqqRNOyNzo+SzuuN1TdkbbF6iKWwFr990VE41P+n6bRQPkj6 CN7IJQkGNWD13IcE7pjIxdouomIM89MvCsf9KhXd9Buadb/Z+lYa812ZbPkYnqa10ngg 24XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947748; x=1789552548; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9OZ5HJWp6Sq+cHPfV8As3zDywlXal3eQY4p7OoCaDH4=; b=VZuGU2Q5xRSGGlPXvHRoiAqx6OrbV5WtJ+s36QiBxAFAQ8OUE7V1wf5X2GKUzCqHjh vVudKo3UzYFk6QtFgZmKSc440KwysVkasJktnv1TZVPjjeX6XXePHxJC9m2QAzYv8Dyt 3ky6k/+Jeyjp4fwEg6yUW8pqN73g3DD5wOUF7M3jQwzXUA5siWe2FBYkgnz5aypWw28x aETIQgjYm14J3QTYCE2sepqoHx/lmtuqYgkJtnoCuoFxPhJB1ftO1/BM5FHjPLRFq+o0 YI1UTr0vZ/lBTeX1voZx7tflLMM5f3eN8KoQhvfFPhqgpIu1BFqVkDCaPzDCA7Qir+Kb 1/JA== X-Forwarded-Encrypted: i=1; AKwUvBwuG/4CMY5Du+k/e7okTDX4wk1IdUZTM34UBtwUnS2TaTklCgQwN9RTe/X0Ily7G/OSEg6WMudAZejLk+A=@vger.kernel.org X-Gm-Message-State: AFuF++m5r4fSDzkcBR5yo2jmx91789gze9A2HtAYxX/shNFde+u12ZWw z9/tUaXMGvxOTZhypfoCRdHH5vY8k35v/b1qziBGsOHlYFhSuA7R6sT/xiCdxIBPLQ== X-Gm-Gg: AYBFou3Cj1kyLLGMkdG5Sg6X9AjgLY9RixcXUEnlmQLyS7VamMYinwgrlIcLQBCJqcC Td1Qw/Bh5qhk3BlW7TU0Jztl5JtSC0JYFQY1PuZWhDMhdG0RI1izEownsXMRJFxSLxKILgWN7Hh /9HB1FXX306Zgfevz5qT2JnACZHv154KvZkLgnoPknyqfcid333Qv2OZtLzc5V9AjM6lLlqjj0P XGERoS/1IALekJyeKIItBQrsFHfpVugUClNRwIbI0oXxrUD23HGyclLAgaKfyxBji5xXq35nt// wgChOuKRKvoj/wM4liGBu6BHL4DS2wEgMxSzEkbdh5DZUStlxNs97u4SbPPIuG++Yb27KXCbLa7 LcmaoAJzfZNZDq/EwdyxKwGApBeyEw6w7YSnEpTRGBh0sRxd+xvlW+Skhi9qXcgUDqLF6XIun83 0CXZmnmAjmpC1v9F/mqxSWsxCGkWCnsd4+CpYb0NRSIlI+A5Dl2jUcHPMOwY2QosTCf74+w/693 9+XWMyTAS3aO0kpcxFqI7Iw81mrN0geYE2/iWDCJ0C3fkZjyFHHZQPEbskOUBdWq0A0Uw== X-Received: by 2002:a05:690c:5885:b0:854:9062:73ae with SMTP id 00721157ae682-87f29d02667mr18443457b3.33.1788947747384; Wed, 09 Sep 2026 02:55:47 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714ab77c5csm107163607b3.32.2026.09.09.02.55.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:55:46 -0700 (PDT) Date: Wed, 9 Sep 2026 02:55:41 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 07/26] mm/fbatch: LRU_NEXT_ACTIVATE bit to optimize folio_activate() In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement an equivalent to the old __lru_cache_activate_folio() optimization, to activate a folio recently put in the lru_add fbatch, without having to put it through the lru_activate fbatch too. Neither lruvec lock nor lru bit can guard this safely and efficiently, so resort to try_cmpxchg() on a further, LRU_NEXT_ACTIVATE bit in folio->lru_next. Signed-off-by: Hugh Dickins --- include/linux/mm_inline.h | 4 ++++ mm/folio.c | 23 ++++++++++++++++++++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 8420b1276535..8f5efadf9c7c 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -346,6 +346,7 @@ static inline void folio_migrate_refs(struct folio *new= , const struct folio *old enum { LRU_NEXT_NEVER_TAIL =3D 0, /* Used by a tail's compound_head */ LRU_NEXT_BATCHED =3D 1, /* Not used by any aligned pointer */ + LRU_NEXT_ACTIVATE, NR_LRU_NEXT_FLAGS }; =20 @@ -358,6 +359,9 @@ bool lru_add_del_folio(struct folio *folio) if (!(lru_next & BIT(LRU_NEXT_BATCHED))) return false; =20 + if (lru_next & BIT(LRU_NEXT_ACTIVATE)) + folio_set_active(folio); + WRITE_ONCE(folio->lru.next, LIST_POISON1); /* BUG_ON(folio->lru_next & BIT(LRU_NEXT_BATCHED)); */ =20 diff --git a/mm/folio.c b/mm/folio.c index a18d8ef6afd5..0b75c3b69d5a 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -256,15 +256,32 @@ static void lru_activate(struct lruvec *lruvec, struc= t folio *folio) =20 void folio_activate(struct folio *folio) { + unsigned long lru_next; + if (folio_test_active(folio) || folio_test_unevictable(folio) || !folio_test_lru(folio)) return; =20 /* - * XXX: It is curiously difficult to recreate safely the old - * __lru_cache_activate_folio() optimization (folio_set_active() - * directly if it's on the local lru_add fbatch): revisit later. + * This optimization is intended for the common case of folio + * having been recently added to this CPU's lru_add fbatch. + * But since other CPUs can now take it at any instant (after + * a folio_test_clear_lru()), and we may be migrated to another + * CPU, it is simplest just to extend the optimization to all CPUs. + * + * folio_set_active() would be unsafe without the lruvec lock, and + * a folio_test_clear_lru() here might cause a racing drain of the + * lru_add fbatch to skip its lru_add(): so use try_cmpxchg(). */ + lru_next =3D READ_ONCE(folio->lru_next); + while (lru_next & BIT(LRU_NEXT_BATCHED)) { + if (lru_next & BIT(LRU_NEXT_ACTIVATE)) + return; + if (try_cmpxchg(&folio->lru_next, &lru_next, + lru_next | BIT(LRU_NEXT_ACTIVATE))) + return; + } + folio_batch_add_and_move(folio, lru_activate); } =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f44.google.com (mail-yx1-f44.google.com [74.125.224.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8D0E49DB8C for ; Wed, 9 Sep 2026 09:57:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947868; cv=none; b=gxAnx+FsiAnE+qfOkzIFQJFGvdehSpV5pOQlh+3BoldwTzX0c15xuUS5Y63sB/hgTRfOq7P7zgL19BWIT9UjKSwKtmHN5sGAbB08I3FGRMxpK4Nt0JyqKAIOk/vDCFOJgWcJGluB74bJjkdz00aZGHoxYIZFCJ2zNb6a8kDR9aM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947868; c=relaxed/simple; bh=Lu6+2Sh9UbWS2A6fpr98+wTn38kozrsF+/qIOZDiezw=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=fe+3HQF4xYAyk3tIlinrYIKa1KyiqLexHyrdPF5oJHRhnmTdahIm41PZ30hWIfburLA0kGsuS0rqxBzUVHq8hCPPX74qY6MUZJhYCxQWcI4IsCLrCH/Ta8Mhid0kyWOk9L0dshIk5hi8FOoYVkrCQdgSeqZMsva1Tgsqe6IPZxk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QnamKCBk; arc=none smtp.client-ip=74.125.224.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QnamKCBk" Received: by mail-yx1-f44.google.com with SMTP id 956f58d0204a3-66e64e6e3fbso4986792d50.2 for ; Wed, 09 Sep 2026 02:57:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947866; x=1789552666; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=si69YHQMirkBMxmzc+pJ5uD7PWHLsBw1JDPVVCPJL9Q=; b=QnamKCBkEqLIrqImoayAM9p1VZMZ8YT8+y8dSP2LqKDcPC/n2hhSijYK9bBIplx2OQ pneC5kej/S9qexwR2SUqbkNWLBEH9iqroXM/IWMFFC9P78HSbe9SdM0cDBngpFVbotsS PREWPXFYuBilXpQjPREHxdLoTy6/QRe17EMFvvUg/Ua940aF90yufuhHnKPBSW1I0on6 53In+vLsgzOVPdznlCbxpMXl2p3k5cOF4EM3EPx265gfpg+JBJjS4ErDTIw4wsOv4Twu 9v+zZ9amdQ1e5cWb9vGH9D4EfiXz28foC21Flvs3Uq5buiVltF97n35t9pgHXh4i/H/v 1DOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947866; x=1789552666; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=si69YHQMirkBMxmzc+pJ5uD7PWHLsBw1JDPVVCPJL9Q=; b=UcEd5dLzqGaBwcna22z9MSlfTpa48x3ir4sPkuoXDu1L0EDoMQ0068X5oNZ496JR/P u+nRxwfOyFmdPr0IXpIMKdiir2TTkctBtJvu6ViRkT9suJCdHVuety4DCzceJ3uyjLnA hF8ewKvPPm7DEInakpZf6h77z6l1Dx/EoaLZAvRHc2BHBU4PLXmbz+YTRUecle/6unh1 CSFyu4IifZHPzmWnn565Q5yxmGbPOshPSK//tA+Z47/Kgn1IoIxjDbFJEfrbeKSQH4E3 et6qMesD8IKOhb+fHFCopd956gr9S1CWazcvxuyuwVQZWqypuGaKUfQTTwV4/TF77EwB /yag== X-Forwarded-Encrypted: i=1; AKwUvBy7JVXeV0vF5FxOi/sj4Cd2cLBRsXLEsp/aDmNF8YHBwaHJrQz/qxRXwavVi3ltNb+xrDz7sCC8xbwri48=@vger.kernel.org X-Gm-Message-State: AFuF++kx0LFCJV8pH/Pm9lTrvmPHqkM4XTmaycwvYO5EqxpO626Dqzod 1GsP0LUVfn1nMtYSak/NFdmZYeTQvTxAM8qSakf9ABf7y7ldtGLu3nxhgTJXXs8SaA== X-Gm-Gg: AYBFou1U7w+kOGGYr555YX5K0G9NeyR6DKF94g4oDf7VMfqp9HcvJJQzErV9XTs7XP+ SYEWNmZBEvDHkxFSVm+6WkovHa4P8Ae3ea5yBLN1nKFt/kqRKdtTHXgHUD/rlDearS/XORfO32/ TAmqXxqmX5qkNnLuA0Fw7WhwRzgES0h6bM/JCntD1RWaVSOut6lkBV2e4SWUl/MfkS5GNYzUPNW pYRadjcCd0R8I9c9Z6pmUk1tcGEQ3ULjZHmNB0r3io1GfgtVsjjJWMFsgzAJQzrI+kedlPXXcX1 /yo4uNhay4ZGW85/QMQFkKBhFMZpli6R15TU+3EGvrhjrPWlGoNk0hfb3SX00d17vBttjcN+zg0 KMafKp0VLOZrcQHPBDi5hfZAK4GuvOInzJJbzH0VUt6CTxx5uawTACtNXxkoPFbf63/MTarvoZM BqNyO2kqarNO9vmAgyiC66t362isxZgTfJ4mjnaVwxTnsW5zhrwbOT+IUA6aDYNwUCBdagzuspw 7rr9yGJZ6/HiqseEtIzjRIs0yqRSyL8i4xGc7zLfG+YOBepROR2AW/NUwQ= X-Received: by 2002:a05:690e:13cc:b0:66f:c6c6:aab8 with SMTP id 956f58d0204a3-66fc6c6bd7emr7410637d50.15.1788947864938; Wed, 09 Sep 2026 02:57:44 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb47a8894sm12054613d50.1.2026.09.09.02.57.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:57:44 -0700 (PDT) Date: Wed, 9 Sep 2026 02:57:39 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 08/26] mm/fbatch: replace mlock_new_folio() by __folio_add_lru(,mlockit) In-Reply-To: Message-ID: <6757bf3c-599f-4ef4-85d3-d572e3bb59fe@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace mlock_new_folio(), working on mm/mlock.c's mlock_fbatch, by __folio_add_lru(,mlockit), working on mm/folio.c's lru_add fbatch: folio moved to lruvec by lru_add(), with its mlocking incidental. Remove old comment about not needing smp_mb__after_atomic() from lru_add(): but that is a detail which will need to be reconsidered. Initialize mlock_count earlier, when adding to fbatch rather than when adding to lruvec. mlock_count count in 2s, with the low bit set to distinguish it from lru.prev. This helps when an mlocked folio is put back early by compaction, but will enable further optimization next. Change mlock_count from unsigned int to long: long to match pointer without endian concerns, signed for better treatment of those rare cases when final munlocks precede their still batched mlocks. This is an intermediate, poorly tested review stage: mlock_new_folio() code removed from mm/mlock.c, remaining code there updated to respect the new mlock_count accounting, but not considered beyond that. Signed-off-by: Hugh Dickins Reviewed-by: Vlastimil Babka (SUSE) --- Documentation/mm/unevictable-lru.rst | 2 +- include/linux/mm_types.h | 10 +++-- include/linux/swap.h | 6 ++- mm/folio.c | 59 ++++++++++++------------ mm/huge_memory.c | 2 +- mm/internal.h | 4 +- mm/mlock.c | 67 ++++------------------------ 7 files changed, 53 insertions(+), 97 deletions(-) diff --git a/Documentation/mm/unevictable-lru.rst b/Documentation/mm/unevic= table-lru.rst index 8d11fe6a0854..45b453226336 100644 --- a/Documentation/mm/unevictable-lru.rst +++ b/Documentation/mm/unevictable-lru.rst @@ -314,7 +314,7 @@ For each PTE (or PMD) being faulted into a VMA, the pag= e add rmap function calls mlock_vma_folio(), which calls mlock_folio() when the VMA is VM_LOCK= ED (unless it is a PTE mapping of a part of a transparent huge page). Or when it is a newly allocated anonymous page, folio_add_lru_vma() calls -mlock_new_folio() instead: similar to mlock_folio(), but can make better +__folio_add_lru(mlockit) instead: similar to mlock_folio(), but can make b= etter judgments, since this page is held exclusively and known not to be on LRU = yet. =20 mlock_folio() sets PG_mlocked immediately, then places the page on the CPU= 's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index fe6220b97cf3..772e3acda8c2 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -86,7 +86,7 @@ struct page { * means the other users of this union MUST NOT use the bit to * avoid collision and false-positive PageTail(). * Bit 1 of the first word is used by page_is_pfmemalloc(). - * Bit 1 of the first word (lru_next) is also used by folio_add_lru(). + * Bit 1 of the first word (lru_next) is also used by __folio_add_lru(). */ union { struct { /* Page cache and anonymous pages */ @@ -361,7 +361,7 @@ typedef unsigned short mm_id_t; * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. * @lru: Least Recently Used list; tracks how recently this folio was used. - * @mlock_count: Number of times this folio has been pinned by mlock(). + * @mlock_count: Number of times this folio has been pinned by mlock() *2 = +1 * @mapping: The file this page belongs to, or refers to the anon_vma for * anonymous memory. * @index: Offset within the file, in units of pages. For anonymous memor= y, @@ -415,7 +415,7 @@ struct folio { struct { unsigned long lru_next; /* public: */ - unsigned int mlock_count; + long mlock_count; /* private: */ }; /* public: */ @@ -509,6 +509,10 @@ struct folio { }; }; =20 +/* folio's mlock_count is doubled, low bit set to distinguish from lru.pre= v */ +#define MLOCK_COUNT_0 1 /* Bit not set in any aligned pointer */ +#define MLOCK_COUNT_1 2 /* Increment or decrement mlock_count */ + #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct page, pg) =3D=3D offsetof(struct folio, fl)) FOLIO_MATCH(flags, flags); diff --git a/include/linux/swap.h b/include/linux/swap.h index 5658a1634b85..a6fad5127118 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -294,7 +294,11 @@ extern unsigned long totalreserve_pages; #define nr_free_pages() global_zone_page_state(NR_FREE_PAGES) =20 /* linux/mm/folio.c */ -void folio_add_lru(struct folio *folio); +void __folio_add_lru(struct folio *folio, bool mlockit); +static inline void folio_add_lru(struct folio *folio) +{ + __folio_add_lru(folio, false); +} void folio_mark_accessed(struct folio *folio); void lru_add_drain_all(void); =20 diff --git a/mm/folio.c b/mm/folio.c index 0b75c3b69d5a..18e97923e527 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -113,31 +113,12 @@ static void lru_add(struct lruvec *lruvec, struct fol= io *folio) =20 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); =20 - /* - * Is an smp_mb__after_atomic() still required here, before - * folio_evictable() tests the mlocked flag, to rule out the possibility - * of stranding an evictable folio on an unevictable LRU? I think - * not, because __munlock_folio() only clears the mlocked flag - * while the LRU lock is held. - * - * (That is not true of __page_cache_release(), and not necessarily - * true of folios_put(): but those only clear the mlocked flag after - * folio_put_testzero() has excluded any other users of the folio.) - */ if (folio_evictable(folio)) { if (was_unevictable) __count_vm_events(UNEVICTABLE_PGRESCUED, nr_pages); } else { folio_clear_active(folio); folio_set_unevictable(folio); - /* - * folio->mlock_count =3D !!folio_test_mlocked(folio)? - * But that leaves __mlock_folio() in doubt whether another - * actor has already counted the mlock or not. Err on the - * safe side, underestimate, let page reclaim fix it, rather - * than leaving a page on the unevictable LRU indefinitely. - */ - folio->mlock_count =3D 0; if (!was_unevictable) __count_vm_events(UNEVICTABLE_PGCULLED, nr_pages); } @@ -383,15 +364,16 @@ void folio_mark_accessed(struct folio *folio) EXPORT_SYMBOL(folio_mark_accessed); =20 /** - * folio_add_lru - Add a folio to an LRU list. + * __folio_add_lru - Add a folio to an LRU list. * @folio: The folio to be added to the LRU. + * @mlockit: Mark the folio as mlocked. * * Queue the folio for addition to the LRU. The decision on whether * to add the page to the [in]active [file|anon] list is deferred until the * folio_batch is drained. This gives a chance for the caller of folio_add= _lru() - * have the folio added to the active list using folio_mark_accessed(). + * to have the folio added to the active list using folio_mark_accessed(). */ -void folio_add_lru(struct folio *folio) +void __folio_add_lru(struct folio *folio, bool mlockit) { struct folio_batch *fbatch; unsigned long lru_next; @@ -426,6 +408,27 @@ void folio_add_lru(struct folio *folio) lru_next |=3D BIT(LRU_NEXT_BATCHED); folio->lru_next =3D lru_next; =20 + if (mlockit) { + long nr_pages =3D folio_nr_pages(folio); + + folio_set_mlocked(folio); + folio->mlock_count =3D MLOCK_COUNT_0 + MLOCK_COUNT_1; + zone_stat_mod_folio(folio, NR_MLOCK, nr_pages); + __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); + } else if (folio_test_mlocked(folio)) { + /* + * A folio is being put back while mlocked. If mlock_count + * has not been overwritten by use of lru.prev, believe it. + * Otherwise, since there may be __mlock_folio()s to come + * through, initialize it to the safer 0 rather than to 1. + */ + if (!(folio->mlock_count & MLOCK_COUNT_0)) + folio->mlock_count =3D MLOCK_COUNT_0; + } else { + /* Initialize this field, which the page allocator did not */ + folio->mlock_count =3D MLOCK_COUNT_0; + } + full =3D !folio_batch_add(fbatch, folio); =20 /* Ensure folio->lru_next visible to folio_test_clear_lru() callers */ @@ -437,24 +440,20 @@ void folio_add_lru(struct folio *folio) =20 local_unlock(&cpu_fbatches.lock); } -EXPORT_SYMBOL(folio_add_lru); +EXPORT_SYMBOL(__folio_add_lru); =20 /** * folio_add_lru_vma() - Add a folio to the appropriate LRU list for this = VMA. * @folio: The folio to be added to the LRU. * @vma: VMA in which the folio is mapped. * - * If the VMA is mlocked, @folio is added to the unevictable list. + * If the VMA is mlocked, @folio will be added to the unevictable list. * Otherwise, it is treated the same way as folio_add_lru(). */ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) { - VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - - if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) =3D=3D VM_LOCKED)) - mlock_new_folio(folio); - else - folio_add_lru(folio); + __folio_add_lru(folio, + (vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) =3D=3D VM_LOCKED); } =20 /* diff --git a/mm/huge_memory.c b/mm/huge_memory.c index abebd8a23e56..d53e5443b802 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3623,7 +3623,7 @@ static void lru_add_split_folio(struct folio *folio, = struct folio *new_folio, /* head is still on lru (and we have it frozen) */ VM_WARN_ON(!folio_test_lru(folio)); if (folio_test_unevictable(folio)) - new_folio->mlock_count =3D 0; + new_folio->mlock_count =3D MLOCK_COUNT_0; else list_add_tail(&new_folio->lru, &folio->lru); folio_set_lru(new_folio); diff --git a/mm/internal.h b/mm/internal.h index 0d78406eb126..40bd128a58ae 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -974,7 +974,7 @@ folio_within_vma(struct folio *folio, struct vm_area_st= ruct *vma) * * mlock is usually called at the end of folio_add_*_rmap_*(), munlock at * the end of folio_remove_rmap_*(); but new anon folios are managed by - * folio_add_lru_vma() calling mlock_new_folio(). + * folio_add_lru_vma() calling __folio_add_lru(). */ void mlock_folio(struct folio *folio); static inline void mlock_vma_folio(struct folio *folio, @@ -1009,7 +1009,6 @@ static inline void munlock_vma_folio(struct folio *fo= lio, munlock_folio(folio); } =20 -void mlock_new_folio(struct folio *folio); bool need_mlock_drain(int cpu); void mlock_drain_local(void); void mlock_drain_remote(int cpu); @@ -1136,7 +1135,6 @@ static inline bool vma_supports_mlock(const struct vm= _area_struct *vma) =20 #else /* !CONFIG_MMU */ static inline void unmap_mapping_folio(struct folio *folio) { } -static inline void mlock_new_folio(struct folio *folio) { } static inline bool need_mlock_drain(int cpu) { return false; } static inline void mlock_drain_local(void) { } static inline void mlock_drain_remote(int cpu) { } diff --git a/mm/mlock.c b/mm/mlock.c index 929abeac85d0..2c690f18031e 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -85,14 +85,16 @@ static struct lruvec *__mlock_folio(struct folio *folio= , struct lruvec *lruvec) =20 if (folio_test_unevictable(folio)) { if (folio_test_mlocked(folio)) - folio->mlock_count++; + folio->mlock_count +=3D MLOCK_COUNT_1; goto out; } =20 lruvec_del_folio(lruvec, folio); folio_clear_active(folio); folio_set_unevictable(folio); - folio->mlock_count =3D !!folio_test_mlocked(folio); + folio->mlock_count =3D MLOCK_COUNT_0; + if (folio_test_mlocked(folio)) + folio->mlock_count +=3D MLOCK_COUNT_1; lruvec_add_folio(lruvec, folio); __count_vm_events(UNEVICTABLE_PGCULLED, folio_nr_pages(folio)); out: @@ -100,25 +102,6 @@ static struct lruvec *__mlock_folio(struct folio *foli= o, struct lruvec *lruvec) return lruvec; } =20 -static struct lruvec *__mlock_new_folio(struct folio *folio, struct lruvec= *lruvec) -{ - VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - - lruvec =3D folio_lruvec_relock_irq(folio, lruvec); - - /* As above, this is a little surprising, but possible */ - if (unlikely(folio_evictable(folio))) - goto out; - - folio_set_unevictable(folio); - folio->mlock_count =3D !!folio_test_mlocked(folio); - __count_vm_events(UNEVICTABLE_PGCULLED, folio_nr_pages(folio)); -out: - lruvec_add_folio(lruvec, folio); - folio_set_lru(folio); - return lruvec; -} - static struct lruvec *__munlock_folio(struct folio *folio, struct lruvec *= lruvec) { int nr_pages =3D folio_nr_pages(folio); @@ -132,9 +115,9 @@ static struct lruvec *__munlock_folio(struct folio *fol= io, struct lruvec *lruvec =20 if (folio_test_unevictable(folio)) { /* Then mlock_count is maintained, but might undercount */ - if (folio->mlock_count) - folio->mlock_count--; - if (folio->mlock_count) + if (folio->mlock_count > MLOCK_COUNT_0) + folio->mlock_count -=3D MLOCK_COUNT_1; + if (folio->mlock_count > MLOCK_COUNT_0) goto out; } /* else assume that was the last mlock: reclaim will fix it if not */ @@ -165,17 +148,11 @@ static struct lruvec *__munlock_folio(struct folio *f= olio, struct lruvec *lruvec * Flags held in the low bits of a struct folio pointer on the mlock_fbatc= h. */ #define LRU_FOLIO 0x1 -#define NEW_FOLIO 0x2 static inline struct folio *mlock_lru(struct folio *folio) { return (struct folio *)((unsigned long)folio + LRU_FOLIO); } =20 -static inline struct folio *mlock_new(struct folio *folio) -{ - return (struct folio *)((unsigned long)folio + NEW_FOLIO); -} - /* * mlock_folio_batch() is derived from folio_batch_move_lru(): perhaps tha= t can * make use of such folio pointer flags in future, but for now just keep i= t for @@ -192,14 +169,12 @@ static void mlock_folio_batch(struct folio_batch *fba= tch) =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { folio =3D fbatch->folios[i]; - mlock =3D (unsigned long)folio & (LRU_FOLIO | NEW_FOLIO); + mlock =3D (unsigned long)folio & LRU_FOLIO; folio =3D (struct folio *)((unsigned long)folio - mlock); fbatch->folios[i] =3D folio; =20 - if (mlock & LRU_FOLIO) + if (mlock) lruvec =3D __mlock_folio(folio, lruvec); - else if (mlock & NEW_FOLIO) - lruvec =3D __mlock_new_folio(folio, lruvec); else lruvec =3D __munlock_folio(folio, lruvec); } @@ -261,30 +236,6 @@ void mlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } =20 -/** - * mlock_new_folio - mlock a newly allocated folio not yet on LRU - * @folio: folio to be mlocked, either normal or a THP head. - */ -void mlock_new_folio(struct folio *folio) -{ - struct folio_batch *fbatch; - int nr_pages =3D folio_nr_pages(folio); - - local_lock(&mlock_fbatch.lock); - fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); - folio_set_mlocked(folio); - - zone_stat_mod_folio(folio, NR_MLOCK, nr_pages); - __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); - - folio_get(folio); - if (!folio_batch_add(fbatch, mlock_new(folio)) || - true || /* XXX Temporarily disable mlock_new batching */ - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) - mlock_folio_batch(fbatch); - local_unlock(&mlock_fbatch.lock); -} - /** * munlock_folio - munlock a folio * @folio: folio to be munlocked, either normal or a THP head. --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D61FD3D3CEE for ; Wed, 9 Sep 2026 09:59:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947970; cv=none; b=hlZXcB3+P8vrgnEPw05CA9/aSG983uXmKbLuqLiyUhD/yXT7Vgg0MnI/Z5cVIVbSMCHux0aqEqhVWcfsePf2OD2ntP492qRwh9f6MnSmN9jrEvyUFfEHINh27SQFJIvM4RYQULv1IpT1cgrw7XyNMF3MUG6iHx/WmVYL0MX23eY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788947970; c=relaxed/simple; bh=IHzeNW4hzepc7IxdL36p9gKzWS2I1X5tar9DJ0IM6+Q=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=H3KW9ZE80JWv2GN0+CohO5vA2M6JCpGVFzAxSbR6owQmwE/mJf7lcsMynH8ThnZJiVB3kafgl/FncYn5c2M2QZnBZVE1oeAJgGn+C5n9RY+uf0DQ4P6ZoGxL3ZAYFm+wpq2MYtAN6YkXLu0vfRx2y97J/KxXaWUDUrb5qqvf3N4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=muDnN76x; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="muDnN76x" Received: by mail-yx2-f12.google.com with SMTP id 956f58d0204a3-66e4ab2029eso498d50.1 for ; Wed, 09 Sep 2026 02:59:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788947968; x=1789552768; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=lcHKAya+E91RVS5tU8FF9W2nmuTVga+iF8pC6HMIeNs=; b=muDnN76xWRqveKhXb6pdgab8dzilAKxPgXsLv5ubkAmcOYKm8vNv7U8pR5/8YykYhi aSAzBF6FpyMEgq7OF+wtoc88QjekB33PT5nvElUpRgpCnZcjTKtOxqK94Y2bphCgFSdW 4CTtVOei3ir/EajkPxEZci8w1eHkRJwcn2/V1wUHKbhEn333GupLOi/vodZNNHQ5vVh3 Flh1wDoIlAo20UNXDLfbYy9F/BCkf3CLWnbun9NZEZAj+UnAgOoF36GkV4MEsXkb8Qvn ciS6hljlOtzr+xyRLMDRRO1DH3zstEb1E4lLuWenaKA/pl25Lh38fOQoOmJSrECUzgxN MefQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788947968; x=1789552768; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lcHKAya+E91RVS5tU8FF9W2nmuTVga+iF8pC6HMIeNs=; b=jk6JSMFofZgXS+vkVxskKGJE3b5ZsRl2guQVLzYMAAvbdB4o+WE+crkSDkfIhOpfRh G3i3OmwT6TawuD7fSLNhgIu20VYWg1lYEfLkRIwJ9IFYooertnxpuNIZutomn9Zgjymj nZyxZfwsMLAU36FKmLWhXIiqUQqfJTwvcru44suyQ4I2E4DMRDm2zs6/90JdpiF58az/ RJvLJABMnJytaPkwkaBegL95C7jPMyWcaa49t1MbZYmBzNj4L+f/g/rMrUx7a/HEo8dD PK7+2h4B6i/FyQljhNXLFkqsRYri/XXgLu7Njus89aigDgPIT2hyMTpZzAcrmMiiSnum 57wg== X-Forwarded-Encrypted: i=1; AKwUvBzPt1aZiuGEeIeL4Ev5h3o/HPjhT72mBwftpKDnbzR/7BlcxGgEBVYal1Zll/vD/iOPecswYXbE9r7kQus=@vger.kernel.org X-Gm-Message-State: AFuF++mCIqsjTCx3j+4Fis//uKZ8kH9y6tKKZ53T+6MqDN6RuPgfPvHA OMfSLScgAR82guYqhA8guKdXAxVmz9P8XSqeDmckvYd/Q8HAwMksBqj06JCdd+W9JA== X-Gm-Gg: AYBFou0lSd0zS5fSq20//+aOwH7MgW4og8ZFX/vGpvm8lT8lTLdWtdO7Ovf1UWZqPcZ NL6+VUrubZpce/DpyTPA68y6s+cHli+9QDGCQrs80t5ddxT9viE/ZYiwYqt2bj+A20RuobWi9eh d34yatuJvQdVdQeJxcoyUREN0a9HJmiZ+VSwm7+J5ZMz11gnpCrwQLhXGT/JR/Ye1Xue5acLAUy EYQIEhvKI6/wmBCUTBS886rp8Ybz+unhr4ed8u9cdWCXQMHYBABwaVj7bHs36R9ca57CMEWt1e9 mUyPv8wyqSlqXFuchbbPP15gLcJYAH4a9QxJOlwnKEIU2yqH3KMqChS4XpntjutEIXwrjK2TVb9 1l5T+huzGQLcTTRZdTFPrVH1SAwMfxmUQItgW9wUhNBFEQofGjEM1AryJ6XDXpr+F0xgfgRWhT/ u+wM3LnhddjQVU12Nx9dsDlaFeTt7Um+8zvvftn8nEk6bEmUreElWIW9+xOc5wELS0CG2stsT3q o+ZF4wAQKe9GoEhV+JZ8PD7KSLvXYSPEajUP+qfQW24QrINiK090b2ZtCXgjkrf7vJRkA== X-Received: by 2002:a05:690c:31a:b0:87c:a15f:b667 with SMTP id 00721157ae682-88153e6779amr21037b3.19.1788947966891; Wed, 09 Sep 2026 02:59:26 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714c1e1eacsm107507057b3.49.2026.09.09.02.59.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 02:59:25 -0700 (PDT) Date: Wed, 9 Sep 2026 02:59:21 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 09/26] mm/fbatch: restore mlock+munlock batching, without extra ref In-Reply-To: Message-ID: <69ba633c-8fd5-1f4b-2e1e-b9b15b8946be@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Update mlock_folio(), munlock_folio() and their fbatch callouts and helpers, to do folio_try_get()s at batch processing time, instead of holding a folio reference all the while in mlock_fbatch: as in folio.c. But more interesting is the use of mod_mlock_count(), using try_cmpxchg() to update folio->mlock_count safely when possible (now when on lru_add fbatch as well as when unevictable). While __mlock_folio() is as hard to think about as before, __munlock_folio() simpler because munlock_folio() can adjust mlock_count itself without clear_lru() or lruvec lock, and so do the folio_test_clear_mlocked() immediately for itself (without which unevictable_pgs_cleared was likely to appear high, when it should be 0 or low to indicate good mlock health). __munlock_folio() is safe for use even when the unreferenced folio has been freed and reused. It appears that __mlock_folio() could affect a folio which has been freed and reused, but only if it is reused as an mlocked folio, in which case its mlock_count is spuriously incremented (but usually a spurious munlock decrement will follow). How grave is this? If unevictable_pgs_cleared remains low, not so bad. I've gone back and forth on whether to move mlock_fbatch and these functions into mm/folio.c: for now they stay here in mm/mlock.c. Signed-off-by: Hugh Dickins --- mm/mlock.c | 147 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 86 insertions(+), 61 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index 2c690f18031e..1050010bbe0b 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -58,6 +58,20 @@ EXPORT_SYMBOL(can_do_mlock); * indicate the unevictable state. */ =20 +static long mod_mlock_count(struct folio *folio, long incdec) +{ + long mlock_count =3D READ_ONCE(folio->mlock_count); + + while (mlock_count & MLOCK_COUNT_0) { + if (mlock_count + incdec < MLOCK_COUNT_0) + return MLOCK_COUNT_0; + if (try_cmpxchg(&folio->mlock_count, &mlock_count, + mlock_count + incdec)) + return mlock_count + incdec; + } + return 0; +} + static struct lruvec *__mlock_folio(struct folio *folio, struct lruvec *lr= uvec) { /* There is nothing more we can do while it's off LRU */ @@ -65,6 +79,7 @@ static struct lruvec *__mlock_folio(struct folio *folio, = struct lruvec *lruvec) return lruvec; =20 lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + lruvec_del_folio(lruvec, folio); =20 if (unlikely(folio_evictable(folio))) { /* @@ -73,92 +88,82 @@ static struct lruvec *__mlock_folio(struct folio *folio= , struct lruvec *lruvec) * folio be unevictable? I'm not sure, but move it now if so. */ if (folio_test_unevictable(folio)) { - lruvec_del_folio(lruvec, folio); folio_clear_unevictable(folio); - lruvec_add_folio(lruvec, folio); - __count_vm_events(UNEVICTABLE_PGRESCUED, folio_nr_pages(folio)); } goto out; } =20 + /* + * Something to keep in mind when studying the arithmetic here: + * we only come to __mlock_folio() when mlock_folio() could not + * mod_mlock_count() itself; but by the time this is processed, + * the folio may have already been munlocked, or another mlock + * already marked it as unevictable and so mod_mlock_countable. + * And don't forget that a folio may be unevictable for reasons + * other than mlocked (hence the folio_evictable() check above). + */ + if (folio_test_unevictable(folio)) { if (folio_test_mlocked(folio)) - folio->mlock_count +=3D MLOCK_COUNT_1; + mod_mlock_count(folio, MLOCK_COUNT_1); goto out; } =20 - lruvec_del_folio(lruvec, folio); folio_clear_active(folio); folio_set_unevictable(folio); - folio->mlock_count =3D MLOCK_COUNT_0; - if (folio_test_mlocked(folio)) - folio->mlock_count +=3D MLOCK_COUNT_1; - lruvec_add_folio(lruvec, folio); __count_vm_events(UNEVICTABLE_PGCULLED, folio_nr_pages(folio)); + + if (!folio_test_mlocked(folio)) + folio->mlock_count =3D MLOCK_COUNT_0; + else if (!mod_mlock_count(folio, MLOCK_COUNT_1)) + folio->mlock_count =3D MLOCK_COUNT_0 + MLOCK_COUNT_1; out: + lruvec_add_folio(lruvec, folio); folio_set_lru(folio); return lruvec; } =20 static struct lruvec *__munlock_folio(struct folio *folio, struct lruvec *= lruvec) { - int nr_pages =3D folio_nr_pages(folio); - bool isolated =3D false; + long nr_pages =3D folio_nr_pages(folio); =20 - if (!folio_test_clear_lru(folio)) - goto munlock; - - isolated =3D true; - lruvec =3D folio_lruvec_relock_irq(folio, lruvec); - - if (folio_test_unevictable(folio)) { - /* Then mlock_count is maintained, but might undercount */ - if (folio->mlock_count > MLOCK_COUNT_0) - folio->mlock_count -=3D MLOCK_COUNT_1; - if (folio->mlock_count > MLOCK_COUNT_0) - goto out; - } - /* else assume that was the last mlock: reclaim will fix it if not */ - -munlock: - if (folio_test_clear_mlocked(folio)) { - zone_stat_mod_folio(folio, NR_MLOCK, -nr_pages); - if (isolated || !folio_test_unevictable(folio)) - __count_vm_events(UNEVICTABLE_PGMUNLOCKED, nr_pages); - else + /* There is nothing more we can do while it's off LRU */ + if (!folio_test_clear_lru(folio)) { + if (folio_test_unevictable(folio) && folio_evictable(folio)) __count_vm_events(UNEVICTABLE_PGSTRANDED, nr_pages); + /* But whoever puts it back on LRU should rescue it */ + return lruvec; } =20 - /* folio_evictable() has to be checked *after* clearing Mlocked */ - if (isolated && folio_test_unevictable(folio) && folio_evictable(folio)) { - lruvec_del_folio(lruvec, folio); + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + lruvec_del_folio(lruvec, folio); + + if (folio_test_unevictable(folio) && folio_evictable(folio)) { folio_clear_unevictable(folio); - lruvec_add_folio(lruvec, folio); __count_vm_events(UNEVICTABLE_PGRESCUED, nr_pages); } -out: - if (isolated) - folio_set_lru(folio); + + lruvec_add_folio(lruvec, folio); + folio_set_lru(folio); return lruvec; } =20 /* - * Flags held in the low bits of a struct folio pointer on the mlock_fbatc= h. + * Flag held in the low bits of a struct folio pointer on the mlock_fbatch. */ -#define LRU_FOLIO 0x1 -static inline struct folio *mlock_lru(struct folio *folio) +#define MLOCK_FLAG 0x1 +static inline struct folio *mlock_flagged(struct folio *folio) { - return (struct folio *)((unsigned long)folio + LRU_FOLIO); + return (struct folio *)((unsigned long)folio + MLOCK_FLAG); } =20 /* * mlock_folio_batch() is derived from folio_batch_move_lru(): perhaps tha= t can * make use of such folio pointer flags in future, but for now just keep i= t for - * mlock. We could use three separate folio batches instead, but one feels - * better (munlocking a full folio batch does not need to drain mlocking f= olio - * batches first). + * mlock. We could use separate folio batches instead, but one feels bett= er + * (munlocking a full folio batch does not need to drain mlocking batch fi= rst). */ static void mlock_folio_batch(struct folio_batch *fbatch) { @@ -169,10 +174,15 @@ static void mlock_folio_batch(struct folio_batch *fba= tch) =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { folio =3D fbatch->folios[i]; - mlock =3D (unsigned long)folio & LRU_FOLIO; + mlock =3D (unsigned long)folio & MLOCK_FLAG; folio =3D (struct folio *)((unsigned long)folio - mlock); fbatch->folios[i] =3D folio; =20 + if (!folio_try_get(folio)) { + fbatch->folios[i] =3D NULL; + continue; + } + if (mlock) lruvec =3D __mlock_folio(folio, lruvec); else @@ -218,19 +228,25 @@ void mlock_folio(struct folio *folio) { struct folio_batch *fbatch; =20 - local_lock(&mlock_fbatch.lock); - fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); - if (!folio_test_set_mlocked(folio)) { - int nr_pages =3D folio_nr_pages(folio); + long nr_pages =3D folio_nr_pages(folio); =20 zone_stat_mod_folio(folio, NR_MLOCK, nr_pages); - __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); + count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); } =20 - folio_get(folio); - if (!folio_batch_add(fbatch, mlock_lru(folio)) || - true || /* XXX Temporarily disable mlock batching */ + /* + * No more to do if mlock_count is maintained: either the folio + * is on an lru_add fbatch, and will be moved to unevictable in + * due course, or it's already counted as unevictable: no need + * for an mlock_fbatch entry below. + */ + if (mod_mlock_count(folio, MLOCK_COUNT_1)) + return; + + local_lock(&mlock_fbatch.lock); + fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); + if (!folio_batch_add(fbatch, mlock_flagged(folio)) || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); @@ -244,15 +260,24 @@ void munlock_folio(struct folio *folio) { struct folio_batch *fbatch; =20 + /* + * No more to do if mlock_count is maintained and still raised. + * But if mlock_count is unmaintained, we might need to queue an + * munlock fbatch entry, just to cancel an undequeued mlock entry? + */ + if (mod_mlock_count(folio, -MLOCK_COUNT_1) > MLOCK_COUNT_0) + return; + + if (folio_test_clear_mlocked(folio)) { + long nr_pages =3D folio_nr_pages(folio); + + zone_stat_mod_folio(folio, NR_MLOCK, -nr_pages); + count_vm_events(UNEVICTABLE_PGMUNLOCKED, nr_pages); + } + local_lock(&mlock_fbatch.lock); fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); - /* - * folio_test_clear_mlocked(folio) must be left to __munlock_folio(), - * which will check whether the folio is multiply mlocked. - */ - folio_get(folio); if (!folio_batch_add(fbatch, folio) || - true || /* XXX Temporarily disable munlock batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f44.google.com (mail-yx1-f44.google.com [74.125.224.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 944333EDE63 for ; Wed, 9 Sep 2026 10:01:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948093; cv=none; b=EKlmT6XyK9ZqWmJ/ZzRSHfsPxZHmSOpm6QNcLrsBdkhD/EaTdoLbWkiBIAiynb+Rog+xecf3CKmwH6EQhP+upoSt6CCNGBrk1ailXa29VbnRvNukwezEb7seOisrJkfQuGl0TgMfAw6xtXfJJT0WCYcNfM/+X7NrSTSDfMW0xD4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948093; c=relaxed/simple; bh=XRPintYB66K90WWvosFbfNFMrCSstXNSaa8RFhDin48=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=hoSYBqXyOVzgXxoK8pgUd7yb5wszXOAaF2PcKxSPuIWe9rkMOxHa5I+8m3lMReYQDebPX9T2npOMUK4BcuJgB4VIXu9dojfh/P7QuE3wYgNikNYsEy4wUBiNZElOxfY8k08T0Y5KKoZDuSnNMFmPKQ14lvAO5eV88FcuiNN6mKw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=knKN5ddo; arc=none smtp.client-ip=74.125.224.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="knKN5ddo" Received: by mail-yx1-f44.google.com with SMTP id 956f58d0204a3-66fcf87897fso4303946d50.3 for ; Wed, 09 Sep 2026 03:01:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948089; x=1789552889; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=uNBljcBd5pVR40/qSQbRzR+DUQkxJ6419XWxuVBt6mg=; b=knKN5ddo70WoGAqNT1cXYO4RFeqwvEyrgjfzjOTWh0ZI7z0eldN3H9OZHWMzP8Acyc tkz5Nq+NpL9Cl5yMr8vTo0h9DtLwMvjbjnDgVE8UFTKEmvfFzDvpIlPVvcrimFFtQsEo 9LMLjq2kA6FBn+PaMr1Wioy/GDYPYiJ+jn72tqftRAgcEk9VbaqC4hfgdzSPCARhEotX w4SSP7FiMBXRm2/0SbJmABzHUA1B8TZXd/3bFMQTALw7vLkROtUu6a2BAfQX7j4rcz1J dtvgs9uG01uy8plVKUXdLdoz0XvRykaSAj+NBOUpnmtxaC9UCWL3SAKZbwYtnXRJGMsw df8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948089; x=1789552889; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uNBljcBd5pVR40/qSQbRzR+DUQkxJ6419XWxuVBt6mg=; b=lz4Fbf08LbxyyDXl/9108WGuFPIZ2f/uFIkw0qMeTZ0a4RhERsV2Fq+FWzj2IJ+pi8 z4EUZ0wYWzfJW4URx9EpjwdxutiDMEPDBNg1FVFgMWVzzYVqqFBBGGqnDBxzURRUtSoT CcsXFakjtt2bRMmSU+TG/EHahcfsa1YLSucyroNRy9GyeOhjUI2zfr5eVGhLgkWCtZ+3 kmNZp8kp1boJdLvaNNLSEZyQVQOxj2y+xoITw1QNclRpLVsXJRNQmjxt47dVa1ZUbz+b nA9b8FmaqkQTn/Z7niqG7jXFvRrDEIV1s6IQRt6wFyo80RXuFHpO7FfXAI/ss9nBmif7 C5JA== X-Forwarded-Encrypted: i=1; AKwUvBxa3u74kMDRWZ6nlTuADHk6GqDJ33CEQVIY7OmZbSW2lh3gYofoHvrMdhFkVSVv0qE3MlO1fgvOAURiDVQ=@vger.kernel.org X-Gm-Message-State: AFuF++lEB3/wXxX7kNWTQ1EwyPcai3rjn+twy/QhXJjVEbKJxDIHcWsw 1xt7vfIXulydb0jChsP/oezY5FeK1eo+oOWi161pkB9vNoyVrwv2I5/UxdGoVjT/wg== X-Gm-Gg: AYBFou3GCFDaEi1xRbR0Rs1qQbitSLCD87DA3IFAyMdxV8EgH5hBVSR24DJ0ihFcgUt 3MF0mu1WTry4BDs92SnetwR8EGOOyAXv2MbnB4saPbCKlbrCYoBiHdOFEsDcogS6OGtGyqe2XZT v40wWgB/jEbd2qTM5FTmMH9EZr0eNnyghJWCTB+UsuL2Cy3zNa028KiFahmALJzMq6S/wRyWDOp L12wO84k4s4F8YCjKBZjeM7CF4b0blU3J09jS73UWty1NIa5Q7jX4+d/kM8hBqoF5xP4be8CZ8k GloXHsT0j8AcZurkrfLDw+8pf3NK8buqDWBSHUFPtxPSQmkFe6eM5uMafKb6U+XAbhyNKLMlOsv MhB4nY1e9sFELrWoHrRkwaBFOkecSVddvWYsiMXG9i17olkSI5nwM4YnW4xf7MwWymmPqVhf+KC OlG7kJvOxQl/AmPZeFieaCbOf+1mYRGuRoIPnFCDK2V4hb+QU/wpZHr278eQ9sYSpPtpUt6OLSe xk0SXwSIicT59kybSMNmtovJ9Ib+mPYtteyh1/PGniIqe3BXSn9WiTHl38= X-Received: by 2002:a05:690c:88:b0:864:6f4d:ef7f with SMTP id 00721157ae682-87121acbbc5mr139590577b3.7.1788948088169; Wed, 09 Sep 2026 03:01:28 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714a0b654dsm107264887b3.25.2026.09.09.03.01.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:01:26 -0700 (PDT) Date: Wed, 9 Sep 2026 03:01:22 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 10/26] mm/fbatch: remove several uses of mlock_drain_local() In-Reply-To: Message-ID: <9633684f-e5c4-d92b-8295-5cccaa18fc36@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove several uses of mlock_drain_local(): they were workarounds for when that sequence of operations tended to leave folios in mlock_fbatch with debilitating raised refcount (and no good if the task changed CPU). Signed-off-by: Hugh Dickins --- mm/huge_memory.c | 2 -- mm/migrate.c | 2 -- mm/rmap.c | 4 ---- 3 files changed, 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d53e5443b802..3628b5bd1359 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3570,8 +3570,6 @@ static bool __discard_anon_folio_pmd_locked(struct vm= _area_struct *vma, folio_remove_rmap_pmd(folio, pmd_page(orig_pmd), vma); zap_deposited_table(mm, pmdp); add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); folio_put(folio); =20 return true; diff --git a/mm/migrate.c b/mm/migrate.c index 15b45832bcfa..1939d6ed19c9 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -447,8 +447,6 @@ static bool remove_migration_pte(struct folio *folio, folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } - if (READ_ONCE(vma->vm_flags) & VM_LOCKED) - mlock_drain_local(); =20 trace_remove_migration_pte(pvmw.address, pte_val(pte), compound_order(new)); diff --git a/mm/rmap.c b/mm/rmap.c index d1819fd69938..1ee42d60d77d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2396,8 +2396,6 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, } finish_unmap: folio_remove_rmap_ptes(folio, page, nr_pages, vma); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); folio_put_refs(folio, nr_pages); =20 /* @@ -2771,8 +2769,6 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, hugetlb_remove_rmap(folio); else folio_remove_rmap_pte(folio, subpage, vma); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); folio_put(folio); } =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f42.google.com (mail-yx1-f42.google.com [74.125.224.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A11B4349AD for ; Wed, 9 Sep 2026 10:03:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948222; cv=none; b=L09R0atabjUrMv82G1zhk4JqYANfGoZ9HJqnYG+nQpt+cWMbYCrr28EPrqy2aCOkzbuLTD8+YGzWLkrIlCkETincBHEn9TgPBBWaO88Vtq0/wOdxvVsUxneU7IsADv+mi8s35fMOW7K2u5RlbGHldi5iJSgrYYFlJHjk1UgbZ8Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948222; c=relaxed/simple; bh=haV5ijKKeNFA+cVGf0/itoGu5pSxbzf5S7kuITcT64g=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=b4rHqQQm5Gq7VBycMFxFVmLWu3gctvj15Qjfn0UmGz56vh78ao9hPYqZQJm7zuS2BRP847d0L75v/K8zP3R7JNUWdWQj6V/JFawsOtlC0oDBRG3EWGweMmqlWP/XSXE8HLC9qqdgNSd2ENECga9LNdc/UUjYcW3knLB/MGQT7sM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SE9CepmM; arc=none smtp.client-ip=74.125.224.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SE9CepmM" Received: by mail-yx1-f42.google.com with SMTP id 956f58d0204a3-66d258d1effso743758d50.2 for ; Wed, 09 Sep 2026 03:03:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948217; x=1789553017; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=OZLYBDT0mJ+F0yYdyxDNynq3EZCGij4ebq9WOsT9s4c=; b=SE9CepmMPaMBUxXW9zwLm5VleuwwDDnZs5aoA/3cHA99HTqzSrYaHCkyA4mwA0PaUO IW56pnPa4ZrK+KgkGF/gJABWCv9tL04f3tg9ZUtLbcMKg3ecVXVPkn3uzAIM5s4jv07Z Vny3d6oSrB0xtGUwknVbuMY5gVK61X8YO6EMFddtk3wzY+GhS07f9q1teS1N9M7V5EHv Pyn65qz1PJd0JxOg2kHxsdM3ffsGUJs/pJH1br5Xd6Jaeci4tYTxdpdXl0VHPe5flF5F RMdUJ0CTitLvnvLhcnpKwz70fcXNQs3ZUKTZ9zUCCC8Ltp1XJ/diErqyIK+pXhI+TqCx Qehg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948217; x=1789553017; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=OZLYBDT0mJ+F0yYdyxDNynq3EZCGij4ebq9WOsT9s4c=; b=NO45AlAvNm8izexX/bhoU+P4Vm99eDAP5Dq5l7075ceKDqrR2kTZTLMhMPY/PC+tkk dh8WHKsN/s84SfBUFAtOVyJuko4K3jPgdHeJQZQRWz87tOQqvUZHsBPlvGtAngyZ6IlG 3i2qV3imnCLpSs4o0k9WevziQ/9XGgeWvtXZFLH3tv9ciSUDtyXa0JPA1CNSd28LTlwZ Yxfnd6j3FAlB0lwCSQM8j+R+wJEhGhuatYWQDo68/zMAZ3+J5iZrU0/UJnIyqD8z9lVD ut+jie976yO/2MHs4bfDoyzRavVZ1mwv2216hnX8qLdAZsauW6RpxPYSkjNHlLiXiCgn AAYQ== X-Forwarded-Encrypted: i=1; AKwUvBx5os0T4DavJO6Z6u2SaTu+27fZ+rXSghWKWQo757vAKtCwCLmwZEFcAIs5v7vp7RVDHKdESMkJ41/Y42U=@vger.kernel.org X-Gm-Message-State: AFuF++kyhNXtqQqjvab9UKKwvktqmFS/q2YGJeIA2h5W0P0zJqxBfDJE PdRW/DNeD5V1x2BnXHeYhW48zfxsuwR4kuxLHAdWVoPZlmkLWiJaMj3l3WN/QmzybA== X-Gm-Gg: AYBFou0MJP9WIQO3bs+liL3FzI+3DonjgrJWZ8NR9PaBaSTQXV7ZaBuwBpzCyvEn2XU 2+q3tcLf5lDwxX37qof0w9AWRsgg0SdsOn6Ue8Mi9zGtOjea4Pw63zBYcmVspo+hBGXiv5RHWoJ 2vs8SnfW6LGCvR0oDR1S4TaysoWIRD8Y7Z1FY8W2ZVpei6Zc85BSYsq+NlP7md8t1UJMfYutvjS VxkXCgDyUjcHLahkQlxSWusDGKSIJK1KRFGWIaMSJFaKSJONQHQF6+d0V6HhoJpo40UfF/Q9HK9 WbdROaM0gn80pgzz/8H4O1ZZfFSJJHABDZMOF4zI3AEd8Pcs7Bq0bBXXdxs2uX0S7KkL0oV3qVh lUPInviw9pxli/8nkGx5v1fFMWYzFx94VinuEbXng8Q1VeOzWHAjAeYMfZV/dCM6l7X3F6reDE/ 2sE8jq7OVXODTnK/Geb4yHo+Tgo2DJ225YWf0MpSEBw5M3yf/e+J3AROeGNPcccAxp4yZWdVHbj wYBu/6jsHC+RJ8KIT299qsjp6lg/qiOWQjahuZ52KEisrMQ3uPxY9wF4nmDXdmAl9+S1w== X-Received: by 2002:a05:690e:4384:b0:66f:8b35:dedd with SMTP id 956f58d0204a3-66fb58f2dd2mr7477525d50.6.1788948216887; Wed, 09 Sep 2026 03:03:36 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb48ee9easm11934983d50.8.2026.09.09.03.03.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:03:35 -0700 (PDT) Date: Wed, 9 Sep 2026 03:03:30 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 11/26] mm/fbatch: remove migration's PAGE_WAS_MLOCKED lru_add_drain() In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A welcome side-effect of mm/mlock.c's mod_mlock_count() succeeding on folios on the per-cpu lru_add fbatch, is that migrate_folio_move() no longer has to lru_add_drain() before remove_migration_ptes() restores a PAGE_WAS_MLOCKED mlock_count: so remove PAGE_WAS_MLOCKED altogether. (Re the "We would like to do something similar for the old page, when unsuccessful" comment above it: that may be easier now, but involve some rearrangement: not researched, so just leave the comment as is.) However: it was uncommon before, for isolate_migratepages_block() to meet a folio on LRU, with suitable refcount, marked mlocked but not yet unevictable: so that got missed when sysctl_compact_unevictable_allowed should not allow it. But now it's a more common case, so check if the folio will become unevictable, by using !folio_evictable() instead. That is more expensive than testing mlocked and unevictable bits: but it is consistent with the decision made once the fbatch gets drained, and is_inaccessible further down depends on the right is_unevictable. Optimize those checks better if they show a regression in practice. Signed-off-by: Hugh Dickins --- mm/compaction.c | 2 +- mm/migrate.c | 8 +------- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9e045a90ba21..1290b9170cb2 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1113,7 +1113,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, if (!folio_test_lru(folio)) goto isolate_fail_put; =20 - is_unevictable =3D folio_test_unevictable(folio); + is_unevictable =3D !folio_evictable(folio); =20 /* Compaction might skip unevictable pages but CMA takes them */ if (!(mode & ISOLATE_UNEVICTABLE) && is_unevictable) diff --git a/mm/migrate.c b/mm/migrate.c index 1939d6ed19c9..364015cbef33 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1147,8 +1147,7 @@ static int move_to_new_folio(struct folio *dst, struc= t folio *src, */ enum { FOLIO_WAS_MAPPED =3D BIT(0), - FOLIO_WAS_MLOCKED =3D BIT(1), - FOLIO_OLD_STATES =3D FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED, + FOLIO_OLD_STATES =3D FOLIO_WAS_MAPPED, }; =20 static void __migrate_folio_record(struct folio *dst, @@ -1258,8 +1257,6 @@ static int migrate_folio_unmap(new_folio_t get_new_fo= lio, folio_lock(src); } locked =3D true; - if (folio_test_mlocked(src)) - old_folio_state |=3D FOLIO_WAS_MLOCKED; =20 if (folio_test_writeback(src)) { /* @@ -1410,9 +1407,6 @@ static int migrate_folio_move(free_folio_t put_new_fo= lio, unsigned long private, * isolated from the unevictable LRU: but this case is the easiest. */ folio_add_lru(dst); - if (old_folio_state & FOLIO_WAS_MLOCKED) - lru_add_drain(); - if (old_folio_state & FOLIO_WAS_MAPPED) remove_migration_ptes(src, dst, 0); =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0202649F120 for ; Wed, 9 Sep 2026 10:05:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948325; cv=none; b=r5ix6BT2WQA84u0RgrPMHywRUB5yZxaHWp/PnLYBcKMD0pNhAsLO1FJ+3s1O/Q7stBpHio0SCFbVGAWqm6yteUXcdhrPeNO/vNHt/eM79tKRqKyMRdTcbkOfq6XtRZMgC9mVAoGrX2GqQjK3ukvL1LoTvatE2SmWNVbi8X7JnCA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948325; c=relaxed/simple; bh=qMr1bVSK6brdsAbJBgTYmdPhEJSJ4vqK0Yk38H2rNMM=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=QKUUW/X0O64nXjpGO145lflyL2Y+O8jaxZb9Ebs73/72k3wU42RUhe6moMPFWf49L2Z9SylDYovDVR771cgcaA/Xh0nuZ1YcSChIFqWJzgXyeCDh2EhgtC46+QJJYKoEJLIuL4R2ZoR0cu9kGF4YYvEiuUQpdF13i+sZubgrtPM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qJIdpKn1; arc=none smtp.client-ip=209.85.128.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qJIdpKn1" Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-8565d77c277so73379707b3.0 for ; Wed, 09 Sep 2026 03:05:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948323; x=1789553123; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=yFQHd4CD2vPDTEJCwRK4KnJHHa5RS0l8x4Ius5m53Xw=; b=qJIdpKn1TDNl8g9xLqgYhGyLTBFmZwLT3uxh4xLIzoB6uOXZ/kkb6TCBUi6sGuyltQ v+Z3Zv2H16MDHZkRD67zgpNvUxTMXYVsrHpgDheD6iGS6mJYgn6KYYTqakiEnpnBvgtd rFGNLbZJq3Z549hPDiaZ2IVFgxtbuSvWkrT7/EmPXP/oUODZbGq8dLOk+2HXp18a2Q/S cfxGp7GsdIQTy8bur5gKlM2+NtSyAdXuqQj2J1AUsqyml8eQWrmyQ7n18yLytFBGh0uP KCqqbNOELV+f/RDtWz2WBhaYkKkxWjWidVHWY6gN7tgXQoR8KJHAmTqQ1Be9uJqc88/j koww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948323; x=1789553123; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=yFQHd4CD2vPDTEJCwRK4KnJHHa5RS0l8x4Ius5m53Xw=; b=Jdhxo8jyGZYJQk9nIbK/2GixjJIDwD0IMs6tP2ExN1oXjhewbf+Czm3o35LD9Qp+C9 00oMkU40pzuPgjxVU69VtW4tmPzO8un1HGBPXG6Z241pLM3yPB7k8zKw8TzKthKJrwYM FU4j9LP4hpfnHM+YQmik5QIe3lOBnSCfIgNrqqVHxeDYiLFlBTUMXxzm553ZessrW0cD jVJhZcvJnJzOuDF+AjuSnRpYznTQnUd0zyfHOUY9UNUUfyFLqKOh3kHuD+yR/FeboJJc 4D6M4kEl2eENQRfaj+7R6daL5jxHVGveHiTXyLO8Q4BckqO5PBHAMwTb4/Paoef3JCbL wZJw== X-Forwarded-Encrypted: i=1; AKwUvBx6S3dMDZBn6yWRkOcegw50hiRr2Gb9Bxv7RIHxhOijFMrQDzOsUzt6SDDdhBY2XjmNfCEHZpY/KGqL/4k=@vger.kernel.org X-Gm-Message-State: AFuF++lLS6Vr7OPqzi6h37MQ2y7ZWgovKWtS6ukUQJxSMNd2hMB1fvCt 0LOjEGMhFEJYyQA9gTUfvBHuQfo7I1JE3GUwqIt5eyyhRmHtSHgODhSbUy9XnsbuZg== X-Gm-Gg: AYBFou1T/gTNsmv3w3WikRRvVDgps5pgTef4hftH6vpEwEkLEUzG5ZXOjpYdy0T/IdW GmINV2e279O816hUqhz/kzDCTGcOPtjkLUJloowXmrH2+m0P9dJxpe3ARZ8Bsl+WfT8cFiLnmsc SGEMTYT+7P7Yl9giPs0dJ2kdkzNbykHN4rNoTP9UNugSPNiHLzyxXgfU2xv7jYV2xq/TRBgUA5J JZJg6ZcaWuuIpoelyYfCsDO/wIcaha7p5xDlmNwfbIvuO/2QZP65l/1ieFHWC/Flts6q69N9vHn xUa1uR9o97uRSxq0aQHGVRc7i+fKu0oNlLDKO1SBpBzseJO9qV/uB13XHeCGXvmIH2Lw3CdxKw4 EDv8NFqV2/r9uFUIB8f/hFfOmZQtbIhtssde1Bjdny2E7OIrAOK0FXRCaV2L1CH5t2dyK14QLNy s918e+rAZzkoMYH8REMNGw9ifU4jzTISUQKEZvciNUnh8/bH2uk3xMGbktCjdGlApzVEgjLG77w kmqjGa5I5WqJfTYnbhDWye05I12HV9hVBvCqxJTEhJz/Fs9qTvyWInJ6MU= X-Received: by 2002:a05:690c:88:b0:864:6f4d:ef7f with SMTP id 00721157ae682-87121acbbc5mr139779397b3.7.1788948322156; Wed, 09 Sep 2026 03:05:22 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714befac64sm107718397b3.47.2026.09.09.03.05.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:05:21 -0700 (PDT) Date: Wed, 9 Sep 2026 03:05:16 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 12/26] mm/fbatch: remove percpu_pvec_drained and folios_put() In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the percpu_pvec_drained field from folio_batch, and its only use in __folio_batch_release(): remove that now pointless lru_add_drain(). Which leaves __folio_batch_release() as an exported name for folios_put() which is itself just a wrapper for folios_put_refs(): mm/mlock.c and mm/folio.c don't need such a wrapper, just say folios_put_refs(,NULL). Or should folios_put() be the export? But __folio_batch_release() is what drivers/gpu and net/sunrpc are using: don't change them in this series. Signed-off-by: Hugh Dickins --- include/linux/folio_batch.h | 2 -- include/linux/mm.h | 18 ------------------ mm/folio.c | 17 +++-------------- mm/mlock.c | 2 +- 4 files changed, 4 insertions(+), 35 deletions(-) diff --git a/include/linux/folio_batch.h b/include/linux/folio_batch.h index e1cc8ae023f1..a3337f70e109 100644 --- a/include/linux/folio_batch.h +++ b/include/linux/folio_batch.h @@ -27,7 +27,6 @@ struct folio; struct folio_batch { unsigned char nr; unsigned char i; - bool percpu_pvec_drained; struct folio *folios[FOLIO_BATCH_SIZE]; }; =20 @@ -41,7 +40,6 @@ static inline void folio_batch_init(struct folio_batch *f= batch) { fbatch->nr =3D 0; fbatch->i =3D 0; - fbatch->percpu_pvec_drained =3D false; } =20 static inline void folio_batch_reinit(struct folio_batch *fbatch) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd09c438fa23..942a9d9ed5c8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2201,24 +2201,6 @@ typedef union { =20 void release_pages(release_pages_arg, int nr); =20 -/** - * folios_put - Decrement the reference count on an array of folios. - * @folios: The folios. - * - * Like folio_put(), but for a batch of folios. This is more efficient - * than writing the loop yourself as it will optimise the locks which need - * to be taken if the folios are freed. The folios batch is returned - * empty and ready to be reused for another batch; there is no need to - * reinitialise it. - * - * Context: May be called in process or interrupt context, but not in NMI - * context. May be called while holding a spinlock. - */ -static inline void folios_put(struct folio_batch *folios) -{ - folios_put_refs(folios, NULL); -} - static inline void put_page(struct page *page) { struct folio *folio =3D page_folio(page); diff --git a/mm/folio.c b/mm/folio.c index 18e97923e527..5021639c5494 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -160,7 +160,7 @@ static void folio_batch_move_lru(struct folio_batch *fb= atch, move_fn_t move_fn) =20 if (lruvec) lruvec_unlock_irqrestore(lruvec, flags); - folios_put(fbatch); + folios_put_refs(fbatch, NULL); } =20 static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch, @@ -1043,22 +1043,11 @@ void release_pages(release_pages_arg arg, int nr) EXPORT_SYMBOL(release_pages); =20 /* - * The folios which we're about to release may be in the deferred lru-addi= tion - * queues. That would prevent them from really being freed right now. Th= at's - * OK from a correctness point of view but is inefficient - those folios m= ay be - * cache-warm and we want to give them back to the page allocator ASAP. - * - * So __folio_batch_release() will drain those queues here. - * folio_batch_move_lru() calls folios_put() directly to avoid - * mutual recursion. + * This used to optimize with a drain before putting: no longer helpful. */ void __folio_batch_release(struct folio_batch *fbatch) { - if (!fbatch->percpu_pvec_drained) { - lru_add_drain(); - fbatch->percpu_pvec_drained =3D true; - } - folios_put(fbatch); + folios_put_refs(fbatch, NULL); } EXPORT_SYMBOL(__folio_batch_release); =20 diff --git a/mm/mlock.c b/mm/mlock.c index 1050010bbe0b..97134eff6b56 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -191,7 +191,7 @@ static void mlock_folio_batch(struct folio_batch *fbatc= h) =20 if (lruvec) lruvec_unlock_irq(lruvec); - folios_put(fbatch); + folios_put_refs(fbatch, NULL); } =20 void mlock_drain_local(void) --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f41.google.com (mail-yx1-f41.google.com [74.125.224.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B51CE4B049B for ; Wed, 9 Sep 2026 10:08:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948515; cv=none; b=qz4j66OEuAXoR0wgeliS8wP+1DeY182f8gz5J8eYILSi0uxaBWCJAVo4YZYVahjIas22a6qSwGu+sl6kPZrQOQuknonmNgxCGCmPZ3/PuRcGV6xQemdKxUwEjMLZ/M0Orcd3YxmGlppFT18SJi562I1XMgOCmsPDPYOvayoH/R8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948515; c=relaxed/simple; bh=sZg+Qi8C9DDFoJlNdlSunXk4grOh7FnoNGg+LOx/5FY=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=B4RPshiQEIMtCjTF0dqLAyfNNVmPqjAFS4QqgeZzX33VhAPTL5+WWaBa0hpPOxp6lfuOWCbuhDFdyiEaJCCEl6eDgPObacGFgqs9EVr+vgniE4tQDasU8HawkpFl0s6Sjq6h4K8QSJ9+1c5M31QTM3cILycsKGKx/4Eih7AnK+c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XVIvxlSM; arc=none smtp.client-ip=74.125.224.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XVIvxlSM" Received: by mail-yx1-f41.google.com with SMTP id 956f58d0204a3-66fc2844f0eso5733256d50.1 for ; Wed, 09 Sep 2026 03:08:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948512; x=1789553312; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=XZP7Izy7OCFS/wAqshdE6rdYOBHqDiQ7cVSskEfKxxg=; b=XVIvxlSM1uhJcR7djRV0gNPFg06Td/5yyfzj7kt0zuSN/nq7YJAe0pznxOUwic5Z96 T5VO9mPvRCQlVw/dF65cWvu8wcZMwtxrzKgwXH4EweWIcis0ZV3/o5E0dfO0zimGo1Xk VvqOk1cK37CEjIvxOrgqIQDawXqHwwpXhjr8Wqg4Oy+b8pHlz20dQM/FCk6P3I3F8jH8 tqoIFc0l+BO9af50GeytgtVNGofxtSb+FDbiVz/A7fo9nPBS4Akx+0g7wVZ0RVd1t7Q9 UYZ69KeAjdGuX82EEqs57Qnbh/wb/WUts1rr2lGC+45bEbm/H6mjmNk2F0qyJhyvpJWu wIdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948512; x=1789553312; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XZP7Izy7OCFS/wAqshdE6rdYOBHqDiQ7cVSskEfKxxg=; b=QMAdKdSCm3OVJ8qMAq4fsmYbl4XpUE8JnhzSlNnRlITVyU09axN7hoJ1uOE6mRm/DO 8wJZxDz1DY2NqCsVPwSahggYls3VH6mZtC5Z6FEJ2J893IHg8Sxbg7qYcQepihOuRQOt vKQmnboyidHUnG+RVwRBzPVco8ueJ0GZrAWDBlMYq1zdU+kA/oJs3a6squNyzWCtmNoy pqkeh0vri+E9aO3LUNAUIzhHNvPg3HudzgUXnstXcllheANoMZc31+fQcsqyhbtC7oQw VS4UPiKhplWhtVcckac/XVANES9901rHaZNP9O0DX0QUsOKII8FC6qIYgfmHS4LfE9Z6 /8xg== X-Forwarded-Encrypted: i=1; AKwUvBwrtQIBp3kHwc2Sd++zj95XRbr5gViwAzqKbJD8QLERP4Lj6J5LhSm/YFljiemhRRUF3paUr8RSDdmVs6c=@vger.kernel.org X-Gm-Message-State: AFuF++nufISI3xzGylyEwtxPDpiJkdv4Ht4Ll6DBxeMKlrtH54D41Zvt 1FaImdxSXUh3gl7CcCo+narTPseymnEh24zC3V5MCKNrkmB5Ltq4La4Y959Ll0n1qA== X-Gm-Gg: AYBFou0XVXlenAniz4CMnQ05stzzAgpyz8UM91N1RkPKNneDFRNEWmGxPVf9Gx9ubDL BZfx+5lzZBBGeQ0aBvgWVR2rjNQqPnwLOC7lsDUWMJCSk2lON6HXQwI+c0iP6gudfWQ1xGipTkp 0gOnoBjQjPAfK6OkwFkUwWiSNaW4wE00KnZSu685tZ/DX7KDm8xDu4z8bj2QaBPU9ANYkVoJz7P ZZa1KR8ZIfwk05bM9YYDVz6BmmUWScMJI/ruMuV2iGGtsK5hZ07ltPLdJejBdfV/pqh88ZD5bEU viJuZQ+Wp8UBbJHlwWQzqv03LUfedBurloGYj6AgnTZylkS+PgB9AVDr6ydtFGQkKIEePgqlwWl qjg27Zb9ERnJUsahMjxjUFDhlBzbo30cppBV/vimQnRsSZ/DhWXU5wttZ7RMzGcclreSPZRjeX0 yrX1hQDviP/iiOSpTUIq8EjFn+YKCWP6Osuv85w+C2ktFlTYfsrhQfsCZsozG80O6O+fnwMaIap OKAYuGIbJtwyhcYAxkPs/n+Zm+gGN7CfOqLElgY41/aBlf5OUkVK8ER9kg= X-Received: by 2002:a05:690e:12c8:b0:66f:bdef:7d95 with SMTP id 956f58d0204a3-66fbdef8a7bmr8856733d50.20.1788948510918; Wed, 09 Sep 2026 03:08:30 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb47a8b8fsm12160676d50.2.2026.09.09.03.08.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:08:29 -0700 (PDT) Date: Wed, 9 Sep 2026 03:08:23 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 13/26] mm/fbatch: no lru_add_drain to collect_longterm_unpinnable_folios() In-Reply-To: Message-ID: <716d267e-11ea-f400-2cfe-d8775e6ac027@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" collect_longterm_unpinnable_folios() has no use for lru_add_drain() nor lru_add_drain_all(), now that the per-cpu fbatch references are gone. Replace the recently added lru_cache_drain_for_folio() by a temporary stub, so any callers outside mm may build independently of this series. Signed-off-by: Hugh Dickins --- include/linux/swap.h | 9 +++++---- mm/folio.c | 48 -------------------------------------------- mm/gup.c | 9 --------- 3 files changed, 5 insertions(+), 61 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index a6fad5127118..0052a6890435 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -304,11 +304,12 @@ void lru_add_drain_all(void); =20 enum lru_cache_drained { LRU_CACHE_NOT_DRAINED, - LRU_CACHE_DRAINED, - LRU_CACHE_DRAINED_ALL, }; -void lru_cache_drain_for_folio(const struct folio *folio, - unsigned int extra_refs, enum lru_cache_drained *drained); +static inline void lru_cache_drain_for_folio(const struct folio *folio, + unsigned int extra_refs, enum lru_cache_drained *drained) +{ + /* Temporary stub for callers' build independent of mm/fbatch series */ +} =20 /* linux/mm/folio-compat.c */ void mark_page_accessed(struct page *page); diff --git a/mm/folio.c b/mm/folio.c index 5021639c5494..d66aa2469e86 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -33,7 +33,6 @@ #include #include #include -#include =20 #include "internal.h" #include "page_alloc.h" @@ -843,53 +842,6 @@ void lru_add_drain_all(void) } #endif /* CONFIG_SMP */ =20 -/** - * lru_cache_drain_for_folio() - drain LRU caches if the caches might hold - * folio references - * @folio: The folio. - * @extra_refs: Extra folio references held by the caller. - * @drained: Drain status for batch folio processing. - * - * Drain LRU caches if the caches might hold folio references. Start - * with a local LRU cache drain, to then drain LRU caches on all CPUs if - * local draining was insufficient. - * - * This function detects LRU cache references by comparing the folio refco= unt - * with the sum of the expected folio refcount + extra references held by = the - * caller. Note that we cannot rely on PG_lru to reliably detect all LRU - * cache references, and there are rare scenarios (concurrent folio (un)ma= pping) - * where this function might miss detecting LRU cache references. - * - * If @drained is not NULL, the function will avoid re-draining LRU caches - * when processing multiple folios in a row. In that case, the variable - * @drained points at must be initialized to LRU_CACHE_NOT_DRAINED before - * the first invocation by the caller. - */ -void lru_cache_drain_for_folio(const struct folio *folio, - unsigned int extra_refs, enum lru_cache_drained *drained) -{ - if (!folio_may_be_lru_cached(folio)) - return; - - if (!drained || *drained =3D=3D LRU_CACHE_NOT_DRAINED) { - if (folio_ref_count(folio) =3D=3D - folio_expected_ref_count(folio) + extra_refs) - return; - lru_add_drain(); - if (drained) - *drained =3D LRU_CACHE_DRAINED; - } - if (!drained || *drained =3D=3D LRU_CACHE_DRAINED) { - if (folio_ref_count(folio) =3D=3D - folio_expected_ref_count(folio) + extra_refs) - return; - lru_add_drain_all(); - if (drained) - *drained =3D LRU_CACHE_DRAINED_ALL; - } -} -EXPORT_SYMBOL_FOR_KVM(lru_cache_drain_for_folio); - atomic_t lru_disable_count =3D ATOMIC_INIT(0); =20 /* diff --git a/mm/gup.c b/mm/gup.c index eb898ea1ee22..fa17f99cbb60 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2266,14 +2266,12 @@ static unsigned long collect_longterm_unpinnable_fo= lios( struct list_head *movable_folio_list, struct pages_or_folios *pofs) { - enum lru_cache_drained drained =3D LRU_CACHE_NOT_DRAINED; unsigned long collected =3D 0; struct folio *folio; long i =3D 0; =20 for (folio =3D pofs_get_folio(pofs, i); folio; folio =3D pofs_next_folio(folio, pofs, &i)) { - const int pin_refs =3D folio_has_pincount(folio) ? 1 : GUP_PIN_COUNTING_= BIAS; =20 if (folio_is_longterm_pinnable(folio)) continue; @@ -2288,13 +2286,6 @@ static unsigned long collect_longterm_unpinnable_fol= ios( continue; } =20 - /* - * We drain not only to make the folio_isolate_lru() succeed, - * but also to remove any other folio references from LRU - * caches. - */ - lru_cache_drain_for_folio(folio, pin_refs, &drained); - if (!folio_isolate_lru(folio)) continue; =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f49.google.com (mail-yx1-f49.google.com [74.125.224.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 179044B2CA6 for ; Wed, 9 Sep 2026 10:10:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948628; cv=none; b=tIg9jLR9axpXYIbjNb6qO2dG5xaOjkprzf8TENxwrJNnxXfDvFstOEtOGdqyc2F41gtjbt5j9o/YJ/Pd3dwpLywRiNe9DWa+nhY7K56Un5G1YhUIfcWsSOlADIMFwd7w4a5vOIeT2SLn7QzQzLQlQNlTWps6hDcRNAzkMcmPIVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948628; c=relaxed/simple; bh=1gndqcwixIf69ERl83wZ/ahlq6SVahr40TbASWBQcgI=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=YhuT+A42l+U+pzQ9Agycxj+T9KLYoiiIBWk0VEtyzCvCoYvAR5qISp5dZ2bH7lXj4D6YaQMjZq504VaS7++bXVT41UMyFO6jaG2KjDPiBYFj5yPBXhP+I5D2GppXkQHqQiHevnjI52GeymSSPLahYlZXhjlvTyXBW2ByemMomVk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Bgff6UGd; arc=none smtp.client-ip=74.125.224.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Bgff6UGd" Received: by mail-yx1-f49.google.com with SMTP id 956f58d0204a3-66fb8a78ad2so6947908d50.2 for ; Wed, 09 Sep 2026 03:10:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948625; x=1789553425; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=lsHaeoqxnMhy2GB3rUAu2bzr8hj8izpjrzp8bJJzBnk=; b=Bgff6UGd6F/BxFyng4Kl/LyCtan7qCWYfurAm16dHjBKg/yW/+EtIiHOldFnUPsRy9 LaK3d/O6kMKriCN/VVi/dvv+tdmDyzQJdGzAk+Jue4tQkTS3ABctBnM8dev21qLBxlev UliqiGTMuPygDF7R613arm5jJYiO4tVpEoUWk2V1CSrw94a8RFIqnhttpr4AflIQbk/Z NoGUgHHTEoDXU4DAU7Y81b6VuqZnehJY5C1AIOlh3KQseylO9XxUIpZvO1RpgvWw2TEV qiIrxuhxLDHpL5q+jGjTFzLrMdInp6PfYEz2zvvhYvsDoZ+vtg3xKiP+amY+WOI3s0BC pwrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948625; x=1789553425; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lsHaeoqxnMhy2GB3rUAu2bzr8hj8izpjrzp8bJJzBnk=; b=dAf9+6iCMQSu+GYWK/K5eUi/t/GkTsfrwGO05COKulQeZNSKFnqq6c+bPGpuMm+HMF 0As7OepEy/8wRTmPpVd3lI4hJ1o5EDZ4/eDNLnEYw3eROp37dz6E8vK0wsP+a4122S8h o3U1VfL+2x6Y5Yy7LZyY5+NExcyuwnUmxpRfe+ziUP+8oyOxaqaqZocwhicV8Evfk0+6 4gqpYSnCtT6GJ/cssqGtL+mTmOh/aTbToGJK4gHRrj6Q5X/hbJ6vKheIelqQEQjg8SZ2 GK/eeur2qqliHSfMDNOMOMzvDBSLiReCFX9v0PCWrVmKHw28h11NceaXJ5CkLwuKG7wF YEfA== X-Forwarded-Encrypted: i=1; AKwUvBzr+bt3eIi3vmEucl2bfYJ9Qt6YJkMaHggQEuJuypcL/Yjeij1krHeCRqrC5AsZctYxSHiu9qVV/sDXLVE=@vger.kernel.org X-Gm-Message-State: AFuF++nsjE5MwH7/DthmOMYPO0qRODWzl4XxxIx5ecx5n3Bf9JC25Yut JveOHr5QehiQHJIgDLk+ciYPl84C9MeCGfS2uxFiypbUTq9uondGgMrnAejVuG/CQQ== X-Gm-Gg: AYBFou1sSU1+yvJrbY6p+LVxNzYxXYSmUJjbyvc1ZJ3FrnZPxZHg5aSM4mjzvxKdJHI YKM0EWV1An93hodT57rrREWyEOQ9KXmCkPtm7mo8TGNoWrS6ACjyGmdvl05Gtv+5yHPK5AvkIkj OTwyyFD3cke8n5Np+aAwgAllIKkiizkA87U8fU4ZvsBxnOuf+sGVTWPAjB8K/6B/nlLUihefAfL mx7ZBExlcosEUbpFzJ/srOfnF74IPVyIcYlGihIE9o9VStfEMsWWjnfHh97JhFasR1nofa1eLr8 v4OaYGbedyUs9VymSwk6NcZToQQj3xUiCIl+s+BXDATpFF15BEWtOP4kEq6RlHVjXpfIHe5Olwz ZZND01a6OIOUTYXvl96NrGMWpE3K4QCrWBnYL+4Nj4nY6DvXAU4oTRj9qDlTotYPwOdhP4AscMH 7nOe/uQ/BeZ26Iy7LqVRAXMvJQg5Vg9jshBNYzc/+OPI0pkPIrLzFT2aYRWLi34N49zGiKlgrS4 hFf5yhQyQ+dymJTSsW0jqKDYbhiL1SBo498Be7Gw7IKoJg1LvTYPWZTt68= X-Received: by 2002:a53:ad49:0:b0:66f:c1bc:4095 with SMTP id 956f58d0204a3-66fc1bc4852mr7414444d50.88.1788948624105; Wed, 09 Sep 2026 03:10:24 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb48b374bsm12096623d50.3.2026.09.09.03.10.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:10:22 -0700 (PDT) Date: Wed, 9 Sep 2026 03:10:17 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 14/26] mm/fbatch: no lru_add_drain() nor _all() for memfd_wait_for_pins() In-Reply-To: Message-ID: <5b393933-b097-9d70-be09-2d00c8d62f6b@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memfd_tag_pins() used lru_add_drain(), memfd_wait_for_pins() then used lru_add_drain_all(): remove those calls, no longer useful now that the per-cpu fbatch references are gone. Signed-off-by: Hugh Dickins --- mm/memfd.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/mm/memfd.c b/mm/memfd.c index c708d92533f4..064f24dcddaa 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -40,8 +40,6 @@ static void memfd_tag_pins(struct xa_state *xas) struct folio *folio; int latency =3D 0; =20 - lru_add_drain(); - xas_lock_irq(xas); xas_for_each(xas, folio, ULONG_MAX) { if (!xa_is_value(folio) && memfd_folio_has_extra_refs(folio)) @@ -168,9 +166,7 @@ static int memfd_wait_for_pins(struct address_space *ma= pping) if (!xas_marked(&xas, MEMFD_TAG_PINNED)) break; =20 - if (!scan) - lru_add_drain_all(); - else if (schedule_timeout_killable((HZ << scan) / 200)) + if (scan && schedule_timeout_killable((HZ << scan) / 200)) scan =3D LAST_SCAN; =20 xas_set(&xas, 0); --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E03C14B7158 for ; Wed, 9 Sep 2026 10:12:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948766; cv=none; b=gTxZTmhnoY5/Fl4E7rrKXt6l3q9NeXs3uYafigNviju2wWgnMDjTHLOmnph2tnbROl3d1+sY54if7qob3a5+qSmuigkI+cgtZXeBzeF+eX5otdrOwTTe8zoFg4MFTh/fcGPDT7+jvs3xujseSIXIb0nGfMHFFyFx218H4APC/qM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948766; c=relaxed/simple; bh=veK2eDLRVGTOGqy7tLB4YUJ+s0qY+28Zl0Q1bFSRbiU=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=hwiLy1s1XVkv6VLQyJhBX25Q9aveBHQ7qfRzx8eS3rvFpK8TBC1NIX0x5SZpkPdgZl0P/PYAf7BxrjFlysuwx3Yz46kE5JopdDBb3V8e7GYcYoHcmVOzA66LmIAVGZ6sHtM0ehLgPJEtwKBdhCk8Ak7iayC5/9zjGeASR/wN7aM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mO/6uVOR; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mO/6uVOR" Received: by mail-yx2-f12.google.com with SMTP id 956f58d0204a3-66e4aa8e6bdso1184847d50.1 for ; Wed, 09 Sep 2026 03:12:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948763; x=1789553563; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=bX03VdslkeGAuZnZEXnLni54ZZj7BtxRxhR+YMJ/+OA=; b=mO/6uVOR1LytX7R1AEc6xw8mKUHiLBuvD5LchX2wHlRXdFmIoI4MoQqBIbTy3bwHsG e3bYpTA+zqhbhomqadUhQdNsmOwpbaTEq+uYRATB7A1Qr6wARQfIEuPVUv6ViDBN6vnH R+2rLRJ1GxMHIGCVzzJ4eC4qT1F7Yz8mL9CpvJWRwI6BrI1iQoYbS7pe4dFCIWjh9mK4 bvqg8H9QnJYjNhSjxmYLHm7zzbaRmYyW18RyqmYlq2jBDh/04Jnqa8mPYKpC8b6EuzLL poiC5S0mXDQQbJ/bhIDYdrY0kd77QsDbdZOR47LIIAwn6pjezMNckQtuUDwwI9ZJi4Aj IhUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948763; x=1789553563; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bX03VdslkeGAuZnZEXnLni54ZZj7BtxRxhR+YMJ/+OA=; b=MP6Kn85jbIuWp5eQw4ChB2Vtj9wLOR7/SdBkcZDq6VW4p6NWxWjEe4iyUGGmxVOYO4 fqdWbeUYCQBfDmfGVjGzyYabTWSvSsXndv25hMm5xHP5xX6jpvzdOfT6/imKU/ex5Ede awmUH4MdGyqyHV3ZfsDdQ6SgwfLxnRQ+YH63ZXolzwm7da15OeQUAdZXGou7s5Iyc877 lmXnf6+7J4SZmKK0TM0nFUMDiCa2dJOmhafDI2VwVWtvfKZHC2mNESii1KsKdUFJfmEr MU6I62cmeiUSq5Qnh7Ccw/2Ilhgm8JhCeCXAysv0FUUF8a0iIlNIag8ncXXpP0icZG4C h8YQ== X-Forwarded-Encrypted: i=1; AKwUvByi+8nY+98ko2voAQHgwizwY1X+Hmb3m238wlJpY3BJRZiBFNhFN7zWCXOpmDrKXlDsU/X3/rtGkkI3xLE=@vger.kernel.org X-Gm-Message-State: AFuF++mh6yc8AwWrvezpT7iohHSsMKMZtwiaUOA/BlO9k65CY0/mLYKU tqmzarj1xAQ89bBt9R6f/zhdD2H0cKEbUPhN/v6AwIR7DPRQJSqK6d4JGIkhA6Gf2w== X-Gm-Gg: AYBFou2DIaDehW02DNXXqbpct71R6wzDRrC/HIw50LZvlRjAu0YFxboTKt6X3oVm61A DuR19bLWl0dArTBCvvJ/EXsl29BYewU3mknKj8e+LievF+Q9lJ+LuKD93sVbFtVHx8Kb3M8Ml6X Zomw/sjG0e8MUl416pMHo79rbtUYr+3Eu9Zlz9lJGCqmynBT19WIGtPea/2P5N+PjbbcadPOj6D SB+Cm7Jq9sbpvdqoF4XRMfFEe9HOePlik5yBzQAxF4sI+5c5wdMWofyFaL84ywK/4ZRItCq4p2q yJjR0q+LOMhJxqPJKcExDEfRnd7jvTOtYeq0DsAwZikoRCPud7N0vdOSgn6vAl2IqYPRTHHZOZd HTrP3KmI6kg/o6cWCBOQ91KrTMBX1pZUwOCvj3MaXIO4Styb6t0GgbHqSiNnnBXq9a4SBIxHbx5 6R9coUlsz1oLCGE3DoCiIsrRbEkIeLAheUzbz1Sd1CrOLPc9X6gRMwu11PzkKZy2GWG5uzo37YL ojdTARP/Tf6SpC7q2Nc9DU2mhSp1QiyvEAOvpj4Qw7HPpND4neU/OBCdxI= X-Received: by 2002:a05:690e:1243:b0:66f:d290:6336 with SMTP id 956f58d0204a3-670f4914bc5mr2669673d50.10.1788948762060; Wed, 09 Sep 2026 03:12:42 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-670febedc44sm2574235d50.11.2026.09.09.03.12.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:12:41 -0700 (PDT) Date: Wed, 9 Sep 2026 03:12:36 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 15/26] mm/fbatch: remove shake_folio() shake_page() from memory-failure In-Reply-To: Message-ID: <29067356-073c-2c36-a78b-7d7d4920aae3@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" shake_folio()'s lru_add_drain_all() no longer serves a purpose, now that the per-cpu fbatch references are gone. Are the retries in get_any_page() then still useful? Not obvious, so keep them. Signed-off-by: Hugh Dickins Acked-by: Miaohe Lin --- mm/hwpoison-inject.c | 1 - mm/internal.h | 1 - mm/memory-failure.c | 38 +------------------------------------- 3 files changed, 1 insertion(+), 39 deletions(-) diff --git a/mm/hwpoison-inject.c b/mm/hwpoison-inject.c index a11222572f97..9eab4b7d25b2 100644 --- a/mm/hwpoison-inject.c +++ b/mm/hwpoison-inject.c @@ -118,7 +118,6 @@ static int hwpoison_inject(void *data, u64 val) if (!hwpoison_filter_enable) goto inject; =20 - shake_folio(folio); /* * This implies unable to support non-LRU pages except free page. */ diff --git a/mm/internal.h b/mm/internal.h index 40bd128a58ae..6700fff13675 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1171,7 +1171,6 @@ static inline bool node_reclaim_enabled(void) */ #ifdef CONFIG_MEMORY_FAILURE int unmap_poisoned_folio(struct folio *folio, unsigned long pfn, bool must= _kill); -void shake_folio(struct folio *folio); typedef int hwpoison_filter_func_t(struct page *p); void hwpoison_filter_register(hwpoison_filter_func_t *filter); void hwpoison_filter_unregister(void); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index a8b03e2920ba..d0abbe13447e 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -310,30 +310,6 @@ static int kill_proc(struct to_kill *tk, unsigned long= pfn, int flags) return ret; } =20 -/* - * Unknown page type encountered. Try to check whether it can turn PageLRU= by - * lru_add_drain_all. - */ -void shake_folio(struct folio *folio) -{ - if (folio_test_hugetlb(folio)) - return; - /* - * TODO: Could shrink slab caches here if a lightweight range-based - * shrinker will be available. - */ - if (folio_test_slab(folio)) - return; - - lru_add_drain_all(); -} -EXPORT_SYMBOL_GPL(shake_folio); - -static void shake_page(struct page *page) -{ - shake_folio(page_folio(page)); -} - static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma, unsigned long address) { @@ -1459,10 +1435,8 @@ static int get_any_page(struct page *p, unsigned lon= g flags) * We raced with (possibly temporary) unhandlable * page, retry. */ - if (pass++ < GET_PAGE_MAX_RETRY_NUM) { - shake_page(p); + if (pass++ < GET_PAGE_MAX_RETRY_NUM) goto try_again; - } ret =3D -EIO; goto out; } @@ -1477,7 +1451,6 @@ static int get_any_page(struct page *p, unsigned long= flags) */ if (pass++ < GET_PAGE_MAX_RETRY_NUM) { put_page(p); - shake_page(p); count_increased =3D false; goto try_again; } @@ -1627,7 +1600,6 @@ static bool hwpoison_user_mappings(struct folio *foli= o, struct page *p, LIST_HEAD(tokill); bool unmap_success; bool forcekill; - bool mlocked =3D folio_test_mlocked(folio); =20 /* * Here we are interested only in user-mapped pages, so skip any @@ -1658,13 +1630,6 @@ static bool hwpoison_user_mappings(struct folio *fol= io, struct page *p, pr_err("%#lx: failed to unmap page (folio mapcount=3D%d)\n", pfn, folio_mapcount(folio)); =20 - /* - * try_to_unmap() might put mlocked page in lru cache, so call - * shake_page() again to ensure that it's flushed. - */ - if (mlocked) - shake_folio(folio); - /* * Now that the dirty bit has been propagated to the * struct page and all unmaps done we can decide if @@ -2554,7 +2519,6 @@ int memory_failure(unsigned long pfn, int flags) * The check (unnecessarily) ignores LRU pages being isolated and * walked by the page reclaim code, however that's not a big loss. */ - shake_folio(folio); =20 folio_lock(folio); =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f53.google.com (mail-yx1-f53.google.com [74.125.224.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C1FF4CB8C0 for ; Wed, 9 Sep 2026 10:14:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948868; cv=none; b=Jd1kZYFp4D23UztWg5Z8SPbyXz+dMdiVeBWe6DvKHovDC9yCoKlS6sSRTUi1o1xevHkU6FfO+MsI8APp/8d3kbg075JMkJtIUw2pLogGjwuNhIGWWGrpst6ChIxgMvYi5XlCUj+IehpY4/wsANE80kMFqoV2tJf14n/qHcSfblk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948868; c=relaxed/simple; bh=iYBl1TBcq8XbAN5lWXQ5xeG2cPfd6X78ZvXFz11jkXE=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=K2zBdioKAQJB2SfmIf13mEnHKj9fsb6205cJ7MfhqPN18c5X83cLte4f4/PVdQXxvDPT2P8sOrmZio2w8YThHPB7kId3KkDXerkhCpQdL9yco4wtcrYmd1UkOCpOY+zNl9EG1yRcf2zxieSSabFttABzCV9ilgyOAmybTwIbf/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PRYPqwdZ; arc=none smtp.client-ip=74.125.224.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PRYPqwdZ" Received: by mail-yx1-f53.google.com with SMTP id 956f58d0204a3-66c70f69d3fso5341226d50.1 for ; Wed, 09 Sep 2026 03:14:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948858; x=1789553658; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=cx1TfB9GnW58nOxDn37ATPgkJR6erGAjBm2BqNqkY28=; b=PRYPqwdZUo/IB9RDXendGZtCja/iYJ1a6PM7GvuYIFcVs73+Aq2REekIyiRl7fjjAk LRp04gPqNFcbyURy/5NcHJapf/ediVuCV9FJ1gJom6LcsEPMy9CDFhkqZ8YDn8ZOT1Ha x4O5p+TtNQEe1tpXqurl4XGFaXkVfFZhpr/nW/5B/zHmkgqGY/bx0ycLpnRgdmH7gU0q KPRfakb0o+JxLzBDZeilT8v9QkXEYKrAImhMZJKQUmLO1sKS50VNsZuJAQNQ3jAtlJZF /hj5X3ycMERjXWMG5IKdk6DNw5Vm752upXAONuac2+tmtxbZxrS+V6RMe0RLe8rdCk66 7aSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948858; x=1789553658; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=cx1TfB9GnW58nOxDn37ATPgkJR6erGAjBm2BqNqkY28=; b=N4NFlWL/kABJOWARl+00FzTy3z47x31awTyf+00nxLpuIq4vQDgzHuvDtBW6mMo/+L PjEglD3wK0kryW1X7EarPY7wIAkAmymfl/vMB61LFU3dSWhZbsLNoaNKWY/XmnQp4yht sgOop/JsAGWJ5r7dfRc82V8Zg4H4WmAxVHHwgFX57488Z7HjnnO1gzUpfGThCrgonUtY fMo9yPdU0oeTCxmYurjADKVigM1FY6chF3jrshJyr6WTuUVZ+OzkiKFlHP+od6jkP5+T L4iYwj2EndEY/VY5pXW45HTRIbHu1qGIshiIfRbifx9dPVva97ph3jbVt4vFP3CrIxfp m58A== X-Forwarded-Encrypted: i=1; AKwUvBz9v5UA1xlCcgjEmgEiP2xRK256v6Z6+nozzcUQanxiCRoUmKt4dywgj0W9W1KPjV0YL+F0NXGRpNNo6F4=@vger.kernel.org X-Gm-Message-State: AFuF++lu58HuOapUqK5tgQ4tdj3nVX8Q4tRJB9ndAuk8l0UJScbETDyS 4N7Nh0tBKQoOqOGSRLvWTKQHlGDr4sD78BXrUoWCTp715ajT/qXRbGIKIcIZ3ShQEg== X-Gm-Gg: AYBFou0kR0jDxl2z3kVravRHdjYOvre3Gyv4dqu4eMdUnIMULKW9TK8SSyZ7ZZgFQKY 8K1zOscEVIMYt277sWC4qq2FTMzk9gzuaohnmZ8NGjTu16H+kDodOBFaD68gue4zrZPAXGAE+gk hsGPIbArzEoo2CZTAuaAalsqftnzH7SPxGPPcJfUG0IgraS1+Zb/4ztgzdI2BAKn00EFpzMX3f5 4ZmYG1I8IDiZqt8w2Lm22eUmZDv9GAFhuhEPuL6cK0MkmPfXMaec+ZoTPnBk52SWbsUx4baX/ks vZdhUfg36+xXDPhfZA7LzC03F7z3iKOGSWvouoFX76WQGsj5G+gNFc/uJxc3tsK9tRzxUwnNPGr Y4v0ALcTsjmMdgGHpC/ylO/92xYpvaSxiVs9/n+NJlFvwSaGgO2p0U9U++wVe8+dG1w9BrBk9em bYC1OzQ9ImvZgXz3oO8gIUbIydBXwAFvJH85LBNYtfOYV2Axzy2jqG7nsIgWTp0D0C6IWTRYfwI YhnhncUs83y1KJ3BQNN//tLrLHZksEG5rgLcq0NxVA5J37FN1LHLsIiV/Q= X-Received: by 2002:a05:690e:4513:10b0:670:f6e8:1f8c with SMTP id 956f58d0204a3-670f6e87f07mr2998845d50.84.1788948857514; Wed, 09 Sep 2026 03:14:17 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb47a8894sm12066216d50.1.2026.09.09.03.14.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:14:16 -0700 (PDT) Date: Wed, 9 Sep 2026 03:14:11 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 16/26] mm/fbatch: remove lru_cache_disable() from NUMA folio migration In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove lru_cache_disable() from mbind(MOVE), migrate_pages and move_pages syscall handling. They do not now benefit from lru_add_drain_all() first; they gain little benefit from invalidating buffer head LRUs first, since 5.0 commit 80409c65e2c6 ("mm: migrate: make buffer_migrate_page_norefs() actually succeed"); and it's a shame that (even without CAP_SYS_NICE) they can prevent concurrent tasks from enjoying the use of those caches. Signed-off-by: Hugh Dickins --- mm/mempolicy.c | 7 ------- mm/migrate.c | 3 --- 2 files changed, 10 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 79053ece02cd..f4d9b5580632 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1343,8 +1343,6 @@ int do_migrate_pages(struct mm_struct *mm, const node= mask_t *from, long err =3D 0; nodemask_t tmp; =20 - lru_cache_disable(); - /* * Find a 'source' bit set in 'tmp' whose corresponding 'dest' * bit in 'to' is not also set in 'tmp'. Clear the found 'source' @@ -1425,7 +1423,6 @@ int do_migrate_pages(struct mm_struct *mm, const node= mask_t *from, break; } =20 - lru_cache_enable(); if (err < 0) return err; return (nr_failed < INT_MAX) ? nr_failed : INT_MAX; @@ -1530,8 +1527,6 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, if (!new) flags |=3D MPOL_MF_DISCONTIG_OK; =20 - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) - lru_cache_disable(); { NODEMASK_SCRATCH(scratch); if (scratch) { @@ -1626,8 +1621,6 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, putback_movable_pages(&pagelist); mpol_out: mpol_put(new); - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) - lru_cache_enable(); return err; } =20 diff --git a/mm/migrate.c b/mm/migrate.c index 364015cbef33..1d2085529cd0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2359,8 +2359,6 @@ static int do_pages_move(struct mm_struct *mm, nodema= sk_t task_nodes, int start, i; int err =3D 0, err1; =20 - lru_cache_disable(); - for (i =3D start =3D 0; i < nr_pages; i++) { const void __user *p; int node; @@ -2439,7 +2437,6 @@ static int do_pages_move(struct mm_struct *mm, nodema= sk_t task_nodes, if (err >=3D 0) err =3D err1; out: - lru_cache_enable(); return err; } =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B876244C50E for ; Wed, 9 Sep 2026 10:16:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948999; cv=none; b=DXasutAbrYKoHwC0DfdXw8NT8m0I7KOhZ9PbJyADlHZGONZIm9Kpo21H2Fh4WOdquI77Jp4RU9YH0qNH5GLyZnInYWFh+dNk4B6LW4fnhQR1Aih97lcN22V7nZAieMLteuq0uHpxO7dEoYyR0sCYOcI89O6o3ChjwuJ1KhHej0k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788948999; c=relaxed/simple; bh=KEiTze2su0A1vp1BBKEvxCvYQVjhfCLRRoQ/gxsZ1F8=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=GFybT38RcGCvVlL4vvEtAKwjg7qUjODfbflVYCjEfQtvvkZ1qjxsUxW9w9mIqQWgk7za1DY+chbmK0tO/PW8RB3BFcvCjce4jY2IMpxUEPJOg6VDZ+gx+s4NxnOBac3zlC7GSQSBUi+NE7Eh4X2qh+j1fBAjMxv6hRYy+FDVn1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hwyrAzI+; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hwyrAzI+" Received: by mail-yx2-f12.google.com with SMTP id 00721157ae682-85d46e4cdcaso8046277b3.3 for ; Wed, 09 Sep 2026 03:16:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788948982; x=1789553782; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Ovd1KkFuD5rQg51sP3+p0bllsBxKk8b6qXB17QYO0U0=; b=hwyrAzI+wvUIgVjo6Lx52WtUrdUvT5rei03O1Ho/0BoeeXp8scCzjMRS+8dh4HtB/M qEcFwK+SzHFY66p/gIZuPAyijePIhadcM49qboerkYuSo+xJVo7vGaulp5qeP9dw8a9g ROJX/n97KeyHT7Rz4/TnCtz7kazdE2QSaIHxgjr9PTtBIQ4mB0j5GPM4bRgLFKRPu0Bb u8b9Ovnho6DBsEHFWNRDE/4FVSP4JoxayygM9QL3IyV/yP1JNNINe15Yl1FxspCm75bJ ++vDAjdbxpjWtLcYFicLrpBC0SVgMpADfE4NAtuJyDRwa/Ax0L/qdSz8dg/a6gzPwAet YJOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788948982; x=1789553782; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Ovd1KkFuD5rQg51sP3+p0bllsBxKk8b6qXB17QYO0U0=; b=sW3xQ1uKydW5GkYFtqSCQYVfYOa+oQAK/E20+4KLPBDtUTlh4VUtkNayP9wq9+lFO2 j5CgEdkIkL/3YMSCt/1A3OpHfGcHicyz+LbRwt1aem2RWVfJyASP+sOMQU81GKasio6E NuLJQyIAbOIuFDJYL0V2eRSkgveOg4CieZrmfagJeM9gQphDBXLqfmqcgQqXUBiLM58t dM3pMtE1x9A/JVIUGqfaJAFy+7zekQsBwIFSgKQtdAIKMmm4diN6f3KDRwkT32/aJm4k 2lTfeWetlvcUu81SzaqAUlq30jbUWq7cEI0KykSOxbTfdnA82GUgYbyfjMxGyIftBWL5 Bz2A== X-Forwarded-Encrypted: i=1; AKwUvBwKdukLjeSdBGELJ7uuqyBP/PbnAnGxAYUNtnsf5mhCpRXcLW5LSf9h1WSynW+lGOe9TkVHcc0j0LTZ6NE=@vger.kernel.org X-Gm-Message-State: AFuF++n+HtrKtv1yv4YAdeXmQhbM5alFfq9ZcGSzKQUyjCDOQtrHoWbQ jRCURRe8KfoN+KjJDRiexmy6wyoeLN/oTIvlBMfKUl7ic8CfjxfR34qwqGcY1qPcXA== X-Gm-Gg: AYBFou2UaOU4rL+/h5GiD+jiE5czVJPNIZ4uF5NbVDysyeZeGMkyjFMgSZJ/XpHOS/J QN3pLPykEGIq9Rh7VA8c9tZCdpchMCpdyhWBDG/eXzEqqxAWSHrncrHTejaYI49NXZyfWI/sRuS u5M+rTWlxlK1G03NlK+toVXDg0dQzmja6/8aB2XXwg1xbybYjr2GYb29ZkJi6B1q+mPwIr7T57s VbhJ2VwcHGP5cT4UWDiVcA/ldEE6XqSHVveiyjDAnBaLTQXxe/iFVk7I0uv6dyziIy9MnKFFS0I JXZbfEFlgZ5mmNllrSolYZlbUSPcGojPGovkD7R+7tYK9Baab1NRJFZrTuKVJ5ybaYsxDC9ZsfI xLJFddiTfRfKIhoSo+u86AFQa/YdQMOO0rH4p/U7aNVNp+aEFSLt45qZcaBaHhLcq5OChTYotVH PAaHAzR/CSERxf6eKMRjTHwX52m+PJRMETQlo3eaVva61hjELCY9/1UpA5Oh2HE6FCZoFp6cs6m buNaKF08AjHCubq/UCWhwtZQ850jeR3j4+PoBQefOY9/CdUDOGxLcCqup4= X-Received: by 2002:a05:690e:484b:b0:670:f743:6eb5 with SMTP id 956f58d0204a3-671030a9c66mr1404195d50.19.1788948981459; Wed, 09 Sep 2026 03:16:21 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb4912962sm11717181d50.12.2026.09.09.03.16.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:16:20 -0700 (PDT) Date: Wed, 9 Sep 2026 03:16:16 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 17/26] mm/fbatch: no lru_cache_disable() in __alloc_contig_migrate_range() In-Reply-To: Message-ID: <84ed3bf8-da32-3447-0577-052c5d1a56b2@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove lru_cache_disable() from __alloc_contig_migrate_range(). It does not now benefit from lru_add_drain_all() first; and gains little benefit from invalidating buffer head LRUs first, since 5.0 commit 80409c65e2c6 ("mm: migrate: make buffer_migrate_page_norefs() actually succeed"). This will be more controversial. lru_cache_disable()+lru_cache_enable() were brought in for CMA page migration, see 5.13 commit d479960e44f2 ("mm: disable LRU pagevec during the migration temporarily") through 8cc621d2f45d ("mm: fs: invalidate BH LRU during page migration") - I guess the testing there must have been on a 4.19-based Android kernel, without 5.0's buffer_migrate_page_norefs(). It's possible that invalidating BH LRUs perhaps 0 times, perhaps N times, will average out worse than invalidating 1 time and stopping everyone else; or that folio_test_clear_lru() failures manifest more than before (note how folio migration has retries on raised refcount, but isolate_migratepages_block() no retry on failed test_clear_lru). But let's give this a try and look out for regressions. Don't delete lru_cache_disable() yet: leaving stale folio pointers in the per-cpu fbatches, with folio_try_get() yet to come on them, would be bad for memory hotremoval: the lru_cache_disable() in offline_pages() protects from that. Signed-off-by: Hugh Dickins --- mm/page_alloc.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 12fac9084c48..3e085a25a776 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7147,8 +7147,6 @@ static int __alloc_contig_migrate_range(struct compac= t_control *cc, .reason =3D MR_CONTIG_RANGE, }; =20 - lru_cache_disable(); - while (pfn < end || !list_empty(&cc->migratepages)) { if (fatal_signal_pending(current)) { ret =3D -EINTR; @@ -7182,7 +7180,6 @@ static int __alloc_contig_migrate_range(struct compac= t_control *cc, break; } =20 - lru_cache_enable(); if (ret < 0) { if (!(cc->gfp_mask & __GFP_NOWARN) && ret =3D=3D -EBUSY) alloc_contig_dump_pages(&cc->migratepages); --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f48.google.com (mail-yx1-f48.google.com [74.125.224.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A125F4E80AC for ; Wed, 9 Sep 2026 10:18:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949100; cv=none; b=CM4v+CU/JWQNDaAu1yxi1Dy3kpIwuIurRdx0N+ptRUI8XZq/5PFl3InU1Kd6PL+AqDj9QTKYpoIdZETle+6FuYes/X3Th6NY37IgULRYMDQTS1b10OoJhe//5Ew26Q6SB2HnRb++Xdq0LgA8idKCVZ7cI0cWLfCchUF2D7qGH38= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949100; c=relaxed/simple; bh=BAoHBLGej7CKU513wAnhWH+90yYgW8iGSWBxT+OzRiU=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=u+fcvONKduXftY99uVqoPKK3Q1xW1gkk4L865EFdMSHpwwTzAfIKHiEa5aDutNrpffO/WI6FbIBcdRyho9yt90H/dSN2wWA6c34iMmEv9Cbu76WW9TtoiNvZNNjarNqwIi+wy7aP+S8YkuDolcX8+teKzD7udP0EpFhS3t9eUWA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=L/ZR8I0o; arc=none smtp.client-ip=74.125.224.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="L/ZR8I0o" Received: by mail-yx1-f48.google.com with SMTP id 956f58d0204a3-66fcf87897fso4327790d50.3 for ; Wed, 09 Sep 2026 03:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788949093; x=1789553893; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Vqq1bLSj+HyikJMGO2+S/zPhjWI8z39VCLxfhn2Q3b0=; b=L/ZR8I0o54sm99u0HJWpQa6I95wZu5qeb6NPYlTf2EcCcBqivBJ1PhksUmu5XREHaB hK2CSzQbzRHRUYRQzB3Gcp97Bo2NNPRXj8STiVrOoROM37tnbee3r8n9QVcsnMPdkQqL H64tr7/hEt7OTFFBJxWH7KYsBPYM2LyF6ETt1WOCG0WVb4L/Ge3byNnyptRfT/WOlGxI RJEkGbKcXzelwuCkI6nGz8aXBY6MmxH2hUWdGnQGMd24VPxq3COAmgGbrpkaNWfSluP9 zSTZuboa0YXAVhBGA7xdRBq/FbI3GOHX9eI2CmVShgCBmrLvyRwOprmpIDaMh2u1bgHQ pXvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788949093; x=1789553893; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Vqq1bLSj+HyikJMGO2+S/zPhjWI8z39VCLxfhn2Q3b0=; b=NRAnwnRq7rsi/V2lI1j5RK+K58K6pOagriy6daHpkSEiQXBkPlX5WLpp3Ea1gvL/BW uzW/w4H7iJxwKofBKKCuqPwx4lwRFj1IXLKgeKyy3HgzUy5aHHypq7iepKDbOGJJI1XD I8P0iQy6LXche4Bo/N7QlVdx181thm0RA3i9f2xl2lNZo7odSk0EPw6UZwW9kKBFM83w +pjZ4o+VZtX+5udSVfinWD+pv/8CGgBSjgqX0R+TCmoYTp4A1k64L3e2FsGefTivDC2F jR3qVOs9gUne7oNLcwEb6OxOFhTwztyPwusF4AktYdnFJ632g5UF/2aakEv4vKsjq5P0 utGg== X-Forwarded-Encrypted: i=1; AKwUvBxM+nKNQRjMgjuMZgOj320CkpuLQbUPayuT+xChIaU4rm43zqOJs4syVs/sBOos067dLWGaOn9Sh/OOz0Y=@vger.kernel.org X-Gm-Message-State: AFuF++nyxsEOHEYnkyQjNcDE3Ig2WR/xjiIBjNpRJBqPSERZlKXIxdZ/ dKIOzTXbP8UQw5p1YYUTeXLPkMMFFa3Ffj3HEIva2zEaSepIQWpIDQltMtat0w7tCw== X-Gm-Gg: AYBFou26IG0w1oSNQ/zNVt4znySA2WmnSb6aX24wM8sN+7m4LpeGeKzIUfnbrAVuvC5 +6LDHzg9T+6Cly+9y8Mt+/ZAQhtR84TBNShehYMt9ii1lQoyFycEnuoebIekUlJxE4uniuBnk1+ R1HS/iXjiUwlsV4xKQCJ9G4NkOJIgZ1A7YzJQwsC7bChj+minDaHvqVj8UOblselys0x9+oxyJS FS/IM+/ACvz0019JKHDZgtOfKRniSQV+rn1L4ZC6A3uwOLK4eD7KovLhGY5DRvdkgS2n18X2Iuh 19JnCgJIyfUXc6RzxsBc9UFmo0ghzJSIaRQru6KOZlgruOErv6khZawFzQYBX5fxfRkBixEb2Gv SMG7SYE5++q//rbMB9ezaR742UHEEygYFt6MSEZlKqKcjkI5hNiEtpwuCmHXi3EPFAqQYUgnFsi Xpxm2e7VMfYRqYSAOazKUXz1r4/FQw7PyhrmWNsnxJ5s7iOZCHsBUQXZFCHbR9ztRMsf5bqtD1N 3Bt8wZZ+BFZ8/Az3SqL1eOA9Mmq1IyiGTVh0wsxejEx7L4TCm8Up1aLAt4= X-Received: by 2002:a53:ce92:0:b0:66c:c638:8e88 with SMTP id 956f58d0204a3-66fb58dcc41mr7649439d50.2.1788949092561; Wed, 09 Sep 2026 03:18:12 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb48d6ea5sm12216161d50.7.2026.09.09.03.18.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:18:11 -0700 (PDT) Date: Wed, 9 Sep 2026 03:18:07 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 18/26] mm/fbatch: remove lru_add_drain() and _all() calls from various In-Reply-To: Message-ID: <3af1563f-1db3-5974-659a-41463ad7a4c9@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Splitting into little patches gets tedious: now that folios on per-cpu fbatches no longer hold an extra reference (and have the lru flag set), most calls to lru_add_drain(), and more importantly lru_add_drain_all(), should be removed. Remove lru_add_drain() from compact_zone(). Remove lru_add_drain_all() from compact_nodes() and compact_store(). Keep lru_add_drain_cpu_zone() in compact_zone(): it was always a bit of a hack, a cheap way to get a local_lock() around pcp drain_pages_zone(). Remove lru_add_drain() and lru_add_drain_all() from generic_fadvise( POSIX_FADV_DONTNEED), but still retry once on failure; remove outdated comment line from mapping_try_invalidate(). Remove every lru_add_drain() and lru_add_drain_all() from khugepaged.c. Remove lru_add_drain_all() from KSM's scan_get_next_rmap_item() restart. Remove lru_add_drain() from wp_can_reuse_anon_folio() and do_swap_page(). Remove lru_add_drain(), lru_add_drain_all() from migrate_device_unmap(). Keep lru_add_drain()s in mm/gup.c populate_vma_page_range() and faultin_page_range(): good housekeeping after a bulk operation. Keep lru_add_drain()s in mm/madvise.c: the ones after a bulk op probably do want to "Push any new pages onto the LRU now", and update the stats; the ones before a bulk op may be trying to stabilize initial conditions, or to minimize draining under ptlock. Keep lru_add_drain_all() in memcg-v1's mem_cgroup_force_empty(): it probably does want to push onto the LRU and update the stats. Keep lru_add_drain() after bulk op in mlock_vma_pages_range(), partly for "Unevictable kB" accuracy; but remove the lru_add_drain() before it. Keep lru_add_drain()s in swap_cluster_readahead(), swap_vma_readahead(): they do want to "Push any new pages onto the LRU now" (akpm 2.5.46). Keep lru_add_drain()s and lru_add_drain_all() throughout mm/vmscan.c: it works on LRUs, so it does need folios to be on an actual LRU. Signed-off-by: Hugh Dickins --- mm/compaction.c | 12 +----------- mm/fadvise.c | 17 +---------------- mm/khugepaged.c | 11 ----------- mm/ksm.c | 12 ------------ mm/memory.c | 14 ++------------ mm/migrate_device.c | 9 --------- mm/mlock.c | 1 - mm/truncate.c | 1 - 8 files changed, 4 insertions(+), 73 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 1290b9170cb2..5944094253b0 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2652,9 +2652,6 @@ compact_zone(struct compact_control *cc, struct captu= re_control *capc) =20 trace_mm_compaction_begin(cc, start_pfn, end_pfn, sync); =20 - /* lru_add_drain_all could be expensive with involving other CPUs */ - lru_add_drain(); - while ((ret =3D compact_finished(cc)) =3D=3D COMPACT_CONTINUE) { int err; unsigned long iteration_start_pfn =3D cc->migrate_pfn; @@ -2969,9 +2966,6 @@ static int compact_nodes(void) { int ret, nid; =20 - /* Flush pending updates to the LRU lists */ - lru_add_drain_all(); - for_each_online_node(nid) { ret =3D compact_node(NODE_DATA(nid), false); if (ret) @@ -3036,12 +3030,8 @@ static ssize_t compact_store(struct device *dev, { int nid =3D dev->id; =20 - if (nid >=3D 0 && nid < nr_node_ids && node_online(nid)) { - /* Flush pending updates to the LRU lists */ - lru_add_drain_all(); - + if (nid >=3D 0 && nid < nr_node_ids && node_online(nid)) compact_node(NODE_DATA(nid), false); - } =20 return count; } diff --git a/mm/fadvise.c b/mm/fadvise.c index b63fe21416ff..f788a5020384 100644 --- a/mm/fadvise.c +++ b/mm/fadvise.c @@ -143,27 +143,12 @@ int generic_fadvise(struct file *file, loff_t offset,= loff_t len, int advice) if (end_index >=3D start_index) { unsigned long nr_failed =3D 0; =20 - /* - * It's common to FADV_DONTNEED right after - * the read or write that instantiates the - * pages, in which case there will be some - * sitting on the local LRU cache. Try to - * avoid the expensive remote drain and the - * second cache tree walk below by flushing - * them out right away. - */ - lru_add_drain(); - mapping_try_invalidate(mapping, start_index, end_index, &nr_failed); - /* - * The failures may be due to the folio being - * in the LRU cache of a remote CPU. Drain all - * caches and try again. + * Retry if any failures, in case they were transient. */ if (nr_failed) { - lru_add_drain_all(); invalidate_mapping_pages(mapping, start_index, end_index); } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 75639298efc2..d09f848d3591 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1236,10 +1236,6 @@ static enum scan_result __collapse_huge_page_swapin(= struct mm_struct *mm, if (pte) pte_unmap(pte); =20 - /* Drain LRU cache to remove extra pin on the swapped in pages */ - if (swapped_in) - lru_add_drain(); - result =3D SCAN_SUCCEED; out: trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result, @@ -2320,8 +2316,6 @@ static enum scan_result collapse_file(struct mm_struc= t *mm, unsigned long addr, result =3D SCAN_FAIL; goto xa_unlocked; } - /* drain lru cache to help folio_isolate_lru() */ - lru_add_drain(); } else if (folio_trylock(folio)) { folio_get(folio); xas_unlock_irq(&xas); @@ -2335,8 +2329,6 @@ static enum scan_result collapse_file(struct mm_struc= t *mm, unsigned long addr, page_cache_sync_readahead(mapping, &file->f_ra, file, index, end - index); - /* drain lru cache to help folio_isolate_lru() */ - lru_add_drain(); folio =3D filemap_lock_folio(mapping, index); if (IS_ERR(folio)) { result =3D SCAN_FAIL; @@ -2975,8 +2967,6 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) bool wait =3D true; enum scan_result result =3D SCAN_SUCCEED; =20 - lru_add_drain_all(); - cc->progress =3D 0; while (true) { cond_resched(); @@ -3208,7 +3198,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc->progress =3D 0; =20 mmgrab(mm); - lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { enum scan_result result =3D SCAN_FAIL; diff --git a/mm/ksm.c b/mm/ksm.c index 49d48d1e0998..890e2c51106d 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2625,18 +2625,6 @@ static struct ksm_rmap_item *scan_get_next_rmap_item= (struct page **page) advisor_start_scan(); trace_ksm_start_scan(ksm_scan.seqnr, ksm_rmap_items); =20 - /* - * A number of pages can hang around indefinitely in per-cpu - * LRU cache, raised page count preventing write_protect_page - * from merging them. Though it doesn't really matter much, - * it is puzzling to see some stuck in pages_volatile until - * other activity jostles them out, and they also prevented - * LTP's KSM test from succeeding deterministically; so drain - * them here (here rather than on entry to ksm_do_scan(), - * so we don't IPI too often when pages_to_scan is set low). - */ - lru_add_drain_all(); - /* * Whereas stale stable_nodes on the stable_tree itself * get pruned in the regular course of stable_tree_search(), diff --git a/mm/memory.c b/mm/memory.c index 8b0c2c735d3d..e9e2199ba5e9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4313,9 +4313,6 @@ static bool __wp_can_reuse_large_anon_folio(struct fo= lio *folio, static bool wp_can_reuse_anon_folio(struct folio *folio, struct vm_area_struct *vma) { - const bool maybe_in_lru_cache =3D !folio_test_lru(folio); - const bool in_swapcache =3D folio_test_swapcache(folio); - if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio)) return __wp_can_reuse_large_anon_folio(folio, vma); =20 @@ -4326,16 +4323,9 @@ static bool wp_can_reuse_anon_folio(struct folio *fo= lio, * * KSM doesn't necessarily raise the folio refcount. */ - if (folio_test_ksm(folio) || - folio_ref_count(folio) > 1 + maybe_in_lru_cache + in_swapcache) + if (folio_test_ksm(folio)) return false; - if (maybe_in_lru_cache) - /* - * We cannot easily detect+handle references from - * remote LRU caches or references to LRU folios. - */ - lru_add_drain(); - if (folio_ref_count(folio) > 1 + in_swapcache) + if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) return false; if (!folio_trylock(folio)) return false; diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 009bfa8b212d..7ece4ca0654a 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -575,11 +575,8 @@ static unsigned long migrate_device_unmap(unsigned lon= g *src_pfns, struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; unsigned long i, restore =3D 0; - bool allow_drain =3D true; unsigned long unmapped =3D 0; =20 - lru_add_drain(); - for (i =3D 0; i < npages; ) { struct page *page =3D migrate_pfn_to_page(src_pfns[i]); struct folio *folio; @@ -600,12 +597,6 @@ static unsigned long migrate_device_unmap(unsigned lon= g *src_pfns, =20 /* ZONE_DEVICE folios are not on LRU */ if (!folio_is_zone_device(folio)) { - if (!folio_test_lru(folio) && allow_drain) { - /* Drain CPU's lru cache */ - lru_add_drain_all(); - allow_drain =3D false; - } - if (!folio_isolate_lru(folio)) { src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; restore++; diff --git a/mm/mlock.c b/mm/mlock.c index 97134eff6b56..971430e6251e 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -424,7 +424,6 @@ static void mlock_vma_pages_range(struct vm_area_struct= *vma, vma_start_write(vma); vma_flags_reset_once(vma, new_vma_flags); =20 - lru_add_drain(); walk_page_range_vma(vma, start, end, &mlock_walk_ops, NULL); lru_add_drain(); =20 diff --git a/mm/truncate.c b/mm/truncate.c index 4151f7a167e3..7ea5d513f76a 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -571,7 +571,6 @@ unsigned long mapping_try_invalidate(struct address_spa= ce *mapping, */ if (!ret) { deactivate_file_folio(folio); - /* Likely in the lru cache of a remote CPU */ if (nr_failed) (*nr_failed)++; } --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47335498914 for ; Wed, 9 Sep 2026 10:20:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949234; cv=none; b=jWTZfaQUWFhZ586h17mhcBLFzyP1pfZHaC6dWKM/EtWPXhIX+kMx8iPEdvxbembBHuEo9R2kpqF/NaWbEVF+OhJiXyuO18YHrxnvzlVi/3bSJ1Z51FeMViVyy+Dx1X9qMJAAzCaoH3t/bhiG5IcK/7bM5PKcdDOHKeTC9zoTlbQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949234; c=relaxed/simple; bh=rPfdB3AFtBKPw/1Uy2omaI9Rtdatgf/EWcjp6PUwumw=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=qDu6ux+b6S3LiMNnq459t+6irAlEE//HENdNW7vSERMn0tui986SlxZI8qND3qj3NP6JjnxpGTv34DPC4XPIcKoPrOEfbr7LaXp7rXKJzxG1KDQAtEwz1dIms6eIzcvAumrFEvxngtSMf2Zy8WvAaWpypiy0aVj6UD+9h7EfzZE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tT9Fz2UG; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tT9Fz2UG" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-85a50f6a7f7so78076197b3.2 for ; Wed, 09 Sep 2026 03:20:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788949231; x=1789554031; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=4U3+AfBAvn6B1ULZHKx9lvgAuWRYBVO83SclUxbDGr8=; b=tT9Fz2UGnxETlTmbyHAwZYaiPWS5B0cx5KG3uw+zIkiDXB4fxFV8HteHjg1CMJvcNE 9Bv0QNlYapPVNxSE1zPdcr2zLNj/dQrH48dgU4rjjuv8PSeMBv3uc3+aKfxjwcnAFTTx Frgio4lW8raBbVZkfM5QT2i9GZaj2Miey2MReEoWuQfgvVOa9CsxC/HeLX0MzmYlWq06 IGFKtqqJMntut2fy7aSV8czLI24+O+PB5sKQxQxtcI+Ge3ATfkB+jB1htGM7dGQPG2vS vn9BkOnMOIGre7A5K/55B2TwyQQA65k8AFHVQIpF/B1FUOiztBt1LXVQZ9BBca79zezB 5L7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788949231; x=1789554031; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4U3+AfBAvn6B1ULZHKx9lvgAuWRYBVO83SclUxbDGr8=; b=jgZV9H/cHsC180Dqt6bWpMUOFFrLdS5FYsOQx1cZ1DDBr1PebUGLHVaDKb7e3yOidF GO2OdhliBOlHlRSYiq30s5v06u/qAT/i8y2+mjUygOhTPDZ7ck6XTeTlC/ZrJUW6i03Q V5sQ0q3MaB1Pdxr3ZEnaNZrcU1YFEvK0ucXk6mtVH58fkEQaNkvwyd5bAf2rrgLB8MUl 2iL8osq5Nf9WBetwc/RDyqqSVMpJXNC1aClsd9cYc/3nkzTTiUel0zYmWKBBee8csy+y gH+iFgAeDa9iQiNaPf7XKSazXUuJz7jAJ1Shr9ITaNWN08IbeVAzVOq1sr+BLrR9Of/p 8mTw== X-Forwarded-Encrypted: i=1; AKwUvBwCkABLKhPZ/DieOWKP5LJZZwgp+mHmmlnndPAbaI6q6JvvDxzmohBsC8hDBM/KZYZJANARaHFCMcHoU2A=@vger.kernel.org X-Gm-Message-State: AFuF++lNpU6mTZhuuVVh60bK10ODpPcabjAv0bFlVRRO5vhL9TpdUAc1 Ycw4bbOcn3InaBIsA64xCMlBTAlI94dvX6R7N/w5dcMWkDYDNbnyGSKOTX/RZjJbEQ== X-Gm-Gg: AYBFou20iszaYtw74xjWH9/lm/us+RugwCLZ9/KYM/cnJ3TWrDIWiQOD0J543N15rbs SgyLZ3P/OZPJBk/p5FawmCnFtVcXh9/ONirsDeKLGPtIx+V2Lik1Rie9zOiF9pUhoBVi4xPdUrI rvUxphTWp7Zbj6kBMSygx3HpaQ+kYEv1WeHrJUKL8gnmeJkM0/KLRFGd/mQeSMEFk1IYj6sq9Gl mZnqrBwF0qMUVb+cSP1L+0HUGDB+bgzmlqJKaluj0zqwYE2xjYXVHdxyRNx+K5a2BWSlLaloL/q Cgt14hbBczhviZ53UYk5pdtEJyMYunCcFG47XWS0UyPbFkzNV3lXUEbWukbXFsUWr5JFVG1eZ+F 3RENLAzfU64vRnn7E0pmY2qmf8wR8xavxkTQSzkQEVJmisuhv3FcS08393onwGGQWP3XPvFumXW 30B7e86zEgVjGqw/iYEBOxmw6nVHCVEBW6aVag2U8evf9sNHf/cwt398Z1b/MLp/3hokMXiLbZh rp95fZsOY1UgVKu2SjkBA+jhUpk6yXEwnZkjuK6ReSkFsLL6nGmqesm+iw= X-Received: by 2002:a05:690c:e3ec:b0:877:e47:f399 with SMTP id 00721157ae682-8770e480a9bmr69779907b3.23.1788949230466; Wed, 09 Sep 2026 03:20:30 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714a0b6179sm106432667b3.23.2026.09.09.03.20.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:20:29 -0700 (PDT) Date: Wed, 9 Sep 2026 03:20:24 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 19/26] mm/fbatch: vm/stat_refresh include lru_add_drain() on each cpu In-Reply-To: Message-ID: <48e09d89-27d7-60cb-a0ef-d2a5ea74d90f@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Reading or writing /proc/sys/vm/stat_refresh has long been a way for testing to force an immediate refresh of /proc/vmstat numbers: it seems appropriate that it should also call lru_add_drain() on each cpu that it visits, before collecting that processor's diffs. Add that in refresh_vm_stats(). I wanted it in refresh_cpu_vm_stats(), which would include it in the regular (default HZ) vmstat_update() too: which might be a wise precaution when proposing to eliminate almost all the old calls to lru_add_drain_all(). However, that would go against its "strives to only access node local memory" cacheline care, and bring it into lruvec lock contention: which would probably displease its authors. Signed-off-by: Hugh Dickins --- mm/vmstat.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/vmstat.c b/mm/vmstat.c index cb57714539fb..d66cb67f55ce 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1990,6 +1990,7 @@ static int vmstat_late_init_done; #ifdef CONFIG_PROC_FS static void refresh_vm_stats(struct work_struct *work) { + lru_add_drain(); refresh_cpu_vm_stats(true); } =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EE9A380FF6 for ; Wed, 9 Sep 2026 10:23:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949392; cv=none; b=bqowU3FVsqn4vO2WgMzsLdrmM392XAXeyDiaYdfW6lJqJCvuUgZHWNYSCCNRWK+R5iTpcTg0uEL5LnT6u9Oril7VEn5EEzVEjBwp8JbnfUdMRnlWTLF3eAj887eIYbjaQXXw8sk04khSWthfyPGXIyDlt2005QmSgX3Yb8drI3U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949392; c=relaxed/simple; bh=RrFIEm8SrJEjhBjzApoefNfd8aQaR4Oxfr/yDuZhFPU=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=BMVs++R1uQZwCSUbxzf90jjMmTW5njhW2eDS/2z4WJAUlia3Tz6wAD78CvXayuhMCjLsLjB/hDvbXlalfAalhJJSWFJRwR+4oNjF5R4dNPY1aHyF/d1Dm2UP8f2DwpEmJs21+wf2hmeWKxWeFTitT0dME5McvA8V5enAh7rLKLk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZoRzlaCR; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZoRzlaCR" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-857ff9fef54so52702827b3.1 for ; Wed, 09 Sep 2026 03:23:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788949388; x=1789554188; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=2UY30k1UeQzYu1OvDSaJuzIQ8I7z52NjmS+xQ0qJeoY=; b=ZoRzlaCRFXhH6gCaS3eEcmQTSTKHzTmJKAFmIlJChPlLxnQWPDvmcYyCo34ZtQ4JwK 9liXSr9XQyn6eDXCJA0PhPpYangmdjaKltp8Z2mbwnKEma2qNUWjNKz+LESYTRm312fu Sc2okpFRLO+NTb1xQex6y7ciauOwGUOmKDcTtFcBSI2lASywJ8vrz8UZEMl0loeyEdqa akAq9L9rAqzMHpqT0nf3pI09liWo+VnpxJ6a6l3ebqBnbZE3pYUqEUc1ioBb925HCRSY dTbxog+PSoprsbaLQk5FE6QVEYobawHzFMMEdh36UGK8SY6HxDgd20i6s3LNBr9kC9E1 5F1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788949388; x=1789554188; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=2UY30k1UeQzYu1OvDSaJuzIQ8I7z52NjmS+xQ0qJeoY=; b=h1tG041J+n7t6fh1HgMvJ/bGcX/Yuu9AbFt4nwPvtT9uXqML0Qh1U8VrCwLkhfn/60 ebQgnibZTSXnn6tp5TZYKSETsSSZ+HbmmFpJyVU7qpiqvet0acuG5UCQ2brKm2Uw0LtZ lHv+RyRpNp9x9YXt4xwHMgFSNGN0lFhRJCdc9DO2TdRUql2PfHwXoW61ThCvyy4PUjSp Fk5yPg/8RjElJ8v7rjlg0fnUz14JtQoh4h7ae+PiOb6r1uit5YLvSBmNz7V7QPO5fE8E oXMkP4pBMiQ0DDqhlhj5aOF3p13HC1+8IKO5eYuTY476PcCSMX26VqfNN/2mQndmiFGc qrdw== X-Forwarded-Encrypted: i=1; AKwUvBwgv38MJYhkkfZHB26SrNN8bKlG50YV79QcQPAofLynywHFLXIC8Jh0odBYrWXV1557p9ikun7Jb+dQgUI=@vger.kernel.org X-Gm-Message-State: AFuF++kKLuKuRZZ8FpL4KnI+XOmgQoaGvms5mC3VICwd5yzsRppUGdVm 6kqzKljc1trqUYVUho2Z1BLDYwUOlhoYc/XLNhazkcagYYbrDchHsYOWHIANJNEMKw== X-Gm-Gg: AYBFou0ifIaeC46psyH2GmLajWfr8wPxJJ3zYTDUKE2C5fV7odP4muaXV2gmP3XGQrS pAESmLvDnNmd2d/bQpfUs6cHsLj7DA6VvE6Im0/J4mOty1p0QbOm5h+O/irZTnHMAxueyVpF/74 BhSrvHn2p7i/OdSQN4tLWnkZjOc9RO95TDD3QEtQx8Cmws9n7FmTE/87JOWC2VLV44uYWi8pBxN Bna2uLT8bXA2JUBpNEs3zyJmPqgrdrbMMDbeZKSQKdRDwWC/wT8z0sh1YcWLjP440+RgvfQdIRV VmUJX2xR9c0ftMNNhML1B4Ql0OULfVcj2daDVRFJcnC5TJ/LMGvDfCcNXnDE/TpDLJ9rjFiDucl MaSzFYp87IIy/CSx9AknMrfCFrKi+X9VoXhWDyzEOJ1IiIqOZRgOpbI5R0vZpi8eK27wLNzhT65 OmbsmwrPeITp53KsRVVVhy0ZucnseA7WEn9F2XJFbWP8U03cunAGfFrzQTyQxUmLutJGC3WOe6+ aD/8rdvdgIUoTOAjS1FkjiIRADsZO+ELmzwhiJRlX8XNm7RJFeUTx3LhG0= X-Received: by 2002:a05:690c:4b85:b0:870:c600:686c with SMTP id 00721157ae682-87126aa77d1mr116385687b3.18.1788949387337; Wed, 09 Sep 2026 03:23:07 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714b62503dsm108473437b3.38.2026.09.09.03.23.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:23:05 -0700 (PDT) Date: Wed, 9 Sep 2026 03:23:00 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 20/26] s390/fbatch: no lru_add_drain_all() in s390_wiggle_split_folio() In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" s390_wiggle_split_folio() has no good reason to lru_add_drain_all(), now that the per-cpu fbatch references are gone. Signed-off-by: Hugh Dickins --- arch/s390/kernel/uv.c | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c index dc14ebc0105b..120a467026a5 100644 --- a/arch/s390/kernel/uv.c +++ b/arch/s390/kernel/uv.c @@ -364,7 +364,6 @@ int s390_wiggle_split_folio(struct mm_struct *mm, struc= t folio *folio) =20 lockdep_assert_not_held(&mm->mmap_lock); folio_wait_writeback(folio); - lru_add_drain_all(); =20 if (!folio_test_large(folio)) return 0; --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7BCB46AA9F for ; Wed, 9 Sep 2026 10:25:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949514; cv=none; b=nOLoheRDwtwNcePNnEuSKBKqOp1LCvwPW9YHk/G92fzY5dB61nCcW/+jl9eBX5kBxc1mJRjUlF5FZ8zTRgYo8qM9tbdv1uPIJIJOkY1lFEbQASbWpxMJ2dh45c3bp/P6W5s0JYfxdgSP1Jipnlk1bFBQ5xNdYCyIUMjWyoVYbQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949514; c=relaxed/simple; bh=PA7/Pbx5AEThbU1dExwqFT9w9coLuAoGrR7Ue2+wOTE=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=ZcC/ejYr9sRHE+bFeDPFdvLVxtvgrbSanzbiFlv2WUn5OHAfLvujXrOGzTNi1RFB4qCCWQKk/jIooarbhQ9PWAQyyOQXX+zYIX9uVvxAcGc0LNAjk/dosLssoG283qpnYoRDFZ6QF2ezn5uJtIRJR+C5qVJ6PaCf7lhGH71SwoM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sMACMHU8; arc=none smtp.client-ip=209.85.128.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sMACMHU8" Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-836c8bde2dcso37594507b3.0 for ; Wed, 09 Sep 2026 03:25:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788949511; x=1789554311; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=tbn1aAX73v4jfLOAScY/xk2YxlfOe4UtWahA1Ug2y2k=; b=sMACMHU8bUBRcevNAt4rnxkr3JHUKboqxgklcBwQXRmJ4F65NgGz0OoXJ1+1SqoDup va3cGoWVD+QdFyHTN315iCwqEyTQMhvn08fWj1aZIOD+a7tAE+SuBkm9Na1vq8daGX0n BpltVo6dID4JgABrOfcolkDET2tDT6Oppk0J/PH9RZxLPddxlD9ARY+GODw/i/WgSpI3 Rz3EaF4SPDHXEYV+Az/iF6Y5gppFgVWwQdH0hHhwz+pvdE4SkCSu+uI1Gixk7kgdALkq FmnLa3oRYEIYZEjQgxK5nhvRIqcoLDn98EBrxj4Z7tuxJPalZYBNbcFJmP6SOU0kLkuz gh9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788949511; x=1789554311; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tbn1aAX73v4jfLOAScY/xk2YxlfOe4UtWahA1Ug2y2k=; b=monsV3BCrEgnF5bAHmCoWKnZA0h9evF5nk5jqhk0WUpQE6bS9RJQNFT+K+/xifYzou asmj2B7v5ipiZhubtGcWGSyfDLVzneD1MJ7YbyPn0zzPFZfKzohHeIfTB58WG4+4NgMN f4/a+49KZd3zON4ZlQeX0aDjU52a7nYxMxi1GhW4U1CcOxKNv2tK2QQ9p3i3OIta2ANk jNmqQmf+meTmTpWpELWNzz0qO8OmMHBI5myvIgEBXt4vBdH8e/irJIa++ioI1Ih9gNUz MOq5xl7bfMgyxpSNjsH+BMM89Xpyt8KeYfJjvFQFy62Zq6T+nG3fs9Nkn9mpMLoTDUM5 t/pg== X-Forwarded-Encrypted: i=1; AKwUvBxdRFnqKgLHvq2oySwV07CUSKmED94q1YDduie60PeP/Ahl771hAvt1f/krMbcfKet6rj9fWL9QW92Y6Jk=@vger.kernel.org X-Gm-Message-State: AFuF++lpQYC/+hTuIQ0haZsFHo0vYKYjRueDSMFwxYNFEx8eza4uL1gL aJvJmwteMrmYVpKuECjU+n/Jk8cFyPHHQdCTCrFgeqRhhcdN9I4XH4l92vxPcTKbvA== X-Gm-Gg: AYBFou1hVoJ9FgeZne885h0ibyvLCnhWHIdO3FQBfqzx2/1JPfgATsoZLZ2Oa5qWbuT o1J4uEztMHRVl3iTsPHLQ6ISFeRzcfHBckwahcj3LnDrRVjDGQBddXjxfCKdDCUSCni5hTtMa2z bovJoxMhbhDkZ3czrj8secFi41GqCo+lg7M1MB3WQSKLkHaLgiH16hDIaPQJGWbcnDgJ6yoaYLe F5ELYwwcDcwAaL/cpW4/tHPWWzRC3pruool3+AI4fKI6VhmALS/6b7sY3JR4O48Yu+2jUKwJqUz dFsjbaFoVzNKFTmWx+/At/FjQfjJmpKJGQHAbWqiX9dGk8eUTg+uZJGzLpSlweNKJcywCmUtMGh ioDU5wn4as2RyXPdVwPRd2MzLrNN7LAkAZO6flWVfeQvnWqJUYJdROzupV5LBieOlcmF/4U/Xpl AEo/Yy/2u5ODLbS+CcIRQrGA6gQkRtAhBnXlFkegTF9/CrFSMshuXcueByuNLX0CTlk0fdKBQEI +mlUI0qmJh/02FMlnrDxgLh7VPFF0EqGUXjk16auc4Z64Hj6nORwjigaOxRoN0+qBxTEw== X-Received: by 2002:a05:690c:62c4:b0:870:96e4:6716 with SMTP id 00721157ae682-87127b36981mr130530337b3.28.1788949510969; Wed, 09 Sep 2026 03:25:10 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-87149315162sm107287417b3.12.2026.09.09.03.25.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:25:09 -0700 (PDT) Date: Wed, 9 Sep 2026 03:25:05 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 21/26] block/fbatch: no lru_add_drain_all() in invalidate_bdev() In-Reply-To: Message-ID: <0debd9f6-0ff8-31b0-4830-3a81ba56754a@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" invalidate_bdev() has no good reason to lru_add_drain_all(), now that the per-cpu fbatch references are gone; but keep doing invalidate_bh_lrus(). Signed-off-by: Hugh Dickins --- block/bdev.c | 1 - 1 file changed, 1 deletion(-) diff --git a/block/bdev.c b/block/bdev.c index cd8323083740..853545b29b7f 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -97,7 +97,6 @@ void invalidate_bdev(struct block_device *bdev) =20 if (mapping->nrpages) { invalidate_bh_lrus(); - lru_add_drain_all(); /* make sure all lru add caches are flushed */ invalidate_mapping_pages(mapping, 0, -1); } } --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f46.google.com (mail-yx1-f46.google.com [74.125.224.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECF7251E452 for ; Wed, 9 Sep 2026 10:27:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949671; cv=none; b=gmoLiy3JUY+7GHzhi9Kdl3vR1HvcrUQcLP1UGVdpeg7UMQq+RhVJwQbf35vIUHYn78n4PGYBlXGrN7xj5gxDJdGi74jcayTjfbbFZ/Etg7ZcY577LIjWo59adjaXeFzBwP4obsHPl2MW/F+80W7tSy7Erx2glXwtSAQ1IT3eJHM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949671; c=relaxed/simple; bh=apGggQcJp2WV1URiHg8Asmh4iwaxGu1pF3vH60oXvTw=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=WH9r0/C9BhXMpy8MCOEHEcVoiyYvombOfJsg+Z+LoqbboCzmjVd6qyQjfkYkWfTjle+qAGfB4ZjtftSAfn1m4FMYMqE3SFrZt0Zayx5vy3q9UgdQaM6IWYmL2RteFhKqqweOaGFDIGJOCf6tiEd48k9RF9OKKwCp6LztSe83DNI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GxGDP+Z6; arc=none smtp.client-ip=74.125.224.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GxGDP+Z6" Received: by mail-yx1-f46.google.com with SMTP id 956f58d0204a3-66fb2fa79f2so4081594d50.0 for ; Wed, 09 Sep 2026 03:27:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788949667; x=1789554467; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0mZ5Gk4/1Pu5bK3S+WjW2BlP4j/qxWTHvLM7mSgUCfk=; b=GxGDP+Z6oWxTSLrBtRAZa+DEityV8+saaWCj3GSm6C/hkO9ldhEfZILDAbNdoqixva J6c3/yfN3eNLcBhTKVSm/yaIjiQDK5sH06oFIzpl1p4Y0mVY7WnZZDXRbJAXU8kgmYk9 A86jXkrLjNjN1LhmOCmobsBWYi/oJ+prJgffhPv4HaVxTmB2IMeTef1ZfZgriVtyAywV pA974nCln5Fv0zd59QRdRhjQ3+Jw7i2HwnCcUIMyKfKwLKG98Yg3z9RRTinyEpzcCW+X FOuutXxJfgEEA4ZsOziq+sniLVIoimIma1ZCtnS+HrK5T5Im3FwHtWt7fl7fmKA1Ai1e PISg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788949667; x=1789554467; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0mZ5Gk4/1Pu5bK3S+WjW2BlP4j/qxWTHvLM7mSgUCfk=; b=mLlyZGSg0QwZfrJeFLlPceElskqXQ0nei73K78pj8D3CRdlweWXZHUS3hFgvybkI71 PHb3I91aJfBeqQbu1G0uCVHzCOoRxenvQ62DDJbIfInvEJsPLDJyfd+JmFKR/qmzdNwf 6dNT+/pE/t8yO4UKv0GBlXuV9oNhgwyd2Y+mGq2cpTYG8UBYaBLRNyUF3o0dxfeMqdVB N5qo0ws7QCYxpRzIAr3ZigT5XdYnGjPH6GpeBjOGSsamjFggYAMgVk5Y6bhbrzihgFhX 9BJB6hLVyX1M4Qtys/gX37zRNjXdnyG0X41jzalF3WVx3LSn2X/PxTbykmRd8RXVdVQY phRg== X-Forwarded-Encrypted: i=1; AKwUvBz71jGl7/2eG/KfMUoldfOG6taE9c44J/z2kBKGx+tEtWxXca5S24VRqT4cSnAat03FcTo4Huq/NxzHGLc=@vger.kernel.org X-Gm-Message-State: AFuF++mMcCOkVvOIa2b1Bi380wHJo9G482tg8lLDf6WNdur9re2A5qIK 173NvWg4L5CyqXqNPJd73TCmqJAu6x3ia/I5DNpdw5L8n6NwGE1y6eyfqdbpfSxJUA== X-Gm-Gg: AYBFou0EJa9JF7qkTFXuBSvGOq+P2E0Po0g706MP9wG0v3H+TYkLpaMVGa/O7FqhEvi +suJ08iVQi7TnHqHOGef5+A7baVb35Y0twRc00ewRpCCkE7MEmbdtvwf9AhZGXsnsI5e6IzMGIz 129IKu4FSnd+oNULv/WwyTzhUtERZc7acXUagY1jBwXyslZeLfhFdkgCfC+e0ciVryVaiwON7lz l1lOHWmvQ19pUTHPfQhhexYiGK57Gi75z336KP4zzDCVxQIjvQltFcj4/ji9EgZZZaXjQ3CrOzT 9ehXlvV/PPErGV4R5aKm+p1YAtDTZ2MvnicS5SdqP6ZDPN33t2CyifG/rJRl3JmpBjDnTZwm7w3 HMmkEJPDn76J61WzVK7iNewIVewgarQAHIzoI7V4psXcQ7/uLFGZ4UMZRwNAmH2ANNGXjPakWr+ pBFubDA8XlEX2ElKPJQ8ZCw6FfIc7kRnchdD/REFKDMTFX1fxSDeEj0U1tvdTQKhKTJbpxsDOOi RTdK8VSayhnbOP3a4BFb4ShZwN9nIFZK5LggOUs5xgbQfAa3ZaICmAg5tGCNYJI5IG39Q== X-Received: by 2002:a05:690e:48b:b0:66f:c1bc:4097 with SMTP id 956f58d0204a3-66fc1bc4874mr7597179d50.90.1788949666224; Wed, 09 Sep 2026 03:27:46 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-670febedc44sm2584559d50.11.2026.09.09.03.27.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:27:45 -0700 (PDT) Date: Wed, 9 Sep 2026 03:27:41 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 22/26] fs/fbatch: drop_caches invalidate_bh_lrus() not lru_add_drain_all() In-Reply-To: Message-ID: <75e9b5b0-6a70-06df-ba0e-4d7bf841c0cf@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" drop_caches has no good reason to lru_add_drain_all(), now that the per-cpu fbatch references are gone; but it may have good reason to do the invalidate_bh_lrus() part of it. Signed-off-by: Hugh Dickins --- fs/drop_caches.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/drop_caches.c b/fs/drop_caches.c index 49f56a598ecb..054af78e6a2e 100644 --- a/fs/drop_caches.c +++ b/fs/drop_caches.c @@ -10,7 +10,7 @@ #include #include #include -#include +#include #include "internal.h" =20 /* A global variable is a bit ugly, but it keeps the code simple */ @@ -60,7 +60,7 @@ static int drop_caches_sysctl_handler(const struct ctl_ta= ble *table, int write, static int stfu; =20 if (sysctl_drop_caches & 1) { - lru_add_drain_all(); + invalidate_bh_lrus(); iterate_supers(drop_pagecache_sb, NULL); count_vm_event(DROP_PAGECACHE); } --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E45494BCACA for ; Wed, 9 Sep 2026 10:30:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949860; cv=none; b=PQBJXfjaHCnCRJKbSFuOFIKD7Mbvw2sEs+Vp+m4l+h2UG4LF6i559nqgSfq/WSrcaiOlc7rGOtplHd9eVxN36d2vIRZBrldozvFuLWRDs+UpKPl4lTrR14cctH1aruZ4CfTNND9lD0XXVowFCIsfMBI6ONSJv7tgWUCpMoaUsVU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949860; c=relaxed/simple; bh=e6+P0w42WSank7bJyBRxJ0I/o2AdvAByGwMedNmCY/Y=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=cKrKMWKGjFA4um/h7foZUaxWsgv14NHLJzsy5UWFQZXu7d2I9PH9qefhOUsoAMoZax8/KToFVd9D5Zktv4LOQX5ZTJ45iMxIAB/p2RlCdlad6RdfRGI56IyCbVWg3fXRoxsjRoomc3WtZDFIK3toA/1lhkK5Mpq+6xm3CZ+QbKY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dnL/0r+o; arc=none smtp.client-ip=209.85.128.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dnL/0r+o" Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-86d43cdee51so70273487b3.2 for ; Wed, 09 Sep 2026 03:30:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788949858; x=1789554658; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=xikkFdSddcZk7EZo4mTtVn7i5ie1aP6lqhxPtSw7MtA=; b=dnL/0r+om2LqYB9UWE6tkpdxIFAuAA70GZwyzBcqbjutXXb8HacxXnzq40Z7FLVc7D CsOGNNuLraa8Ye+UWFNxQLLysNo6ShwSpEpNaYyBF5h4KPkuwctUWSGDijbRS/GV6JQm QZFjVrOUF64ePn+xzLq1r98qqSC7BMmeYCW5UKBwB8Bus4PLL7vjqJzCOfHLkddxhtei o0gVwgxkyL0LbI12y+ZF0AwReh/ktCAWly79eYoR19C0eg33Qem0lJl9VSw41d+LlHjb /mGNIwm1h2TtZTnmayVZYgjeEaXPRTFhpgMtXetL38UDHNBAuregBL9hsCaIaL0XA+NM vbNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788949858; x=1789554658; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xikkFdSddcZk7EZo4mTtVn7i5ie1aP6lqhxPtSw7MtA=; b=CKlGOImGfiMJiiYo4Qkfn7mOdsmoLARVObeDMIuDf1yXxzbUwC+xyWlzLKm0pUcBME 8w00CYmZ2mRnBhAfIGscz0nRvIREVHFim2izFK5AgnfgKpV0z5nkVTGWpSufgjMuihUh mPR7O1i0nVFu2bQWxRHjTegJnTQVihTIT75jUlFchEgEMKsFE600k1hyxImH/6H9Mk5Y GS5TxiheSw7qfV+34d8yeKVf0aDeKz06pTp4zFHCKvRUQE6hoTE/O142cpInD19x8xqy zXxFMc88ApW/Rg1rmsYr6+zwlr8GgxyrrKT2NuwM+JaNTsITOhMADum63aQMjfzkcgBA oE4A== X-Forwarded-Encrypted: i=1; AKwUvBzv6kzoHHoLs2Jqu3c8dOM9F/hEcAqCpJu9UnLKO65sRMqzhoPx9JevRwih0oiW2KvIeFV3E2z9mYad2gA=@vger.kernel.org X-Gm-Message-State: AFuF++mw29Gs1MOjHil8Y2m30IkVnQtNZ9x5eathYIZ7C1Ed1X2l/IlR 24uZRL4yq4WrAultbsMVCgq6CBhoLeYZW7sKUudbNqSd5HZrYhSo6KOBG8Sc/qnM/w== X-Gm-Gg: AYBFou2b4lW3FXa7clv0J/gnZYbZQRiu0zNbj8NIqaNeI3XT+Y8vsiCypQ0nWvU1jbc g5tD3cYvrbz0xh1fPZxGVRTrE2/zjDjI0yQm/ARMBXkCG0jXJxl3he6505nWaRimXfvlPtYbTMt KhIXZO+a0UYcAZXFKGzZZdeiAVdYQgpLmO8Sp6ENGc+c/BjIlKiZ+rgh8HQM0ZGNgs7H55qE0Lt o2tr4VvNroMHCnkImYNlA+rJF+py8PHb8WEUFdTRx/xSm7CJbCXl3FKio27/FPzRjHJxokFQ0g7 cwn2St1ecbejNfqNKoFMSP3GJET7h5OPWa8hpoM1Z0nmxhum0W3W7PVzQ37qctH3P9XS79Q62ai rzFlD3kOAtSAGMxWQwGldLXAmYlAI2XLkzgBmpeg0ED9oA0UfS9oNZ+qj5N1C7BKI3klWpe631w 5tI/TTY2+nH3b2XTPqGhT+HrXvexQwOXI8yv2v33GR5DSZA34mCXlpJYUcvzhyH2kVkHY3thLgw k5wxnUm5HjBn/GWORLEzyOF5MaokwCKzI5IKfKW71X0YKoesqspPw== X-Received: by 2002:a05:690c:c047:b0:845:1064:5c03 with SMTP id 00721157ae682-8712b59ce6dmr98838127b3.23.1788949857100; Wed, 09 Sep 2026 03:30:57 -0700 (PDT) Received: from [192.168.1.242] (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714b52f641sm106441207b3.37.2026.09.09.03.30.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:30:56 -0700 (PDT) Date: Wed, 9 Sep 2026 03:30:51 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 23/26] fs,mm/fbatch: use invalidate_bh_lrus() not invalidate_bh_lrus_cpu() In-Reply-To: Message-ID: <56a1dba3-1dd4-e4dd-a38b-3fdee74a690e@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Whilst there can be a case for including an invalidate_bh_lrus_cpu() in lru_add_drain_all()'s workqueued visits to remote CPUs, wouldn't it now be a preferable cleanup to remove that alternative, and stick with the smp_call_function_many_cond()-based invalidate_bh_lrus() throughout? And fix the UP lru_add_drain_all() to include an invalidate_bh_lrus(), which went missing when 5.15 commit 243418e3925d ("mm: fs: invalidate bh_lrus for only cold path") took that out of lru_add_drain(). Signed-off-by: Hugh Dickins --- fs/buffer.c | 16 +--------------- include/linux/buffer_head.h | 4 ---- mm/folio.c | 25 ++++++------------------- 3 files changed, 7 insertions(+), 38 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index ed966fa73b1b..7d114e5b9c62 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1427,7 +1427,7 @@ static void invalidate_bh_lru(void *arg) put_cpu_var(bh_lrus); } =20 -bool has_bh_in_lru(int cpu, void *dummy) +static bool has_bh_in_lru(int cpu, void *dummy) { struct bh_lru *b =3D per_cpu_ptr(&bh_lrus, cpu); int i; @@ -1446,20 +1446,6 @@ void invalidate_bh_lrus(void) } EXPORT_SYMBOL_GPL(invalidate_bh_lrus); =20 -/* - * It's called from workqueue context so we need a bh_lru_lock to close - * the race with preemption/irq. - */ -void invalidate_bh_lrus_cpu(void) -{ - struct bh_lru *b; - - bh_lru_lock(); - b =3D this_cpu_ptr(&bh_lrus); - __invalidate_bh_lrus(b); - bh_lru_unlock(); -} - void folio_set_bh(struct buffer_head *bh, struct folio *folio, unsigned long offset) { diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index fd2c7115c054..f19f9e80be8f 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -518,8 +518,6 @@ bool mmb_has_buffers(struct mapping_metadata_bhs *mmb); void mmb_invalidate(struct mapping_metadata_bhs *mmb); int mmb_sync(struct mapping_metadata_bhs *mmb); void invalidate_bh_lrus(void); -void invalidate_bh_lrus_cpu(void); -bool has_bh_in_lru(int cpu, void *dummy); extern int buffer_heads_over_limit; =20 #else /* CONFIG_BUFFER_HEAD */ @@ -528,8 +526,6 @@ static inline void buffer_init(void) {} static inline bool try_to_free_buffers(struct folio *folio) { return true;= } static inline int mmb_sync(struct mapping_metadata_bhs *mmb) { return 0; } static inline void invalidate_bh_lrus(void) {} -static inline void invalidate_bh_lrus_cpu(void) {} -static inline bool has_bh_in_lru(int cpu, void *dummy) { return false; } #define buffer_heads_over_limit 0 =20 #endif /* CONFIG_BUFFER_HEAD */ diff --git a/mm/folio.c b/mm/folio.c index d66aa2469e86..dbad66e805e2 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -681,21 +681,6 @@ void lru_add_drain(void) mlock_drain_local(); } =20 -/* - * It's called from per-cpu workqueue context in SMP case so - * lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on - * the same cpu. It shouldn't be a problem in !SMP case since - * the core is only one and the locks will disable preemption. - */ -static void lru_add_and_bh_lrus_drain(void) -{ - local_lock(&cpu_fbatches.lock); - lru_add_drain_cpu(smp_processor_id()); - local_unlock(&cpu_fbatches.lock); - invalidate_bh_lrus_cpu(); - mlock_drain_local(); -} - void lru_add_drain_cpu_zone(struct zone *zone) { local_lock(&cpu_fbatches.lock); @@ -711,7 +696,7 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain= _work); =20 static void lru_add_drain_per_cpu(struct work_struct *dummy) { - lru_add_and_bh_lrus_drain(); + lru_add_drain(); } =20 static bool cpu_needs_drain(unsigned int cpu) @@ -724,8 +709,7 @@ static bool cpu_needs_drain(unsigned int cpu) folio_batch_count(&fbatches->lru_move_tail) || folio_batch_count(&fbatches->lru_deactivate_file) || folio_batch_count(&fbatches->lru_deactivate) || - need_mlock_drain(cpu)) || - has_bh_in_lru(cpu, NULL); + need_mlock_drain(cpu)); } =20 /* @@ -824,6 +808,8 @@ static inline void __lru_add_drain_all(bool force_all_c= pus) } } =20 + invalidate_bh_lrus(); + for_each_cpu(cpu, &has_work) flush_work(&per_cpu(lru_add_drain_work, cpu)); =20 @@ -839,6 +825,7 @@ void lru_add_drain_all(void) void lru_add_drain_all(void) { lru_add_drain(); + invalidate_bh_lrus(); } #endif /* CONFIG_SMP */ =20 @@ -872,7 +859,7 @@ void lru_cache_disable(void) #ifdef CONFIG_SMP __lru_add_drain_all(true); #else - lru_add_and_bh_lrus_drain(); + lru_add_drain_all(); #endif } =20 --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx2-f12.google.com (mail-yx2-f12.google.com [74.125.224.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED4CE4F55BE for ; Wed, 9 Sep 2026 10:33:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949999; cv=none; b=mD/9oppUzNQA1o35elp26Q/4AbPrmgdjKaFQ18NvLNC2p2wjNuiTywt8/ujo+nai6J+v79ZH159+cjUoyc8iLYbU/jwajjiJkxsM7WurMDdavIFAlCBs+FoW9x3PYPXB72jhajZrtM7batBSOTijmqvrqnGkL/1ENWDa4IAShj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788949999; c=relaxed/simple; bh=ZvWWFyEwjUxkBs81Ot3sUxd33dtMx+r31MyIHQ1994E=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=VbsYQh0314wDLdJLjlnA7ip4MFt4whT5oVvWfT/yKNPRDVN6eN767nUYJ/9BGisSX17eSQlhPVO0dqoRG9N4IBg2i1kXwtXnokupgYWiV0FvLiXabZ7nRV3DyvF6iDv4LjcsShYylpLU0m7rQeoS81IHjX+pon1dwHE7+M4xcD8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QCnG7i4m; arc=none smtp.client-ip=74.125.224.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QCnG7i4m" Received: by mail-yx2-f12.google.com with SMTP id 00721157ae682-85d46e4cdcdso9727187b3.2 for ; Wed, 09 Sep 2026 03:33:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788949997; x=1789554797; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9XKIfsbwB2WTm2tTUcIYzMdzj76RrLfvaApBm113T1U=; b=QCnG7i4mq53r5LnAlbiV6CtpVnw/JuxmF3bJrKNXDm7Lbe29gAQb7nZ3YIRFWeqO6g NiXqMOGxiNfs6guNJefOLDwsd2a7SyZUsPRubvEklCBIwjNusqzcqexsPW1jxIb3c6zi x0SGys1LkOIxjnhcBZEzuKPdUqYwwsg16RcQFIhPKbO/cnful0vwmh3FGZyDxbkB07kR 3dfVYxodOJuq7uHuEg5tGeAo6DKRrv0KAJi2VMxYBkry9QpZAd5F86ywkgRqng7bYHHT JDDjQch1YO8lI0cRvQffL9gbPgnCH1/LJ7zeqUhyl4Lhf2FjVONxpR9z4s0LE/H92nS/ WDVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788949997; x=1789554797; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9XKIfsbwB2WTm2tTUcIYzMdzj76RrLfvaApBm113T1U=; b=jmdisye70jPh2CEk759dPuYqR8vXKX9pDFAlpcONE5sqOP5SqZHsLPOouEQwma+cy8 eghdam96wU6muJtm7LhhAgDHZqMkdFMxAQkXnZ7pNh3pz7wBY9ZSzzKdVAFbhjXm4wNH IhXFKnqngyHcxQ3RGOUeOYgjlgzHp7pr7NF6pg9mED+Jte36ysDM9YjJgoQxQjPRQRb+ sVkmfR9aSp4g4W7L8U3ewT2SgCEW37PEmmWPzrNUX6cQvDmqzmpYNNBzcD/MEeyumF7r blx4BcpKxP647252gJ6u2hrAbWMmwGtVpDS80X52m7xjnsUcwKMaBlccZQbkxpo/Bksk HnPQ== X-Forwarded-Encrypted: i=1; AKwUvByfb0NbvSmjJh4vWdeSY82FKoh5IaLmVyCkiNfOLhLwaN6m7eLKHDufe+HwM2Yi0dtX0ApCUAASl5clNNc=@vger.kernel.org X-Gm-Message-State: AFuF++kSzVrHqF+WXx+fU+FSrv43xzhIVYqKp/sQGJkxFdx1DLQm7eYn gEGanT5wRuEXjn5/SbDVClrq5QNQrTqgyEI8exCtyNdWw0N7kLu8sH/UA4cekHN1FA== X-Gm-Gg: AYBFou2/Pyj7T0fnXvWbFdFimWiesxJemQeSOYLFXhkVaxAiU6+8ee5C/QYx63yv7WA gVtVo3g4XadVTpIVl/4eJdEmochEBph4i3WDk5agG1mNVRjuQwM6paYTS9QXw9TS9sJn2Bml/TE klvnJBBYmXvVT5y23nS0q1pcEMoBejTTjHpCl55Czdfo+9Euiqwn02uWGofqV9ql3Gj5y+Y1+l+ tKc5G4tvA7Ufn+bpjReXWoE9FPB9MKnAGU9oAUftqrLu5etPZGnoENpgEAK91lNz/jPu8Jo6lIa 4v/8LLoEgnIVs2y32LELmK5dyJenhRET+COnKU1Wd2WPWvXDbJPlNmV9Dsw6bp18a4Do0yW8kH/ 4LAH4OqMNYcDGAEE7ng4PUVwSzcSUDpgUO2jQe5pu805a7x+U3hV3W+0z5ASKLTwLasOAh9E1HJ Zq/4Ozyo7k/lsFnwDI/s0YbridaZ7y42otUyE6yAVxuWoLRwc5dem5zTOa3H3aBBSDuzArv4IPF YRs37W0HQVz9sbBDDTIzh0Pi3czjnOhjEwZ0M2/d6ArjlmPJ1YL4pM= X-Received: by 2002:a05:690c:9:b0:836:ec9b:b468 with SMTP id 00721157ae682-87f2a0de993mr24243117b3.34.1788949996112; Wed, 09 Sep 2026 03:33:16 -0700 (PDT) Received: from [192.168.1.242] (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714b720d82sm107878567b3.41.2026.09.09.03.33.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:33:15 -0700 (PDT) Date: Wed, 9 Sep 2026 03:33:10 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 24/26] fs,mm/fbatch: lru_cache_disable() keep off buffer_head lrus only In-Reply-To: Message-ID: <4a8ab88a-689f-d4d1-adf9-2498a1b3246d@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the per-cpu fbatch folio references are gone, there seems to be no excuse for lru_cache_disable() there - other than offline_pages() needing to lru_add_drain_all() to erase stale pointers from the fbatches. Remove lru_cache_disabled() checks from all except bh_lru_install(): assuming that lru_cache_disable() might have value in preventing repeated calls to invalidate_bh_lrus() when migrating folios in memory hotremoval. So move all that from mm/folio.c to fs/buffer.c: but I can't see how any of the paranoid synchronize_rcu_expedited() business is needed now (or even before) - leave it out. And back at the mm end, lru_add_drain_all() does not need that force_all_cpus either - it stopped forcing all cpus in 5.18 commit ff042f4a9b05 ("mm: lru_cache_disable: replace work queue synchronization with synchronize_rcu"). Signed-off-by: Hugh Dickins --- fs/buffer.c | 24 +++++++++++++++--- include/linux/buffer_head.h | 4 +++ include/linux/swap.h | 7 ------ mm/folio.c | 49 ++++--------------------------------- mm/internal.h | 6 ----- mm/memory_hotplug.c | 4 +++ mm/mlock.c | 4 +-- 7 files changed, 36 insertions(+), 62 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 7d114e5b9c62..7455a11dfc4a 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1200,6 +1200,24 @@ static inline void check_irqs_on(void) #endif } =20 +static atomic_t lru_disable_count =3D ATOMIC_INIT(0); + +void lru_cache_disable(void) +{ + if (atomic_inc_return(&lru_disable_count) =3D=3D 1) + invalidate_bh_lrus(); +} + +static inline bool lru_cache_disabled(void) +{ + return atomic_read(&lru_disable_count); +} + +void lru_cache_enable(void) +{ + atomic_dec(&lru_disable_count); +} + /* * Install a buffer_head into this cpu's LRU. If not already in the LRU, = it is * inserted at the front, and the buffer_head at the back if any is evicte= d. @@ -1215,9 +1233,9 @@ static void bh_lru_install(struct buffer_head *bh) bh_lru_lock(); =20 /* - * the refcount of buffer_head in bh_lru prevents dropping the - * attached page(i.e., try_to_free_buffers) so it could cause - * failing page migration. + * The refcount of buffer_head in bh_lru prevents dropping the + * attached page (i.e., try_to_free_buffers), so it could cause + * repeated calls to invalidate_bh_lrus() during page migration. * Skip putting upcoming bh into bh_lru until migration is done. */ if (lru_cache_disabled() || cpu_is_isolated(smp_processor_id())) { diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index f19f9e80be8f..3b6a41b7932f 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -517,6 +517,8 @@ void mmb_init(struct mapping_metadata_bhs *mmb, struct = address_space *mapping); bool mmb_has_buffers(struct mapping_metadata_bhs *mmb); void mmb_invalidate(struct mapping_metadata_bhs *mmb); int mmb_sync(struct mapping_metadata_bhs *mmb); +void lru_cache_disable(void); +void lru_cache_enable(void); void invalidate_bh_lrus(void); extern int buffer_heads_over_limit; =20 @@ -525,6 +527,8 @@ extern int buffer_heads_over_limit; static inline void buffer_init(void) {} static inline bool try_to_free_buffers(struct folio *folio) { return true;= } static inline int mmb_sync(struct mapping_metadata_bhs *mmb) { return 0; } +static inline void lru_cache_disable(void) {} +static inline void lru_cache_enable(void) {} static inline void invalidate_bh_lrus(void) {} #define buffer_heads_over_limit 0 =20 diff --git a/include/linux/swap.h b/include/linux/swap.h index 0052a6890435..423d9b669149 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -314,13 +314,6 @@ static inline void lru_cache_drain_for_folio(const str= uct folio *folio, /* linux/mm/folio-compat.c */ void mark_page_accessed(struct page *page); =20 -extern atomic_t lru_disable_count; - -static inline bool lru_cache_disabled(void) -{ - return atomic_read(&lru_disable_count); -} - extern unsigned long shrink_all_memory(unsigned long nr_pages); long remove_mapping(struct address_space *mapping, struct folio *folio); =20 diff --git a/mm/folio.c b/mm/folio.c index dbad66e805e2..55ca799a9dcd 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -173,7 +173,7 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, local_lock(&cpu_fbatches.lock); =20 if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + !folio_may_be_lru_cached(folio)) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 if (disable_irq) @@ -434,7 +434,7 @@ void __folio_add_lru(struct folio *folio, bool mlockit) smp_mb__before_atomic(); folio_set_lru(folio); =20 - if (full || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + if (full || !folio_may_be_lru_cached(folio)) folio_batch_move_lru(fbatch, lru_add); =20 local_unlock(&cpu_fbatches.lock); @@ -719,7 +719,7 @@ static bool cpu_needs_drain(unsigned int cpu) * Calling this function with cpu hotplug locks held can actually lead * to obscure indirect dependencies via WQ context. */ -static inline void __lru_add_drain_all(bool force_all_cpus) +void lru_add_drain_all(void) { /* * lru_drain_gen - Global pages generation number @@ -743,7 +743,7 @@ static inline void __lru_add_drain_all(bool force_all_c= pus) if (WARN_ON(!mm_percpu_wq)) return; =20 - trace_mm_lru_add_drain_all_tp(force_all_cpus); + trace_mm_lru_add_drain_all_tp(false); =20 /* * Guarantee folio_batch counter stores visible by this CPU @@ -770,7 +770,7 @@ static inline void __lru_add_drain_all(bool force_all_c= pus) * (C) Exit the draining operation if a newer generation, from another * lru_add_drain_all(), was already scheduled for draining. Check (A). */ - if (unlikely(this_gen !=3D lru_drain_gen && !force_all_cpus)) + if (unlikely(this_gen !=3D lru_drain_gen)) goto done; =20 /* @@ -816,11 +816,6 @@ static inline void __lru_add_drain_all(bool force_all_= cpus) done: mutex_unlock(&lock); } - -void lru_add_drain_all(void) -{ - __lru_add_drain_all(false); -} #else void lru_add_drain_all(void) { @@ -829,40 +824,6 @@ void lru_add_drain_all(void) } #endif /* CONFIG_SMP */ =20 -atomic_t lru_disable_count =3D ATOMIC_INIT(0); - -/* - * lru_cache_disable() needs to be called before we start compiling - * a list of folios to be migrated using folio_isolate_lru(). - * It drains folios on LRU cache and then disable on all cpus until - * lru_cache_enable is called. - * - * Must be paired with a call to lru_cache_enable(). - */ -void lru_cache_disable(void) -{ - atomic_inc(&lru_disable_count); - /* - * Readers of lru_disable_count are protected by either disabling - * preemption or rcu_read_lock: - * - * preempt_disable, local_irq_disable [bh_lru_lock()] - * rcu_read_lock [rt_spin_lock CONFIG_PREEMPT_RT] - * preempt_disable [local_lock !CONFIG_PREEMPT_RT] - * - * Since v5.1 kernel, synchronize_rcu() is guaranteed to wait on - * preempt_disable() regions of code. So any CPU which sees - * lru_disable_count =3D 0 will have exited the critical - * section when synchronize_rcu() returns. - */ - synchronize_rcu_expedited(); -#ifdef CONFIG_SMP - __lru_add_drain_all(true); -#else - lru_add_drain_all(); -#endif -} - /** * folios_put_refs - Reduce the reference count on a batch of folios. * @folios: The folios. diff --git a/mm/internal.h b/mm/internal.h index 6700fff13675..d9930b752153 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -53,12 +53,6 @@ static inline bool folio_may_be_lru_cached(const struct = folio *folio) return !folio_test_large(folio); } =20 -static inline void lru_cache_enable(void) -{ - atomic_dec(&lru_disable_count); -} - -void lru_cache_disable(void); void lru_add_drain(void); void lru_add_drain_cpu(int cpu); void lru_add_drain_cpu_zone(struct zone *zone); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 226ab9cb078a..19756c45b5e7 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -2095,6 +2096,9 @@ int offline_pages(unsigned long start_pfn, unsigned l= ong nr_pages, =20 } while (ret); =20 + /* Remove any instances of the freed pages from per-cpu fbatches. */ + lru_add_drain_all(); + /* Mark all sections offline and remove free pages from the buddy. */ managed_pages =3D __offline_isolated_pages(start_pfn, end_pfn); pr_debug("Offlined Pages %ld\n", nr_pages); diff --git a/mm/mlock.c b/mm/mlock.c index 971430e6251e..a3cfdb274fc7 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -247,7 +247,7 @@ void mlock_folio(struct folio *folio) local_lock(&mlock_fbatch.lock); fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); if (!folio_batch_add(fbatch, mlock_flagged(folio)) || - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + !folio_may_be_lru_cached(folio)) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); } @@ -278,7 +278,7 @@ void munlock_folio(struct folio *folio) local_lock(&mlock_fbatch.lock); fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); if (!folio_batch_add(fbatch, folio) || - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + !folio_may_be_lru_cached(folio)) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); } --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 272465221C9 for ; Wed, 9 Sep 2026 10:35:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788950152; cv=none; b=S1IYCVPDvGcwrwBB6qHCmqNZDLVUpAFJe/nabEcg7RhuME1Z+lcWG+uAxHULUoaze5eskUJi/TMp6IZ9SCojJCa9hbreR5XQq9pU8ohqrl8IZzXxTC4mmXwXOq+gAW6wIH7X/531d1J4nhzZgFd3FhJ6btryqNpLP1XlZmbJGtA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788950152; c=relaxed/simple; bh=RILLo1OM8wUjyazDx0uYb0ZEOlSa7QCNuPJs8t0VB8g=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=hDD9pw8j0EIjSGWp4A+KuWHupNfksZKSOzqxc+XIYvVABRMSs2K5jp8IsuPEJ79NuIrBykkh1v8qGkvJyswImKnCSMXkarzpfX/OUreQhSwMoXnbPwKCaPPntCjvxaewc8xxsnOwG/1b0cx5vidKDcu1xJmgPra/CH9xPrOVkkg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Y7JxXRY9; arc=none smtp.client-ip=209.85.128.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Y7JxXRY9" Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-861a2ae9c51so67081607b3.0 for ; Wed, 09 Sep 2026 03:35:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788950150; x=1789554950; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=01Cz/m/rw5yJ2Ele+bEMQyVaRjGaQszVRRuNtZjuAys=; b=Y7JxXRY9lDl6xdfun4kcEv86OTrEQM55JJNMOAS8dS9x5sWlAJSxDXreUGfqQPbcED XLQ9/6f3pvZLbK708V2enROPzlPuXEvHEfQ/GnIh82qtIw7Lo1M7yFN7pJJRDk66KwNM WImd6FUjuy/FxntUCo9elqCDafrWOPGP2ies8hazChJ7X79X9kFm3cTDSdPFAWQVWhHS xidR9NaO0mxL8ydAVtn/4c7S2u2ZbV/c1lGaUjhNJby3ij170xuebuagtoMCx3hY6wJc fqa64N/b1umfpyLtF2U6dMZCE393S8+Thee29sMSB5YV5tLMSvUGQA7Z/cjxlFHJ11Lb cwTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788950150; x=1789554950; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=01Cz/m/rw5yJ2Ele+bEMQyVaRjGaQszVRRuNtZjuAys=; b=fa8RauNe3yWgEvXRxLpC/MCl6LZhTq7BRcPTkR6qbsRpHWnuf8n/mXcyApL81RvugX cn/qc+Z8+gTIsBEBtYKX6i1vC0RHgXt/JAk1BSVLxVV42GdyxbQ/UOEsKca0UyhwOWRS R3DZJA7m3XG+VcFnH2F71jPzBtnI0a/1YFtRLSLeovVMKJIf8ptBhBQ1Pu+R1WPAP7bR //pPj6VBQBwAVcURUgB/SVg7dg+felq6hmVJJsAStE/6YzHgNELsfYRl/aY0OoFEkpYy oJNJMbzt9OT07tPZ5nz1kk81G6tO6evxsaZh2N2JDX+/I46IdhPT71PgafUZsjf4yMWl F9tQ== X-Forwarded-Encrypted: i=1; AKwUvBxGLg+Cl+L7DPngMtjS8I+aDYm7lZMDcwndVFd5xPq0ePKJrAcL41HSrOdcPj7a3rnihNvT1IFp+qpsCwo=@vger.kernel.org X-Gm-Message-State: AFuF++nXf9pS3Hye+HQrLVMoDQv4O/YfnJG0eKQF/6MXyUPlqXjAMM5O TSHG7tQm6iTv49GcD9u2s2IDwX83iuEmZ0ofnm8jliC8mlaf64q6Yv9lK9sN8yJPpA== X-Gm-Gg: AYBFou363dGz8Fj1unjiC757lsrwPndu+vDvaSQs66wZtIB07kVJU9oAGNmNiCIcl6n LgFS/RT9H+remHyyP/00H6mPs1G8Eu73Sav/MtrJOhKVPfn27O0mhJhq2bjlRfXAtXiN2gbhrHs CG0YvUL1eX0p5aCe1ObhtcpdTc5+qso4D1PQXuHuW2FuyjjHO4f4sW7O+7llU3GJ2oozRD6PE6D yynt7dEXj1ngHsk+3HquUHz+9xRzKhY0Aj3m/phhYpF5GpGtlaQvwu4JVk4g6XKzJkX3+qyCxYo WUMz7RJkVJkz0a3o22VbsZ94qcQCuO52d+at5+lNm1pr7rxMnqfpoKWmbdc7m7G2msiSgDyYgyA FOMdOopkb9w3WoMAxTbRg38AHkqm601TLjtjAGcjV6I3lTfZYs7BYZ3sYVYmfR6kSg9VXqBugr0 dG09mU4i+odMeprNfSPfFhkazR45YbnywhhOQcrDYZ69SLRea7sTKbkr6H+0vNOawMMqJ4LGz1Z PeeHABwUGw+Ru7oAJAURPSjk0WtkMlpLtMz/yQ6ugxTdpTI7GOI6axB7RpwgJQi X-Received: by 2002:a05:690c:c50d:b0:873:ea3b:6c08 with SMTP id 00721157ae682-873ea3b77c9mr91382717b3.1.1788950148979; Wed, 09 Sep 2026 03:35:48 -0700 (PDT) Received: from [192.168.1.242] (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-87143ca6ddesm107684497b3.2.2026.09.09.03.35.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:35:47 -0700 (PDT) Date: Wed, 9 Sep 2026 03:35:42 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 25/26] mm/fbatch: move lru_add_drain_all() declaration to mm/internal.h In-Reply-To: Message-ID: <26b1d94a-497a-fd6d-e700-342ccb65c045@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Only mm now uses lru_add_drain_all(), in forced reclaim and in memory hotremoval: help to keep it that way by moving its declaration alongside that of lru_add_drain() in mm/internal.h. Signed-off-by: Hugh Dickins --- include/linux/swap.h | 1 - mm/internal.h | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 423d9b669149..dfa6314c0cee 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -300,7 +300,6 @@ static inline void folio_add_lru(struct folio *folio) __folio_add_lru(folio, false); } void folio_mark_accessed(struct folio *folio); -void lru_add_drain_all(void); =20 enum lru_cache_drained { LRU_CACHE_NOT_DRAINED, diff --git a/mm/internal.h b/mm/internal.h index d9930b752153..f28fd310455c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -54,6 +54,7 @@ static inline bool folio_may_be_lru_cached(const struct f= olio *folio) } =20 void lru_add_drain(void); +void lru_add_drain_all(void); void lru_add_drain_cpu(int cpu); void lru_add_drain_cpu_zone(struct zone *zone); void folio_deactivate(struct folio *folio); --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yx1-f54.google.com (mail-yx1-f54.google.com [74.125.224.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA88751FCD0 for ; Wed, 9 Sep 2026 10:38:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788950284; cv=none; b=qa83STqSAzP23tUyXTijSjg9iM8kN5mPQu377ivJyHJiySG1cJIH2hnwOJ0GNDuTQxIpKF9INxzfYojzUNOtB6YHJ7CX+LAIwsn93gyqBYBeILPKR4Zygcc6MZI5hFMGZcZW26e0a8DGcCb4VKKyfww12R1sKYhlXEAN2ZyNQ/o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788950284; c=relaxed/simple; bh=G/+0ifcYc6SQaMip/fcKwW0z1nfyT6dvA3rvG1KtE7g=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=M+NQQLf/B9N2JZEX+08GQBbHNmlzBFvXBH0YVO/PaXF9i7o0c9GJCKL4VeSaj/95L2sURrOLA7TVleYsX5Vzu3e1Sz+4aojukSym0BkdjxqH7JJA6VulFqVNqAGX15cz9nUgHSoiQJ47uX2EhYICWTI8kwcGCarfU67o5zn7X88= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dlladz9v; arc=none smtp.client-ip=74.125.224.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dlladz9v" Received: by mail-yx1-f54.google.com with SMTP id 956f58d0204a3-66fdf2a9aacso3725248d50.0 for ; Wed, 09 Sep 2026 03:38:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788950282; x=1789555082; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ubgJuILilCUv+JDm5jbfScMjR0I7p8athUKDiI1DXU4=; b=dlladz9v903VmREmuBo/W8xz9HvP2F4cjlFXBE7pBWEhJnjx0VhA45vaIX64TS0YUA 8R63dwujaHs0glLZ6fMrR908UpfZxOgzlu64ik9WDs6OEPZiuTNHYTaPxvA4HoCPPFFc ui/keBGH/qlewdQIP08QxKbElMaN2urO0KbcRcaKnxcuNl6kFqSuqNhr1T+2qys7syKs 19aLGiDvTzd7nIklqcmAsgqICRMtmVcmJHH50PvRxkESYxtm+SGw+4o9u7r5hiM3alRw 1O5uAV/vA90QN5RGlzxDIkzec2aAzwjVWjihT0T5hMdABoZNgn6NcN+isZaKKM36IuYS Ebgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788950282; x=1789555082; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ubgJuILilCUv+JDm5jbfScMjR0I7p8athUKDiI1DXU4=; b=Py/HYZ0PzbKCs3+xlb/1LAzS2t8IBkluDeYKc2InvBKfOe76lYrkgIqnmH23piKJNA IXBDWPfZ3EDO++eCYKd6BRt+RdsDlsFFOUObUrgAWye8lVoGphed66pVdq6Qr4mE1hpS Bw5xgXnLB15Vl7CKEUm6Oste1tJfpJ1NYE18O0f+s9/etoNRUCpjYqKRIbYxN5CJQmFj 85hXITBuhBXUlnz0dIcihLXT8Xd34VdrXME+G4xDbiv6DmzmDuEqnAK7FAO7pNzZTgRt XtAcDc2Q11hjIJa0WbMFylPiDpNyWx3XEox4AP8i9GX/UCuaUebY6AASN897n4Ev4Xf9 EWaQ== X-Forwarded-Encrypted: i=1; AKwUvBxE37zVEQRngTD4psMNk/EUqbKtGJuxxGdMBVqCS4gzYTYkAkl0yHDqClerIqA6SHLWq3dBuq3CeppkLXI=@vger.kernel.org X-Gm-Message-State: AFuF++myM6TeKA5qxVSH+DobixklfYb0lS7NyY2uaHpXvUPTsfvo3TPE MHe54dXmQEwCxvsF1mwfW8gobZPzm2QOu8JzWNI0iMVC/J8EZFb1Jsp73rUyX+F4bA== X-Gm-Gg: AYBFou1v7kZ0jtqxJew+Nh1UR5WCR5jLuUehTqX0oVfItDSgVkH1sFQ0Ct42F7G4UFH CgdcU8+OF5RVrgwOvMCRIREBclxsP4W5vnQH2/42yF5HZqQkwMXYhLRo8or52+G/Laa+jpEUlP7 kmxSgcYBha0H+J0njYhmQ3oWsQGF4tlcHTP4qBsU867IW1RSHLNVeaiPfL6eTOE0z4SCgvsZDJS kLNXlM/y2O0nm4+5GcOo3jwn5iPQOJWmfr3aPcdn1+SsVCarq2LeNVJaxPs0uMRP3dLjqirPHvu et2R0QxpXjQG1HFNSclcUacXugyc/C6hbSN1hAE0LGY08alrqyLeq0YjcsHDyGTyeZgUFmCG1Hn 79+sVmqJSjFHYMJ5SknCp+N7mxrN2zi9EgZMaSz50d2RtI5E2EABVAVPKbhen+RgDwWC+qPwdDw S5JijysuTrh+MXWYzBiQK0arDCBDS2WAB/qSaPViRymGKGYhpRxySqneBnd+Ua14zp0QG/4seb+ DwN31iMAVb78A5RJjfLNunEgCXzU0gwKtt2ZvzQRX8Xwnj88BrIlw== X-Received: by 2002:a53:a6c1:0:b0:66f:79dd:19b4 with SMTP id 956f58d0204a3-66fb5a81fd0mr8721653d50.33.1788950280907; Wed, 09 Sep 2026 03:38:00 -0700 (PDT) Received: from [192.168.1.242] (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66fb49763bfsm12116197d50.19.2026.09.09.03.37.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:37:59 -0700 (PDT) Date: Wed, 9 Sep 2026 03:37:55 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 26/26] mm/fbatch: drop reference inside the loop when draining In-Reply-To: Message-ID: <5af5eb5a-2a18-268f-d86b-5dd079a748be@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" folio_batch_move_lru() and mlock_folio_batch() used folios_put_refs() after their loop: but that's counter-productive, to batch up dropping all the references acquired within the loop. Now folio_put_testzero() inside the loop, where we also already hold (bar races) the right lock to remove the folio from its lruvec. This should be much friendlier to compaction, in the common case when the folio is still in use, to bring it back to expected refs sooner; but I suspect that when the folio is freed, compaction won't accept it until it gets to be PageBuddy later on? Respect the comment in mm/vmscan.c move_folios_to_lru(): could be done differently, but it's not worth optimizing the case when we cleared lru, since it also has to handle the case when we failed to clear it. Signed-off-by: Hugh Dickins --- mm/folio.c | 23 +++++++++++++++++------ mm/mlock.c | 27 +++++++++++++++++++++------ 2 files changed, 38 insertions(+), 12 deletions(-) diff --git a/mm/folio.c b/mm/folio.c index 55ca799a9dcd..7c545e4c090d 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -128,20 +128,18 @@ static void lru_add(struct lruvec *lruvec, struct fol= io *folio) =20 static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t mov= e_fn) { - int i; + int i, j =3D 0; struct lruvec *lruvec =3D NULL; unsigned long flags =3D 0; =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { struct folio *folio =3D fbatch->folios[i]; =20 - if (!folio_try_get(folio)) { - fbatch->folios[i] =3D NULL; + if (!folio_try_get(folio)) continue; - } =20 if (!folio_test_clear_lru(folio)) - continue; + goto restored_lru; =20 /* Do not add to LRU if it has already been added */ if (move_fn =3D=3D lru_add && !lru_add_del_folio(folio)) @@ -155,11 +153,24 @@ static void folio_batch_move_lru(struct folio_batch *= fbatch, move_fn_t move_fn) lruvec_add_folio(lruvec, folio); restore_lru: folio_set_lru(folio); + /* See mm/vmscan.c move_folios_to_lru() comment on ordering */ +restored_lru: + if (unlikely(folio_put_testzero(folio))) { + folio_unqueue_deferred_split(folio); + __page_cache_release(folio, &lruvec, &flags); + fbatch->folios[j++] =3D folio; + } } =20 if (lruvec) lruvec_unlock_irqrestore(lruvec, flags); - folios_put_refs(fbatch, NULL); + if (unlikely(j)) { + fbatch->nr =3D j; + mem_cgroup_uncharge_folios(fbatch); + free_unref_folios(fbatch); + } else { + folio_batch_reinit(fbatch); + } } =20 static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch, diff --git a/mm/mlock.c b/mm/mlock.c index a3cfdb274fc7..c528cf7135bf 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -27,6 +27,7 @@ #include =20 #include "internal.h" +#include "page_alloc.h" =20 struct mlock_fbatch { local_lock_t lock; @@ -170,28 +171,42 @@ static void mlock_folio_batch(struct folio_batch *fba= tch) struct lruvec *lruvec =3D NULL; unsigned long mlock; struct folio *folio; - int i; + int i, j =3D 0; =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { folio =3D fbatch->folios[i]; mlock =3D (unsigned long)folio & MLOCK_FLAG; folio =3D (struct folio *)((unsigned long)folio - mlock); - fbatch->folios[i] =3D folio; =20 - if (!folio_try_get(folio)) { - fbatch->folios[i] =3D NULL; + if (!folio_try_get(folio)) continue; - } =20 if (mlock) lruvec =3D __mlock_folio(folio, lruvec); else lruvec =3D __munlock_folio(folio, lruvec); + + if (unlikely(folio_put_testzero(folio))) { + folio_unqueue_deferred_split(folio); + /* __page_cache_release() without irqflags */ + if (folio_test_lru(folio)) { + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + lruvec_del_folio(lruvec, folio); + __folio_clear_lru_flags(folio); + } + fbatch->folios[j++] =3D folio; + } } =20 if (lruvec) lruvec_unlock_irq(lruvec); - folios_put_refs(fbatch, NULL); + if (unlikely(j)) { + fbatch->nr =3D j; + mem_cgroup_uncharge_folios(fbatch); + free_unref_folios(fbatch); + } else { + folio_batch_reinit(fbatch); + } } =20 void mlock_drain_local(void) --=20 2.51.0 From nobody Fri Sep 25 19:19:53 2026 Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16052521223 for ; Wed, 9 Sep 2026 10:41:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788950469; cv=none; b=qidVUgGzSM43/yq+b2aAGZd5FKcsqYWbv+CPiEfQcTQe4x6MSVlHLOERc6wj99coE2InxyzA4bn85bYrio5rzZ/6krjvB95q5miA0RdLgwkIyrdj0iVyQSP2IcOZ8nbZLa7sufamw641xYMYMk2oYfAr2B3XWepIG2o8X5OmyEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788950469; c=relaxed/simple; bh=wCi3pUhWZvq93WjLW4ifiEYeMwDAkqD4CUEmoBl4gEM=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=RZi4A6kKDEzngNB4Dobf50rPmtYVK1ZJqqUIUOtGnAIgvthWKS77gMc9srTMMuYVAtD5UrBf8v7BHrgyEAytUxUkApVkS65TPcp8oKPFXpptoDUr5K2EvhJGTYrSZfBU29YmCpN9A2BKuyLJN3EoaSEPB+8yZ+xEmG+3QRDo0BI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uFQXFG0h; arc=none smtp.client-ip=209.85.128.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uFQXFG0h" Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-878ed0f17bfso34330427b3.1 for ; Wed, 09 Sep 2026 03:41:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788950467; x=1789555267; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=PGN+ybR37+qvtDqf9cFN47582j44el3q/H5/GSGFvrU=; b=uFQXFG0hXoimqiWby6dfpdSvwp1ywbxkEFfD0FeBMdu6eVcVqX/qatRdy77+gn1t28 pr3ZJH6p0J62kW32d+tHDLRyOhkmZTfuuM3+aI5rapDZserz+Tnrlkf+ZySE48FhYCtr dNzHT8cG7CB++qH1YjPu0SEFSiDn7z2fU4bqPNI6ym0ohv0YhN3m/4v8A/UVzxCO8iXf opoWMkuRstsNVFOQsaGR6flUmhiIQ93WPekU7RlOjR8zRh4AhDkKbI6dkHFA/Ic6uhYI /wvnbtUiVr9xbHNaCoFf+PEN+tG44AJHK0+HlLZ+FMG/tCQsQL/0S+MmOIJyyOU5ARYT GGkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788950467; x=1789555267; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=PGN+ybR37+qvtDqf9cFN47582j44el3q/H5/GSGFvrU=; b=DfA6aTKH6ktHdmMlPchryrmw0fkGqeT8dwPhjNZ8/k6GNzyod16yHuodpmX5q8F5O7 koTjx25HjXHkwdBjAnd+bCq+hHMn01KtHP9qY+2uZhE3VK8tavCqwI6qo3PEX4dbNChy BUb1Bj0DGBfFPd6GsMTDhLWsLLRuQPtOr+uzPHdHZiTGY/qYrhCh6FQz+hkXLUVBoCLF 3beXlu591cAjonULs4FGV0zTaCJ5gltPrFl6cZYkzdrHXJzwLxuPb6dnkoUi3BsqqNYB Ox8zJAOORM8Z34eqQtvBB15n49A7l5IbE+s12A4VA8ibrY3zKXS5T0sQAb+/WEHpwpV0 OtKQ== X-Forwarded-Encrypted: i=1; AKwUvBxtvIp78ZNhIeYChB0GHCcopwOsWYmVLLEfYaMR5mw6MAnpzecJR52+VYYUEiS5povN6eqz6z3bPxMUmoM=@vger.kernel.org X-Gm-Message-State: AFuF++lk0hI6FMkN21eerlEMmrYOVfuLfzQkbYQ5O4XApBhBsZcPMqmP l8J9m09vTjqZKeXHOmJHcSJj/OISbpBh/C+Oqn1wZIHN13IFvwqDz/IqHNB6td1kjQ== X-Gm-Gg: AYBFou31PIxe8kRr8uVHmkZYYctEjmIeO9if+bX2CUQt4NX6deDSfqwxmXmW6OMZneP RLYaL5PKJXOs4LiM2+A/QG0CQTcjj+yzWrFaxEN7PDF1L6K0PyXd0iNXBgaViJE8eKvr1KfkF7m g/eNA2LN58jM7EIoamCsz4XeYuHRXgLcjfwld/L/2GfOiqGQB7ulqbt5HrNOVTK2vbRO+Pp8OE1 IxuDTn1q8wf+JKYzZyuIgf9z+28UjhnTnjZ93A61pp86/BSraXiyZELyDbMj9k4X6kpOjGWmI11 ira/TxWTLuBKdYzV5t71lZ7u5ydslwkJI6c/MhsCBYZUCb+0H5oP816AJe2wf02aJHYREX1WJ64 XoUxbUntG2LQ5pB5S6h4TgDF3FB0sbWaG0WXhrO2BD2xX6NyGIc+p7+qsgnn/tO1/uuf4uDOETX XRkTt95nIZiQZM8Bzbrcv9XXaO92oQZHrPYT2nB4FmZ3IvYWUAoDQa/igKLnv7D8xvgRpv+qHCK LA4Zf6A7DRTj13vTA0pQ7wBZ8kS1Ta3pXP7bX4EEELEqhimlRd4BPmrVaQ= X-Received: by 2002:a05:690c:e3e8:b0:81e:9826:942c with SMTP id 00721157ae682-8711c7488c4mr132269777b3.0.1788950466181; Wed, 09 Sep 2026 03:41:06 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8714ab77c4dsm108323097b3.34.2026.09.09.03.41.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2026 03:41:05 -0700 (PDT) Date: Wed, 9 Sep 2026 03:41:01 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 27/26] mm/fbatch: paranoid folio vmstats in folio_batch_move_lru() In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Save mapping and index when adding folio to per-cpu fbatch, check them when draining the fbatch, show folio counts at the end of /proc/vmstat when CONFIG_DEBUG_VM=3Dy. This is a work-in-progress hack, not intended for kernel release: hence the 27/26 leaving it out of the series; but a useful framework for study and reassurance on stale fbatch entries. Could be extended to the mlock_fbatch, but probably nothing new to learn there. May be more useful if separate stats kept for the lru_add fbatch and (together) the other four - their characteristics are different. Needs refinement: the current numbers raise more questions than give answers. The folio_hoisted, folio_changed, folio_already numbers mostly record nothing worse than LRU-eligible folio pages getting freed before drain, and soon recycled for fresh LRU-eligible folios: no harm there; but these stats do not verify that. For example, after hours of kernel builds on tmpfs while swapping: folio_not_got 65750323 folio_off_lru 357827 folio_hoisted 10458363 folio_matched 2888574439 folio_swapped 113626282 folio_changed 11558942 folio_already 12717475 folio_skipped 705 Signed-off-by: Hugh Dickins --- include/linux/vm_event_item.h | 10 ++++ mm/folio.c | 101 +++++++++++++++++++++++++++++++--- mm/vmstat.c | 10 ++++ 3 files changed, 112 insertions(+), 9 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 2628ccda076a..4572463fb65f 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -179,6 +179,16 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NRSWPIN, NRSWPOUT, #endif /* CONFIG_SWAP */ +#ifdef CONFIG_DEBUG_VM + FOLIO_NOT_GOT, + FOLIO_OFF_LRU, + FOLIO_HOISTED, + FOLIO_MATCHED, + FOLIO_SWAPPED, + FOLIO_CHANGED, + FOLIO_ALREADY, + FOLIO_SKIPPED, +#endif /* CONFIG_DEBUG_VM */ NR_VM_EVENT_ITEMS }; =20 diff --git a/mm/folio.c b/mm/folio.c index 7c545e4c090d..9f0cb755595a 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -40,19 +40,35 @@ #define CREATE_TRACE_POINTS #include =20 +#ifdef CONFIG_DEBUG_VM +#define PARANOID_SIZE FOLIO_BATCH_SIZE +#else +#define PARANOID_SIZE 0 +#endif + +struct mapix { + struct address_space *mapping; + pgoff_t index; +}; + struct cpu_fbatches { /* - * The following folio batches are grouped together because they are prot= ected - * by disabling preemption (and interrupts remain enabled). + * The following folio batches are grouped together because they are + * protected by disabling preemption (and interrupts remain enabled). */ local_lock_t lock; struct folio_batch lru_add; + struct mapix mapix0[PARANOID_SIZE]; struct folio_batch lru_activate; + struct mapix mapix1[PARANOID_SIZE]; struct folio_batch lru_deactivate_file; + struct mapix mapix2[PARANOID_SIZE]; struct folio_batch lru_deactivate; + struct mapix mapix3[PARANOID_SIZE]; /* Protecting the following batches which require disabling interrupts */ local_lock_t lock_irq; struct folio_batch lru_move_tail; + struct mapix mapix4[PARANOID_SIZE]; }; =20 static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) =3D { @@ -60,6 +76,57 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches)= =3D { .lock_irq =3D INIT_LOCAL_LOCK(lock_irq), }; =20 +#if PARANOID_SIZE +static void paranoid_count(enum vm_event_item folio_counter) +{ + count_vm_event(folio_counter); +} + +static void paranoid_save_mapix(struct folio_batch *fbatch) +{ + int i =3D fbatch->nr - 1; /* folio has just been added */ + struct mapix *mapix; + + mapix =3D (struct mapix *)(fbatch->folios + ARRAY_SIZE(fbatch->folios)); + mapix +=3D i; + mapix->mapping =3D fbatch->folios[i]->mapping; + mapix->index =3D fbatch->folios[i]->index; +} + +static void paranoid_check_mapix(struct folio_batch *fbatch, int i) +{ + struct folio *folio =3D fbatch->folios[i]; + enum vm_event_item folio_counter; + struct mapix *mapix; + unsigned long lru_next =3D READ_ONCE(folio->lru_next); + + if (lru_next & BIT(LRU_NEXT_BATCHED)) { + lru_next ^=3D (unsigned long)&fbatch->folios[i]; + if (lru_next & ~(BIT(NR_LRU_NEXT_FLAGS) - 1)) + paranoid_count(FOLIO_HOISTED); + } + + mapix =3D (struct mapix *)(fbatch->folios + ARRAY_SIZE(fbatch->folios)); + mapix +=3D i; + if (folio->mapping =3D=3D mapix->mapping && folio->index =3D=3D mapix->in= dex) { + folio_counter =3D FOLIO_MATCHED; + } else if (!folio->mapping || !mapix->mapping) { + /* Truncated? Swapcache? Needs more cleverness */ + if (folio->index =3D=3D mapix->index) + folio_counter =3D FOLIO_MATCHED; + else + folio_counter =3D FOLIO_SWAPPED; + } else { + folio_counter =3D FOLIO_CHANGED; + } + paranoid_count(folio_counter); +} +#else +static void paranoid_count(int event) { } +static void paranoid_save_mapix(struct folio_batch *fbatch) { } +static void paranoid_check_mapix(struct folio_batch *fbatch, int i) { } +#endif + static void __page_cache_release(struct folio *folio, struct lruvec **lruv= ecp, unsigned long *flagsp) { @@ -135,22 +202,32 @@ static void folio_batch_move_lru(struct folio_batch *= fbatch, move_fn_t move_fn) for (i =3D 0; i < folio_batch_count(fbatch); i++) { struct folio *folio =3D fbatch->folios[i]; =20 - if (!folio_try_get(folio)) + if (!folio_try_get(folio)) { + paranoid_count(FOLIO_NOT_GOT); continue; + } =20 - if (!folio_test_clear_lru(folio)) + if (!folio_test_clear_lru(folio)) { + paranoid_count(FOLIO_OFF_LRU); goto restored_lru; + } + + paranoid_check_mapix(fbatch, i); =20 /* Do not add to LRU if it has already been added */ - if (move_fn =3D=3D lru_add && !lru_add_del_folio(folio)) + if (move_fn =3D=3D lru_add && !lru_add_del_folio(folio)) { + paranoid_count(FOLIO_ALREADY); goto restore_lru; + } =20 folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); =20 /* Do add to LRU if not already there (move_fn skipped) */ - if (lru_add_del_folio(folio)) + if (lru_add_del_folio(folio)) { lruvec_add_folio(lruvec, folio); + paranoid_count(FOLIO_SKIPPED); + } restore_lru: folio_set_lru(folio); /* See mm/vmscan.c move_folios_to_lru() comment on ordering */ @@ -176,16 +253,21 @@ static void folio_batch_move_lru(struct folio_batch *= fbatch, move_fn_t move_fn) static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch, struct folio *folio, move_fn_t move_fn, bool disable_irq) { + struct folio_batch *cpu_fbatch; unsigned long flags; + bool full; =20 if (disable_irq) local_lock_irqsave(&cpu_fbatches.lock_irq, flags); else local_lock(&cpu_fbatches.lock); =20 - if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || - !folio_may_be_lru_cached(folio)) - folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); + cpu_fbatch =3D this_cpu_ptr(fbatch); + full =3D !folio_batch_add(cpu_fbatch, folio); + paranoid_save_mapix(cpu_fbatch); + + if (full || !folio_may_be_lru_cached(folio)) + folio_batch_move_lru(cpu_fbatch, move_fn); =20 if (disable_irq) local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags); @@ -440,6 +522,7 @@ void __folio_add_lru(struct folio *folio, bool mlockit) } =20 full =3D !folio_batch_add(fbatch, folio); + paranoid_save_mapix(fbatch); =20 /* Ensure folio->lru_next visible to folio_test_clear_lru() callers */ smp_mb__before_atomic(); diff --git a/mm/vmstat.c b/mm/vmstat.c index d66cb67f55ce..cc61ac10782a 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1507,6 +1507,16 @@ const char * const vmstat_text[] =3D { [I(NRSWPIN)] =3D "nrswpin", [I(NRSWPOUT)] =3D "nrswpout", #endif /* CONFIG_SWAP */ +#ifdef CONFIG_DEBUG_VM + [I(FOLIO_NOT_GOT)] =3D "folio_not_got", + [I(FOLIO_OFF_LRU)] =3D "folio_off_lru", + [I(FOLIO_HOISTED)] =3D "folio_hoisted", + [I(FOLIO_MATCHED)] =3D "folio_matched", + [I(FOLIO_SWAPPED)] =3D "folio_swapped", + [I(FOLIO_CHANGED)] =3D "folio_changed", + [I(FOLIO_ALREADY)] =3D "folio_already", + [I(FOLIO_SKIPPED)] =3D "folio_skipped", +#endif /* CONFIG_DEBUG_VM */ #undef I #endif /* CONFIG_VM_EVENT_COUNTERS */ }; --=20 2.51.0