From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yx1-f44.google.com (mail-yx1-f44.google.com [74.125.224.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50E6D430CC6 for ; Mon, 24 Aug 2026 13:52:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787579570; cv=none; b=Hfw330W3aQQcYyliHcvwY1JVQUqpL8OqA/OkLgmYXkbPaIg9IlPkXpaNuDa8YyCh4JuMR2bH+E7zT814N35xV/F1cYa/0gIds6yGB7vOlEZ644B/dty5CzJWy5zbQIKnsSqOzZ9/MmD19aW7NquJJbQR8S6nUrqajqk2vMrdMPo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787579570; c=relaxed/simple; bh=INQpY4vkmtPsJDoCs8IyTHaF6Ew7wSWtqA9unZfAs40=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=YHrW0j4MQ3ds73NYUVYOH9lvXJ50s8XyuK6Ib63RTNuMkQOiGxWFDDYNRVETUSa10WwF7rYimeSGkKbn1mgGdgGfv/JS22j7rClpUqej/Bk93jjsFFv18hvDdy3Aa5tSH/fFThFJzJckmCq+JQkWClySUu3e8grVIIoywbfHdSM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hu6X3jGD; arc=none smtp.client-ip=74.125.224.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hu6X3jGD" Received: by mail-yx1-f44.google.com with SMTP id 956f58d0204a3-66ceaade3f1so2138518d50.1 for ; Mon, 24 Aug 2026 06:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787579568; x=1788184368; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=n94V7wbjjjBLybhmkw6p8j8STOH0bCJCKxuuQhNVBOI=; b=hu6X3jGDHFD05LuNeJDhlU+vZjMv+Tu2gevXiip4LPrM1pwMF9fjU48iE3OeoCqad0 sDJg1f9GIkJYdmNuP0hlr3AH5D60sX56d9LSbeDKqG/SVyR21/who9KZKZla2z90wQY6 zD4Mk6ovSgUncPfhddvClr1Y2hX0t3HrSdB4ksSrawyN5AB4ah4U9qbr2kXCVmpe44TA bHJGzyB4wN3le1XTBP/9jQrQK4k3iQw1rIie37nFAGiT02/kwQgl2NWcxaFcc7TvEl7p iJD+7cEGcAzuIizJF44H8m69FlNALUnW/Ys1o2KYXs+PNy+0ZKuNzEvRf9qcbE2Lx3kv waww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787579568; x=1788184368; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=n94V7wbjjjBLybhmkw6p8j8STOH0bCJCKxuuQhNVBOI=; b=Tj5vtJhTX7u2t6VkIV02yA4GaTUiH9ZbpNcFrh5a8QEzfWThiyaFDojarWkRRyVMg8 gmPtN5sCf9GUO24Zx1Bd20bVfvTMvf8DgUb2jNkePPN/9+pptTrgoPqGCEEPvE50UftH iMPJXv9bbgbgInwX1oYRf9fYCPiTsi9XugH5/W+rcXDJh4CzBHgyj2Fw0x2NX8uXCld7 Yfkaf3WiB7sYbM/n2t+pTzO34oFLYvRFCuK37m1lAa0kTU++lR2XSPRrA8SSHghi0Vvf KjJG2VMeNJCFNTk0lTi+wXYssy8FpHQvE23P8vn7tFSIe5bNS9WQc6A3dKXOAIeAiHgV Nwuw== X-Forwarded-Encrypted: i=1; AHgh+RrnUAIqiIQmI/7ziYRAc2ZSe4xnlhu3fwv3YlgtGg80a6JXCCgCzCxHKTm3F5gOJ1N605Q/1bfHiR69ZVg=@vger.kernel.org X-Gm-Message-State: AFuF++kvkCZA13Gfo3sUnVXqZ2KS7cFVwBLJowWqcN2BEkzRbp6ggVb4 5Zob/BFbA9PR1RWVD5BtP2tuMrxgjijy9ygnJJhTWFb+lzEnoH2XhsSf8OpSXOLfHQ== X-Gm-Gg: AR+sD12e/zzH35cCYPgJ+5abDp1+cIzYdsv5MtD4mJYhtqTF7fgouQ43Hm7TqY+7121 7QafYOoPnB+jPdtJgVonN8JR8WBOSdHW2w+v0Nbk5Uv+m9ajbpI4Qoo69juUKX6rq8TX6mxjZWm xL8bMFWvQpEsg1dHwajDcfgRyIdaEqyVABghLOnnMsNWbQgf4bPFjbEOkBiuYoCsmqZXedtd5Ry dyLaH7jEDPDWFcvPul9ab+/0NRtlb8/QYlCaP1DJePBkE8O38xMmVGIMXGTiixFskvP1w1Gqm2H 9mb8R5vMrnTOWmjw5Nb7h4+EzDsac8cmJniY/ykFbHrWS/VHkYl2mt2aanIg/HLhHBKdmxCC4fk Dg+XzjWqRJThDv9LwULDM1l3oIbIL2F3ulvdsD5bsp/OmmGl8rNKNzUxWSxr3GanPAQCGu6tMBV d2GmB+zT4urdP4ycbXpuEG3kTaglQGyQ+hhdNSP1v2h0Hg5lAeEvnb8G0B6o1aNRqe92GmlqDYQ ZY4uV/UKUsp0QfqMNmsM+UA9SSatm7Pyi3BJNNPsvPASPqOfJwB87R0E7cdmVYlK5TZ6Rw= X-Received: by 2002:a53:cdcf:0:b0:667:b314:aea4 with SMTP id 956f58d0204a3-66ce4a56db4mr7162361d50.16.1787579567501; Mon, 24 Aug 2026 06:52:47 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cf4624f47sm3724257d50.4.2026.08.24.06.52.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 06:52:46 -0700 (PDT) Date: Mon, 24 Aug 2026 06:52:41 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 01/25] mm/fbatch: remove !CONFIG_SMP special case of folio_activate() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <3f18ab7d-c12a-d2a3-4d3e-f9908f94f27d@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" 3.0 commit eb709b0d062e ("mm: batch activate_page() to reduce lock contention") brought in an ifdef CONFIG_SMP around activate batching: https://lore.kernel.org/linux-mm/20100805140755.501af8a7.akpm@linux-foundat= ion.org/ shows a sensitivity to bloat that day, not any incompatibility with UP. No other batching here has a UP alternative, and it's a bit confusing: simplify mm/folio.c a little by removing it now. Certainly we can reduce UP bloat (and/or 32-bit bloat) by, say, lowering FOLIO_BATCH_SIZE from 31: traditionally 16, 14, 15, then raised to 31 by 6.9 commit 9cecde80aae0 ("mm: increase folio batch size"); or by giving just the static per-cpu folio batches a type of their own with a smaller array size on UP (1? or a little batching worthwhile even on UP?). But not right now, it's orthogonal to this series. And I suspect that the old ifdef led to lru_activate being placed last, whereas it's usually the second most popular fbatch: move it there, to match cpu_needs_drain() comment "Check these in order of likelihood that they're not zero". Signed-off-by: Hugh Dickins Acked-by: David Hildenbrand (Arm) Reviewed-by: Vlastimil Babka (SUSE) --- mm/folio.c | 40 ++++++---------------------------------- 1 file changed, 6 insertions(+), 34 deletions(-) diff --git a/mm/folio.c b/mm/folio.c index d2937600cf72..62b96c9ce19e 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -47,12 +47,10 @@ struct cpu_fbatches { */ local_lock_t lock; struct folio_batch lru_add; + struct folio_batch lru_activate; struct folio_batch lru_deactivate_file; struct folio_batch lru_deactivate; struct folio_batch lru_lazyfree; -#ifdef CONFIG_SMP - struct folio_batch lru_activate; -#endif /* Protecting the following batches which require disabling interrupts */ local_lock_t lock_irq; struct folio_batch lru_move_tail; @@ -349,15 +347,6 @@ static void lru_activate(struct lruvec *lruvec, struct= folio *folio) count_memcg_events(lruvec_memcg(lruvec), PGACTIVATE, nr_pages); } =20 -#ifdef CONFIG_SMP -static void folio_activate_drain(int cpu) -{ - struct folio_batch *fbatch =3D &per_cpu(cpu_fbatches.lru_activate, cpu); - - if (folio_batch_count(fbatch)) - folio_batch_move_lru(fbatch, lru_activate); -} - void folio_activate(struct folio *folio) { if (folio_test_active(folio) || folio_test_unevictable(folio) || @@ -367,25 +356,6 @@ void folio_activate(struct folio *folio) folio_batch_add_and_move(folio, lru_activate); } =20 -#else -static inline void folio_activate_drain(int cpu) -{ -} - -void folio_activate(struct folio *folio) -{ - struct lruvec *lruvec; - - if (!folio_test_clear_lru(folio)) - return; - - lruvec =3D folio_lruvec_lock_irq(folio); - lru_activate(lruvec, folio); - lruvec_unlock_irq(lruvec); - folio_set_lru(folio); -} -#endif - static void __lru_cache_activate_folio(struct folio *folio) { struct folio_batch *fbatch; @@ -694,6 +664,10 @@ void lru_add_drain_cpu(int cpu) trace_mm_lru_add_drain_tp(cpu, nr_folios); } =20 + fbatch =3D &fbatches->lru_activate; + if (folio_batch_count(fbatch)) + folio_batch_move_lru(fbatch, lru_activate); + fbatch =3D &fbatches->lru_move_tail; /* Disabling interrupts below acts as a compiler barrier. */ if (data_race(folio_batch_count(fbatch))) { @@ -716,8 +690,6 @@ void lru_add_drain_cpu(int cpu) fbatch =3D &fbatches->lru_lazyfree; if (folio_batch_count(fbatch)) folio_batch_move_lru(fbatch, lru_lazyfree); - - folio_activate_drain(cpu); } =20 /** @@ -825,11 +797,11 @@ static bool cpu_needs_drain(unsigned int cpu) =20 /* Check these in order of likelihood that they're not zero */ return data_race(folio_batch_count(&fbatches->lru_add) || + folio_batch_count(&fbatches->lru_activate) || folio_batch_count(&fbatches->lru_move_tail) || folio_batch_count(&fbatches->lru_deactivate_file) || folio_batch_count(&fbatches->lru_deactivate) || folio_batch_count(&fbatches->lru_lazyfree) || - folio_batch_count(&fbatches->lru_activate) || need_mlock_drain(cpu)) || has_bh_in_lru(cpu, NULL); } --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53AEE4302FC for ; Mon, 24 Aug 2026 13:55:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787579728; cv=none; b=cfduQoWoVnFx2fovHWmMB8fFGHLZwZ8diosSvuTRvtlbcRYrzRhGRuDbH2C5rJxEJVilFJHG9TSUfcfaAXKCd7hgjqP4xJ+iS1XUP+lm9fjPzRB/wKlDu7xAlcrAgysyk8kh7K6iHW+7Yui7QRLysmS5nVvxEKkhzHtWIGfwJ58= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787579728; c=relaxed/simple; bh=1pawnKHS5KFrhVDfUohuGCOCrDfc8Tr2JRyh3/rR6Ds=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=RvkZoZABw8LlvrV9q2d09YLcHLwDFoAIAqSHRV81bULkUVHpStLd1Il+rC8RysHRLfpn589cyzKw5Pyk88HxX0Klr9H4JOLcWb/Uw6kmFs//9/lJmNEtzCqq2aOh4gXaS+TUy2yjFW382UqjYWCv7O//Mnu58wUknkvJDbkHI4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PlMN4MUf; arc=none smtp.client-ip=209.85.128.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PlMN4MUf" Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-7dbcb505578so40276117b3.3 for ; Mon, 24 Aug 2026 06:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787579726; x=1788184526; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=kV9CvWZgKJ3OzoIjmFQaCybgDiXpPYv7ZoBPddJ/7AM=; b=PlMN4MUfQ5D2mw3zw54NNsyU6D1AE/pj/NAap5TEvjl/8Il4WZgDO/o6tmzc3Cf0Ex GM3rmEJX8vEjCe7tJhBk6yw+1bcSO7NZwe4YEaknjFYwvkVlC1LGFGTtQ26xJYkc9VY6 wmrPI7f7ng1vy3ryztuamFEyFdcVJM/ltTzEgqVvhxJLJCV5OEMXeaZRpxo7lL5O3Xc4 iV1qRGn0ublc4Ls1wZ+iU5aryP/gtsSMkitbvn/qiUV9eGFCOPDCTTRD2ZQhFsvwPRGn Ei8NTeZb2pya9AODihemEt1u3ieDeQ6zuDO3TD6haLdtE5xxVVa0mOWNliJGgdy4xnr6 mjRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787579726; x=1788184526; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=kV9CvWZgKJ3OzoIjmFQaCybgDiXpPYv7ZoBPddJ/7AM=; b=JUqm9poFPIHOf69niY2qc15cCcONmrihY2JLbZeCJQUkUGQpWejEWonpLYm+MNEIvz gk62wn/5phoa/AR8gLvhZRJajghP17NVeCe/4Jqdo9m4GQzriGKslZpttRFy57VIM/tF uH6TKjo5HeXhnK+uOFPils/rtoXjAa2110xJny4mKVGRMZvB5uUNjXLiQfJ4D9+WygnD Njxx+SxnrqbYRfeHTxNAVHj8bD21NF6iq+ddfikZsa44StE4Xs1ixmMbP9GTgRPKn6nx r4CfJS6h8nlLSBj35zmFcWeC3yqbWKc4iZJhjgE0/CIrdnegN4g51eRhQMaZXmcBDuvR 7RcQ== X-Forwarded-Encrypted: i=1; AHgh+RrIte7sNi8QBdgEEVE8QQVM7eSEm6pQmxxqNb1kZG94j9uI1vz5v9dNEE3lcsccsAw4cSuFYok95T9HDps=@vger.kernel.org X-Gm-Message-State: AFuF++nWTO9PbjTYv0x5lKPa2K976kO24JG+NlgQHdyqQXiVlzdzCoK1 VaabTmz+YSe4S4vUg8epiw084dmPb2NIGDGzvgbINcXxpY5by77rXpp2TuLTYi9DXg== X-Gm-Gg: AR+sD10BpksWkDUj1bDFCzpKtwFRxRJvkuqfDoY9jmV5qZzFA+RWtgu82HjTooBz9Jk MhZWK9/itwLquiBMkZmj4+HUEJRNeifooD8Xuyf4yU2R+8mcyYqr7saKi9McpdFkWy6drsXfZ20 IBwF7VAxOUCQoe/VTHviq6GRS/1i4Ug4dZsU+DelLU7Xzx2lQ7nP5otHTfQ/T+A+8CrSm0lbVVJ HGDiNI0EhZjwEZ5agYrRuWqY3yOyx1iShVgRCS0WSXdGbl/fxILP9NKznHFRhzlqOKNDm7FgYwA wgEaxv3q6q5upu3EYW2/fR1nz/7vL8vMbcumMfxHuhx6Vrt0x6G91fkNn8zfqsiPX3BWgfvftN8 XynyZh0oV8HDwYIWnCe+tmsECzqDyFvXXNBt5Mz9LuPXrWaFGbvFCDRYQPuaaJbHY8aX2HHJgYq 3lqngp+aCVJFTxkx7BXlAhWiK3NkhjYz5CzXC8ZBpzeJOZHnii4ToQzl1dgyjucypQ0ozJhCNNH iC1XFiFpCqHQPA5OVGsSlX7BygVOrgbSfMhu5yBrxA4WSSOXkd/7juTnvBD X-Received: by 2002:a05:690c:e3e5:b0:844:7e98:fe01 with SMTP id 00721157ae682-849f18a8409mr92007567b3.5.1787579725434; Mon, 24 Aug 2026 06:55:25 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-853ff053a0esm3017297b3.0.2026.08.24.06.55.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 06:55:23 -0700 (PDT) Date: Mon, 24 Aug 2026 06:55:18 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Andrew Morton , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 02/25] mm/fbatch: allow folios_put_refs() to skip xa_is_value() entries In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <44c7773e-5806-d24a-83c9-f39b3a134138@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let folios_put_refs() (hence folio_batch_release()) skip xa_is_value() entries, and therefore remove unneeded folio_batch_remove_exceptionals(). It made some sense when introduced in 3.1 for shmem swap entries only, but workingset shadows popularized exceptional entries in 3.15, and it's silly for so many sites to be squashing exceptionals out of the fbatch, merely to suit an inadequacy in folios_put_refs(). But remove exceptionals on leaving truncate_folio_batch_exceptionals(), one of whose callers then passes the fbatch on to others less tolerant. No longer essential to this series, since 7.2 commit 9669b87065a6 ("mm/lruvec: preemptively free dead folios during lru_add drain") allowed folios_put_refs() to skip NULLs; but still an improvement. Signed-off-by: Hugh Dickins Acked-by: David Hildenbrand (Arm) Reviewed-by: Vlastimil Babka (SUSE) --- include/linux/folio_batch.h | 5 +---- mm/folio.c | 25 ++++--------------------- mm/shmem.c | 2 -- mm/truncate.c | 16 ++++++++++------ 4 files changed, 15 insertions(+), 33 deletions(-) diff --git a/include/linux/folio_batch.h b/include/linux/folio_batch.h index b45946adc50b..e1cc8ae023f1 100644 --- a/include/linux/folio_batch.h +++ b/include/linux/folio_batch.h @@ -22,8 +22,7 @@ struct folio; * The folio_batch is used to amortise the cost of retrieving and * operating on a set of folios. The order of folios in the batch may be * significant (eg delete_from_page_cache_batch()). Some users of the - * folio_batch store "exceptional" entries in it which can be removed - * by calling folio_batch_remove_exceptionals(). + * folio_batch store "exceptional" (xa_is_value) entries in it too. */ struct folio_batch { unsigned char nr; @@ -100,6 +99,4 @@ static inline void folio_batch_release(struct folio_batc= h *fbatch) if (folio_batch_count(fbatch)) __folio_batch_release(fbatch); } - -void folio_batch_remove_exceptionals(struct folio_batch *fbatch); #endif /* _LINUX_FOLIO_BATCH_H */ diff --git a/mm/folio.c b/mm/folio.c index 62b96c9ce19e..b2bce6b77498 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -983,6 +983,10 @@ void folios_put_refs(struct folio_batch *folios, unsig= ned int *refs) if (!folio) continue; =20 + /* Skip any "exceptional" (workingset or shmem swap) entry. */ + if (xa_is_value(folio)) + continue; + if (is_huge_zero_folio(folio)) continue; =20 @@ -1088,27 +1092,6 @@ void __folio_batch_release(struct folio_batch *fbatc= h) } EXPORT_SYMBOL(__folio_batch_release); =20 -/** - * folio_batch_remove_exceptionals() - Prune non-folios from a batch. - * @fbatch: The batch to prune - * - * find_get_entries() fills a batch with both folios and shadow/swap/DAX - * entries. This function prunes all the non-folio entries from @fbatch - * without leaving holes, so that it can be passed on to folio-only batch - * operations. - */ -void folio_batch_remove_exceptionals(struct folio_batch *fbatch) -{ - unsigned int i, j; - - for (i =3D 0, j =3D 0; i < folio_batch_count(fbatch); i++) { - struct folio *folio =3D fbatch->folios[i]; - if (!xa_is_value(folio)) - fbatch->folios[j++] =3D folio; - } - fbatch->nr =3D j; -} - #ifdef CONFIG_MEMCG static void lruvec_reparent_lru(struct lruvec *child_lruvec, struct lruvec *parent_lruvec, diff --git a/mm/shmem.c b/mm/shmem.c index 89a1495e55f7..3911721e1e55 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1156,7 +1156,6 @@ static void shmem_undo_range(struct inode *inode, lof= f_t lstart, uoff_t lend, truncate_inode_folio(mapping, folio); folio_unlock(folio); } - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched(); } @@ -1276,7 +1275,6 @@ static void shmem_undo_range(struct inode *inode, lof= f_t lstart, uoff_t lend, } folio_unlock(folio); } - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); } =20 diff --git a/mm/truncate.c b/mm/truncate.c index b58ba940be47..4151f7a167e3 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -53,7 +53,7 @@ static void clear_shadow_entries(struct address_space *ma= pping, /* * Unconditionally remove exceptional entries. Usually called from truncate * path. Note that the folio_batch may be altered by this function by remo= ving - * exceptional entries similar to what folio_batch_remove_exceptionals() d= oes. + * exceptional entries. * Please note that indices[] has entries in ascending order as guaranteed= by * either find_get_entries() or find_lock_entries(). */ @@ -95,7 +95,7 @@ static void truncate_folio_batch_exceptionals(struct addr= ess_space *mapping, dax_delete_mapping_entry(mapping, indices[i]); } } - goto out; + goto squash; } =20 xas_set(&xas, indices[j]); @@ -113,8 +113,14 @@ static void truncate_folio_batch_exceptionals(struct a= ddress_space *mapping, if (mapping_shrinkable(mapping)) inode_lru_list_add(mapping->host); spin_unlock(&mapping->host->i_lock); -out: - folio_batch_remove_exceptionals(fbatch); + +squash: + for (i =3D j + 1; i < nr; i++) { + folio =3D fbatch->folios[i]; + if (!xa_is_value(folio)) + fbatch->folios[j++] =3D folio; + } + fbatch->nr =3D j; } =20 /** @@ -575,7 +581,6 @@ unsigned long mapping_try_invalidate(struct address_spa= ce *mapping, if (xa_has_values) clear_shadow_entries(mapping, indices[0], indices[nr-1]); =20 - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched(); } @@ -732,7 +737,6 @@ int invalidate_inode_pages2_range(struct address_space = *mapping, if (xa_has_values) clear_shadow_entries(mapping, indices[0], indices[nr-1]); =20 - folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); cond_resched(); } --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yx1-f51.google.com (mail-yx1-f51.google.com [74.125.224.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5226E432E88 for ; Mon, 24 Aug 2026 13:59:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787579944; cv=none; b=dXSpRKhFpyMteKxqKMTcaqyKAPDaK9dRPRyq0o06lMZqtXvPrStG1Iw4awKp5COe3pyRJD4gDQPFsJJ/r5dBg/lE0pRhjHXmk3h4rqGKx21D2RSyyXP4iiKQLANrCsw7jFTgi32FBj3SnBDGbxJnyBb3oJCzhLqW3BjPUSPPSY0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787579944; c=relaxed/simple; bh=VyXPha22ahQxc4HIlWxvGyHCTBNhScu/VLpt/eHz32A=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=HoPRJVf+l433ExWCOzv+Mia63iawSfUVlLHKmvBCNjxhE0JJSUcX+Kkfr87Jv988aYsi00dbDZl9OD0NXivYUjxkg+DALPzLPvRzLcJKJPnsIvQarqwtZ5B4ib3k+uNDWZLIu4pTmgnVUbb9nvQUKgQ91G1W3weyYQq1G0dp3vM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EFNBQcft; arc=none smtp.client-ip=74.125.224.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EFNBQcft" Received: by mail-yx1-f51.google.com with SMTP id 956f58d0204a3-66c744a00edso2126775d50.2 for ; Mon, 24 Aug 2026 06:59:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787579940; x=1788184740; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=8DePtEtd6bTjy8gbPPUEJydq7WmfU3cohWIB8nVvJ4g=; b=EFNBQcftqm3mUxQIObkidigTaDAlKV7NioBcsr6/HJ/6XySrMXtY0aI4mbeNkNC2gZ BBY3gb0o1DKjAcMAE/mq+uxnsfL55YIg7SQLA8tg+ObId46P1xR+I7f3M9vXrCXo6z9F fN7LxcHgSQDb/L6LCZbq/Y0NvwzcKRU/TVcsssudYspXWq1EznzHpxnm9edhZzIVtW2n iBJxUkRowhpuWMAC/r03p9cuCxHiGIuvjWdXuLfN5ch3nyhrUxPXWMOcIXdz98/UH8mQ EIH4FFWeveO5cluEDLxFm0z9keEAqarc+nc4DZEeVREAPeYYLUqxiuGJzG5qIL5Y2FC+ hezg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787579940; x=1788184740; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8DePtEtd6bTjy8gbPPUEJydq7WmfU3cohWIB8nVvJ4g=; b=AikiuQ7FwCvTDAq37RO+FuCN+WVOwbcJs1VGAI/51KAD+9LgA/xNnfLgEtjJOMj/ep 5Wz0uFde0Af17eVjOgGfNIfJ4XlS7fiutu+mtXYKTlEE7hRLoanlR0jCJp/fWt7QF9Vq jOUduG+c5kRD8iX0+khFeHkK7Bcn7/LBMD7fIGr7kuwNL65iVDfWO/wn5/W5k5G89N0Q xi5XF02MmO/SrBgQtOQuqL8FQGUQpcHJM/sHbr6YfEMMB4CVkbnsnqh69mwpP5xSrVk4 V2Jgm5miEelcn1NNWFfd6zykSY4DKAjK0si3cRq7T0j2sCuo5OuYGUd45QtJQ+nKrlXj +0Dg== X-Forwarded-Encrypted: i=1; AHgh+RpaIPXfwtWj/Hh/p4hw23iVFPd4XKyPI7p+FII7XKVy+hMUKQ8ZJ01IH3cfPMSiioLIrD6iMUn/vD4UH4A=@vger.kernel.org X-Gm-Message-State: AFuF++kJIIJh/gxhOC1dPoNf+I2E965UcFc+5UdyTgMEdjauYl0CDYMm Lj1bl9lFkQ/VKxvlpub5HLTFZK2L37/SNxGLgEQZvwH1G1/A5jT6WI5PdiB4SaMW6w== X-Gm-Gg: AR+sD102sS/wkOQLIYybKFVzKiQ945E7TBXobTuw+u1je9QbXZUa+IPKK7aVssLMUx+ VnmL3Lup/X3RPP/AZOwcW4bGV/jbMrlLImuwKL1BjVxMNWF5+U4c+r7sUylDJXMjgyj4aA0gDnb F9+dvg0ajHgCbvuGJKJCLEVmpBrOAd7mP8Fm9yGmJWvO5IfC/zrnA89v3pQZanIBMjJjTvbIJTY tgczG/UwQlR8MfNpWfTijgZX5Ma9feC9t6GvoCOxePDnIgPhepXfwCK0g3tPNwDbIDttexG3gkG 71XvlSW+t6x2cSUFf1ZwvZk1IG9HnKDGJD0HHcX56gD/XnKXLVXaX302br4X0u/Bn1CH1uxJX1w NqZc7Jd1QigE/Mbw/GxVNi9CkVNu+5KHYCOy17HMu8kH4l/MNPy68WihvsLZMeFDBwZRmUmia0g /cX1Q4w40CqUvf2XpiYEHCK0WyQ0kG/dm/PqIv+5rMdUwc0l7bn6a/YDv1LPmQJAorgj8mlOfix /zqsWwKcHh9UtBLENP7wyUx9DfOLo1vXqXT51uiiJZ7x/zGigJh8eITQnQ= X-Received: by 2002:a53:ac9c:0:b0:66c:fb45:77e2 with SMTP id 956f58d0204a3-66cfb457872mr3574386d50.28.1787579939500; Mon, 24 Aug 2026 06:58:59 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84ca68c1b14sm34163727b3.21.2026.08.24.06.58.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 06:58:58 -0700 (PDT) Date: Mon, 24 Aug 2026 06:58:54 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 03/25] mm/fbatch: temporarily disable lazyfree and mlock+munlock batching In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <9593975e-8732-999a-73d1-37fd332eb840@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It will not matter if we occasionally get activation or deactivation wrong; but a mistaken lazyfree (MADV_FREE) would likely cause dataloss. So disable its batching while reworking the per-cpu fbatch handling, then re-enable it with more thought afterwards. Similarly disable mlock+munlock batching temporarily: they will need some redesign before re-enabling. Just insert one disabling line for now, leaving the rest of the code as it was, for consideration later. Signed-off-by: Hugh Dickins Reviewed-by: Vlastimil Babka (SUSE) --- mm/folio.c | 2 ++ mm/mlock.c | 3 +++ 2 files changed, 5 insertions(+) diff --git a/mm/folio.c b/mm/folio.c index b2bce6b77498..a7010ae3edff 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -218,6 +218,8 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, local_lock(&cpu_fbatches.lock); =20 if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || + /* XXX Temporarily disable lazyfree batching */ + fbatch =3D=3D &cpu_fbatches.lru_lazyfree || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 diff --git a/mm/mlock.c b/mm/mlock.c index efa6716e4dfb..14c02e155d68 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -255,6 +255,7 @@ void mlock_folio(struct folio *folio) =20 folio_get(folio); if (!folio_batch_add(fbatch, mlock_lru(folio)) || + true || /* XXX Temporarily disable mlock batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); @@ -278,6 +279,7 @@ void mlock_new_folio(struct folio *folio) =20 folio_get(folio); if (!folio_batch_add(fbatch, mlock_new(folio)) || + true || /* XXX Temporarily disable mlock_new batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); @@ -299,6 +301,7 @@ void munlock_folio(struct folio *folio) */ folio_get(folio); if (!folio_batch_add(fbatch, folio) || + true || /* XXX Temporarily disable munlock batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D85E3438BA for ; Mon, 24 Aug 2026 14:01:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580090; cv=none; b=uSCxKUlHwgmUUYH8K9Oqye3R/cZcLwxs+DlG8eUqO7200iwA2+As8Ex0nbWMU2yNUoNZo/4eDp9YyQfMjcFBQhKclqqZCu5H+vuG9/K8W2Z3LPish7BVhSbBT70HV9liyvG9TxMqk4sQw2rpAuEGqife6CoJelKv3oAu+atbFpE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580090; c=relaxed/simple; bh=/pbJ1sO8Ra8ej9Se9y+dT8qIv3r62mtB11/3S/mbr/M=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=gnev88DLSzu/MCHp/TrPrz9ebNQaXacItoyFVvEB9IpEb+qBB02rKtoJyDMENt7HIVy4cqs7kIr0yV/CZTFXuJFF1yl08/B6eOOXrF/sAeLe+43D+22YCpROu3fjHcoE1iHhRPnPMzF8H9MWtCtMPFy6xwO8seYjXebQJYKqAO8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WXusWK62; arc=none smtp.client-ip=209.85.128.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WXusWK62" Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-7dbcb505578so40388677b3.3 for ; Mon, 24 Aug 2026 07:01:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787580087; x=1788184887; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=F3uJrLcgO0TGgb78nKF0vmj4EM5IUhGHLplUqQEbOiM=; b=WXusWK62U/1OkkqLZFgAfI4LXG1JGNyU+9TKWgxZWt0vfvVh1VhDaz0/EvlF2Y5lRB ZglsWxQGbwevKQodRx9PFNtqmpNlvYbE/HPAyEblcDqBgZGEMMD8l2J7Ot5JTRXy8zDC Z+stuRMH7L6ktr9x9Zm55avNMribdwGESU5ZOGsn+/p81t+ilUg4P9j+OMznd/s7BX4n FEKCmsQEp3uigYEtB+VLgzOEaU0GLKvdXl58ScGg9KfW5/8GMksvvBmcjFflWy3PCQlr ZJW7HBAA8COOz2z0MwT5GlFeJsvs+NSdhm2H6ka3jyS0+zr8cF7UWrLNGv1j1MVAiVX2 oLsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787580087; x=1788184887; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=F3uJrLcgO0TGgb78nKF0vmj4EM5IUhGHLplUqQEbOiM=; b=DO2nqLN2qJJB5vE+St7MZG1zLF9suJncpNwn9pYPHQmR/eda+ZYYtvqXq4GWavXdWr I6FAOpPeEklNShdfNOfaSVPdAqJzaC0qR8XvfOhgc0rDIT4FtOD2GKYY0XTUb4FV4hh1 zI+ffEIVyTbghr4+dFM1ZQlUmWOmoNCHrQRAxXt9iyt+MX8/OBIg9IuZqEJjfyUhqgCV lUcWzx9HxVOhlJlpeZUovLQbD1TCsEcuX+zjiUCJE3J72RfCWRA8Z/mqwstyF4A1B7l+ kRyJK7XI0f7ksNgoGe3ERMfU8QNOUw67VvZ31pwT5PyAdqFFQPqf6k7o2q8wveziEe5+ YPHQ== X-Forwarded-Encrypted: i=1; AHgh+RoAPT9HNCLb0dG0rthzWxNX12H1ahTflUuGpcS+A5PBVmtLb3WmUsyG5yQcLx/WKvsb3Rz3oH6vSSANbHs=@vger.kernel.org X-Gm-Message-State: AFuF++kYKHvsZyAlC5RkX6SFHTsvuZOaby43XsVo2LJL+phGArlZrEjD 1wy5HPKdEo/isg1i+o1gHOdjLqKq8b7eLo85xgqH49mNxbl3AMMx1un4C+ocm7EKGA== X-Gm-Gg: AR+sD114woRUflJK5wDg+MaudarMMziEtdiS4QxG4dWdD52e3x59EY7J5AZ0ypuvoWn GfH2DMlBynsRjb7av2+H2uxShWgpy1a/ytTc3z6zgL/QQdgIMvoAwFx2hB4nPDk6T5ucBRSswM7 AWx6yHrMZBGHUuAVCImHvjObJtIS9CGxjD3NKESOW30mViWymdThp13cVfC+JwwgKlcQOnhZTQm xeSgpxizZwMfC+hBpXdVyNZXRFd4ppNuUDPXu7/XlrAAByQyAK8IwEnMJ6AWzuSjBvBDgsewDzj zl/IW3G6WvILgNYtB6YUMFlTZGI4oQr4v+8LBDP+rpWhxVlNrDMSrIQZySIyLsrQy6NhsrEmfWZ 1YNfjPly2ewRWxCsKhcxunvRqyrT+PflX1y7MXvxS76q3tpoCtU3ecuDOpHM/usnnJwHIn5CPQ+ yv1N6hzDPEttJkC5DscEtmPBFFn8xgvzswh1TcQ6c6wq2kRUClfdZ4Og57yphOe3gNS1P2BcYzy YDdLE5P/1uPlaukNyp6iRVLXUm3FrQxk7bA/T7LM9hIdraG X-Received: by 2002:a05:690c:568a:20b0:844:9f61:92d9 with SMTP id 00721157ae682-849f67acaa8mr75062557b3.17.1787580086310; Mon, 24 Aug 2026 07:01:26 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-85426caa5e9sm1557737b3.34.2026.08.24.07.01.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:01:25 -0700 (PDT) Date: Mon, 24 Aug 2026 07:01:20 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 04/25] mm/fbatch: lru bit set, no extra ref, while folio on per-cpu fbatch In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Treat folios on a per-cpu fbatch as if they were already on the lruvec: with PG_lru set, without holding an extra reference. This will enable the removal of most lru_add_drain() and lru_add_drain_all() calls soon. Recognize such a folio by 0x02 set in the folio->lru.next pointer by folio_add_lru(). Then lruvec_del_folio() (aided by "lru_add_del_folio") can pretend to unlink it, and folio_batch_move_lru()'s lru_add case can check whether one of the others has already moved it to lruvec. Let folio->lru.next point to the lru_add fbatch entry, but this is now just for debugging: it seemed to be important for folio_batch_move_lru() to distinguish fresh from stale entries, but then it turned out that it has to processs them identically. Activate, deactivates and move_tail, holding no reference on the folio, might come to act on a stale folio when the fbatch is drained: but it's acquired by try_get and test_clear_lru, so safe even when suboptimal. Reclaim is not an exact science, and there have been no complaints of missed actions since 5.11 commit fc574c23558c ("mm/swap.c: serialize memcg changes in pagevec_lru_move_fn") introduced the TestClearPageLRU protocol: so don't expect complaints of a few surprisingly taken actions. Signed-off-by: Hugh Dickins --- include/linux/mm_inline.h | 19 ++++++ include/linux/mm_types.h | 4 +- mm/folio.c | 119 +++++++++++++------------------------- mm/huge_memory.c | 6 +- 4 files changed, 66 insertions(+), 82 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 621c8653d8f7..1ecaf2ef9f2b 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -343,6 +343,23 @@ static inline void folio_migrate_refs(struct folio *ne= w, const struct folio *old } #endif /* CONFIG_LRU_GEN */ =20 +enum { + LRU_NEXT_NEVER_TAIL =3D 0, /* Used by a tail's compound_head */ + LRU_NEXT_BATCHED =3D 1, /* Not used by any aligned pointer */ + NR_LRU_NEXT_FLAGS +}; + +static __always_inline +bool lru_add_del_folio(struct folio *folio) +{ + /* BUG_ON(folio_test_lru(folio)); */ + if (!(folio->lru_next & BIT(LRU_NEXT_BATCHED))) + return false; + folio->lru.next =3D LIST_POISON1; + /* BUG_ON(folio->lru_next & BIT(LRU_NEXT_BATCHED)); */ + return true; +} + static __always_inline void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) { @@ -384,6 +401,8 @@ void lruvec_del_folio(struct lruvec *lruvec, struct fol= io *folio) =20 if (lru_gen_del_folio(lruvec, folio, false)) return; + if (lru_add_del_folio(folio)) + return; =20 if (lru !=3D LRU_UNEVICTABLE) list_del(&folio->lru); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 939b5ea8c9e0..2b1a1f983a91 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -410,10 +410,8 @@ struct folio { union { struct list_head lru; /* private: avoid cluttering the output */ - /* For the Unevictable "LRU list" slot */ struct { - /* Avoid compound_info */ - void *__filler; + unsigned long lru_next; /* public: */ unsigned int mlock_count; /* private: */ diff --git a/mm/folio.c b/mm/folio.c index a7010ae3edff..88e3ebd7e652 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -151,57 +151,34 @@ static void folio_batch_move_lru(struct folio_batch *= fbatch, move_fn_t move_fn) int i; struct lruvec *lruvec =3D NULL; unsigned long flags =3D 0; - struct folio_batch free_fbatch; - bool is_lru_add =3D (move_fn =3D=3D lru_add); - - /* - * If we're adding to the LRU, preemptively filter dead folios. Use - * this dedicated folio batch for temp storage and deferred cleanup. - */ - if (is_lru_add) - folio_batch_init(&free_fbatch); =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { struct folio *folio =3D fbatch->folios[i]; =20 - /* block memcg migration while the folio moves between lru */ - if (!is_lru_add && !folio_test_clear_lru(folio)) - continue; - - /* - * Filter dead folios by moving them from the add batch to the temp - * batch for freeing after this loop. - * - * We're bypassing normal cleanup. Clear flags that are not - * applicable to dead folios. - * - * Since the folio may be part of a huge page, unqueue from - * deferred split list to avoid a dangling list entry. - */ - if (is_lru_add && folio_ref_freeze(folio, 1)) { - __folio_clear_active(folio); - __folio_clear_unevictable(folio); - folio_unqueue_deferred_split(folio); + if (!folio_try_get(folio)) { fbatch->folios[i] =3D NULL; - folio_batch_add(&free_fbatch, folio); continue; } =20 + if (!folio_test_clear_lru(folio)) + continue; + + /* Do not add to LRU if it has already been added */ + if (move_fn =3D=3D lru_add && !lru_add_del_folio(folio)) + goto restore_lru; + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); =20 + /* Do add to LRU if not already there (move_fn skipped) */ + if (lru_add_del_folio(folio)) + lruvec_add_folio(lruvec, folio); +restore_lru: folio_set_lru(folio); } =20 if (lruvec) lruvec_unlock_irqrestore(lruvec, flags); - - /* Cleanup filtered dead folios. */ - if (is_lru_add) { - mem_cgroup_uncharge_folios(&free_fbatch); - free_unref_folios(&free_fbatch); - } - folios_put(fbatch); } =20 @@ -210,8 +187,6 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, { unsigned long flags; =20 - folio_get(folio); - if (disable_irq) local_lock_irqsave(&cpu_fbatches.lock_irq, flags); else @@ -339,7 +314,6 @@ static void lru_activate(struct lruvec *lruvec, struct = folio *folio) if (folio_test_active(folio) || folio_test_unevictable(folio)) return; =20 - lruvec_del_folio(lruvec, folio); folio_set_active(folio); lruvec_add_folio(lruvec, folio); @@ -355,37 +329,12 @@ void folio_activate(struct folio *folio) !folio_test_lru(folio)) return; =20 - folio_batch_add_and_move(folio, lru_activate); -} - -static void __lru_cache_activate_folio(struct folio *folio) -{ - struct folio_batch *fbatch; - int i; - - local_lock(&cpu_fbatches.lock); - fbatch =3D this_cpu_ptr(&cpu_fbatches.lru_add); - /* - * Search backwards on the optimistic assumption that the folio being - * activated has just been added to this batch. Note that only - * the local batch is examined as a !LRU folio could be in the - * process of being released, reclaimed, migrated or on a remote - * batch that is currently being drained. Furthermore, marking - * a remote batch's folio active potentially hits a race where - * a folio is marked active just after it is added to the inactive - * list causing accounting errors and BUG_ON checks to trigger. + * XXX: It is curiously difficult to recreate safely the old + * __lru_cache_activate_folio() optimization (folio_set_active() + * directly if it's on the local lru_add fbatch): revisit later. */ - for (i =3D folio_batch_count(fbatch) - 1; i >=3D 0; i--) { - struct folio *batch_folio =3D fbatch->folios[i]; - - if (batch_folio =3D=3D folio) { - folio_set_active(folio); - break; - } - } - - local_unlock(&cpu_fbatches.lock); + folio_batch_add_and_move(folio, lru_activate); } =20 #ifdef CONFIG_LRU_GEN @@ -476,16 +425,7 @@ void folio_mark_accessed(struct folio *folio) * unevictable page accessed has no effect. */ } else if (!folio_test_active(folio)) { - /* - * If the folio is on the LRU, queue it for activation via - * cpu_fbatches.lru_activate. Otherwise, assume the folio is in a - * folio_batch, mark it active and it'll be moved to the active - * LRU on the next drain. - */ - if (folio_test_lru(folio)) - folio_activate(folio); - else - __lru_cache_activate_folio(folio); + folio_activate(folio); folio_clear_referenced(folio); workingset_activation(folio); } @@ -505,6 +445,10 @@ EXPORT_SYMBOL(folio_mark_accessed); */ void folio_add_lru(struct folio *folio) { + struct folio_batch *fbatch; + unsigned long lru_next; + bool full; + VM_BUG_ON_FOLIO(folio_test_active(folio) && folio_test_unevictable(folio), folio); VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); @@ -524,7 +468,26 @@ void folio_add_lru(struct folio *folio) folio_mark_accessed(folio); } =20 - folio_batch_add_and_move(folio, lru_add); + local_lock(&cpu_fbatches.lock); + fbatch =3D this_cpu_ptr(&cpu_fbatches.lru_add); + + /* Storing this address is only for debugging */ + lru_next =3D (unsigned long)&fbatch->folios[fbatch->nr]; + /* This mask will do nothing on 64-bit */ + lru_next &=3D ~(BIT(NR_LRU_NEXT_FLAGS) - 1); + lru_next |=3D BIT(LRU_NEXT_BATCHED); + folio->lru_next =3D lru_next; + + full =3D !folio_batch_add(fbatch, folio); + + /* Ensure folio->lru_next visible to folio_test_clear_lru() callers */ + smp_mb__before_atomic(); + folio_set_lru(folio); + + if (full || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + folio_batch_move_lru(fbatch, lru_add); + + local_unlock(&cpu_fbatches.lock); } EXPORT_SYMBOL(folio_add_lru); =20 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 644d6905b49c..98b1d0ea50f0 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3993,8 +3993,12 @@ static int __folio_freeze_and_split_unmapped(struct = folio *folio, unsigned int n } =20 /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - if (do_lru) + if (do_lru) { lruvec =3D folio_lruvec_lock(folio); + /* Move from fbatch to lruvec before lru_add_split_folio()s */ + if (lru_add_del_folio(folio)) + lruvec_add_folio(lruvec, folio); + } =20 ret =3D __split_unmapped_folio(folio, new_order, split_at, xas, mapping, split_type); --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DECAB430CE5 for ; Mon, 24 Aug 2026 14:04:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580242; cv=none; b=AvdjnDI7iiDMQSmNLrbBmdhRO+bMmhwvKCI/2HezaDlLRm/3EoIiPBxWXDRJsdiOWupzyCkpKN61Zr50RJlXb9UHrsyC/67mll2fAMFJ5/IPMFnTF51ARcky0dKOv72HSQUuEMID4He2PGLwjuIFwSGcVUQr7X04z5wjkuFp/JM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580242; c=relaxed/simple; bh=LN4S+uhNeXEkUev4MtTwRUrSM4Q5tR/xe+5lbgofppg=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=fLAg7Ulhz0oL4OLr0MM2PwTjAEKIJubFBiO1omBQEqGDj3qzayO3n9eSdNquqOAQjxaHgoPZaJ4uWSAYED/cjdPlKttkqTLv5H55Msyw8p5CD+R8F49fsS3s8Xt62Vne8hkYipw06TT1gOtFGLYJ/2LLa/ARXJmpo+eMZPjS+xQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rtl52Ag6; arc=none smtp.client-ip=209.85.128.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rtl52Ag6" Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-836cb2fa1bcso52633077b3.1 for ; Mon, 24 Aug 2026 07:04:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787580240; x=1788185040; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=gLqnUltfGMwKBemLbA/06+FuqdHtLjtQ/H9OD8DkJLA=; b=rtl52Ag6X87b4zggXTJD7rgDtSYz03nSHK2+Pnjivk9XsTrG/xr0dPmc00ofbZztON G6/piNskoQTvXOwyCzz2tTj1fcti2eCb3O/cKFy5ejNiy/nZspB4YvWAf7ad7c1GbCx3 8Vxd3w/SWRN9LlPKXU70AdbcYilK3QfI9rtujvxBb96jDNmSNgrbbecDXiIdROQTVOW1 GzDdjcg48xiPXr/AKK1YS2VMOJ01t5Z8qMiMb496nMlIpIJYFFB+Pt8l/iiMXhommpWQ fa6ZCXVE1OwyXsL5fcId8kProLFoFo9xNuM9Qtd5EE2JLDfMHQn9cEeAXo1bN6wAY1nR b/+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787580240; x=1788185040; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=gLqnUltfGMwKBemLbA/06+FuqdHtLjtQ/H9OD8DkJLA=; b=qOP/z9ZoaTLXhRTxCx38KOZ3j5RRaO+Myy5YaiNm7ymIvvuZdyCmKH58nCEAnYHTdR Go32B2d1xQ/CMk6GNPR5h+DAJ5WQrIYu2uhnEkGNFFBqy1U1yTBG6lA8WG47gCEtPWz6 h0seCgavAjyRpkSmlS+Kb0n7/KCo7LB6Ezbb96qujxytxfqHWuu9+xb+lQnisNKYNbeG CCG8dje2md0z0MIk5tprbVYQ4xyQK/QF1WSZbz6YqQCP9HX/NgVzmbwc7NBZehONsPCn NqVGGkBEHzdzbNHXnRWXEQ/efQg/NwFIq26dTPiIaNWF5wFB/Dn7yW+tD8aeWU0Sb9RR ddMA== X-Forwarded-Encrypted: i=1; AHgh+RrHIfhqhDyjIMIv59YyzrdCObfXvPhbIjXdUHUIZ0ZZN7YDnyQElWXnByFVgfOZxy2ilS64tRRRsBSxO1Q=@vger.kernel.org X-Gm-Message-State: AFuF++ksg8AtVtD4UNLV0jpoAWfGO53E2QcYc9xrYzbhvoqOBAdLBmBK hLt+PH/KA/Q9ZaXzDxgw2SHnQ6PyTd5OFRS/4qDytbybaISA+hcXsnKR1FG4FRkYrA== X-Gm-Gg: AR+sD11+dIkoKe5sHgisJI2DMXvHsuyLB9qWiIY2VM18GQIYS09wGy3jX8n60Zlefew l10j+t+nKXrZE1fsPB+iO8pN5migcW/rU5lnCI9F/Z+KebEdsQhrxHUjv/YEzkCcX6h14JsmdtA zctNDGZHcMm5mfy+mfWdyDWdvYO6bdYd15dPywQvVr4T9ArkAZCFbCSENqVvMa7bQHB0bG2UcdT sDqSd+Wp1/vhpaKX8vsLlbPm2EcvwNXjYexhkSy2x1WvYPG3UcFIgI84ftHrX9IbKYgLU0veETG YnQVdLrPqWs3W0owA/mu/AU+I5D2+CH/s4fXzJeR3q9aRk79PIxFQxbAr1FHdjjfI/MFUs6dtvU jLdthWBL3PNRr9L3jGR705ir4MRyVEtqxM4hh26wfAIjsEuU1BEX69IhTZXaKokXxrZgde/aUwi 8C7bMxbhtFyvWa6ysxwTM8ZqghFFT128A4RaOF+qO1+MLRQrFWD8p6VbQvxuSnBPotEJWYTT3cv wq2tWwwZER1HbgQ3+uHJGigpGWigVkmIlo+PIbZyosSE0dk X-Received: by 2002:a05:690c:6004:b0:822:854d:4563 with SMTP id 00721157ae682-84c9682009cmr74201097b3.8.1787580237568; Mon, 24 Aug 2026 07:03:57 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cab851290sm33478997b3.35.2026.08.24.07.03.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:03:56 -0700 (PDT) Date: Mon, 24 Aug 2026 07:03:52 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 05/25] mm/fbatch: lru_add_del_folio()+folio_add_lru() after clear_lru() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <9c538ff2-7150-1021-df1e-fbc10f0bb92a@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Most callers of folio_test_clear_lru() then proceed to remove the folio from its lru, and add it back at the end when they're done (if still in use). But isolate_migratepages_block() and check_move_unevictable_pages() sometimes decide against, and release immediately with a folio_set_lru(). Which usually works fine: but there's now a small chance that while they held the folio with lru bit cleared, an lru_add fbatch drain came along, and had to skip that folio because its lru bit was transiently cleared (previously, the lru_add fbatch drain relied on finding lru bit never yet set). This risks leaving that folio off lru, unreclaimable until freed. Fix such cases by trying lru_add_del_folio() (which only takes action and returns true if the folio was on an lru_add fbatch), then folio_add_lru() if it succeeded: invalidating the old fbatch slot, appending in a new one. Signed-off-by: Hugh Dickins --- mm/compaction.c | 13 +++++++++++-- mm/vmscan.c | 16 +++++++++------- 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index a049415512c6..9e045a90ba21 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1204,7 +1204,13 @@ isolate_migratepages_block(struct compact_control *c= c, unsigned long low_pfn, !cc->alloc_contig)) { low_pfn +=3D folio_nr_pages(folio) - 1; nr_scanned +=3D folio_nr_pages(folio) - 1; - folio_set_lru(folio); + if (lru_add_del_folio(folio)) { + lruvec_unlock_irqrestore(locked, flags); + folio_add_lru(folio); + locked =3D NULL; + } else { + folio_set_lru(folio); + } goto isolate_fail_put; } } @@ -1293,7 +1299,10 @@ isolate_migratepages_block(struct compact_control *c= c, unsigned long low_pfn, if (locked) lruvec_unlock_irqrestore(locked, flags); if (folio) { - folio_set_lru(folio); + if (lru_add_del_folio(folio)) + folio_add_lru(folio); + else + folio_set_lru(folio); folio_put(folio); } =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index 4ca9775ceee8..7da12ffbd3bd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -8021,17 +8021,19 @@ void check_move_unevictable_folios(struct folio_bat= ch *fbatch) folio_clear_unevictable(folio); lruvec_add_folio(lruvec, folio); pgrescued +=3D nr_pages; + } else if (lru_add_del_folio(folio)) { + lruvec_unlock_irq(lruvec); + folio_add_lru(folio); + lruvec =3D NULL; } - folio_set_lru(folio); + if (lruvec) + folio_set_lru(folio); } =20 - if (lruvec) { - __count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); - __count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); + if (lruvec) lruvec_unlock_irq(lruvec); - } else if (pgscanned) { - count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); - } + count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); + count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); } EXPORT_SYMBOL_GPL(check_move_unevictable_folios); =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAD54438011 for ; Mon, 24 Aug 2026 14:06:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580400; cv=none; b=ebNZAwY5DlL09R9b5DW2rZ86tJ183AQP5hvSLgWv5cMxvwmdUHMa8x6YjjLXOJiNtn5ifnvYPXsNtmdBv+/dmDB4oDlOs8TXZmg7g+dg/SVQ9jaEgxL3FHLvzoY/7lziXcmHqXvyJg1ZWEodqCVJX2imHnhIo9Z1+zLFmORawpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580400; c=relaxed/simple; bh=DLxNbafGeyCEB6Cj0RzgQIU/rPeLdjUktAZ9VhvQqDY=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=X8cssVlCyGFuc44Gh71q+calCeph243IXeGKcv7tVG6yGueWfs/GgYmxMrkkGGfoXaBhvTyKTIlTrHIFx7lkz3Z12/xfcJTPnvi+O1FpkB1KDYTq3+0SXZ4NUKqGAoND7i3UXfZx4Pwle2geh+x6F05+5BwYALVHubcSFSqaGQc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rwVlQZ/Y; arc=none smtp.client-ip=209.85.128.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rwVlQZ/Y" Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-836c436a6b3so47061567b3.0 for ; Mon, 24 Aug 2026 07:06:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787580395; x=1788185195; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9t24td1SIXy4MUg0Hzwh5Pqt3oG+R8+H8KjX8Zm4YEA=; b=rwVlQZ/Y3WoQgBowJD0tTNwiBqu2dt4ZKs5sqjrHuEGHqWq/Smwlmx18nzap4cG0T5 /EXh18q+u0o77xqnrfOE+5iV2/plBqkvEcpadBR5GAAn+L/IjI6DdKkk13z+M+RhN7ox 6TBoBIHMmkBbaFDgrgv0v2lGY0zq7hejzCGg40x/PIDm2aNykOhrNmw6XwdtpLmO0yhe 8nNVP/LTVl5fnjY2C+xF54H8AMSN+HDZD/WQe68k09jRF+18pF65ixcpGboDxrZgxTqE LA4E4nebhCl4Tn5u5p45XXzieSq5M2+GARodli0D5hDOyMpgVKjMBC/XuIuEMmnz3dEl XfoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787580395; x=1788185195; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9t24td1SIXy4MUg0Hzwh5Pqt3oG+R8+H8KjX8Zm4YEA=; b=NAq+iyiJLIBWu8A3yhdMthf0TTGSDAUqiM+auzXB139S7irKhP46kdYzBUUF3YLcNs cAfqk57x7gyOJm8uQuZqR/qsZrTTzE1oFUsfTsPPSjb6tncOStGnb2yCiIhltLsW+Rv4 JIwMsYVxCjhfsUHwTzrf/gRPI8tEC2UcZCd0PtCs2LM21j+6MZfuppXyjLzjBDXY0I2B rKi/EVHL5li9TjthgAv5NchTawjNUq0uDbAywJmAP9e/5DVwGbfcpP6RtESCMDrZ53vO axf1alhuKGxorFTqpXE5N3b44ZOroGD70WAa8+Pt/jvCZK5AQ+bpmmFdoDROlF0pP9ta I2Mw== X-Forwarded-Encrypted: i=1; AHgh+RrwWGaK6FAJ3k64smPYJtEx2ljX4XIzHnh8356pM/I1hEXFSXCN3HZJ8IE9Xzm1QrSc/IpaAGuH1kbypmk=@vger.kernel.org X-Gm-Message-State: AFuF++lmlZR1hcg+EMXV+vnkpHcfzu0bGafb+WF1GOTP3kTg1vnQtEho vBxJtF5zxTv4gHDyg6SIXqgpqNBqclUQBJGY9E0VMR5pYP7STmWXMp+UF3dWQNiP+soFFb3vpZ0 xpAeTV9btpbM= X-Gm-Gg: AR+sD12BtAK5o1kKmSvSY/RAOq3qSzMe5CmQllmDu9YfMbVLhBy1obbkG+nx+Kyw6ej r2oxxZnLHw0pEbU8PjDgA9GC1Tti8DeAXpComMiU5OHG4CgfYrc8sPTxaxzmNayOhJzeu3o21wT zgH+qivP5LWM0QnG6+Gma6geYWwdvZU8+XD5xXANZMnZVOaNj/NBd4OTpQWC783x76AI9EZH19Y hX4HIzA2Ij8tekTLuo/vEyKRQzkUEuyaMjpZ/Erj7qTAnOHp7GU84azDMhMcsbt6WW6Gnk8rbGk QZ0GYIIVhmNWUMYbMauBw7jATahUPzS9W/cYmUNNWmIiruu9L077NBuBzPPCsRB6rE4TD0G5RxW fWEP0s2jNFDGRvkFYiJk0BGb7gQJ5qYxCjLZED472/w2TbWblK5UJ5RR62KltQUAajWjxzmswJv Pm/K1U1UyGew9+HlkvfmqGRETwUivoLJA7Ok3Jp0/tLO1uCmurYtf/TCU8S5w/InWLqvrvOC3Oh FXowPM5Oy4SjlxNeQLMhtFQMTwEvwg13feUE7lIlMCtVoPn X-Received: by 2002:a05:690c:e1c5:20b0:80c:2874:67c6 with SMTP id 00721157ae682-849f5a0fb48mr73986717b3.24.1787580394257; Mon, 24 Aug 2026 07:06:34 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84caacaf1e9sm34320857b3.23.2026.08.24.07.06.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:06:31 -0700 (PDT) Date: Mon, 24 Aug 2026 07:06:26 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 06/25] mm/fbatch: fbatch_drain_lazyfree(onstack fbatch) before ptl unlock In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <7155e86c-17f7-77d0-21dd-2d267f384a72@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Re-enable lazyfree batching for MADV_FREE. But it's not safe now to leave potentially stale (then reused) folios in a per-cpu fbatch for lazyfree. Instead, madvise_free_pte_range() keep an fbatch on its stack, and drain it each time before dropping pagetable lock, while the folios are secure. Ignore folio_may_be_lru_cached() and lru_cache_disabled(): limitations irrelevant to this fbatch drained under spinlock (even if RT); though in practice madvise_free_huge_pmd() does have to drain every time. Signed-off-by: Hugh Dickins --- include/linux/huge_mm.h | 6 ++++-- mm/folio.c | 34 +++++++++++++++++++++------------- mm/huge_memory.c | 6 ++++-- mm/internal.h | 3 ++- mm/madvise.c | 9 +++++++-- 5 files changed, 38 insertions(+), 20 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c745f7ad2298..d50906327d1d 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -24,9 +24,11 @@ static inline void huge_pud_set_accessed(struct vm_fault= *vmf, pud_t orig_pud) } #endif =20 -vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf); +struct folio_batch; bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *= vma, - pmd_t *pmd, unsigned long addr, unsigned long next); + pmd_t *pmd, unsigned long addr, unsigned long next, + struct folio_batch *fbatch); +vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf); bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_= t *pmd, unsigned long addr); int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t= *pud, diff --git a/mm/folio.c b/mm/folio.c index 88e3ebd7e652..e76868c95acc 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -50,7 +50,6 @@ struct cpu_fbatches { struct folio_batch lru_activate; struct folio_batch lru_deactivate_file; struct folio_batch lru_deactivate; - struct folio_batch lru_lazyfree; /* Protecting the following batches which require disabling interrupts */ local_lock_t lock_irq; struct folio_batch lru_move_tail; @@ -193,8 +192,6 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, local_lock(&cpu_fbatches.lock); =20 if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || - /* XXX Temporarily disable lazyfree batching */ - fbatch =3D=3D &cpu_fbatches.lru_lazyfree || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 @@ -651,10 +648,6 @@ void lru_add_drain_cpu(int cpu) fbatch =3D &fbatches->lru_deactivate; if (folio_batch_count(fbatch)) folio_batch_move_lru(fbatch, lru_deactivate); - - fbatch =3D &fbatches->lru_lazyfree; - if (folio_batch_count(fbatch)) - folio_batch_move_lru(fbatch, lru_lazyfree); } =20 /** @@ -700,19 +693,35 @@ void folio_deactivate(struct folio *folio) =20 /** * folio_mark_lazyfree - make an anon folio lazyfree - * @folio: folio to deactivate + * @fbatch: batch to which folio will be added + * @folio: folio to be lazily freed * - * folio_mark_lazyfree() moves @folio to the inactive file list. - * This is done to accelerate the reclaim of @folio. + * folio_mark_lazyfree() moves @folio to the inactive file list + * via @fbatch. This is done to accelerate the reclaim of @folio. */ -void folio_mark_lazyfree(struct folio *folio) +void folio_mark_lazyfree(struct folio_batch *fbatch, struct folio *folio) { if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) || !folio_test_lru(folio) || folio_test_swapcache(folio) || folio_test_unevictable(folio)) return; =20 - folio_batch_add_and_move(folio, lru_lazyfree); + if (!folio_batch_add(fbatch, folio)) + folio_batch_move_lru(fbatch, lru_lazyfree); +} + +/** + * fbatch_drain_lazyfree - drain the caller's folio batch + * @fbatch: batch of folios to be lazily freed + * + * Must be called before caller drops the page table lock: that is, + * before dropping the last certain reference to the folios in @fbatch. + * It would be very bad to lazyfree a folio after it was freed and reused. + */ +void fbatch_drain_lazyfree(struct folio_batch *fbatch) +{ + if (folio_batch_count(fbatch)) + folio_batch_move_lru(fbatch, lru_lazyfree); } =20 void lru_add_drain(void) @@ -766,7 +775,6 @@ static bool cpu_needs_drain(unsigned int cpu) folio_batch_count(&fbatches->lru_move_tail) || folio_batch_count(&fbatches->lru_deactivate_file) || folio_batch_count(&fbatches->lru_deactivate) || - folio_batch_count(&fbatches->lru_lazyfree) || need_mlock_drain(cpu)) || has_bh_in_lru(cpu, NULL); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 98b1d0ea50f0..b1f315400111 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2356,7 +2356,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * Otherwise, return false. */ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *= vma, - pmd_t *pmd, unsigned long addr, unsigned long next) + pmd_t *pmd, unsigned long addr, unsigned long next, + struct folio_batch *fbatch) { spinlock_t *ptl; pmd_t orig_pmd; @@ -2417,7 +2418,8 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, st= ruct vm_area_struct *vma, tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } =20 - folio_mark_lazyfree(folio); + folio_mark_lazyfree(fbatch, folio); + fbatch_drain_lazyfree(fbatch); ret =3D true; out: spin_unlock(ptl); diff --git a/mm/internal.h b/mm/internal.h index 68db5abd0a4c..ababee1a8872 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -66,7 +66,8 @@ void lru_add_drain(void); void lru_add_drain_cpu(int cpu); void lru_add_drain_cpu_zone(struct zone *zone); void folio_deactivate(struct folio *folio); -void folio_mark_lazyfree(struct folio *folio); +void folio_mark_lazyfree(struct folio_batch *fbatch, struct folio *folio); +void fbatch_drain_lazyfree(struct folio_batch *fbatch); =20 /* mm/vmscan.c */ unsigned long zone_reclaimable_pages(struct zone *zone); diff --git a/mm/madvise.c b/mm/madvise.c index 240d9161ee74..6ef1f489123c 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -657,6 +658,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, struct mmu_gather *tlb =3D walk->private; struct mm_struct *mm =3D tlb->mm; struct vm_area_struct *vma =3D walk->vma; + struct folio_batch fbatch; spinlock_t *ptl; pte_t *start_pte, *pte, ptent; struct folio *folio; @@ -664,9 +666,10 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned= long addr, unsigned long next; int nr, max_nr; =20 + folio_batch_init(&fbatch); next =3D pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) - if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next)) + if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next, &fbatch)) return 0; =20 tlb_change_page_size(tlb, PAGE_SIZE); @@ -724,6 +727,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, continue; folio_get(folio); lazy_mmu_mode_disable(); + fbatch_drain_lazyfree(&fbatch); pte_unmap_unlock(start_pte, ptl); start_pte =3D NULL; err =3D split_folio(folio); @@ -768,13 +772,14 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigne= d long addr, clear_young_dirty_ptes(vma, addr, pte, nr, cydp_flags); tlb_remove_tlb_entries(tlb, pte, nr, addr); } - folio_mark_lazyfree(folio); + folio_mark_lazyfree(&fbatch, folio); } =20 if (nr_swap) add_mm_counter(mm, MM_SWAPENTS, nr_swap); if (start_pte) { lazy_mmu_mode_disable(); + fbatch_drain_lazyfree(&fbatch); pte_unmap_unlock(start_pte, ptl); } cond_resched(); --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yx1-f46.google.com (mail-yx1-f46.google.com [74.125.224.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 873E931F98A for ; Mon, 24 Aug 2026 14:09:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580583; cv=none; b=Pc8OwVU1wLSDiwDFsCWnZlv6bDplEBs+AGQURcqes3c+6ZiqXhfgnlpJSf1V+nqw2lI+xkhuFo47YSCM5JMeVLdOiHVVmtf2Ab1T8jZfm6NJW5tm3OAdmmGuYYkbVArzQW97NeGBPoJedZYUT/79ntLlmJGSNg4MEwnUmvQ12r4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580583; c=relaxed/simple; bh=j2H+JD3zxqAi9f4PV0Xuj1JWeNtTun4i8LwUusPeK7Q=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=J4Gyzq+DV//cyr6TsdU/tXHo4KahsF1qWGrl16g7zhSxW/41cEVBYeY/9iC6ixsPpO8qgo6g0XX1ffaRztO8PrZXISSNBvVNdQwq8PV298ec7/lQQfslM5LpIjM7EUq4A7vd0jv/wCEj+iPPyi8YnLqzW9gQ+4GXcuwR461Vlwc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TKrdMjGK; arc=none smtp.client-ip=74.125.224.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TKrdMjGK" Received: by mail-yx1-f46.google.com with SMTP id 956f58d0204a3-66c711b7f2aso5162848d50.3 for ; Mon, 24 Aug 2026 07:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787580580; x=1788185380; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=lCdAyHrZb4E51CG+1fE1lJJ+Rs+kWhPosCUCdWMIdjI=; b=TKrdMjGKpRZxCkZtmh3UsETVJaI5g0z8tshCjEt8pl6+7wrF/QBWAqFqJS/0rz7stY IccctcRZH/iZ30/+VSZxsXiGAGgFPAk55OilHDQR8ZBm0lE4z8CQxVINkiIpLMzlrZ1K m6q0A+tbUmiWTPmHuM081AFUqw3J50mWeB/lTl0rfrIPrE2JF9jA0hpnckflWEnvVV0r 0fLqnuOMR3zqjkE404a02v+oagExYp50Ru1+obT2lZO6igGKGwa36qxUFblqJ9X6NYPa n4j3J5FcB0p7ZLyog+p4DxRK4khx+Nfc8xGr67dl0+qN7Hb+p+CtSUmKDPO5y4T5M007 BnWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787580580; x=1788185380; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lCdAyHrZb4E51CG+1fE1lJJ+Rs+kWhPosCUCdWMIdjI=; b=CmpdOpJ0AZv2LHKmvJo1K1DoL+GqKJUZ39py3riiU4E0J88rhyIHYZWLWUFzhKxbel qtN1MUTg47fTBx11Ixu/qzBKgUe28f0lhkOTSe/iMcEQ3hVBZIw83W7XywG9IssKlg0u zBowRquS+QJUUpIqzwyqEXlGZWEJuT9MVE44M2u6wz8DSwkfqAdkJcMsPvdYm1aZiCWY NKiOoLFvCLkRvSnDg9iKDBhD3tmG5j42X67FGO4ThM6gz10yOR/3hW8MSmiRqW8AjuDt 9VgdFhMsmk9ISp/qZxZk7SbYhwlW/iCJlYT/kE6elDcxf/eBHEvP3U8/5zRxkq01ZbLU Dn0g== X-Forwarded-Encrypted: i=1; AHgh+RotScdj9zF9aj9Oe0qWOP/Hcizc/6pwepY3oLSl4Z33MX4Z9JlPGS9Nuke3srpjYDmePxLxMPe4if5gVYA=@vger.kernel.org X-Gm-Message-State: AFuF++nMkXDJnxZAu2LUA7qN7HQYvdxoe6QLVDDpme9lBjX2/eLugL26 VPGucfc/dCl/oGyrlSnroQdytM7PCuJtLysVmYJW4L/TqmLjyPQGnyJlTh95DdhG3Q== X-Gm-Gg: AR+sD10gXdNFskaa6nbnQciUzwIRhjnyLTfjeoag1Gy8fVCAWGodOkiK/CKIYstVe6X lxECCIbHpSxh84+W6qADI6iD4zi6Q0eR4EXgefZ+C5lmkQSfUn7QW/9aViq29edr8/deygMERGb p6c3AftPVo6Oe54+ZeB4ICjdse24oAx6zxqeJGNvp7KX0uBONiIORrAo29HBdyrGvQYYyEOgBJ3 IQVRanE/VBr26VALXb6UASJhmY00uH3GZhLVn5k1IOnm0bsaJiaDOEtmkN+YDbndz3bp4Jcicmw hDghQKQ+tU7yU6CHLEeRSAWOCa5WguOwrPxjtSgUI49F1Ywn0U+nRgo+XQjxWhM2uZCM+ASSPCl Wri1sl2HS2U/xr4vyjfISgmqJq3mWXNhOOJJhnJqIW99yHJZokbUulrcZSImte4Tk47l5800hMO 1Rk1saOWvsCm1bY3JjEIPWiM2chH4APffaZd9o5Ffddbx8Qp94d6s30uPNRuqH4tOudcOm1UZlb YdL9dkTqskNy1Vq+GEs0P6fHQnisuKFJZvUAeij8Gy//ZGJ X-Received: by 2002:a05:690c:e1c1:20b0:836:ec3d:b5e6 with SMTP id 00721157ae682-849f5dfebc0mr66916777b3.28.1787580579812; Mon, 24 Aug 2026 07:09:39 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84ca5d252f0sm33979987b3.13.2026.08.24.07.09.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:09:38 -0700 (PDT) Date: Mon, 24 Aug 2026 07:09:34 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 07/25] mm/fbatch: LRU_NEXT_ACTIVATE bit to optimize folio_activate() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <16b39f43-d91e-7b23-900e-90cec13837ff@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement an equivalent to the old __lru_cache_activate_folio() optimization, to activate a folio recently put in the lru_add fbatch, without having to put it through the lru_activate fbatch too. Neither lruvec lock nor lru bit can guard this safely and efficiently, so resort to try_cmpxchg() on a further, LRU_NEXT_ACTIVATE bit in folio->lru_next. Signed-off-by: Hugh Dickins --- include/linux/mm_inline.h | 3 +++ mm/folio.c | 23 ++++++++++++++++++++--- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 1ecaf2ef9f2b..1b54900e87f0 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -346,6 +346,7 @@ static inline void folio_migrate_refs(struct folio *new= , const struct folio *old enum { LRU_NEXT_NEVER_TAIL =3D 0, /* Used by a tail's compound_head */ LRU_NEXT_BATCHED =3D 1, /* Not used by any aligned pointer */ + LRU_NEXT_ACTIVATE, NR_LRU_NEXT_FLAGS }; =20 @@ -355,6 +356,8 @@ bool lru_add_del_folio(struct folio *folio) /* BUG_ON(folio_test_lru(folio)); */ if (!(folio->lru_next & BIT(LRU_NEXT_BATCHED))) return false; + if (folio->lru_next & BIT(LRU_NEXT_ACTIVATE)) + folio_set_active(folio); folio->lru.next =3D LIST_POISON1; /* BUG_ON(folio->lru_next & BIT(LRU_NEXT_BATCHED)); */ return true; diff --git a/mm/folio.c b/mm/folio.c index e76868c95acc..0d8eb9cf5ad5 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -322,15 +322,32 @@ static void lru_activate(struct lruvec *lruvec, struc= t folio *folio) =20 void folio_activate(struct folio *folio) { + unsigned long lru_next; + if (folio_test_active(folio) || folio_test_unevictable(folio) || !folio_test_lru(folio)) return; =20 /* - * XXX: It is curiously difficult to recreate safely the old - * __lru_cache_activate_folio() optimization (folio_set_active() - * directly if it's on the local lru_add fbatch): revisit later. + * This optimization is intended for the common case of folio + * having been recently added to this CPU's lru_add fbatch. + * But since other CPUs can now take it at any instant (after + * a folio_test_clear_lru()), and we may be migrated to another + * CPU, it is simplest just to extend the optimization to all CPUs. + * + * folio_set_active() would be unsafe without the lruvec lock, and + * a folio_test_clear_lru() here might cause a racing drain of the + * lru_add fbatch to skip its lru_add(): so use try_cmpxchg(). */ + lru_next =3D READ_ONCE(folio->lru_next); + while (lru_next & BIT(LRU_NEXT_BATCHED)) { + if (lru_next & BIT(LRU_NEXT_ACTIVATE)) + return; + if (try_cmpxchg(&folio->lru_next, &lru_next, + lru_next | BIT(LRU_NEXT_ACTIVATE))) + return; + } + folio_batch_add_and_move(folio, lru_activate); } =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE77643230D for ; Mon, 24 Aug 2026 14:11:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580721; cv=none; b=fzRJEg6uDqwi8lQ6Ruc4mmfnS8dlX1NjWf61eIXJuL2XNiJyglf0paY+XbiYTu9TF/oUxhyOJqM5NMBWs1Lp7OBBAGAGNIhmSeoN0gzo8v2IXTmZFBPXHiAKiVIojGWnTQctNX/9/N3u+iXj9ANqt116+kMCneH5H/LsunLQtVo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580721; c=relaxed/simple; bh=+0fA+IVzLIwd37UWmLHSvuI1Tv12RLXSll+aIlyQ6d4=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=GppU4tun9RKpkcnsqS/Q5CkMQ5vm0qj+IYdWAPIKmXTDzSECZ3ao8tNUAuW5j+8z8lNNh587PxusH3BrXmZKlN0MeTHz38JzXSe0pUIZhV72XZvrat1trF475TvxuHJl6UWV+HnBzXHlkWc+7DzxDu2H5+fBqQcEuJ5WfO+iJEQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=A0YcQbpG; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="A0YcQbpG" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-836c590b61eso53003807b3.0 for ; Mon, 24 Aug 2026 07:11:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787580719; x=1788185519; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=hDepnQKhG/ALdT6VngAfs3Gdhl8mXkl03WI9zXHFkmc=; b=A0YcQbpGMz6mq26efc2DqHEyIWOaIbyD7Z/jkLzJQcz/NuEYwkA+ZCFJQtPCP/xID2 qpVs9A6H5Ytq6Z+lkwAuwVQGxRBhBzNIk1LHyIxt4cTO73U+nuxIvcbxodTZ0qaxE4/k 2kEK/HI8x4BnrHV8bbcHZmpkMc/55P88Zk+QBIVtg8mvkZO/xhXPA0ryOhZStniQIeUp TIAJlt9FhQ7MDkD0srz0QzdhyKCvUflUJtKmK3lyuOHlnate8NK8flWHYM9nzd/l3loH eV8dgMF7SQdaB3l9ezeR1a+NZu95gC8Jwpbkn9c+8xWZ8TC3nFEf4/RqAcPJ+5qC7wwJ e0eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787580719; x=1788185519; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=hDepnQKhG/ALdT6VngAfs3Gdhl8mXkl03WI9zXHFkmc=; b=AGKDwEPUxGuD4ozEugC9tDA4LrZlIallMUNASmy/8jrtLo+EG6OQ1f+s+r8pl6YECr wXMflLlgt/6HSYRZ9ORhGfySft7lr0U3Dt9pZ1c18zPQjV7Wlesz3cBg2rWmrfVNzW/3 HiKL6oK4xCyVofHxnwL3PElVdS5a83GnZw4ZRNvTW89f2iJBxgrRMpSlGcNueC1VDVNa 6E4oVww6vLd65IGbwlCXWEatH9bygfdwwXDqweVKRs2cNfzbq4yMh2mh5BOxqjbOpa65 1GbE4T4aXJpsgGipzbiXjoLXOQsiOc14WPRij47zEkH63PNgF7C7+smPnu1BEE0oIT4q 6DVg== X-Forwarded-Encrypted: i=1; AHgh+RpMq3BICK9g+K/LYij6PrcvF2gGdw4xNJPdBY3HmcESjqSdL35xsMkRDWeWK9PwV0/OcsbZo1kczjXL+Xk=@vger.kernel.org X-Gm-Message-State: AFuF++mV/kbD1d1Ljc11jV/ORK5CCrJTlHj/wFrN0Wx2gvoWVMl01mrR faJzNOS7rgu8srTj/+nS0REYxpVXTnPNe59Hiva5H3SAOvckikMDsQ3PGFcN4ik4VA== X-Gm-Gg: AR+sD13AM6bE9e8gP3IkE7JHezvllbW5KGd9mZyn7vbsuantaPEfI3EtcWUdeILyLpI NpWXmcRqCrVwoyhZ9uk+iPgzT+UMgv2XpYspl1x8KW9uqMY78Hv8mvY7B21+Cbllq33sxmu7b2X qDopfTrPzQKkMh5Aqvb5ARPzY9w3//tP1K0MUy9ICmM1cOMIj41dWuKmMjovjHzyrWy5ry7Rfk7 JYmfF3gjuj2QzTIuJBS/w3LMqheh7qokYDJiUx0ObdqX1laPX7s3VLwlBm7cmx9tb258oILwCzC d2ZhbLtHJfvb9UInMHfMFa2DW1baLv3vLjISZxlTgtbo3iu5NFVsVG2Z1e39+zdElIlJQItyMU1 aYlgAMLFRXh7tJ1iWDVL4oBsI/yzLINPQuIujsjXr6C2fv4+k5oz1q0LzAZagq9FWtb9whFsKdF K9Rtl82X2p+MUIn2ZfvWLXmJeh69ryB6eUFinrRFcrxwMspX2pRN/v7Zp5HBmTEI4H1sqzzwPxU Cyih8iv+etnpia9PxCuR+2zesv4jheyp6vwZwqROFcxqx/TH1MNd7Zc9BtQ X-Received: by 2002:a05:690c:398:b0:815:bc6a:2e48 with SMTP id 00721157ae682-84c97c883fcmr78457887b3.14.1787580717737; Mon, 24 Aug 2026 07:11:57 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84ca62f6ad5sm33965507b3.18.2026.08.24.07.11.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:11:56 -0700 (PDT) Date: Mon, 24 Aug 2026 07:11:52 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 08/25] mm/fbatch: replace mlock_new_folio() by __folio_add_lru(,mlockit) In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace mlock_new_folio(), working on mm/mlock.c's mlock_fbatch, by __folio_add_lru(,mlockit), working on mm/folio.c's lru_add fbatch: folio moved to lruvec by lru_add(), with its mlocking incidental. Remove old comment about not needing smp_mb__after_atomic() from lru_add(): but that is a detail which will need to be reconsidered. Initialize mlock_count earlier, when adding to fbatch rather than when adding to lruvec. mlock_count count in 2s, with the low bit set to distinguish it from lru.prev. This helps when an mlocked folio is put back early by compaction, but will enable further optimization next. Change mlock_count from unsigned int to long: long to match pointer without endian concerns, signed for better treatment of those rare cases when final munlocks precede their still batched mlocks. This is an intermediate, poorly tested review stage: mlock_new_folio() code removed from mm/mlock.c, remaining code there updated to respect the new mlock_count accounting, but not considered beyond that. Signed-off-by: Hugh Dickins --- Documentation/mm/unevictable-lru.rst | 2 +- include/linux/mm_types.h | 8 +++- include/linux/swap.h | 6 ++- mm/folio.c | 59 ++++++++++++------------ mm/huge_memory.c | 2 +- mm/internal.h | 4 +- mm/mlock.c | 67 ++++------------------------ 7 files changed, 52 insertions(+), 96 deletions(-) diff --git a/Documentation/mm/unevictable-lru.rst b/Documentation/mm/unevic= table-lru.rst index 8d11fe6a0854..45b453226336 100644 --- a/Documentation/mm/unevictable-lru.rst +++ b/Documentation/mm/unevictable-lru.rst @@ -314,7 +314,7 @@ For each PTE (or PMD) being faulted into a VMA, the pag= e add rmap function calls mlock_vma_folio(), which calls mlock_folio() when the VMA is VM_LOCK= ED (unless it is a PTE mapping of a part of a transparent huge page). Or when it is a newly allocated anonymous page, folio_add_lru_vma() calls -mlock_new_folio() instead: similar to mlock_folio(), but can make better +__folio_add_lru(mlockit) instead: similar to mlock_folio(), but can make b= etter judgments, since this page is held exclusively and known not to be on LRU = yet. =20 mlock_folio() sets PG_mlocked immediately, then places the page on the CPU= 's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2b1a1f983a91..049b114aee09 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -359,7 +359,7 @@ typedef unsigned short mm_id_t; * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. * @lru: Least Recently Used list; tracks how recently this folio was used. - * @mlock_count: Number of times this folio has been pinned by mlock(). + * @mlock_count: Number of times this folio has been pinned by mlock() *2 = +1 * @mapping: The file this page belongs to, or refers to the anon_vma for * anonymous memory. * @index: Offset within the file, in units of pages. For anonymous memor= y, @@ -413,7 +413,7 @@ struct folio { struct { unsigned long lru_next; /* public: */ - unsigned int mlock_count; + long mlock_count; /* private: */ }; /* public: */ @@ -507,6 +507,10 @@ struct folio { }; }; =20 +/* folio's mlock_count is doubled, low bit set to distinguish from lru.pre= v */ +#define MLOCK_COUNT_0 1 /* Bit not set in any aligned pointer */ +#define MLOCK_COUNT_1 2 /* Increment or decrement mlock_count */ + #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct page, pg) =3D=3D offsetof(struct folio, fl)) FOLIO_MATCH(flags, flags); diff --git a/include/linux/swap.h b/include/linux/swap.h index 696ed01709c2..f21e1dd6febc 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -294,7 +294,11 @@ extern unsigned long totalreserve_pages; #define nr_free_pages() global_zone_page_state(NR_FREE_PAGES) =20 /* linux/mm/folio.c */ -void folio_add_lru(struct folio *folio); +void __folio_add_lru(struct folio *folio, bool mlockit); +static inline void folio_add_lru(struct folio *folio) +{ + __folio_add_lru(folio, false); +} void folio_mark_accessed(struct folio *folio); void lru_add_drain_all(void); =20 diff --git a/mm/folio.c b/mm/folio.c index 0d8eb9cf5ad5..fa4cf9d7d51b 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -112,31 +112,12 @@ static void lru_add(struct lruvec *lruvec, struct fol= io *folio) =20 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); =20 - /* - * Is an smp_mb__after_atomic() still required here, before - * folio_evictable() tests the mlocked flag, to rule out the possibility - * of stranding an evictable folio on an unevictable LRU? I think - * not, because __munlock_folio() only clears the mlocked flag - * while the LRU lock is held. - * - * (That is not true of __page_cache_release(), and not necessarily - * true of folios_put(): but those only clear the mlocked flag after - * folio_put_testzero() has excluded any other users of the folio.) - */ if (folio_evictable(folio)) { if (was_unevictable) __count_vm_events(UNEVICTABLE_PGRESCUED, nr_pages); } else { folio_clear_active(folio); folio_set_unevictable(folio); - /* - * folio->mlock_count =3D !!folio_test_mlocked(folio)? - * But that leaves __mlock_folio() in doubt whether another - * actor has already counted the mlock or not. Err on the - * safe side, underestimate, let page reclaim fix it, rather - * than leaving a page on the unevictable LRU indefinitely. - */ - folio->mlock_count =3D 0; if (!was_unevictable) __count_vm_events(UNEVICTABLE_PGCULLED, nr_pages); } @@ -449,15 +430,16 @@ void folio_mark_accessed(struct folio *folio) EXPORT_SYMBOL(folio_mark_accessed); =20 /** - * folio_add_lru - Add a folio to an LRU list. + * __folio_add_lru - Add a folio to an LRU list. * @folio: The folio to be added to the LRU. + * @mlockit: Mark the folio as mlocked. * * Queue the folio for addition to the LRU. The decision on whether * to add the page to the [in]active [file|anon] list is deferred until the * folio_batch is drained. This gives a chance for the caller of folio_add= _lru() - * have the folio added to the active list using folio_mark_accessed(). + * to have the folio added to the active list using folio_mark_accessed(). */ -void folio_add_lru(struct folio *folio) +void __folio_add_lru(struct folio *folio, bool mlockit) { struct folio_batch *fbatch; unsigned long lru_next; @@ -492,6 +474,27 @@ void folio_add_lru(struct folio *folio) lru_next |=3D BIT(LRU_NEXT_BATCHED); folio->lru_next =3D lru_next; =20 + if (mlockit) { + long nr_pages =3D folio_nr_pages(folio); + + folio_set_mlocked(folio); + folio->mlock_count =3D MLOCK_COUNT_0 + MLOCK_COUNT_1; + zone_stat_mod_folio(folio, NR_MLOCK, nr_pages); + __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); + } else if (folio_test_mlocked(folio)) { + /* + * A folio is being put back while mlocked. If mlock_count + * has not been overwritten by use of lru.prev, believe it. + * Otherwise, since there may be __mlock_folio()s to come + * through, initialize it to the safer 0 rather than to 1. + */ + if (!(folio->mlock_count & MLOCK_COUNT_0)) + folio->mlock_count =3D MLOCK_COUNT_0; + } else { + /* Initialize this field, which the page allocator did not */ + folio->mlock_count =3D MLOCK_COUNT_0; + } + full =3D !folio_batch_add(fbatch, folio); =20 /* Ensure folio->lru_next visible to folio_test_clear_lru() callers */ @@ -503,24 +506,20 @@ void folio_add_lru(struct folio *folio) =20 local_unlock(&cpu_fbatches.lock); } -EXPORT_SYMBOL(folio_add_lru); +EXPORT_SYMBOL(__folio_add_lru); =20 /** * folio_add_lru_vma() - Add a folio to the appropriate LRU list for this = VMA. * @folio: The folio to be added to the LRU. * @vma: VMA in which the folio is mapped. * - * If the VMA is mlocked, @folio is added to the unevictable list. + * If the VMA is mlocked, @folio will be added to the unevictable list. * Otherwise, it is treated the same way as folio_add_lru(). */ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) { - VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - - if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) =3D=3D VM_LOCKED)) - mlock_new_folio(folio); - else - folio_add_lru(folio); + __folio_add_lru(folio, + (vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) =3D=3D VM_LOCKED); } =20 /* diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b1f315400111..c7bd99592195 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3621,7 +3621,7 @@ static void lru_add_split_folio(struct folio *folio, = struct folio *new_folio, /* head is still on lru (and we have it frozen) */ VM_WARN_ON(!folio_test_lru(folio)); if (folio_test_unevictable(folio)) - new_folio->mlock_count =3D 0; + new_folio->mlock_count =3D MLOCK_COUNT_0; else list_add_tail(&new_folio->lru, &folio->lru); folio_set_lru(new_folio); diff --git a/mm/internal.h b/mm/internal.h index ababee1a8872..ff4bd3a14539 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -976,7 +976,7 @@ folio_within_vma(struct folio *folio, struct vm_area_st= ruct *vma) * * mlock is usually called at the end of folio_add_*_rmap_*(), munlock at * the end of folio_remove_rmap_*(); but new anon folios are managed by - * folio_add_lru_vma() calling mlock_new_folio(). + * folio_add_lru_vma() calling __folio_add_lru(). */ void mlock_folio(struct folio *folio); static inline void mlock_vma_folio(struct folio *folio, @@ -1011,7 +1011,6 @@ static inline void munlock_vma_folio(struct folio *fo= lio, munlock_folio(folio); } =20 -void mlock_new_folio(struct folio *folio); bool need_mlock_drain(int cpu); void mlock_drain_local(void); void mlock_drain_remote(int cpu); @@ -1105,7 +1104,6 @@ static inline bool vma_supports_mlock(const struct vm= _area_struct *vma) =20 #else /* !CONFIG_MMU */ static inline void unmap_mapping_folio(struct folio *folio) { } -static inline void mlock_new_folio(struct folio *folio) { } static inline bool need_mlock_drain(int cpu) { return false; } static inline void mlock_drain_local(void) { } static inline void mlock_drain_remote(int cpu) { } diff --git a/mm/mlock.c b/mm/mlock.c index 14c02e155d68..53d754e82ba2 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -85,14 +85,16 @@ static struct lruvec *__mlock_folio(struct folio *folio= , struct lruvec *lruvec) =20 if (folio_test_unevictable(folio)) { if (folio_test_mlocked(folio)) - folio->mlock_count++; + folio->mlock_count +=3D MLOCK_COUNT_1; goto out; } =20 lruvec_del_folio(lruvec, folio); folio_clear_active(folio); folio_set_unevictable(folio); - folio->mlock_count =3D !!folio_test_mlocked(folio); + folio->mlock_count =3D MLOCK_COUNT_0; + if (folio_test_mlocked(folio)) + folio->mlock_count +=3D MLOCK_COUNT_1; lruvec_add_folio(lruvec, folio); __count_vm_events(UNEVICTABLE_PGCULLED, folio_nr_pages(folio)); out: @@ -100,25 +102,6 @@ static struct lruvec *__mlock_folio(struct folio *foli= o, struct lruvec *lruvec) return lruvec; } =20 -static struct lruvec *__mlock_new_folio(struct folio *folio, struct lruvec= *lruvec) -{ - VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); - - lruvec =3D folio_lruvec_relock_irq(folio, lruvec); - - /* As above, this is a little surprising, but possible */ - if (unlikely(folio_evictable(folio))) - goto out; - - folio_set_unevictable(folio); - folio->mlock_count =3D !!folio_test_mlocked(folio); - __count_vm_events(UNEVICTABLE_PGCULLED, folio_nr_pages(folio)); -out: - lruvec_add_folio(lruvec, folio); - folio_set_lru(folio); - return lruvec; -} - static struct lruvec *__munlock_folio(struct folio *folio, struct lruvec *= lruvec) { int nr_pages =3D folio_nr_pages(folio); @@ -132,9 +115,9 @@ static struct lruvec *__munlock_folio(struct folio *fol= io, struct lruvec *lruvec =20 if (folio_test_unevictable(folio)) { /* Then mlock_count is maintained, but might undercount */ - if (folio->mlock_count) - folio->mlock_count--; - if (folio->mlock_count) + if (folio->mlock_count > MLOCK_COUNT_0) + folio->mlock_count -=3D MLOCK_COUNT_1; + if (folio->mlock_count > MLOCK_COUNT_0) goto out; } /* else assume that was the last mlock: reclaim will fix it if not */ @@ -165,17 +148,11 @@ static struct lruvec *__munlock_folio(struct folio *f= olio, struct lruvec *lruvec * Flags held in the low bits of a struct folio pointer on the mlock_fbatc= h. */ #define LRU_FOLIO 0x1 -#define NEW_FOLIO 0x2 static inline struct folio *mlock_lru(struct folio *folio) { return (struct folio *)((unsigned long)folio + LRU_FOLIO); } =20 -static inline struct folio *mlock_new(struct folio *folio) -{ - return (struct folio *)((unsigned long)folio + NEW_FOLIO); -} - /* * mlock_folio_batch() is derived from folio_batch_move_lru(): perhaps tha= t can * make use of such folio pointer flags in future, but for now just keep i= t for @@ -192,14 +169,12 @@ static void mlock_folio_batch(struct folio_batch *fba= tch) =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { folio =3D fbatch->folios[i]; - mlock =3D (unsigned long)folio & (LRU_FOLIO | NEW_FOLIO); + mlock =3D (unsigned long)folio & LRU_FOLIO; folio =3D (struct folio *)((unsigned long)folio - mlock); fbatch->folios[i] =3D folio; =20 - if (mlock & LRU_FOLIO) + if (mlock) lruvec =3D __mlock_folio(folio, lruvec); - else if (mlock & NEW_FOLIO) - lruvec =3D __mlock_new_folio(folio, lruvec); else lruvec =3D __munlock_folio(folio, lruvec); } @@ -261,30 +236,6 @@ void mlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } =20 -/** - * mlock_new_folio - mlock a newly allocated folio not yet on LRU - * @folio: folio to be mlocked, either normal or a THP head. - */ -void mlock_new_folio(struct folio *folio) -{ - struct folio_batch *fbatch; - int nr_pages =3D folio_nr_pages(folio); - - local_lock(&mlock_fbatch.lock); - fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); - folio_set_mlocked(folio); - - zone_stat_mod_folio(folio, NR_MLOCK, nr_pages); - __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); - - folio_get(folio); - if (!folio_batch_add(fbatch, mlock_new(folio)) || - true || /* XXX Temporarily disable mlock_new batching */ - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) - mlock_folio_batch(fbatch); - local_unlock(&mlock_fbatch.lock); -} - /** * munlock_folio - munlock a folio * @folio: folio to be munlocked, either normal or a THP head. --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B92E43849A for ; Mon, 24 Aug 2026 14:14:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580854; cv=none; b=ufhKLPcis5cnh+9cb8HkN8W3BJanCgNuvT1bMantpYU6ZRh6v9xOfYD+AYp272NMnFJ3Puq/6JRrxdeMZdiZODz9Fy1l5Nlgc/feHl7bLpCzSGeD6KLXwHMqYMFVFH9Mh6zvb+yHJthjrhKroFpHzSUnJpWzlz5y8ZD0FwM7/OQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787580854; c=relaxed/simple; bh=E+/Y8A6JoYUD95BeI1jel2LRvtmdEOwPn6AtXIpcyyU=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=tKz0vGNyBnPpFcISKh5h1KiUE9iNClSWfevYW+Th0gGgkaPkpggz281rJAEwO6fn7D0X+35CRcTGdrp64chZ2DmAIXIi14Nh+B26APRUVau/OMKFAREEZJTB/XKqPfpWQq7uKiXe04B160VNUFC77RZMnPvJkmR/tWI8dCtvGfg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jF6WGMsE; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jF6WGMsE" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-81ed2a06b9eso33422477b3.3 for ; Mon, 24 Aug 2026 07:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787580851; x=1788185651; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Aw7ugmpQOmjY2/CNCHaKDY6XRRUcvMaJ3u0iOskkTBo=; b=jF6WGMsE1yarx5eL7ZoBBJZqhOmbWjKW0yB9y4xpuJrG+VawwYsuCdByYBiDkV1wYj cSbsU1XJK60aytCEVs24D+ph919SyEjVZp6YMkm6hyK6U2nk6aozsG1FOMTT9iyYFcC/ 7lrTJYZCZHT4JVeUMEWTvdb7c8TXmUnCQkU/04354Il1xDzQNrxoHqxy5h1VYisROLaJ l8/ckLm6UfeDfVrmEZAHEyCjbpPwk9KdfmS27LDaux6ayGS9vLQE4eghXwx3eZGiRgFY /4aLL1YMerGmsp/B6O8hH/OS0QrrVrLHfbJGLBBfugp1pBqQzKl3tonA0h28LpFFgHvM 2FYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787580851; x=1788185651; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Aw7ugmpQOmjY2/CNCHaKDY6XRRUcvMaJ3u0iOskkTBo=; b=QPKn7lEoWwCdr5FqehZ5CDZl9rqB7vFboJJAZWAk2PAhdA6T5cJfL/hwTqjGQnIEWu R0JkfOOsRJxCDYkIagXQIJBk8zglC6PwdzqLO7+ZCOc5RmcWFDNsdbWChM4jLz84g7Yd AyTSbmfq2AlpYW4O8/H6i6xdztL3lfrYUwWxfl0T8RMaRDFJ3ZHmKr48nCXxRORAd0ee Q4yuJQjcVxFXTyDPpSnev21zJjad5Q9NZ0bWY2p26ZFJ0in4yyLfr4BHymJO3joKx5kC 9325xtsQLg6P6vAtIvIkm0OzH/FPFXf0UWv/XLs4f10VA6/APbhQAT141jVcLTPjdesW xSVA== X-Forwarded-Encrypted: i=1; AHgh+Rqky164WZdy+BBdKuxp2IUpTlDTkScg7kAt8R4s0BBuZ4mPkIBDNCWHbPRBqUjq8JiTwECQEsjmKeeFZ5k=@vger.kernel.org X-Gm-Message-State: AFuF++lZxsrTU+kHwaVZ1ZKRTv0iEYIr178NmOc0gDIR57L5C4JN0wRy hDMnMYYSuka/GXlKCukV71RJl1DY2CqqbEvXEwjWNfEkVJ99gMZ937PyGT50mWNpJw== X-Gm-Gg: AR+sD11B5yiMkTkbAHf6Ld0/1F1jXJlGXB+vfXlw/3GAH2iZJXo8Cv4kp0CEHAfeCJb SCauvGkg0JxqKYdI8VsQ83zgg0rts2RdDXSTtiJ1tdU3sKgCJzu8PfcGigUDeblCiJ8JngCzVL4 6wUP4VKdYLkebMcC0tSmS/oCoN0mgsqv7OEXP4feqcO9f4fYatqj9ob2/d8aIuDOHhKyqDpTwfI hMSsWYQR6/UaIjJayBBsX5ZiCXLfHrSlnSvHKO3P15P/bUoQ5ZBgekZw6diyw9exQFmbgXFFqvK R7rSlJ/Fj2BolMsEWEh63QGOpDq2sc90GZb4wb8ourzmqXvjmbNY8xc2Ul3wf460zVh8aijXgFJ Lgn3f6TC4xctuJ5ard40XzyVKb7xvkwLoDDphLosyuNITivsGlS6qQvpJuWjzSwcMo6Vf/8dzmH bdrIknLph2mxlgJ529zIt17J4h5GSN8csSTDrjsI97flOUfgNb0Ix9RprvDEX95Q96FQL00jH5F kgTMgiaPzvotyR2IrR+6neVeA2Yy793mQwJ7AtROOOCFlPF X-Received: by 2002:a05:690c:e001:20b0:81d:bc5:4624 with SMTP id 00721157ae682-849f5df6e6amr76092477b3.25.1787580850776; Mon, 24 Aug 2026 07:14:10 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cac4ef4edsm34110127b3.45.2026.08.24.07.14.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:14:10 -0700 (PDT) Date: Mon, 24 Aug 2026 07:14:06 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 09/25] mm/fbatch: restore mlock+munlock batching, without extra ref In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <32d663fb-f192-e0e5-114e-8a92240fb0ac@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Update mlock_folio(), munlock_folio() and their fbatch callouts and helpers, to do folio_try_get()s at batch processing time, instead of holding a folio reference all the while in mlock_fbatch: as in folio.c. But more interesting is the use of mod_mlock_count(), using try_cmpxchg() to update folio->mlock_count safely when possible (now when on lru_add fbatch as well as when unevictable). While __mlock_folio() is as hard to think about as before, __munlock_folio() simpler because munlock_folio() can adjust mlock_count itself without clear_lru() or lruvec lock, and so do the folio_test_clear_mlocked() immediately for itself (without which unevictable_pgs_cleared was likely to appear high, when it should be 0 or low to indicate good mlock health). __munlock_folio() is safe for use even when the unreferenced folio has been freed and reused. It appears that __mlock_folio() could affect a folio which has been freed and reused, but only if it is reused as an mlocked folio, in which case its mlock_count is spuriously incremented (but usually a spurious munlock decrement will follow). How grave is this? If unevictable_pgs_cleared remains low, not so bad. I've gone back and forth on whether to move mlock_fbatch and these functions into mm/folio.c: for now they stay here in mm/mlock.c. Signed-off-by: Hugh Dickins --- mm/mlock.c | 147 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 86 insertions(+), 61 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index 53d754e82ba2..1050010bbe0b 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -58,6 +58,20 @@ EXPORT_SYMBOL(can_do_mlock); * indicate the unevictable state. */ =20 +static long mod_mlock_count(struct folio *folio, long incdec) +{ + long mlock_count =3D READ_ONCE(folio->mlock_count); + + while (mlock_count & MLOCK_COUNT_0) { + if (mlock_count + incdec < MLOCK_COUNT_0) + return MLOCK_COUNT_0; + if (try_cmpxchg(&folio->mlock_count, &mlock_count, + mlock_count + incdec)) + return mlock_count + incdec; + } + return 0; +} + static struct lruvec *__mlock_folio(struct folio *folio, struct lruvec *lr= uvec) { /* There is nothing more we can do while it's off LRU */ @@ -65,6 +79,7 @@ static struct lruvec *__mlock_folio(struct folio *folio, = struct lruvec *lruvec) return lruvec; =20 lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + lruvec_del_folio(lruvec, folio); =20 if (unlikely(folio_evictable(folio))) { /* @@ -73,92 +88,82 @@ static struct lruvec *__mlock_folio(struct folio *folio= , struct lruvec *lruvec) * folio be unevictable? I'm not sure, but move it now if so. */ if (folio_test_unevictable(folio)) { - lruvec_del_folio(lruvec, folio); folio_clear_unevictable(folio); - lruvec_add_folio(lruvec, folio); - __count_vm_events(UNEVICTABLE_PGRESCUED, folio_nr_pages(folio)); } goto out; } =20 + /* + * Something to keep in mind when studying the arithmetic here: + * we only come to __mlock_folio() when mlock_folio() could not + * mod_mlock_count() itself; but by the time this is processed, + * the folio may have already been munlocked, or another mlock + * already marked it as unevictable and so mod_mlock_countable. + * And don't forget that a folio may be unevictable for reasons + * other than mlocked (hence the folio_evictable() check above). + */ + if (folio_test_unevictable(folio)) { if (folio_test_mlocked(folio)) - folio->mlock_count +=3D MLOCK_COUNT_1; + mod_mlock_count(folio, MLOCK_COUNT_1); goto out; } =20 - lruvec_del_folio(lruvec, folio); folio_clear_active(folio); folio_set_unevictable(folio); - folio->mlock_count =3D MLOCK_COUNT_0; - if (folio_test_mlocked(folio)) - folio->mlock_count +=3D MLOCK_COUNT_1; - lruvec_add_folio(lruvec, folio); __count_vm_events(UNEVICTABLE_PGCULLED, folio_nr_pages(folio)); + + if (!folio_test_mlocked(folio)) + folio->mlock_count =3D MLOCK_COUNT_0; + else if (!mod_mlock_count(folio, MLOCK_COUNT_1)) + folio->mlock_count =3D MLOCK_COUNT_0 + MLOCK_COUNT_1; out: + lruvec_add_folio(lruvec, folio); folio_set_lru(folio); return lruvec; } =20 static struct lruvec *__munlock_folio(struct folio *folio, struct lruvec *= lruvec) { - int nr_pages =3D folio_nr_pages(folio); - bool isolated =3D false; + long nr_pages =3D folio_nr_pages(folio); =20 - if (!folio_test_clear_lru(folio)) - goto munlock; - - isolated =3D true; - lruvec =3D folio_lruvec_relock_irq(folio, lruvec); - - if (folio_test_unevictable(folio)) { - /* Then mlock_count is maintained, but might undercount */ - if (folio->mlock_count > MLOCK_COUNT_0) - folio->mlock_count -=3D MLOCK_COUNT_1; - if (folio->mlock_count > MLOCK_COUNT_0) - goto out; - } - /* else assume that was the last mlock: reclaim will fix it if not */ - -munlock: - if (folio_test_clear_mlocked(folio)) { - __zone_stat_mod_folio(folio, NR_MLOCK, -nr_pages); - if (isolated || !folio_test_unevictable(folio)) - __count_vm_events(UNEVICTABLE_PGMUNLOCKED, nr_pages); - else + /* There is nothing more we can do while it's off LRU */ + if (!folio_test_clear_lru(folio)) { + if (folio_test_unevictable(folio) && folio_evictable(folio)) __count_vm_events(UNEVICTABLE_PGSTRANDED, nr_pages); + /* But whoever puts it back on LRU should rescue it */ + return lruvec; } =20 - /* folio_evictable() has to be checked *after* clearing Mlocked */ - if (isolated && folio_test_unevictable(folio) && folio_evictable(folio)) { - lruvec_del_folio(lruvec, folio); + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + lruvec_del_folio(lruvec, folio); + + if (folio_test_unevictable(folio) && folio_evictable(folio)) { folio_clear_unevictable(folio); - lruvec_add_folio(lruvec, folio); __count_vm_events(UNEVICTABLE_PGRESCUED, nr_pages); } -out: - if (isolated) - folio_set_lru(folio); + + lruvec_add_folio(lruvec, folio); + folio_set_lru(folio); return lruvec; } =20 /* - * Flags held in the low bits of a struct folio pointer on the mlock_fbatc= h. + * Flag held in the low bits of a struct folio pointer on the mlock_fbatch. */ -#define LRU_FOLIO 0x1 -static inline struct folio *mlock_lru(struct folio *folio) +#define MLOCK_FLAG 0x1 +static inline struct folio *mlock_flagged(struct folio *folio) { - return (struct folio *)((unsigned long)folio + LRU_FOLIO); + return (struct folio *)((unsigned long)folio + MLOCK_FLAG); } =20 /* * mlock_folio_batch() is derived from folio_batch_move_lru(): perhaps tha= t can * make use of such folio pointer flags in future, but for now just keep i= t for - * mlock. We could use three separate folio batches instead, but one feels - * better (munlocking a full folio batch does not need to drain mlocking f= olio - * batches first). + * mlock. We could use separate folio batches instead, but one feels bett= er + * (munlocking a full folio batch does not need to drain mlocking batch fi= rst). */ static void mlock_folio_batch(struct folio_batch *fbatch) { @@ -169,10 +174,15 @@ static void mlock_folio_batch(struct folio_batch *fba= tch) =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { folio =3D fbatch->folios[i]; - mlock =3D (unsigned long)folio & LRU_FOLIO; + mlock =3D (unsigned long)folio & MLOCK_FLAG; folio =3D (struct folio *)((unsigned long)folio - mlock); fbatch->folios[i] =3D folio; =20 + if (!folio_try_get(folio)) { + fbatch->folios[i] =3D NULL; + continue; + } + if (mlock) lruvec =3D __mlock_folio(folio, lruvec); else @@ -218,19 +228,25 @@ void mlock_folio(struct folio *folio) { struct folio_batch *fbatch; =20 - local_lock(&mlock_fbatch.lock); - fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); - if (!folio_test_set_mlocked(folio)) { - int nr_pages =3D folio_nr_pages(folio); + long nr_pages =3D folio_nr_pages(folio); =20 zone_stat_mod_folio(folio, NR_MLOCK, nr_pages); - __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); + count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages); } =20 - folio_get(folio); - if (!folio_batch_add(fbatch, mlock_lru(folio)) || - true || /* XXX Temporarily disable mlock batching */ + /* + * No more to do if mlock_count is maintained: either the folio + * is on an lru_add fbatch, and will be moved to unevictable in + * due course, or it's already counted as unevictable: no need + * for an mlock_fbatch entry below. + */ + if (mod_mlock_count(folio, MLOCK_COUNT_1)) + return; + + local_lock(&mlock_fbatch.lock); + fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); + if (!folio_batch_add(fbatch, mlock_flagged(folio)) || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); @@ -244,15 +260,24 @@ void munlock_folio(struct folio *folio) { struct folio_batch *fbatch; =20 + /* + * No more to do if mlock_count is maintained and still raised. + * But if mlock_count is unmaintained, we might need to queue an + * munlock fbatch entry, just to cancel an undequeued mlock entry? + */ + if (mod_mlock_count(folio, -MLOCK_COUNT_1) > MLOCK_COUNT_0) + return; + + if (folio_test_clear_mlocked(folio)) { + long nr_pages =3D folio_nr_pages(folio); + + zone_stat_mod_folio(folio, NR_MLOCK, -nr_pages); + count_vm_events(UNEVICTABLE_PGMUNLOCKED, nr_pages); + } + local_lock(&mlock_fbatch.lock); fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); - /* - * folio_test_clear_mlocked(folio) must be left to __munlock_folio(), - * which will check whether the folio is multiply mlocked. - */ - folio_get(folio); if (!folio_batch_add(fbatch, folio) || - true || /* XXX Temporarily disable munlock batching */ !folio_may_be_lru_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 922AF42E406 for ; Mon, 24 Aug 2026 14:16:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581000; cv=none; b=AXL+mW25P41oY6gQWXUh7NEsMzvWxGT7PySTG7XvERSkvYwO4KVcMLwrThfbNIjuyg2rbS0vk5GnkiauuOKd+1Me5hggCbBYmTNGsOfvbQdSVEUJRyNavhkJqKRnuMI24uzm0aGhLdyM6zGqCEfGdowTGczI5yDuwyna/3H2lMg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581000; c=relaxed/simple; bh=QniatQlrXg10kLiLZvm0JWLrvxMekaoyIeWxGsqqyx4=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=VUiVHiHNmjMDnuzSbBAOkw9ZPafePuGnyS6BR36JDziGu8VhVLYCHgItyQXSZ7rhdFCLxhnkp0D8W+d8z/OIX0y0Mo4QnHiDwsepYTe4kd9VVRf3lWwJCMECG3/JZusSVZ6TME++zAzEmXC/cBNRidyex0aUtNAEEMOQSYJORNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HZdUIPxT; arc=none smtp.client-ip=209.85.128.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HZdUIPxT" Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-81dfdbd86d1so45215807b3.1 for ; Mon, 24 Aug 2026 07:16:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787580997; x=1788185797; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=vRQTGUT/r/6sjQRYrNEkhtszeIpWm9eC3zYKRwiMzSE=; b=HZdUIPxTkwxFQa4RpZvCl6gyLZYQw97I5pgVbLbpwnIz9HUpD78bFxBcdtU2Ou7v/a inh9csPo7IuQeVlDn9VnkQe1hsxTf19J0cz3fYhQinK4rkS6aXHMi4GZo9co5DeM7lDU 19HSW/qV/duNgmO3lz5+xtDnTR1t8WLBcAEklnG3q/ujAsJnhHuWRbIkgNF8njMbQ+kv /3N8u1a1otYGBD8t7LnYA2sYGWhDUJKE9poajleKk0YUgT8isZH5R8wZUR8KzxWglbGO dvDtg1S/Z75k/M/PZY922QTXl1FomDedf/vp57R8ltaKEz+Hadzi5SEuZmyP9MZePq// AEbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787580997; x=1788185797; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vRQTGUT/r/6sjQRYrNEkhtszeIpWm9eC3zYKRwiMzSE=; b=RIFGfEUFQu3ESEXdVR/YNfUWUsYyvSzwOuBAVFu/x95SmOqy4Rqu2nI4XChShu9dPq u8NoQ78zMIl3+YwrYdCxW7sK870EfpXBa/+RInNrO7YdhMUJ7ufEHTfW7LY7N9poKNiI LMtz5gO0PVKXebgzbMWKWu/uPDGzjKTs2CI0JoZfa4vganSxYHvYUMGF4sECgnvWhZGK LIS5ggwLemdTlThvP4uQElyVK689hjAPLEsH7gtK+VQW+ns69V5gPftbC4eKNWL+VFZg YH/rAuyp81S+KAGSk7lSJwj2N7E5wTCPlKFP85DnPTTaJK/4JgB6N8DlJSorGFEFaARU uArA== X-Forwarded-Encrypted: i=1; AHgh+RpCCbUL0LRSD+I2p3e3IsvIDR3BfqvZ5Kti+kgLf/sssZBtuzK1zgHBQ6Y4F4CxYYwPEkwdr7GyfxGbFa8=@vger.kernel.org X-Gm-Message-State: AFuF++nj0Gz1TBS9DbNi26GrTLE/4Rjb5a/En9AAHma4Di6DPCEBKP7Z jMcKMTRzSJP/+0biBvVARnubH7JKS9mlL05IU0fSpEVOs5lPg/uugyoML26LSyWxIA== X-Gm-Gg: AR+sD13qQ0epBjnhm0VKD2w+ZzcBcu9e6z76cuuLZjz54VQBZ4XnRJ66xnwY4k6VzVF tmbf+cfCcc9wZQwMcAJIKkoTYYkiieInma850DHjzXVF2lcqsRfooeWbNCDmKTzXeSPD1R/5qjJ aVkq0OOQNd5JOr2kb2YlAu9IdJetfBujsJvnONoXBn6VheUHQqfynpxpaNEuIBeEd6P+nGXHxHb 3kxCSrt7tNHsW49Ek12aaeWS32B/uIi93RiAocq0FvVD7+o3Gwn4P+iY/U6Wjq5CIpwpC+7StSb 6IBrPDfyjdnfF/Ek9+qNyJcq3grcjH2F9ezmBccxiA8wBhHAL+04hBdsBgnLk+J8VTyM+xOJFT3 8v0Oqd4wVJabQFPVMW2ajF4P4iMwIgwaQ2Owwsp+Ry7SerEvjnm29HGB5OgFFncMThIhadCfFj/ UJYY7oYas6KiVxoEoXIHq0aKzwsyqgJatYJqSFvA09S6NCv/2RZ4INdszlsE3U6PLZhUIkYjrk/ z2Ri5zIFqcgD7xk5io/EyOtcwmt5GWjHlBEF6/FO3XoC9C4 X-Received: by 2002:a05:690c:568a:20b0:844:9f61:92d9 with SMTP id 00721157ae682-849f67acaa8mr75561237b3.17.1787580996861; Mon, 24 Aug 2026 07:16:36 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-851e1419e0bsm11879757b3.26.2026.08.24.07.16.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:16:35 -0700 (PDT) Date: Mon, 24 Aug 2026 07:16:30 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 10/25] mm/fbatch: remove several uses of mlock_drain_local() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove several uses of mlock_drain_local(): they were workarounds for when that sequence of operations tended to leave folios in mlock_fbatch with debilitating raised refcount (and no good if the task changed CPU). Signed-off-by: Hugh Dickins --- mm/huge_memory.c | 2 -- mm/migrate.c | 2 -- mm/rmap.c | 4 ---- 3 files changed, 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c7bd99592195..df6bf69a066b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3568,8 +3568,6 @@ static bool __discard_anon_folio_pmd_locked(struct vm= _area_struct *vma, folio_remove_rmap_pmd(folio, pmd_page(orig_pmd), vma); zap_deposited_table(mm, pmdp); add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); folio_put(folio); =20 return true; diff --git a/mm/migrate.c b/mm/migrate.c index 8aaafcea7bc1..6b71b2415d00 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -448,8 +448,6 @@ static bool remove_migration_pte(struct folio *folio, folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } - if (READ_ONCE(vma->vm_flags) & VM_LOCKED) - mlock_drain_local(); =20 trace_remove_migration_pte(pvmw.address, pte_val(pte), compound_order(new)); diff --git a/mm/rmap.c b/mm/rmap.c index 1f72d279ba68..e76823ec4d5e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2390,8 +2390,6 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, } finish_unmap: folio_remove_rmap_ptes(folio, page, nr_pages, vma); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); folio_put_refs(folio, nr_pages); =20 /* @@ -2765,8 +2763,6 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, hugetlb_remove_rmap(folio); else folio_remove_rmap_pte(folio, subpage, vma); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); folio_put(folio); } =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8647433E60 for ; Mon, 24 Aug 2026 14:18:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581112; cv=none; b=A8mkdXPyWJuK+vb4Mqd/yiKG6E6e9H3qUo54cX2xuNnSLUIngEX9vE1zgvW31d18D5xKcMTwgoJAcbBnG/JRMPpPsQyEnObQB5rS8djUJsJPwNTMkgQpsx48dOzmmTMEogweloNlRhO/5ouVFSVtImsKVN3DF1Ingn3IP1Wk2zI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581112; c=relaxed/simple; bh=NXqB4cq2up2lZFmd8LXQWlaLoQrbIuSlvHGTMlBcpMo=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=r6Eb8dsjdL9hnfdFZjpglPmThwf73/I6Np58BZSKh+doWAM79cyr8nZihAHpGJ39ZtwERmy3RUCUlCleJekf+Y7t2G71nfsZXlj3clTWtQH4nNOWIRP5+Smx8FqoCoMX1+n+LbuTT1d64MAP4KFDvnWpNJzouh3zVgp/r1alALA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=iEWFyp2f; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="iEWFyp2f" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-8111c0c7561so52248657b3.3 for ; Mon, 24 Aug 2026 07:18:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787581109; x=1788185909; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=p1RKlgv7gfcdgd0R95YOF5t38k1glOotR90LR4kTqgc=; b=iEWFyp2fObpGuMxuC3AxmU+tN6T5R0BSy85FKMQOYOj0zD6eb+5TZ4Gy9kkRrUDX7l sBfXgKGoB5BlSlq9Vy+xXpe8N12y8g65QcgEfOhdRO+FxdanYmLBwx4XdJJmTcPmDS/Q k8oHz8e4apoTLoFfrpoYRsKudobrDwWUEBbYKJW2o2P4WxYhb0UE1yWHQnxb4DbdB4Z1 MQf0SEv65ZzjkonEMzDdolDCxFGDabCQZg1WG6z5hXp5qimSXeSTii8THmYsmArGmCxI oqSgx3CpTY7Ls0wUnBdq1CQ9s8ZZbJlvOZ9KQbaQJDUQmshkCWvEPZoNw8thg2V5cBJg /TVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787581109; x=1788185909; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=p1RKlgv7gfcdgd0R95YOF5t38k1glOotR90LR4kTqgc=; b=a8Ev6Zz2GlaRy+iAe2fw18eMLqQ5yNg4SJpw7A1b6vhwwVrwihf3oQdQbZNrAT/N2Q T14wDubDWO7armnFCYVj42VCRrB0LIXex+W9Nl75bDxGGgttlkZLjIS+0zvzSeLe2HVc zHcoVEO8zg1ZQ1DqBxw7oP1DRWWM98VKyQGnd9oHOuuUgcmqJ5l4Jgb+Vd2bRo6qJ3V8 ZVwVXMtoAgaF9mqGtekFXXuN6b06uMpLiZs15SkKZHhadTU5HwtbQjjLTSgMF1Ls21VW 30HFcD8BBdgl0zTWNfgoDraG4JEr+dINAEVA68Z/OJOKKAlYCuZ3ydnrnTzAEYGghRfo lizg== X-Forwarded-Encrypted: i=1; AHgh+Ro2haicYlxTXUJ9jBFe9jTPJy7jKLl/IMZrEzeT+hT/vtuudXhfGAbCChmqgHWa89XaiWOE5TNKrRo1KL0=@vger.kernel.org X-Gm-Message-State: AFuF++nSSyy6uUTyThLZfcnSYx/oomK8JZr83Vo8YFvJBWfF0zJ6LStr ualz5VXDckq8gdvXNYmugB53gIvb+Y3mUyjby0z1g10j15Vk6ZzFP5hjhx+Mro+tcA== X-Gm-Gg: AR+sD12B0aVZMI8ezVAJ1yVDD3kD3FZZVYIiJYLhAo0Y+L+py+FV5m2CdnsMQabBlM5 oivoM0mKgSsdcEj/nwlv0ub7K2tX2UkpXSQKIRyPrU963XYlh/ON5DOQRaYKtBt0c2Jc+qt7HOY gaI0f7VsdQjmUKzRFydqDDpWNhkDUJmfgDRsp1tSwIMaHKfS63uPzLDVxC7sEd3V56SEZ2vXB9a APgFBrRqlpRF/F/xvdPJulFBe9+wrct95n/MC2HmOekgWT4y2IgFSkBQVdCxSXh5X8ExyXz2GN4 +/uIs5tKX1K+47NASAq2+0oMbSAhbOd4fW0i6NudLavU5eTWOvJqT1gcRcnjXEtR/uNDGlukoq1 zIXWbHriTs44JdYtqdLuAe6FqYjbyW01RgGUhuhSxoa001wX8vDFLB2z2NVXs2eU9hvyRPUFIrH eEhxHFP8BGQ6k8WPUzSCUyxD18lupwvpUX7/f80qilwJ10s6k41mL/f0VguIlp9QkCr167JWoPt xKDV931GNHJSzrvIio0NSv1ajn+sXx+HUgiqI2w9H0cDvIZ X-Received: by 2002:a05:690c:e0d5:10b0:821:1013:b420 with SMTP id 00721157ae682-849f090e309mr91327927b3.2.1787581108999; Mon, 24 Aug 2026 07:18:28 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8542a633964sm1660667b3.37.2026.08.24.07.18.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:18:28 -0700 (PDT) Date: Mon, 24 Aug 2026 07:18:24 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 11/25] mm/fbatch: remove migration's PAGE_WAS_MLOCKED lru_add_drain() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <6b1ccb51-be6d-5f1b-147c-771e6adaf55c@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A welcome side-effect of mm/mlock.c's mod_mlock_count() succeeding on folios on the per-cpu lru_add fbatch, is that migrate_folio_move() no longer has to lru_add_drain() before remove_migration_ptes() restores a PAGE_WAS_MLOCKED mlock_count: so remove PAGE_WAS_MLOCKED altogether. Re the "We would like to do something similar for the old page, when unsuccessful" comment above it: that may be easier now, but involve some rearrangement: not researched, so just leave the comment as is. Signed-off-by: Hugh Dickins --- mm/migrate.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 6b71b2415d00..534908a0839a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1148,8 +1148,7 @@ static int move_to_new_folio(struct folio *dst, struc= t folio *src, */ enum { FOLIO_WAS_MAPPED =3D BIT(0), - FOLIO_WAS_MLOCKED =3D BIT(1), - FOLIO_OLD_STATES =3D FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED, + FOLIO_OLD_STATES =3D FOLIO_WAS_MAPPED, }; =20 static void __migrate_folio_record(struct folio *dst, @@ -1259,8 +1258,6 @@ static int migrate_folio_unmap(new_folio_t get_new_fo= lio, folio_lock(src); } locked =3D true; - if (folio_test_mlocked(src)) - old_folio_state |=3D FOLIO_WAS_MLOCKED; =20 if (folio_test_writeback(src)) { /* @@ -1411,9 +1408,6 @@ static int migrate_folio_move(free_folio_t put_new_fo= lio, unsigned long private, * isolated from the unevictable LRU: but this case is the easiest. */ folio_add_lru(dst); - if (old_folio_state & FOLIO_WAS_MLOCKED) - lru_add_drain(); - if (old_folio_state & FOLIO_WAS_MAPPED) remove_migration_ptes(src, dst, 0); =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 867CD2F8E99 for ; Mon, 24 Aug 2026 14:20:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581257; cv=none; b=qNRvu61rCF48D6X8jqJyg+zbKw7GlmAIa5/TmwxNl1HSHirfXePRM0i6QvcuJE8PKYH5Bla6I2eCXMlZolPPjyDX7dNK09tmr1+eHjRsHxYsauWbDLJH3KNFgR41qLRbCWTVuDzK8mW6F3U8rPJEoemNILJwPjrlOk+C0pAXKxE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581257; c=relaxed/simple; bh=8h+Tl/oSzEl273fsifSd2ROQHSdSF1jGQXnUSkA/qlo=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=cuVh0wT6sPzvm8uKn7aQ/8Yi8siSd7boXIYz7p2pKZKNDccAI4lUAHPM1Brdc9AjDUwT3JvV6txyMMhMsL7hvIvi9qBTHUhq2cY5p2Kfzw5nu2xORQqpMu+GGgeI+iwTurMDmEY/sXKPi0eEuTTwTKN5w2r2bZ/O+qLUa4hFVnY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MQKTzxBX; arc=none smtp.client-ip=209.85.128.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MQKTzxBX" Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-8200b55dc47so54572437b3.3 for ; Mon, 24 Aug 2026 07:20:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787581254; x=1788186054; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=YRoTlYAeNn0iQ+Akcq71dtDJ5mRi19RmdUI1yCGlg3o=; b=MQKTzxBXRLyXVALWAMfieUw5HptWmyT/+uz5VhS6deH8nU8b1zOs+lAID3sfB/Skr6 n7Ehyqp/XUEB8x17/4+qABrwy+TaxUsKEryBhMSWXfK9wC/7lUifEcxzyPVh2Hf1S4PW BXaasjoFX3mulB7nmbyzAF7O4JdQc0KEggUEsW6CD0QGHg2+jAOOja+a1FhEVuDhAK29 5kM0EZNz99xCkBi0SC12k81e6vr6UrHjBMXlmCe+FqglI3I5zzIcJUKavJqb/DJpIEtn Fvh973ENETGWBjdfEVadgmYcJVw85XXs9QxysAdhG6d/BRg2vwmAwJmWgUUNq6lHm13c Ovcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787581254; x=1788186054; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YRoTlYAeNn0iQ+Akcq71dtDJ5mRi19RmdUI1yCGlg3o=; b=S9otscyJvaVnElvnr89OHGPF7yD878curK2Fo+qE6hJnNWqhW/8bcCtUkFTS0FNH36 k7FWRm59FPQpi7UOimm38Jn/mth7XrwcfDQt7pEWBPs1LCra5I7wjnsc5+SV0r//87Jm uAgU1+Es0kXYIaQqH94VVHOaLotKugBRpx+ztPBISI8K/Q9hh7PrAUzoYbibhYHu6QgI yzifQRK3szqDNkVEZaE0lDfM4aIN0ExQ3RPrCzPayzFHcvLeK2Mnud8lAePFqoIEeo8k h+NIO0oYgec1hwNCF4fdzVWLBkupI4WryxkBL8mmUWCUmNXVuiQDk/SHxpAI6q0E8fOe ITaA== X-Forwarded-Encrypted: i=1; AHgh+RqN+wkDc9QAClgt+3+7XV5v4JoXP0qn/oAeYbtDH9FG6Bpq196f+n2540bwge7gJmu1FsqqrLll2qqEcTw=@vger.kernel.org X-Gm-Message-State: AFuF++mV+ZUmqStgMp8Tfh08yUAqtJ49BDaUaoaDI/qixaEPfYRUKpa6 sMu+PM+NnwHLpRXjFqZy1YKymVxqTDIJxCqNLycizmPlYie98rmqMlw1JRD8SOiSBg== X-Gm-Gg: AR+sD12M+se+bqR0MGtUcEL6bH57a4MwBzg1CAXbVP2+Lcg3whsmgybnIFBbcyrJiY7 V+UrJkaPpVQzMoSR7ibRTh3MQjQC9hx0KhOiqLRsqzfnwNjQxKaLFj6H8wZTTiqx+0e93SMRdoc 8cDNipUA+OqpaVFDlgMc35/nXX02q94wLE21Y1zpO5hhEF4u3JBJy9J0SA/zcf3AVjDhui5uSAO gwTnnByQYMCdFmp7OGU5CCnyhsSGMRQf0tD4KotRzcVgJ/+Xx28d4/6vKWEY1B20T+vOWjwiNPH Sn92Cgbm5sxIduBp+pCeFimxdtTcY6bh/GOPj0JZeyM492Db1ZGOL1ZTQbRN/EZTR/+gXmpUp0X ZpFWvvv6ZcV+MrgiNZO9/S2xVDV50WEWN5HKKiBvPpVgUmORAKjSofM11KQcz0vch0fcv4X6M6t E4O5Zg6WKKXF0khWc0oFJMAjhPEbVBhZKvFkNch+J4WEDQvbzysHPENRj0k9H4DkEa9PNyk6uOt jF+km1bJMWjNEBvsaAWbTdBztsFQ4F3/9aRshU7I3N2uLTC X-Received: by 2002:a05:690c:e217:10b0:81e:a471:e8b2 with SMTP id 00721157ae682-84c95f4df1amr53178007b3.2.1787581253476; Mon, 24 Aug 2026 07:20:53 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cac3f525fsm33909927b3.44.2026.08.24.07.20.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:20:49 -0700 (PDT) Date: Mon, 24 Aug 2026 07:20:44 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 12/25] mm/fbatch: remove percpu_pvec_drained and folios_put() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <3599e5ac-b74f-87ad-ab65-78d24c148fd1@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the percpu_pvec_drained field from folio_batch, and its only use in __folio_batch_release(): remove that now pointless lru_add_drain(). Which leaves __folio_batch_release() as an exported name for folios_put() which is itself just a wrapper for folios_put_refs(): mm/mlock.c and mm/folio.c don't need such a wrapper, just say folios_put_refs(,NULL). Or should folios_put() be the export? But __folio_batch_release() is what drivers/gpu and net/sunrpc are using: don't change them in this series. Signed-off-by: Hugh Dickins --- include/linux/folio_batch.h | 2 -- include/linux/mm.h | 18 ------------------ mm/folio.c | 17 +++-------------- mm/mlock.c | 2 +- 4 files changed, 4 insertions(+), 35 deletions(-) diff --git a/include/linux/folio_batch.h b/include/linux/folio_batch.h index e1cc8ae023f1..a3337f70e109 100644 --- a/include/linux/folio_batch.h +++ b/include/linux/folio_batch.h @@ -27,7 +27,6 @@ struct folio; struct folio_batch { unsigned char nr; unsigned char i; - bool percpu_pvec_drained; struct folio *folios[FOLIO_BATCH_SIZE]; }; =20 @@ -41,7 +40,6 @@ static inline void folio_batch_init(struct folio_batch *f= batch) { fbatch->nr =3D 0; fbatch->i =3D 0; - fbatch->percpu_pvec_drained =3D false; } =20 static inline void folio_batch_reinit(struct folio_batch *fbatch) diff --git a/include/linux/mm.h b/include/linux/mm.h index 87feaa5a2b78..a426f7351787 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2201,24 +2201,6 @@ typedef union { =20 void release_pages(release_pages_arg, int nr); =20 -/** - * folios_put - Decrement the reference count on an array of folios. - * @folios: The folios. - * - * Like folio_put(), but for a batch of folios. This is more efficient - * than writing the loop yourself as it will optimise the locks which need - * to be taken if the folios are freed. The folios batch is returned - * empty and ready to be reused for another batch; there is no need to - * reinitialise it. - * - * Context: May be called in process or interrupt context, but not in NMI - * context. May be called while holding a spinlock. - */ -static inline void folios_put(struct folio_batch *folios) -{ - folios_put_refs(folios, NULL); -} - static inline void put_page(struct page *page) { struct folio *folio =3D page_folio(page); diff --git a/mm/folio.c b/mm/folio.c index fa4cf9d7d51b..782b8245d213 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -159,7 +159,7 @@ static void folio_batch_move_lru(struct folio_batch *fb= atch, move_fn_t move_fn) =20 if (lruvec) lruvec_unlock_irqrestore(lruvec, flags); - folios_put(fbatch); + folios_put_refs(fbatch, NULL); } =20 static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch, @@ -1062,22 +1062,11 @@ void release_pages(release_pages_arg arg, int nr) EXPORT_SYMBOL(release_pages); =20 /* - * The folios which we're about to release may be in the deferred lru-addi= tion - * queues. That would prevent them from really being freed right now. Th= at's - * OK from a correctness point of view but is inefficient - those folios m= ay be - * cache-warm and we want to give them back to the page allocator ASAP. - * - * So __folio_batch_release() will drain those queues here. - * folio_batch_move_lru() calls folios_put() directly to avoid - * mutual recursion. + * This used to optimize with a drain before putting: no longer helpful. */ void __folio_batch_release(struct folio_batch *fbatch) { - if (!fbatch->percpu_pvec_drained) { - lru_add_drain(); - fbatch->percpu_pvec_drained =3D true; - } - folios_put(fbatch); + folios_put_refs(fbatch, NULL); } EXPORT_SYMBOL(__folio_batch_release); =20 diff --git a/mm/mlock.c b/mm/mlock.c index 1050010bbe0b..97134eff6b56 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -191,7 +191,7 @@ static void mlock_folio_batch(struct folio_batch *fbatc= h) =20 if (lruvec) lruvec_unlock_irq(lruvec); - folios_put(fbatch); + folios_put_refs(fbatch, NULL); } =20 void mlock_drain_local(void) --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FF814399E5 for ; Mon, 24 Aug 2026 14:23:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581412; cv=none; b=R6XLDDMTo5Yam8czh6LcY7ALlouUMAVHSQkCzB1uK4hFW/9X9+CCIju4KYrAa/UihvlXTVfvhT4saicwRRKlEUkawdw7C4wD1MmvcKL+S4s+SjAyK3zsPNZPCaKLskQRJuMf/qVELvK2zd/LoEqGTp2pWneAZ+sIF9O5c6E9D84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581412; c=relaxed/simple; bh=6kB5h1HEECPTicOG3LYyogzNQExsOtWLu7kTMzR5cg0=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=JoKT1PdiQ7FRO0z90jCXgGrsLGfTXwXbw3KzthwaV+ce/UEBxVsUpBe7+lAWwpkmJ7AqLn85K+KC1J+CqwbNglOESGvS336P44d0xlLgTuc0FihTMiIy4AgYzQk/Dp4IFsl9CEA1rKiz0OSUH3aa4v93fN1r8/NNFBi1l8QW8T8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qTL5iIIU; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qTL5iIIU" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-836c590b61eso53213617b3.0 for ; Mon, 24 Aug 2026 07:23:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787581409; x=1788186209; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=DAA3AJBckMfL7iDkU5H4UAChzqupIeoXexgfVyLiXUk=; b=qTL5iIIUHVBB9JoHNIpqnDPZ8Tgfan4sEu+QR0jdnehPcdRuSmmSwrTk5iczR8q5DD 600sh1DTir0FNM08emH2W9KFR+c5JnQvIJhyb3T26Cw4B38BIWghn22vvykMM45M388U bJnjrkF6MAh4c/yHo3+o/t/uMp88ct+95h8a6fHmgzcnDUYnTD7ZsU5oDvccIbAxwtl2 WXjIvy7nmjcYv7IeJhvfOCQdnexTlqMx+eN9GT5NvTXc1Pu3tbLAzS6FYPxnkxx6+ZSN Op/Oam74dLn3ZBLZnFPzkvHBG35ayFwnOvUGHFjjlvN6wqCLJ/YOlwnFTazDqJ9O/4LZ BUig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787581409; x=1788186209; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=DAA3AJBckMfL7iDkU5H4UAChzqupIeoXexgfVyLiXUk=; b=PBv39dsYhkp4lxm5Jq21S7hHwQI4QLWrskTOexhHZQNmYKPCwlbMs4xKFr+3DUXvwZ TEQ/aVYr86Mze01AimczUSGiyZfkJ/I7Z3M5/NaKsWfWUmxQTpbm5p2Cy+Q27k8TqlM7 7k+fJxzKAOyc075Gzg16uFtEuGUbdyDQfgdWHFlV7R3G4+7TEmZML0vSJztw+GnchlP/ zCcVo3sP+LNoLlYqWzFbMY42SbdfvCwVqE7IHneVZdOs8QJSSEStxJsVIONbxmJ+JI+k dqE7ayw+4f+PJSoJcTt3bgQSTKzlcp7Iir7Fd6YwXB31OYFQ1ypyMXhs+om8jlSAd+2f HiYQ== X-Forwarded-Encrypted: i=1; AHgh+RoCWuVx9ezigvx7UDnmLP/Vto719Cotgs6Ni5/jcoR+5ZY4DbZ+Nq82K/yP4RBnWxOkPxaPiXaOG2arIzA=@vger.kernel.org X-Gm-Message-State: AFuF++nP8dA+0tGRAiRBqThdCM1y7aQuud7+wDHqqWIU7PRVYsj8ype4 U3ho7RqMTuWZGh1y1CM3mpcxkAvFZN9G1GlXEQDQqMPqNi4Ix1UoujymFgEOr+7Jew== X-Gm-Gg: AR+sD11vS5MIyq9vWcW/XP0a/2IcxsgzeNq0AbJZnpsQROzWcKLRlcRscmpctQSbvS7 QXsDM2aDbAWwK7CzXx7l80tio5CZC7bQ8d72x1HqeowD6A3oabIAhx3X1aAKrD/ywY0ggrBsg1Z /uJleALyxpkgc0YE/BF44omboWdLCYLmO2jikBpJWVPWXM8eREZNT5TYtQ2oRAvu+e+F5RQGjfF kpFxbivPEeZhv+3+VKGotyHC0VHcPqZNs2ji2PU0Hk0WR8JQ5P4v8KYusU3xAypwWAnn++yZT01 4bQ5FXfJcDx1Oa3bxCBJYsyFAZiWUKRKeHqlPJw7mS+mNXCjv8CcJ65UmE5iazrdUkP0qSqkul2 w5HBTlGYIc14YifX39mIjDud2glH4FsuCO/EIW0JSy9XYn7xUQogswAIRemJZNhS0TP2HR45n6T JsJujufXsWFJrNbcontK+iJQ//RpQCNzUeN2n7Jx4sOOKSwSdJJW36fdAopPofYmrUOLvBbF/g8 k1FMN38irMk42yADa/WL+KTaTUBtnEZ9Fh/DmY3Sz7v1KGu X-Received: by 2002:a05:690c:a4db:b0:80d:a249:4854 with SMTP id 00721157ae682-84c99f902c3mr43542587b3.17.1787581408945; Mon, 24 Aug 2026 07:23:28 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cab854483sm33838647b3.33.2026.08.24.07.23.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:23:27 -0700 (PDT) Date: Mon, 24 Aug 2026 07:23:23 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 13/25] mm/fbatch: no lru_add_drain() to collect_longterm_unpinnable_folios() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" collect_longterm_unpinnable_folios() has no use for lru_add_drain() nor lru_add_drain_all(), now that the per-cpu fbatch references are gone. Signed-off-by: Hugh Dickins --- mm/gup.c | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 99902c15703b..9616379ab069 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2268,7 +2268,6 @@ static unsigned long collect_longterm_unpinnable_foli= os( { unsigned long collected =3D 0; struct folio *folio; - int drained =3D 0; long i =3D 0; =20 for (folio =3D pofs_get_folio(pofs, i); folio; @@ -2287,19 +2286,6 @@ static unsigned long collect_longterm_unpinnable_fol= ios( continue; } =20 - if (drained =3D=3D 0 && folio_may_be_lru_cached(folio) && - folio_ref_count(folio) !=3D - folio_expected_ref_count(folio) + 1) { - lru_add_drain(); - drained =3D 1; - } - if (drained =3D=3D 1 && folio_may_be_lru_cached(folio) && - folio_ref_count(folio) !=3D - folio_expected_ref_count(folio) + 1) { - lru_add_drain_all(); - drained =3D 2; - } - if (!folio_isolate_lru(folio)) continue; =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02DAE43BDDD for ; Mon, 24 Aug 2026 14:26:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581564; cv=none; b=VMD/p3cSwSqKm0OKvHwGM4MGcFS62vYdyyCSE/rneeV9b7czhn/SIm3QEBcHwCPRigwrLBRLUiZ8oT295aDq3iaM98okcdVcVVPo7uSdJms0SzIFvMphB+evYRLlunpBXmluFFRj1rj9Xk4gDTnQGpOxC0tf/8lkjPDPLC6XpzY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581564; c=relaxed/simple; bh=1gndqcwixIf69ERl83wZ/ahlq6SVahr40TbASWBQcgI=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=upqgZ7OxqYAN9ZVTmbN4MvwlsTWzOw51DlnSeaSRWDBsQkijcRF+KeYqTZVOW3bzdm6/51CZpKDfz3JWP6zolDoSfYGvzldNVwC7wKIo4Zkljnb/KqHzUhLL/KUvvlwDwo8vmaMAtYT85vSGl3IMSplEfcbps9tN+UG3gTUFGW4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qtyj8YcU; arc=none smtp.client-ip=209.85.128.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qtyj8YcU" Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-836d2861a39so48293137b3.2 for ; Mon, 24 Aug 2026 07:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787581562; x=1788186362; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=lsHaeoqxnMhy2GB3rUAu2bzr8hj8izpjrzp8bJJzBnk=; b=qtyj8YcUWb3nhbWfFR1yFkoOzDGF3NubYDQbAGYRXxSFU9sFzPTnQXiVe1+L8Ya6Ub Y3ABAv8Skeh07ilRLu2LRwBRetxeZ3YjvOwJoSVd+ieEDuHymyt4KK7PyHaLFCez6/yA CmYhhc3hG7OddFHnmw1Zds95SnjpsFeq6aR4Wpgl5qB23W0fCz4t9iCoDsh/RU+pcBqi epj4aR0lPB6ForXt8ft/Mn+wYMS94Ze0kESzmtAXcUmhlDfYh9lsTD46vBBuJsMneRcq DsveRAU02svpn38yxs09ge518OuI/pVdkv3Knis9TnbAIvU4lDhnuCGaGoZiptRruc5f /2dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787581562; x=1788186362; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lsHaeoqxnMhy2GB3rUAu2bzr8hj8izpjrzp8bJJzBnk=; b=tLdUj6q6a9CxX1TfRyAifTKdzXsEV9xvgXJuIJhwp3J5gM0oYkj1Ua1OoPyDZ9jkWO JaHTXt/kKyaAtdDTbKK7MK/vVYAETHddA7xCAi8NY8RBhF3Wg6y2M+JPQuS0HA7dp7zw xC3/CPBsJuI11hYmHWsgC0eByYMWi4PmH/Uir/2hyKDKiI9u5GrATwJ5BX10KkItHG4J ipiVnuRSBGhxRz10s/hQgwFwcDJuyLO+UxRReEPuBRMZi60XgYJu89pnLjlxojgzNy5w B4Ov2jq7C6og1qi3H5ecH7Q/uG92B8MtjOySAtYt9MgCnK7fPZgRUDfkOlmu4CTRfIo8 cRUg== X-Forwarded-Encrypted: i=1; AHgh+RpN2Zagok4JP0WqSXksN2M0ODY0DJhe6d3yVIXEc5LyQlkIJk9bmW1I08D6abbGlJ6yY1wZwnZ+FQFEi6I=@vger.kernel.org X-Gm-Message-State: AFuF++kcuHFsQkv/y+BZimSJIMD5dOK1bnS7Fx53VqnwM/0n7KufGGmS HB1Ol8Tn42hU2llGd0n9wr8QW271r+PhHcxApVg6scUXKZojFzHla1Q0/BXPom6xwg== X-Gm-Gg: AR+sD12slokik4aqp2ksXedT7uiNO8iRaa4F+ttlyazH3i1TQgjDU507YtM5MdTYDAz F3agvmPIkbZnKZqmebUPNXut2bv28Wut5VN4VJPE92F/WGAG+1MxopVzKUn5AOVFN47XWMjp8wF P8eEfw6rB4qbdlXvB2LXhyayT7uVL5H40nK9wkkr+ymtSqMv8NfQpI4KpI7tvwUEHDU5qIIW37x +s++aZswK1wac5W4QqfLUsKtTYNvPRc1jcotqK+arR4Ox3apQYJdlImnB89Ekytj4egBOv4z0Lr 65cyZED72NgXuZn7iplI/yjs7jei7ssl+OVtYTb6Rrt4mZ00OnM72GdB/xI1+/nRhn3/I3H2dyT CWiQNpRiPQRafsMVzJuKvQfeEH+1Tq3tgXAsQ6cOwHdj+meuDj519q0nCyGtnfsEOn+EvZxE15m xJKQO6r0xRu5wppGfL8Q59Kp2vechl8SkOaAiPQTZ7PvUGJRUgkXj2gIkty9XESJ5w+VYsszqEH PW/vLHohQU3c4RnAwnwVaR9dwEK2F5XuKvO1DDGQaBexuHa X-Received: by 2002:a05:690c:c604:b0:821:1288:f820 with SMTP id 00721157ae682-84c95591347mr57984377b3.4.1787581561128; Mon, 24 Aug 2026 07:26:01 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84ca68c1b14sm34720667b3.21.2026.08.24.07.25.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:26:00 -0700 (PDT) Date: Mon, 24 Aug 2026 07:25:56 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 14/25] mm/fbatch: no lru_add_drain() nor _all() for memfd_wait_for_pins() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <4bad9833-60f5-30fd-098f-3a306722833d@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memfd_tag_pins() used lru_add_drain(), memfd_wait_for_pins() then used lru_add_drain_all(): remove those calls, no longer useful now that the per-cpu fbatch references are gone. Signed-off-by: Hugh Dickins --- mm/memfd.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/mm/memfd.c b/mm/memfd.c index c708d92533f4..064f24dcddaa 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -40,8 +40,6 @@ static void memfd_tag_pins(struct xa_state *xas) struct folio *folio; int latency =3D 0; =20 - lru_add_drain(); - xas_lock_irq(xas); xas_for_each(xas, folio, ULONG_MAX) { if (!xa_is_value(folio) && memfd_folio_has_extra_refs(folio)) @@ -168,9 +166,7 @@ static int memfd_wait_for_pins(struct address_space *ma= pping) if (!xas_marked(&xas, MEMFD_TAG_PINNED)) break; =20 - if (!scan) - lru_add_drain_all(); - else if (schedule_timeout_killable((HZ << scan) / 200)) + if (scan && schedule_timeout_killable((HZ << scan) / 200)) scan =3D LAST_SCAN; =20 xas_set(&xas, 0); --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 592AE3B2D1A for ; Mon, 24 Aug 2026 14:27:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581673; cv=none; b=ZYQ9r7W0Nc+y+FvdEA+OtpC/CGzbCSvJpJvaIesOWgA8MLqz+7diqJUKMORKG53w3wu/v74INT7DDSwW3x7Za895qfC5JEXg3WH2FF46LGjI8h2PMPbEt1O8WsM5swSdXbqeFBcRqKjV9fz+Hb7ywj1U5QTk179DyNDSqCT2gXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581673; c=relaxed/simple; bh=K5tiLB2R8cTQP87kSzydxOBlV+RGaP+4Wl5t2pez52c=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=ZAbLZpRVMj3Quu5kU1aXekQjh3FQac+WGNgGO6S6DHwSNyuplE5rUXYEcctwX7L9YkhmYO7jJgKnmGf0GLLuO4kNsUS1AkjYKgasVyR9rDaX21tIvgCL90JZDsLHc2HHb3H6qpsM8O77N6aSd+RbKbW7V25BL25hqy5l48Vq5Ks= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jABRjcEF; arc=none smtp.client-ip=209.85.128.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jABRjcEF" Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-836c4474028so41010067b3.0 for ; Mon, 24 Aug 2026 07:27:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787581671; x=1788186471; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=M66vr5Wj9LfuIyfBX17/YRcr9cErJu0CoiHB0PAp+Bg=; b=jABRjcEF/5TjwJko1iCIOR8dVyjMem4iahZMg4fJIuUiUIgSGqpWMDL3AeDBrOcA1e JUbYqoDsQUUK+OLz5uZmjuKDGbmLy51AYdJ0FsgiME3QQw7FnshKxODGY6pV39e5Mr61 o2AsY5bXVAJrivZfZZvxaW/bYNOE6mbpezsEgSanQquVE9LYUqvT5hWlS2ueqgES2w+r 7cPp0pWQ855xC1rK94DmRjToW6jEvyQ/lZLLLYQYhUHjcc9OIsqIATwmWdW01Vz8KiQ0 T8xydmUk7A0a1wtxF021HYsVou6pNjG0e9UNI66a6YQKXIYjvrWEK19/zaIZUFqYVmLa PN4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787581671; x=1788186471; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=M66vr5Wj9LfuIyfBX17/YRcr9cErJu0CoiHB0PAp+Bg=; b=H9Nf+OCG7a3VHK2LXjQnZND3LRxtjrOCYoIgLWNGLVNf1vv+8FKQYCXkYO8m7dwg+B RvhImCGVZyt6k6uIwTnxKKa3YVRuPwxhF1y2s4wU9ehSme/MYDaQes+tCMF6yuutXRQJ mrFKnz2z83ssxQjmfgbWuV1gTMLIP0AtUmPFtE7BlI+hAmAuHacFKKtt22USkn6SSPFW 2r5BC0AwtLmQxLexNSWOWuckbYNn3zBK/tHB5ycavG+NFJGDRVQ9a2yOf1/geTryr0dH IfFJ6UqCBtsnXiKYga49dqB+D2SF6jKI56k9x9kqiqLUSRcn4MNgkwJa63duZrFWpSyK QSLA== X-Forwarded-Encrypted: i=1; AHgh+RorQkaKKjcWnvf8OYka/t30QQqkOuoFUlZJT9e27KVu9QZDtwJlnr/9BDPLEM3rMDohcq+N6i8mGbopnzk=@vger.kernel.org X-Gm-Message-State: AFuF++lZSTZmqWsRNguzY7vZiArB4U9Z5pR5UQjxQb0TB9UgIdG53HGP wdCh291zabo1nabV/th7iR8Gc2w8YAiorFn8w9qtNiAcyTIkCcua+VFn0Jt9tyzM8g== X-Gm-Gg: AR+sD12IL9LL5X/QfCoELtN5iOOTZGNQG27vS4vP2gQNK5BISEl/dkYFWJcnjGuLpBp trSuNa8Hm3lVjzwPE03URwNuNjFYAUdVFHUb2zYu3HhYl8DsjIcLrYHPfK/c1d2peym4zwzk+yN 4IE3u94xfxk1eXDemwwIJ5vcpXd9lV0+es9w84hZAOYi7jwhsSq3Hbzob5SLLiZxEIisMhg6EJE s1rnJo7J0gXnwUeU/xilcBDfJnt/KJVmE36iFTU7IaZM/riGeggICwX0pzmCwU5bmWI/hINZDOD PDvtSVIcCAMNfNUs8T8BCK1hZNKn2zMAE+R8vDVy3ICJ+Rin2ZmF+G7lY8sGEvmv3c0TLwsqb/1 omi/2wZcHvdA/OreYsBgEAjKlRQ2BI4tmr97/d7IzUCm5KBQoq/1ulT7qUS+469rnbTfhLdur7R wN3VUrO5j9zyQmund2D3T9boosrnS0lOQlajN2TYibrLyIa7QZvxfmKXPM3x1iwVc6rHaE3+XRR /XxP7tu/v9Dzy7Y+ex3rSop5lrV1aLS5A9Gn23qVgZdtmaX X-Received: by 2002:a05:690c:6d81:b0:851:5445:ad5b with SMTP id 00721157ae682-8515445b0cfmr26010647b3.1.1787581670595; Mon, 24 Aug 2026 07:27:50 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cacacb78fsm33749697b3.49.2026.08.24.07.27.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:27:48 -0700 (PDT) Date: Mon, 24 Aug 2026 07:27:44 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 15/25] mm/fbatch: remove shake_folio() shake_page() from memory-failure In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <213f92b1-7ac2-189c-4656-ba7d00cc7666@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" shake_folio()'s lru_add_drain_all() no longer serves a purpose, now that the per-cpu fbatch references are gone. Are the retries in get_any_page() then still useful? Not obvious, so keep them. Signed-off-by: Hugh Dickins Acked-by: Miaohe Lin --- mm/hwpoison-inject.c | 1 - mm/internal.h | 1 - mm/memory-failure.c | 38 +------------------------------------- 3 files changed, 1 insertion(+), 39 deletions(-) diff --git a/mm/hwpoison-inject.c b/mm/hwpoison-inject.c index a11222572f97..9eab4b7d25b2 100644 --- a/mm/hwpoison-inject.c +++ b/mm/hwpoison-inject.c @@ -118,7 +118,6 @@ static int hwpoison_inject(void *data, u64 val) if (!hwpoison_filter_enable) goto inject; =20 - shake_folio(folio); /* * This implies unable to support non-LRU pages except free page. */ diff --git a/mm/internal.h b/mm/internal.h index ff4bd3a14539..9a25552cbd83 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1140,7 +1140,6 @@ static inline bool node_reclaim_enabled(void) */ #ifdef CONFIG_MEMORY_FAILURE int unmap_poisoned_folio(struct folio *folio, unsigned long pfn, bool must= _kill); -void shake_folio(struct folio *folio); typedef int hwpoison_filter_func_t(struct page *p); void hwpoison_filter_register(hwpoison_filter_func_t *filter); void hwpoison_filter_unregister(void); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index aaf14608b30e..2a6a01e260ed 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -310,30 +310,6 @@ static int kill_proc(struct to_kill *tk, unsigned long= pfn, int flags) return ret; } =20 -/* - * Unknown page type encountered. Try to check whether it can turn PageLRU= by - * lru_add_drain_all. - */ -void shake_folio(struct folio *folio) -{ - if (folio_test_hugetlb(folio)) - return; - /* - * TODO: Could shrink slab caches here if a lightweight range-based - * shrinker will be available. - */ - if (folio_test_slab(folio)) - return; - - lru_add_drain_all(); -} -EXPORT_SYMBOL_GPL(shake_folio); - -static void shake_page(struct page *page) -{ - shake_folio(page_folio(page)); -} - static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma, unsigned long address) { @@ -1459,10 +1435,8 @@ static int get_any_page(struct page *p, unsigned lon= g flags) * We raced with (possibly temporary) unhandlable * page, retry. */ - if (pass++ < GET_PAGE_MAX_RETRY_NUM) { - shake_page(p); + if (pass++ < GET_PAGE_MAX_RETRY_NUM) goto try_again; - } ret =3D -EIO; goto out; } @@ -1477,7 +1451,6 @@ static int get_any_page(struct page *p, unsigned long= flags) */ if (pass++ < GET_PAGE_MAX_RETRY_NUM) { put_page(p); - shake_page(p); count_increased =3D false; goto try_again; } @@ -1627,7 +1600,6 @@ static bool hwpoison_user_mappings(struct folio *foli= o, struct page *p, LIST_HEAD(tokill); bool unmap_success; bool forcekill; - bool mlocked =3D folio_test_mlocked(folio); =20 /* * Here we are interested only in user-mapped pages, so skip any @@ -1658,13 +1630,6 @@ static bool hwpoison_user_mappings(struct folio *fol= io, struct page *p, pr_err("%#lx: failed to unmap page (folio mapcount=3D%d)\n", pfn, folio_mapcount(folio)); =20 - /* - * try_to_unmap() might put mlocked page in lru cache, so call - * shake_page() again to ensure that it's flushed. - */ - if (mlocked) - shake_folio(folio); - /* * Now that the dirty bit has been propagated to the * struct page and all unmaps done we can decide if @@ -2554,7 +2519,6 @@ int memory_failure(unsigned long pfn, int flags) * The check (unnecessarily) ignores LRU pages being isolated and * walked by the page reclaim code, however that's not a big loss. */ - shake_folio(folio); =20 folio_lock(folio); =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 950F0349CEF for ; Mon, 24 Aug 2026 14:30:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581814; cv=none; b=dDvNAuHZv3yoeqIOQ+6pMINAOsWXxZdwGN03xwOUdRO3tJjlBieDN/v6irNw884Wd1RJZ31oB6Rva07sV1VlDEFl1sSKBPSOBsBjtO3eLwFxCbMFC2zl/+QDy5CCeMaPudZe8J3heWG1x8eYT8LPJorKL2KxVM+q09IMXk8dTrk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581814; c=relaxed/simple; bh=9DZojNp3mZvt3EkXO4hFRxVx3MJf/eYMKHBe+MyaBCs=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=tQFbwnb6r/aiNh/9qbWDz4IQhCzAGKa7I+tOpnYKYBBAD6nqF50oc+kDP0eXDvO2qYtYsrDzVKvDmyU8DWHRXhSnIR+qKY1NI//kwn8ygsG4wpXU/042at61d29hstt8UWjUhtPd8QYU0Mu9RdqUujOVoxelHEgdMqnNZ8+1Iq8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=c/BVKwms; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="c/BVKwms" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-84c4d4f68c7so38291277b3.2 for ; Mon, 24 Aug 2026 07:30:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787581810; x=1788186610; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=jkA2C/lQgXcypv6Ly6gx3bJF+pwu7LTevogMVJJ14Qg=; b=c/BVKwmskUlMdRTYj31BuO49DkQnZEsVS88LeGTloCLEWl67MFMWY9U6KoXZpWhL40 eRqUhLiCwNUHpoEAUbBakuEAE2RKa+OlK1PdTbgFZGSU6shJu4ef5s3HBzjKEue10BCh UaP72LhLPnPAwjIP/0lrcR7VmeJLeBadr6kBuOisMv3a0sLliDOg1cP7hOGxwMRZjqd4 IOGkyaDIwLoLH9X+/dH6BzXl5D5dmur3c+IMW6zopQSrIza0DobJfZO/OAnCv5ZqcNn5 oyszmgqtJojMd1wJIu+uEorStzGdD/KvpdkMbeUZCx0S0fyCX+422ZIVPjWNd6EqqMj2 Yj6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787581810; x=1788186610; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=jkA2C/lQgXcypv6Ly6gx3bJF+pwu7LTevogMVJJ14Qg=; b=MZ46clKFMpyJzgtklnj0ZY19reHidRaZ4SDEl46xvgjRSnD0di9I3mEaPJP25QFdg4 sqtUsgj86IFbPe58wopyyMIgvUoUeDjKTCWxVR3l32pPmJPz+mM9LfMtLMIDXIYeLBv3 vK1nQnhP3pIsJ2YD5qs2BLVRDTYmrjYk2q4/GM8FiylE6qh5r3DdFYPAU0je8EqV3JCS wWlMl3mQeg4+C8YwShwHT5ofWSFf4pXGQaKrVvn/V48fNe9bpU/Q2/vBcNCVAPxfQUz3 TuZbvWnlhqCfYp6kJb4smRShejzIl+uAhQAKa1VpAM/kpDYIzCaRuXyUhkYtNFg5SRfw w0Qg== X-Forwarded-Encrypted: i=1; AHgh+RrzMmlgfttE/SboE2fWHEgYq1MmtYX+cCLnTaRwGCQnwlVsxUnW/EejJQjzu4enggloiNBsS+NqQIput2Y=@vger.kernel.org X-Gm-Message-State: AFuF++m/gnOaTcQg/2N/PYynHyP48mweoksKzCRwSByvjnO3j+nk6V0w SGnIIV4eHSTio/7j1ARjx1Fzx9n4siZsIuPxqz0WkCvAxJjf10LNkQDQXe9Qi+yVVw== X-Gm-Gg: AR+sD101VYDdodwO/R6wVK3aEdkmgEqChkZHEtAsT1P8Sr8dv1uwOchMA72A8Lg1tnc g+aCkgIS0qC6Jjg3N//+aUgpID0H/QhqpJwgq5+MA6XOHf92ZvQDYCZ1HKudWcnJtL0FRuXvHq3 E8BFLCsl5y3a9I6LhPUAi81ebplh37MgNIMcrFbs64DhxurxGf5/zTIC5TL9eNFfUSEQ8vU0JqZ kTRuYFzNvxtszB4rgwYj8A28cBnyZAiNYMKCgEthXNXzPBcRC3WdQa2FTKEIeoBWfmkrEIaXeWY 4mA5vMZrQhJaD/ry7JNNTFgVoQOfXUamkI3mcNQat5F/nHJhG1IjPjJDlZsOzzsmh3z3oce4WH+ 4Qnpj3lT1zOkIBXu3ftCg78N+lD4PlCqG9VlgHFUSn53PrQEP1RkjWm/w1Uc9BK5jacZQ85CCRZ CJ65nDTaKgcQIQA0Kt4NRi7w5e91VKVnwgKgVwsKBJpfEwSyvyCnKZWPkzrIW4xYlg7OYhSWAO9 3YDgPpREXp+o9Q7kQz0LKEKr5qbMTP+DN7wyqRQuscB69gVkNcfy0AeuBY= X-Received: by 2002:a05:690c:e29a:20b0:81e:eaaa:535b with SMTP id 00721157ae682-849f6c946f8mr72860737b3.35.1787581809761; Mon, 24 Aug 2026 07:30:09 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84caabb9e88sm34055617b3.29.2026.08.24.07.30.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:30:08 -0700 (PDT) Date: Mon, 24 Aug 2026 07:30:04 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 16/25] mm/fbatch: remove lru_cache_disable(() from NUMA folio migration In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove lru_cache_disable() from mbind(MOVE), migrate_pages and move_pages syscall handling. They do not now benefit from lru_add_drain_all() first; they gain little benefit from invalidating buffer head LRUs first, since 5.0 commit 80409c65e2c6 ("mm: migrate: make buffer_migrate_page_norefs() actually succeed"); and it's a shame that (even without CAP_SYS_NICE) they can prevent concurrent tasks from enjoying the use of those caches. Signed-off-by: Hugh Dickins --- mm/mempolicy.c | 7 ------- mm/migrate.c | 3 --- 2 files changed, 10 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 501e0b80d7da..44912d707d57 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1343,8 +1343,6 @@ int do_migrate_pages(struct mm_struct *mm, const node= mask_t *from, long err =3D 0; nodemask_t tmp; =20 - lru_cache_disable(); - /* * Find a 'source' bit set in 'tmp' whose corresponding 'dest' * bit in 'to' is not also set in 'tmp'. Clear the found 'source' @@ -1425,7 +1423,6 @@ int do_migrate_pages(struct mm_struct *mm, const node= mask_t *from, break; } =20 - lru_cache_enable(); if (err < 0) return err; return (nr_failed < INT_MAX) ? nr_failed : INT_MAX; @@ -1530,8 +1527,6 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, if (!new) flags |=3D MPOL_MF_DISCONTIG_OK; =20 - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) - lru_cache_disable(); { NODEMASK_SCRATCH(scratch); if (scratch) { @@ -1626,8 +1621,6 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, putback_movable_pages(&pagelist); mpol_out: mpol_put(new); - if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) - lru_cache_enable(); return err; } =20 diff --git a/mm/migrate.c b/mm/migrate.c index 534908a0839a..bf9fe533a5fb 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2360,8 +2360,6 @@ static int do_pages_move(struct mm_struct *mm, nodema= sk_t task_nodes, int start, i; int err =3D 0, err1; =20 - lru_cache_disable(); - for (i =3D start =3D 0; i < nr_pages; i++) { const void __user *p; int node; @@ -2440,7 +2438,6 @@ static int do_pages_move(struct mm_struct *mm, nodema= sk_t task_nodes, if (err >=3D 0) err =3D err1; out: - lru_cache_enable(); return err; } =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5019A415F22 for ; Mon, 24 Aug 2026 14:32:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581966; cv=none; b=OxhHVUmU4LxCuYDZZ1HaRjCokD8+nKm86ZQeopQITrYvfLV9yykergUcRQRZAJgAPqbFUX60cMQb3WVkHKNro4rPbl6wBtr+Zjzk7i6mDipTC8i7kCn/6bU9ebGItcAeAcYvBMQi4q6QLhWy4T2gjDe3ho7+QfXbTa9e+xcjDXk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787581966; c=relaxed/simple; bh=IyiHBlYBydb0+wjSK5zpFY78pBcCj2Msqig9Vh8hGqc=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=mbbrh97nvv3VrzOtMJZYUJJ2gNIOkmSrX29PbFMJGBs75UhQD9oLR3TJHAV+Z68bSFvSQnE6RDEVEL1k/Ql2dwmYcI9T2oLjKXmxkjV2U9ZvAHetqKL/yp+gusU7kiQyLnuLwOd2km7Qg/51xgz11bYEN1YwSlTZ9YjNJoIVzug= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ec5CR+rg; arc=none smtp.client-ip=209.85.128.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ec5CR+rg" Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-836c8bde2dcso51088377b3.0 for ; Mon, 24 Aug 2026 07:32:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787581964; x=1788186764; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=DObH//Sc3+e2cOll9IRNEEzq8JrLC+/wx5igp55m6yw=; b=Ec5CR+rgyvv3vc2elUSuTy+DPNhomuft4y+vVpja58FKeJsAp1SD1tCZ+JylsdVUsT 2QvrtXoAzIK0B0px26LRaYKPScsSxVw6b9FQ3OuCt9pecz0EKz4/Z2PWaODYVAeB+Tux Z+Q77ns1SUVNKVJsoVMiF81Bl4j0Mklcv4LsQIY2CZtnaoJrMtcEDxqoaJiuYFQH50cN RIRptggEtQ4COT9vmQOtLNHF7I69MxYxAEKMZCS5jTgzCMdFoDOsZppN0yNG/TUkNikU 7KEtSkL27G3ph3RE6pmzS55nNPm2WXCIXF6BXkGSS/SRreo3r2DEuvx3bIkKG6kjhZP6 scAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787581964; x=1788186764; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=DObH//Sc3+e2cOll9IRNEEzq8JrLC+/wx5igp55m6yw=; b=mxA7EBm8WePsrblMg1/0HOv+HQy2RSl0MVGlsCUavngJXtI7fUrMXPrNqvRpCtRO2C UuDMoSZrSR2JwjeIpLonjyL/rPmDhdtfzUjr/c8tNlNL6UpUP8Bx5y9croMcz8ZBHudO r0Z6ZFOHZBsqyqFKU8tzNvJU8ysFDRdmieiAJe/4NX3LG+3FKJN/mdAyXjoSaLV8quYZ AfsQGYp7dKPiY9ebXT4hhPkhlTcfAi0tuzH3iAZAj6fcwWe0PXv9kaJQFUSgMEcYbgep duRbWTBSLvheU+X+cyzafaJEAeZXf2m7+vcDajmSWTfFpPDb1L7lfAMGRlzdUXT4LgYn ZfdQ== X-Forwarded-Encrypted: i=1; AHgh+RrtXH5Z4yhfAlsvgUROWNGSqtm7jFoUfntNaZo/BOiCebG2C5URjE4pCP/Vfi3mxdYreGwlLwOUKgpgww8=@vger.kernel.org X-Gm-Message-State: AFuF++kwpL9ORcBC442HS6dp33rwiJ17uyAiPTNXN6PV9+RMyzQXyeWi lM2r7JksNxEQ4sOHdkz95V5VSCAsO+N4g7b7LLfwVJXgxZO6Cr0dmJCJgY6i/qdXqg== X-Gm-Gg: AR+sD11go8/yd+Uf2/9AFR22dZDN8OKPn5MIah+8N/lcn+okYCG8lcaTHeDXUHklR2f 2GzfmAW3yLr532F0KU8LvQSaQr2Zu/+9EatXarMwiPzDFXfH+02PVfJrs8S/0oeSfLXrfw4wiPo ++Z68CXSkXTrZVhd/4Cfsd3UwGReZQDvLjN9FZz1XWORJI4cSwQqbdPhG5wH3MLpVqBXFaxZbcm f7CAI9oqO/772dL1xvRwDuIEPw4BXf5MTEi9zkbmWc53xIZKjJrWX7oQM++GvrPdX9zuA9yNbnS Iv2Nqt0CL/rqp5759byAeNR76hHZHuAOTFJv7l8hBDkpo45KcdOWL4AmlHxyuqJUF9/7e74gwOW 5OtXmIJICTzAcaRolZX+HHrR6gy621OUptkWfuE7uDdV+c5Q6t7jWoBTqanfgmD9Q7qFVMVHJxY ldLv6u/plJLv/cel1DSjGSL1TocyZJoXIlZdyfg5dqEWcB+SXWXqp1bZyVmExACG2pR1INH9GsH JnDX240EEQtmgaMfO4KR9pyhGmoKj1Ok5tPfklJ5MzsP6+H X-Received: by 2002:a05:690c:ec7:b0:81d:5155:213b with SMTP id 00721157ae682-84c9587c096mr75848627b3.14.1787581963521; Mon, 24 Aug 2026 07:32:43 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cac201719sm34583487b3.43.2026.08.24.07.32.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:32:42 -0700 (PDT) Date: Mon, 24 Aug 2026 07:32:38 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 17/25] mm/fbatch: no lru_cache_disable() in __alloc_contig_migrate_range() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <614c4116-8326-641e-d7bc-534fde0422b9@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove lru_cache_disable() from __alloc_contig_migrate_range(). It does not now benefit from lru_add_drain_all() first; and gains little benefit from invalidating buffer head LRUs first, since 5.0 commit 80409c65e2c6 ("mm: migrate: make buffer_migrate_page_norefs() actually succeed"). This will be more controversial. lru_cache_disable()+lru_cache_enable() were brought in for CMA page migration, see 5.13 commit d479960e44f2 ("mm: disable LRU pagevec during the migration temporarily") through 8cc621d2f45d ("mm: fs: invalidate BH LRU during page migration") - I guess the testing there must have been on a 4.19-based Android kernel, without 5.0's buffer_migrate_page_norefs(). It's possible that invalidating BH LRUs perhaps 0 times, perhaps N times, will average out worse than invalidating 1 time and stopping everyone else; or that folio_test_clear_lru() failures manifest more than before (note how folio migration has retries on raised refcount, but isolate_migratepages_block() no retry on failed test_clear_lru). But let's give this a try and look out for regressions. Don't delete lru_cache_disable() yet: leaving stale folio pointers in the per-cpu fbatches, with folio_try_get() yet to come on them, would be bad for memory hotremoval: the lru_cache_disable() in offline_pages() protects from that. Signed-off-by: Hugh Dickins --- mm/page_alloc.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 083cbcb5bdde..96ab51b78a5f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7135,8 +7135,6 @@ static int __alloc_contig_migrate_range(struct compac= t_control *cc, .reason =3D MR_CONTIG_RANGE, }; =20 - lru_cache_disable(); - while (pfn < end || !list_empty(&cc->migratepages)) { if (fatal_signal_pending(current)) { ret =3D -EINTR; @@ -7170,7 +7168,6 @@ static int __alloc_contig_migrate_range(struct compac= t_control *cc, break; } =20 - lru_cache_enable(); if (ret < 0) { if (!(cc->gfp_mask & __GFP_NOWARN) && ret =3D=3D -EBUSY) alloc_contig_dump_pages(&cc->migratepages); --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yx1-f51.google.com (mail-yx1-f51.google.com [74.125.224.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A46153F326F for ; Mon, 24 Aug 2026 14:34:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582087; cv=none; b=lmkgIrTfOAzwx+5+zJXTlOXeQd2igyWjIx+9TU2fyUqzbtoOYv0qGDdSoZacWPY71aZp10mFt0K0FdyPh2Z0Hy3MA+m4kl3jWz5b0Dg1dWhR33IelWj0k6jBU5e6AMXI+JrjdUYyaom11qFpA/zFqoD3jDY4KFvyp7xZrt3C58w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582087; c=relaxed/simple; bh=H0gw26kMTBjDSKLBQ5g+X+dYx41gnb8peREE6F6XyqQ=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=fZkztZob+XjoYQRcfQvWZYmSWCPIqaPffAbmtJzsKC359EiecpHLUpXfHxtnqbLh0ZBnShVnEBOy7yeHuyY8Y+PbBSXr6oTr/YthBriTmVgmR64ywaf/ganZdp6e6Lx4uTUyjPwRHWM6o43cIaSdLQUrOOauLsveQut5il3cozI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Rr8iH3ZU; arc=none smtp.client-ip=74.125.224.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Rr8iH3ZU" Received: by mail-yx1-f51.google.com with SMTP id 956f58d0204a3-66c67a73eb7so6790037d50.0 for ; Mon, 24 Aug 2026 07:34:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787582084; x=1788186884; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=03W8AdXTU8nvNZrlTGS6pFe21yyktV4JnRO2xEfU4uM=; b=Rr8iH3ZUq28RosshzS3vyZrlK7Pzq1o/nuoVy0jor82SLJUwkIdMl4YgoriS52hyxi vtgKLbYhwQv26xmHwTnHadaSK+gkieS8ME8xipcT96ukKDc5fL1a3QCXgQsVoJiL0FVg g/xzYTiMvly/xn3PHRwqnClD2xKQyn7dNFTW9sZRYUlj/8cKFx4Q8ewyld4x6M0PDSRM NRy9f6FHFC4kHDFHimmW5lyVhNGziKUN8qNCwnDdrksaMHdfNfdcB6TEk/cAnahdbFsP 9mi2qGNoM9FU6xwDv3Psw0zZfINlytuNIgYQTf5ll/GbcJMwALkjG6KPfbWfDp+n+QCN omqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787582084; x=1788186884; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=03W8AdXTU8nvNZrlTGS6pFe21yyktV4JnRO2xEfU4uM=; b=MXmNA1NQc8xr/sGQ9c8AOVzg2szT89fzdxEpiFa7g5zviZPXhUx2MF8sK2TbWcYECp CJSIP+ewEZQlZY9x7uamIeRWFMzZdtCjszDxtMAjR+YMuN2dHbA1RUXjcUht4WVYDB6B KK8RPDzM8JaRn1GMQ3NbtqBTVlaxxtoICnjmYM6ARBouO/g4pMpm6RDmLBuNM56488ye R53c1JgqGMrUY4HW1ZBU2NhyoRyWnR869FnNq44hh6RkIx6jSxqvHz4R7P09ujOWHAui L9ns3PVQqNWwMr9ygrkl3awrslY9NoqWRNCusHlg2xGpMGDVlThuPYn9Ixta1Iacn0U2 LjoA== X-Forwarded-Encrypted: i=1; AHgh+RpJesj1j8nUX8NUdPJIIiIrko1c8z58mWrWLtuwc6X5HiIn1McYKlMd+gnzRoAURyiAcdRSyr83aufuN14=@vger.kernel.org X-Gm-Message-State: AFuF++kPftpXNwJX9dV67vqUuPOlHIGEGPps3KSDesYBC8GNYinokTQq 2tEMxanEUwduBWEx/hGjUq1BiLzNwK4Gpxm0+aBzey30vP+eWuqR6NyVJPmZpcU6LA== X-Gm-Gg: AR+sD10cm0rPj7bz/keyeR++P8RJuOEX13nxwdaaEEo9PBb0YOtMtRiWxJukxF7bvhm GvIrfGJGdesFj5ezKAHYODVxVU3H3f3mA7zEvyQCO6j89wD8M+7V3VhJPE3ltj++FdB0VwLbHMw kHhiszSC+NFC6h0st5zEiOryIBHcwPTc+jkC2ZgYd8sd+PtJ5lQFOadBR06ik+A5hZ9qmmOLMg1 0X6TvHvm9KfttjcvOl6nYZwoJ3ec+Z4aBzEvmX4d82ews6f5H8NzIGxsKX0A6AUIgXbEfaQpNfp 58Anh42et50zYvd9J/TiIZvGbipEW1nyZWT7+6gaVA5Re7jiXz74YYgCVNBoEwXezY+gJHXGfYJ fuAlr3n4UmIcUV5Ng5FlBtnR4TZSu4nsw/I/U7PfnKVBDBMMrciyYk94xoPv70EYzSkw4cISnqq eYXpe5I54ZN5/5mcA+/TPFydxozIuzuth7Q+mm0WUMB2c99HDuSa3UI4Fptins+CGDP86Jv2bO2 NaukM5sjIxvPgqz20zDAP/D3rwVeHzLSC4E6qxD+sB0pmYEK3gjeIFJlg8= X-Received: by 2002:a53:bd04:0:b0:668:1e79:116e with SMTP id 956f58d0204a3-66ce5bb3350mr6460594d50.27.1787582083708; Mon, 24 Aug 2026 07:34:43 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cf4662219sm3783915d50.7.2026.08.24.07.34.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:34:42 -0700 (PDT) Date: Mon, 24 Aug 2026 07:34:38 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 18/25] mm/fbatch: remove lru_add_drain() and _all() calls from various In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <0b9e175f-edf5-630f-afd5-e71fb4e9d376@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Splitting into little patches gets tedious: now that folios on per-cpu fbatches no longer hold an extra reference (and have the lru flag set), most calls to lru_add_drain(), and more importantly lru_add_drain_all(), should be removed. Remove lru_add_drain() from compact_zone(). Remove lru_add_drain_all() from compact_nodes() and compact_store(). Keep lru_add_drain_cpu_zone() in compact_zone(): it was always a bit of a hack, a cheap way to get a local_lock() around pcp drain_pages_zone(). Remove lru_add_drain() and lru_add_drain_all() from generic_fadvise( POSIX_FADV_DONTNEED), but still retry once on failure; remove outdated comment line from mapping_try_invalidate(). Remove every lru_add_drain() and lru_add_drain_all() from khugepaged.c. Remove lru_add_drain_all() from KSM's scan_get_next_rmap_item() restart. Remove lru_add_drain() from wp_can_reuse_anon_folio() and do_swap_page(). Remove lru_add_drain(), lru_add_drain_all() from migrate_device_unmap(). Keep lru_add_drain()s in mm/gup.c populate_vma_page_range() and faultin_page_range(): good housekeeping after a bulk operation. Keep lru_add_drain()s in mm/madvise.c: the ones after a bulk op probably do want to "Push any new pages onto the LRU now", and update the stats; the ones before a bulk op may be trying to stabilize initial conditions, or to minimize draining under ptlock. Keep lru_add_drain_all() in memcg-v1's mem_cgroup_force_empty(): it probably does want to push onto the LRU and update the stats. Keep lru_add_drain() after bulk op in mlock_vma_pages_range(), partly for "Unevictable kB" accuracy; but remove the lru_add_drain() before it. Keep lru_add_drain()s in swap_cluster_readahead(), swap_vma_readahead(): they do want to "Push any new pages onto the LRU now" (akpm 2.5.46). Keep lru_add_drain()s and lru_add_drain_all() throughout mm/vmscan.c: it works on LRUs, so it does need folios to be on an actual LRU. Signed-off-by: Hugh Dickins --- mm/compaction.c | 12 +----------- mm/fadvise.c | 17 +---------------- mm/khugepaged.c | 11 ----------- mm/ksm.c | 12 ------------ mm/memory.c | 14 ++------------ mm/migrate_device.c | 9 --------- mm/mlock.c | 1 - mm/truncate.c | 1 - 8 files changed, 4 insertions(+), 73 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9e045a90ba21..8ed375b08f3e 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2652,9 +2652,6 @@ compact_zone(struct compact_control *cc, struct captu= re_control *capc) =20 trace_mm_compaction_begin(cc, start_pfn, end_pfn, sync); =20 - /* lru_add_drain_all could be expensive with involving other CPUs */ - lru_add_drain(); - while ((ret =3D compact_finished(cc)) =3D=3D COMPACT_CONTINUE) { int err; unsigned long iteration_start_pfn =3D cc->migrate_pfn; @@ -2969,9 +2966,6 @@ static int compact_nodes(void) { int ret, nid; =20 - /* Flush pending updates to the LRU lists */ - lru_add_drain_all(); - for_each_online_node(nid) { ret =3D compact_node(NODE_DATA(nid), false); if (ret) @@ -3036,12 +3030,8 @@ static ssize_t compact_store(struct device *dev, { int nid =3D dev->id; =20 - if (nid >=3D 0 && nid < nr_node_ids && node_online(nid)) { - /* Flush pending updates to the LRU lists */ - lru_add_drain_all(); - + if (nid >=3D 0 && nid < nr_node_ids && node_online(nid)) compact_node(NODE_DATA(nid), false); - } =20 return count; } diff --git a/mm/fadvise.c b/mm/fadvise.c index b63fe21416ff..f788a5020384 100644 --- a/mm/fadvise.c +++ b/mm/fadvise.c @@ -143,27 +143,12 @@ int generic_fadvise(struct file *file, loff_t offset,= loff_t len, int advice) if (end_index >=3D start_index) { unsigned long nr_failed =3D 0; =20 - /* - * It's common to FADV_DONTNEED right after - * the read or write that instantiates the - * pages, in which case there will be some - * sitting on the local LRU cache. Try to - * avoid the expensive remote drain and the - * second cache tree walk below by flushing - * them out right away. - */ - lru_add_drain(); - mapping_try_invalidate(mapping, start_index, end_index, &nr_failed); - /* - * The failures may be due to the folio being - * in the LRU cache of a remote CPU. Drain all - * caches and try again. + * Retry if any failures, in case they were transient. */ if (nr_failed) { - lru_add_drain_all(); invalidate_mapping_pages(mapping, start_index, end_index); } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b237f6e7662a..904a45c11fca 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1225,10 +1225,6 @@ static enum scan_result __collapse_huge_page_swapin(= struct mm_struct *mm, if (pte) pte_unmap(pte); =20 - /* Drain LRU cache to remove extra pin on the swapped in pages */ - if (swapped_in) - lru_add_drain(); - result =3D SCAN_SUCCEED; out: trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result, @@ -2325,8 +2321,6 @@ static enum scan_result collapse_file(struct mm_struc= t *mm, unsigned long addr, result =3D SCAN_FAIL; goto xa_unlocked; } - /* drain lru cache to help folio_isolate_lru() */ - lru_add_drain(); } else if (folio_trylock(folio)) { folio_get(folio); xas_unlock_irq(&xas); @@ -2340,8 +2334,6 @@ static enum scan_result collapse_file(struct mm_struc= t *mm, unsigned long addr, page_cache_sync_readahead(mapping, &file->f_ra, file, index, end - index); - /* drain lru cache to help folio_isolate_lru() */ - lru_add_drain(); folio =3D filemap_lock_folio(mapping, index); if (IS_ERR(folio)) { result =3D SCAN_FAIL; @@ -2981,8 +2973,6 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) bool wait =3D true; enum scan_result result =3D SCAN_SUCCEED; =20 - lru_add_drain_all(); - cc->progress =3D 0; while (true) { cond_resched(); @@ -3214,7 +3204,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc->progress =3D 0; =20 mmgrab(mm); - lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { enum scan_result result =3D SCAN_FAIL; diff --git a/mm/ksm.c b/mm/ksm.c index b4142746777e..5bae928620ad 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2626,18 +2626,6 @@ static struct ksm_rmap_item *scan_get_next_rmap_item= (struct page **page) advisor_start_scan(); trace_ksm_start_scan(ksm_scan.seqnr, ksm_rmap_items); =20 - /* - * A number of pages can hang around indefinitely in per-cpu - * LRU cache, raised page count preventing write_protect_page - * from merging them. Though it doesn't really matter much, - * it is puzzling to see some stuck in pages_volatile until - * other activity jostles them out, and they also prevented - * LTP's KSM test from succeeding deterministically; so drain - * them here (here rather than on entry to ksm_do_scan(), - * so we don't IPI too often when pages_to_scan is set low). - */ - lru_add_drain_all(); - /* * Whereas stale stable_nodes on the stable_tree itself * get pruned in the regular course of stable_tree_search(), diff --git a/mm/memory.c b/mm/memory.c index 8da0f945141b..1478bee69e30 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4251,9 +4251,6 @@ static bool __wp_can_reuse_large_anon_folio(struct fo= lio *folio, static bool wp_can_reuse_anon_folio(struct folio *folio, struct vm_area_struct *vma) { - const bool maybe_in_lru_cache =3D !folio_test_lru(folio); - const bool in_swapcache =3D folio_test_swapcache(folio); - if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && folio_test_large(folio)) return __wp_can_reuse_large_anon_folio(folio, vma); =20 @@ -4264,16 +4261,9 @@ static bool wp_can_reuse_anon_folio(struct folio *fo= lio, * * KSM doesn't necessarily raise the folio refcount. */ - if (folio_test_ksm(folio) || - folio_ref_count(folio) > 1 + maybe_in_lru_cache + in_swapcache) + if (folio_test_ksm(folio)) return false; - if (maybe_in_lru_cache) - /* - * We cannot easily detect+handle references from - * remote LRU caches or references to LRU folios. - */ - lru_add_drain(); - if (folio_ref_count(folio) > 1 + in_swapcache) + if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) return false; if (!folio_trylock(folio)) return false; diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 18d097c38853..d3f865904c48 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -575,11 +575,8 @@ static unsigned long migrate_device_unmap(unsigned lon= g *src_pfns, struct folio *fault_folio =3D fault_page ? page_folio(fault_page) : NULL; unsigned long i, restore =3D 0; - bool allow_drain =3D true; unsigned long unmapped =3D 0; =20 - lru_add_drain(); - for (i =3D 0; i < npages; ) { struct page *page =3D migrate_pfn_to_page(src_pfns[i]); struct folio *folio; @@ -600,12 +597,6 @@ static unsigned long migrate_device_unmap(unsigned lon= g *src_pfns, =20 /* ZONE_DEVICE folios are not on LRU */ if (!folio_is_zone_device(folio)) { - if (!folio_test_lru(folio) && allow_drain) { - /* Drain CPU's lru cache */ - lru_add_drain_all(); - allow_drain =3D false; - } - if (!folio_isolate_lru(folio)) { src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; restore++; diff --git a/mm/mlock.c b/mm/mlock.c index 97134eff6b56..971430e6251e 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -424,7 +424,6 @@ static void mlock_vma_pages_range(struct vm_area_struct= *vma, vma_start_write(vma); vma_flags_reset_once(vma, new_vma_flags); =20 - lru_add_drain(); walk_page_range_vma(vma, start, end, &mlock_walk_ops, NULL); lru_add_drain(); =20 diff --git a/mm/truncate.c b/mm/truncate.c index 4151f7a167e3..7ea5d513f76a 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -571,7 +571,6 @@ unsigned long mapping_try_invalidate(struct address_spa= ce *mapping, */ if (!ret) { deactivate_file_folio(folio); - /* Likely in the lru cache of a remote CPU */ if (nr_failed) (*nr_failed)++; } --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yx1-f50.google.com (mail-yx1-f50.google.com [74.125.224.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC338438011 for ; Mon, 24 Aug 2026 14:37:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582225; cv=none; b=J+KkKzKdPgYTJcWn2hz2FkR3Pcgl9JhRkZ8+oLmQrULGu9Uzp4yJFA8MbPk9Igg8SwgLMKOhG61DK7QHTZW+3+G+KEq1ZJWwZGbhpLpv+S3MdSKfyJBVmKh7lqalaKe+V8cqNn5wRkpHQf4EWe8Amq0Gyx3kYqS12E7BXEJjD2M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582225; c=relaxed/simple; bh=1YRYVGkuKNtQk+ObMqK/890RkRJk46OHbYzz71c/FoY=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=skTAz/xT5vNaajjXPeYvS7sT/2g4ZD00+yGCXuxaEPa2oBwtlMfg3UrGnCOWXHIX9XcCVh0oFvI8VFY+5nA4xEVRXXx/N6ZbLuXq2r42b/UMxKGJ0pz7xQsnDjoDr6TiuxMQ77EYyGul/2KGxZimJJAMSgnJGNRgsXRDaZcj2JA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CpoTD5X2; arc=none smtp.client-ip=74.125.224.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CpoTD5X2" Received: by mail-yx1-f50.google.com with SMTP id 956f58d0204a3-66d06b5f023so1612382d50.2 for ; Mon, 24 Aug 2026 07:37:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787582222; x=1788187022; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=dFFCpD+eyjMYiOlaAWIMnRn59i/x4sm2LgO1kNi90A0=; b=CpoTD5X2Yap6uyOE19vYISXLZLUPaFug+oYxYnFCxjdDvc/IoO+SW/2rh9+xSgQaCo MxsDfnJdHR4y4UsTLKJ9tB5Eu3l/k9lwrum05vtjfTL6VFXGra1GIDZWaDJSjrAM6nwj aOIzSjO/H69XukuT82R+NGw3hyFDx3YegL7S5PRdlzPOdtEoyB/qTVRrtdGa1MwzAcxh H1PNXBt9gLhpHKRMVy8RP/HaVXlhrOHTI2ZXz6NUWxzGW5RulmZ2LrtgJTuYaj4fWNl6 4RUJpUUkoPUMOzaHzrUOYgmftSCaIQ6reFW+xKhc8L7+qgiEv8+BI4UcNtYj9curBKdR T/gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787582222; x=1788187022; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dFFCpD+eyjMYiOlaAWIMnRn59i/x4sm2LgO1kNi90A0=; b=Yh4zcNSpE+qYy7GkpE/oNUOau8It0BbRXrOCaKOGt5wZA800Onj3J/ElbXyzSabueC nlPN/XGU6QtBqghodpZmjtDGZXmtmsU21emy2HDs/pgvTbsxoM6BcT6fgvUAjlgxH5J1 IWQOkR9ZEnI3LFjjci4zzSCmIppjIJ4DGgSs0UPbmpbHbBiABdcwPzoScmJBd53LJruh Mjf22/ugWXnzDKuGLbS4y1ithnzVnLnQv95weK33dC19CAsPgx5CA65PErVHrfhva6lD CaMED/oo/47Tdi9Vqj4HWD1cMuRpcAphykaMEwKyExWNVKx6MxHb6z5ZWxUw7wFn4G+0 YkkQ== X-Forwarded-Encrypted: i=1; AHgh+Ro7svYwCgiE1/7ZlU0r/cz+lXyTOoTKoJSEB1i1ODFtfy0acNYnoaMb+E5DXs8xQI0wNCSHIYfSxRhxhdU=@vger.kernel.org X-Gm-Message-State: AFuF++lidkqIAtknMUqOk3N62SJ5XrhjccYhMtVIOdOAgnvKnntcFBgu 4aYslRrhno8ud1MIFJfHm2s9dUO6A+ye2rY6hNSqLx1e2q0cb0OEF5zIjqfkawd/5A== X-Gm-Gg: AR+sD114w12KH+xcWEZ4DpmA9GdazK7vB5Ng3BH7Kw3kwUtqIw+khPTht/73ueKYD3H 0vWZzgu8SmYGC5Drx4hAuToOnQwvxAGQ2DER4RoVeLZfi5O1ovZExeYR25AQbqZQz5B4A0AdCSs pqj7poecYq69KDRQph2Uiyy7ACBHcM/IVA0xCPAynZe0+vy5fvLIHCavmVjh9kYB/cUrC1ab/kh 7BhIo6ItMjWrxMc+/HmSEtJOSOZ5AvPkIhWLiovnKC5y0LFhJCTvrPrWAKR9JyUpMwHckwJANlS NIS2zZI95r1iKdXVb7AVfdQ5OQ0oyR3nMxPXkIvyDircPdxZthGJ5izzCY9eVT40GatxpqSpkN7 grTqPmXEKYPLsdxEiL8EHowuJLDQzyUQRLlMTklKnFsVHcpeJMK5oaSAPPZ1rXfpi6IXSYW0Ly+ T2/HM4o+8PYiJBzNeubf08oeN15v0SrmTGsgFLEXBiKvtw4QmlMl5cqMaNCkU30d3+QNKa0xusX s8IO6uSP5/ywniHdFDDBxMtbuFVjaws/Yc8Ik9bO878O64F X-Received: by 2002:a05:690e:1309:b0:66d:7b7:e36c with SMTP id 956f58d0204a3-66d07b7ea83mr2357073d50.48.1787582221803; Mon, 24 Aug 2026 07:37:01 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cf46460aesm3729534d50.6.2026.08.24.07.36.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:37:01 -0700 (PDT) Date: Mon, 24 Aug 2026 07:36:57 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 19/25] mm/fbatch: vm/stat_refresh include lru_add_drain() on each cpu In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <509da837-bd5c-2090-d26d-0002fef0b1ac@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Reading or writing /proc/sys/vm/stat_refresh has long been a way for testing to force an immediate refresh of /proc/vmstat numbers: it seems appropriate that it should also call lru_add_drain() on each cpu that it visits, before collecting that processor's diffs. Add that in refresh_vm_stats(). I wanted it in refresh_cpu_vm_stats(), which would include it in the regular (default HZ) vmstat_update() too: which might be a wise precaution when proposing to eliminate almost all the old calls to lru_add_drain_all(). However, that would go against its "strives to only access node local memory" cacheline care, and bring it into lruvec lock contention: which would probably displease its authors. Signed-off-by: Hugh Dickins --- mm/vmstat.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/vmstat.c b/mm/vmstat.c index 3b5cb1031f72..0f75488687b5 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1973,6 +1973,7 @@ static int vmstat_late_init_done; #ifdef CONFIG_PROC_FS static void refresh_vm_stats(struct work_struct *work) { + lru_add_drain(); refresh_cpu_vm_stats(true); } =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 108F043C076 for ; Mon, 24 Aug 2026 14:39:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582360; cv=none; b=LK0H+B7qjLcngTvUNk+EB08XJ52UYZSSpRreVcibY6SlauxpgIKBOgQu+V5401me9knRk9UcmQp34vgZ0+76yABtaZzgiMp9DygKwlRK+djrmo93p+vj+pbs1CsbfO+eg6cp9yb0l5wD0UNiTaa0ciGY4K5SAlU+znc+seG/x3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582360; c=relaxed/simple; bh=RrFIEm8SrJEjhBjzApoefNfd8aQaR4Oxfr/yDuZhFPU=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=r1wPV3AaoPizBQxN9i7HrTnQw2RwKVSrprmM+CVhvmt0IujjPM4W5Ovu+WvYZVFSHerywr3k2GWc6s7s/1sfwDCykNDvdN9nVjBPz+HvtlLD76ZKKqVSzBCj4NeMoMhlYoOrbcyUm2D1D449zbCkUrHlgxPtFafDi9/oY6gjwLk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vHmlLxfi; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vHmlLxfi" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-836eac5682bso63495147b3.0 for ; Mon, 24 Aug 2026 07:39:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787582358; x=1788187158; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=2UY30k1UeQzYu1OvDSaJuzIQ8I7z52NjmS+xQ0qJeoY=; b=vHmlLxfitfZqNVju4Fys+i9kMW93CHbXhaQ4TweDYrYsnvNOrqfA4WWNkrT0GV/92H PYd56DwBcbzMkwVcAQkmr6KBnZVJJFOZsXEUcuPVnKXZwJKmJbGLyjkZerz1wOXyeEbD an7AWJQPNPrc/LaH7IVqH7T9ScABFmmDf224G9ZCReXmuCjOBYVw5TBayV+7kijVrrhf Jf2qsaDgt8SLMxtG6ZtvLErEWIVT61s7PJbodg/Md5vdN5JMfweDOaTuOU82gVunF2Ff 9tMKMIW/oJYKvQXAbp7KTKpCVjXuhACi7jzA9Ay0MgByz8RsaI30eSuu6zLynm4l1OHD tZZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787582358; x=1788187158; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=2UY30k1UeQzYu1OvDSaJuzIQ8I7z52NjmS+xQ0qJeoY=; b=a6SRhCjmK/yJJIn0AjgavvS9DzmD7qGc4JsI7AKv0IvtC1MhEGmuXg8Qzg8aV/5ziC 7bXE7zaKJVxz9a6V8hnGtT2iW7ZaHGoVVkGNc6Hs0CE/if97ItHSb/NRiLxlodQAx9M0 EfMgy4SzlUqC8OS488K8KXaF8Ipvd2boTeztNM//JX+4g5GHEj0wkZo3IV+q8irv7cLB Zfl8rGKxKE/F5XIVij9jjRhBfvhlXrUIe1g90niQBPWXf+hKTsZq8s/UTNTaNCRcaoRS swhW7O+UtUoIuWF8mTnLzt5hdleQYfNhx+DzLPa/qQbquaPBAvmuXIjw6XTMoxc5N68Q MhnQ== X-Forwarded-Encrypted: i=1; AHgh+RqEUEzzqdJAPPVrLaV6OU4pGmZ+jN+QRLfYo1E1JXOrFy1qquwqkFhkObK7Nx6kgqC37FSghuT5WiBTi7k=@vger.kernel.org X-Gm-Message-State: AFuF++l6OOJTbpP81TMFMoCJUAkWDGTZf9jBSDbh0yEKksr2HUfs0MT0 JknEh298RP6Ftu2ZkXiVne+M3O7Vp9rB6AdG7jbeeHRwqvEkfpCTs9PNPkBaf2e+jA== X-Gm-Gg: AR+sD13u1amrl7QBI144AWDfr8UZn7VhNiL8OJGBtdlVcAtaEtafHTQ1/eg+KTV+stm CVwcg/ZZYKPZtbpOv8Fb4U2VltrTIGkTy+t86qvnGWA1DB+/EYESxKA1gOt3XtnyOXYG3doG5Ug E8vNI5oxAVtGNgnzTT7wcJeKdcwn2tgIEO0NHgDFr4gKq7X5NCxKkLtl4NSsY94xYHbJD0jdR4R x2nuRrQ9vH3eTwCTsYCrm3aMyq7q2kkn98Ryor8+jqD+zTbPTIOouEh0IffAxCTTq82GBIrIbFc 933RB3TGBpju8gWH2VEhz2FPTguvJtPmDpP4tr3dvwfPfuq9NbseXY1fKVj56hesWNNz/6U94Us ACC1dLZFWmk0RXyJQhnlNUALMzec63WLxWOddPAE3P6FX17wVosjn6y/PfeAzy7+mLKQL/7wJ40 EWd4dFiyMvpufvQPUgzL/3k0JvmOa8QXKcoFBiVXhsG34XFudTcqKGaHHe0w/I4ubPTfmwvj8Nm AKiPBGJr6FCSGEbLfB00k3p8tsafipOmQPG9wLTzC+t3Ahn X-Received: by 2002:a05:690c:c221:b0:81e:a373:47ad with SMTP id 00721157ae682-847073b93fbmr133529677b3.2.1787582357182; Mon, 24 Aug 2026 07:39:17 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84ca5188982sm35471097b3.10.2026.08.24.07.39.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:39:16 -0700 (PDT) Date: Mon, 24 Aug 2026 07:39:12 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 20/25] s390/fbatch: no lru_add_drain_all() in s390_wiggle_split_folio() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <7bf68e7b-f88c-6ca9-39ab-e9fedab0c8be@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" s390_wiggle_split_folio() has no good reason to lru_add_drain_all(), now that the per-cpu fbatch references are gone. Signed-off-by: Hugh Dickins --- arch/s390/kernel/uv.c | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c index dc14ebc0105b..120a467026a5 100644 --- a/arch/s390/kernel/uv.c +++ b/arch/s390/kernel/uv.c @@ -364,7 +364,6 @@ int s390_wiggle_split_folio(struct mm_struct *mm, struc= t folio *folio) =20 lockdep_assert_not_held(&mm->mmap_lock); folio_wait_writeback(folio); - lru_add_drain_all(); =20 if (!folio_test_large(folio)) return 0; --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yx1-f53.google.com (mail-yx1-f53.google.com [74.125.224.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1DC443B489 for ; Mon, 24 Aug 2026 14:41:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582516; cv=none; b=scFw1Hc8IRfy7qzHZB7XnVMjl0N65YnV/D710soVheriDOdqVIj1SYVR7ZOVTRrJ+3Xen76MAbC2rdcVNxLb55gB3L7FrWIP8pLvr4Ps6lEdff6xw+09w3hOOzqQmo7bg1NUeCTZNSBtghebZ/JfoBG6kyuFsqF/1Hc07LKYtrU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582516; c=relaxed/simple; bh=PA7/Pbx5AEThbU1dExwqFT9w9coLuAoGrR7Ue2+wOTE=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=neaX4My/JIRhhhZEPzMw8Tmh4wiXraBaqidOg5WBSYkzh9g/yNreXfMKImk9RCcEPuBEUkgNAUqlaFXQcHmNjEi1rDERBuDky9qhMXDmYQ1tUXOz4SMgWPTToTUS87nBytLeKLrv58ND2qpCT9YHx7rDa8vdrlK7iWoTr8e++zM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rcFBr73Y; arc=none smtp.client-ip=74.125.224.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rcFBr73Y" Received: by mail-yx1-f53.google.com with SMTP id 956f58d0204a3-66d07ddf077so1567516d50.2 for ; Mon, 24 Aug 2026 07:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787582514; x=1788187314; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=tbn1aAX73v4jfLOAScY/xk2YxlfOe4UtWahA1Ug2y2k=; b=rcFBr73YIy47nKOwBwDuCgz1JK+wC6/lD6EuW9Isgoigtk5Wwk7ahW30oOmczMv4J8 Zs2r5rzqFq1Rdw3M+a9FtgKaYDrU0nNN8dxzGWoMtSuOKvsj+McppwvPEHfT6j+6f/kf OVVby1pB3/oQ8e+TscMluo4qRb6D43Wnj49aw2VcgXFD1VLCJCsQoesu1IcEXSqM5bRT kGxWhiScnSC0h9ug1UZEjOpthm8GS3LE30uHrPDfs4ddXPrxSnIVzyHeHsFc+O39zOW3 2x77IpHhRCR/Bb6g5YZmnvm9xrlh8drOBetRlA4xh0jh7vM4ClvivDELUUpHhYvzpdTS tSDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787582514; x=1788187314; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tbn1aAX73v4jfLOAScY/xk2YxlfOe4UtWahA1Ug2y2k=; b=ClC0eU90WzR85GrkVUQ6ifSHjBCyaY0d8H6ioHJPUwWkRHPxtlg1Pl/s/U6vHwJklH I/eTglkYsorrkLcPoF97h2rtDhpeW3y/uNsvavlCoNqPN2/j5xCvPDv7OH6ugMD9qNTI tdXUdauyD8cqKDOnNk4Ym4GCzWaqjaS7t3tBSUvLrmavN6Q8b4XrOjHlAxpifujKv6Yj tkwGcVyaJlOrZR/nlnXmPwHsr22Sbcw3AWpwBF98xfeFIhH1D30kiz23XURM6yF/QtLH b3LCzlQqJ3wIm/O4D34FXnw5FPvDg1qMK2+2ZIXBE3byIlnKsdKMtbHsUj6FEGwa5j7E sh0w== X-Forwarded-Encrypted: i=1; AHgh+Rqq4YBwHIJC4GGc1Mg8JiX3Ob8cjf46RzT+S3/NPW0jG7iYb/rVjfStFcXoADl1IITkgRxIhfdb6pYxGIc=@vger.kernel.org X-Gm-Message-State: AFuF++m4T6nGrZHP1DdWhBFX5O6O+1PyLbGPa04bjgFzQAp3b7lwK9ip Y2+vB3k9bgA+LlIPjH1s0r+c0dWr4frlJ/q+ovF7Yuc4k2UvaspP2dTeFFCZckOZAw== X-Gm-Gg: AR+sD137AVbotPr7Sj3OReABBYgAiXvur45v0ufzDdBZa1UBmgqQ0A20+bLaMaFXX+J 9b2g+rPU0YbWk95GIVnDO+yFd7QFJR0j5xwNDZatqafJL2nuvHhDkoUxD+2LBNr3J/jQn9fr1nV 5M23VJE4IZz1Dho1YOURidqHoYIwjknZ6gI5grygJl57TJDT4U/z0zc7e4kGdfD4yF8dBoGa/mV ezpCRz8msmlt6Sng9khyyp4x5fy0i1DRiCJD5IEy/kZny5APgEyH5nEXQrx2/F27o2AxUnLkdhj IqB+0b/0Q7b8HR2K05VTk9omexxMOmpv8g+Wp4dnH5IxDjROULg5fYRLc2AZ4bhSrd5srfcXRYY 3m6dX8XEzkaLNxn3iQKn/LbadDrfpIMaFOSMMXP37RKlFWEL1SmFrTdFl4EKLOFq/RwCtdQXxJb Pgry8n+VIruL6o2vl3z7XKXgya2ZZ19eKtegDumIWeXXwEUezvEhXhfbFc9SsOEb3w6XbjvtGcB U5JJx05ksevTeP0c2VSdkKh+4T5nw7sCOluHETkzF/j2kRlY1UUjSvSlmE= X-Received: by 2002:a05:690e:7b1:b0:667:cf87:5838 with SMTP id 956f58d0204a3-66cf222e93amr4903896d50.25.1787582512262; Mon, 24 Aug 2026 07:41:52 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84caabb8b35sm34455727b3.27.2026.08.24.07.41.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:41:51 -0700 (PDT) Date: Mon, 24 Aug 2026 07:41:46 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 21/25] block/fbatch: no lru_add_drain_all() in invalidate_bdev() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <8facb079-aa8a-2973-4882-173e7d06ac07@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" invalidate_bdev() has no good reason to lru_add_drain_all(), now that the per-cpu fbatch references are gone; but keep doing invalidate_bh_lrus(). Signed-off-by: Hugh Dickins --- block/bdev.c | 1 - 1 file changed, 1 deletion(-) diff --git a/block/bdev.c b/block/bdev.c index cd8323083740..853545b29b7f 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -97,7 +97,6 @@ void invalidate_bdev(struct block_device *bdev) =20 if (mapping->nrpages) { invalidate_bh_lrus(); - lru_add_drain_all(); /* make sure all lru add caches are flushed */ invalidate_mapping_pages(mapping, 0, -1); } } --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87FA2456DE0 for ; Mon, 24 Aug 2026 14:44:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582694; cv=none; b=PUSd4LafN5pY+Odbqt9Qz6k5LB3Gz3qaZpjb5ia52BOxjQdVb5LrC0u70oZJ8jr5Oz+j8aJM8qPGyDlpoEeGh+Z2pc7B+51nQyyvqIoiXyksBIttpJbcpKf96UU8Suh7E9sPsVY3rDUh/rMclA4G3wFbMVv+BsA38YdL1HrCMgs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582694; c=relaxed/simple; bh=apGggQcJp2WV1URiHg8Asmh4iwaxGu1pF3vH60oXvTw=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=fody2V7Qf+PqAwOxcqxOIhza2UUc2lgI76UxyyC2+1y9UzLE2y3klvTtJ3P30MujmLHb+3YSR8lUQv4JjhBXVSnY9apmqgfLfIlcynhTzdFIKksBQrzWDLTo5NGk0gtbeHC7iL9P/9Lm8UPK6XlAVc1Z0/AnyPi3LVvqT5AbIJA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vjfA+dv3; arc=none smtp.client-ip=209.85.128.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vjfA+dv3" Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-81ea0b7d137so30765987b3.2 for ; Mon, 24 Aug 2026 07:44:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787582691; x=1788187491; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0mZ5Gk4/1Pu5bK3S+WjW2BlP4j/qxWTHvLM7mSgUCfk=; b=vjfA+dv39bRlVtK0b0GiuzgSC5Dw1NWDJhcPFIX+5i1SXqDN4Poyu8PuB8wxuWaklV vSn+QfqIxrvfnXLqYLZGfBzwXcEU7Ohsog4WOxIHMfRylgketZWsD6uvG8IpdRqaCq36 av+sgXujeDW4AkrUEsvxcrhKTH3v0xkbKO7nJW0gNz3CTAwC5eRpASfwRQGgR4bD9NgD oEhHl4iFJ4T1KC8HnbPrLYQ+L/lEPhnZQhc6Guec/9L+By7XpAyI9eqgFlZZEbxQEml7 eEDt/MZvpYnD2UkNDh7s0MKICzHOW8oxHS7v3v/G1I96pVF2rwpJ5n1u6d7Wjc+48yc0 04hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787582691; x=1788187491; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0mZ5Gk4/1Pu5bK3S+WjW2BlP4j/qxWTHvLM7mSgUCfk=; b=NFE04RUajsSyr2wdWojWAxJCRk2hDEYiYSYXoGHFEuj3KaJ4NtxdSO2HfX7wo/FBmR 4cgKEjfq3rY07J7UpUW/+Ifm+InH+LapCogs86HTD3YJwSlqWTvPt1Gi8iT6N5F26B9D UfU9S5yMjxkxR5Jwp3pK7jXFV0C799X9Qub92cxADcembGjSC7WNvN6VSbniPhozX5XO 9Rv9VJeauaPeYyExYB48uymaMd1ZJDtj2oXshtzXVkBu6ayWkFZdYdwGHbnIoDZhAVZ6 TMIAAMZCkIotb9Ai9mZ4wHI6vfC3L5mTQZnXyvH2Q+DbM9ST0onYRuC9iapdGAx3Yvcb XKIA== X-Forwarded-Encrypted: i=1; AHgh+RrUcRvw/NESycaeFHZueg5U4EEGyCsTeHP/V2M+AFjaIh3AcBsBYRGPF9RlIKHnsGplK/XH0WCh8Kgm890=@vger.kernel.org X-Gm-Message-State: AFuF++nKazH/Vuyj32kTSbWQrL8xFaxLQaLlm85CFPRYADg0J277PPiK wc8CTAjLeEnMhBy0DWeJ3bk6IhSRJA6HmpAwShb0gGqKfpMsL76hdv6N7gYNQAXwvg== X-Gm-Gg: AR+sD137pGJOua6qVEfVnAkQuNHZXBPOB4K7Kw2n3wrDjjqbBExZ5DRCbtXd1KHJQuh R4Vxc9BOlGZUaS7zQgp0FY3zWTWnUc8vwRBpjYIRvXnS4U6GDZ/jwMzn5UOgv1JD1zDdiQ7D26w huigVOkIEdZdPXRZ0hdozUN0/EDG5nUn2bPxsIXf7QmriJtrl72XY167UVesPTAjIllCWF2J/qu /HBcEW/YWhg0Pvwhz4C2QXMrtLEQ0Om0GoaSXvjAy+SHad0khcLyRpq36tS7oVGXnC8SxzCtfd8 KU7ZY2kti/GYM+9/3jMOgCe+bb4Zt3MjqLlYHpfiIJFAUrF+gHMBDrmjYtxuOYbFV68EWJryuKL 6QPWVHZxgaTu3IEwbD5qEI3Cw3zvprMpfwPVU03osza7b2bzdPn0Qw0oN67vshz0pB0FFr7NZzm MUzp9QXH4Za2c/EzpYcJ9EqJqYYpsE2KP8oX+MbenOxRgK/dOwuZyh4wvyAqzn7ghL/0Qfp2tVl NtGJBiQJSv9dbldA4zI4k9TXbrAubrWRYa9JYz03n249sdh X-Received: by 2002:a05:690c:5:b0:853:f007:7557 with SMTP id 00721157ae682-853f016ba69mr7002677b3.19.1787582690568; Mon, 24 Aug 2026 07:44:50 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cacac521asm34834717b3.48.2026.08.24.07.44.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:44:50 -0700 (PDT) Date: Mon, 24 Aug 2026 07:44:46 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 22/25] fs/fbatch: drop_caches invalidate_bh_lrus() not lru_add_drain_all() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <97a16bae-dfe8-b58b-f0c9-6eafb3881195@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" drop_caches has no good reason to lru_add_drain_all(), now that the per-cpu fbatch references are gone; but it may have good reason to do the invalidate_bh_lrus() part of it. Signed-off-by: Hugh Dickins --- fs/drop_caches.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/drop_caches.c b/fs/drop_caches.c index 49f56a598ecb..054af78e6a2e 100644 --- a/fs/drop_caches.c +++ b/fs/drop_caches.c @@ -10,7 +10,7 @@ #include #include #include -#include +#include #include "internal.h" =20 /* A global variable is a bit ugly, but it keeps the code simple */ @@ -60,7 +60,7 @@ static int drop_caches_sysctl_handler(const struct ctl_ta= ble *table, int write, static int stfu; =20 if (sysctl_drop_caches & 1) { - lru_add_drain_all(); + invalidate_bh_lrus(); iterate_supers(drop_pagecache_sb, NULL); count_vm_event(DROP_PAGECACHE); } --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58DC136F8E6 for ; Mon, 24 Aug 2026 14:47:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582849; cv=none; b=XRA2mrgc1eGdLO4Sd+AASQOZSL7bUdDAFDMvZRZzK1F0ntGlkayV9MQq2aoiTIWC3EbQxKiFaCezsPHXBG9NvWwH0q+1GkO7LYQ+y1CXo3B5pcPlynEASpNSjkldXaGtTy/Yj3eDhwROA0sw3BnArFMPuRHAKbDprl6DS7azlhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582849; c=relaxed/simple; bh=ViLLHeiqDr9S1/zgmlJqAYe2z/6nKsf8TuVaGMbVWp8=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=Tt36R+1vUgaP2LWvrtaz68T6ZXf9f3Im9A9GX1M9drmBK+0HP0RcfPPwecrnKN9+L6d42Yadat79dvSKt9nfqOgBAb4NOFZK7zAqA5DMOzELVGsdypfGXNHWzB5wSz+SlmEqlyC1GSoS57Oj7YEIl/o9ubP/tN5V55aVBw1tpN8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EukbmAIJ; arc=none smtp.client-ip=209.85.128.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EukbmAIJ" Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-81ee6b2da98so49565457b3.3 for ; Mon, 24 Aug 2026 07:47:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787582847; x=1788187647; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0ep6UPzJTnZmCqVomVSsdJlCYmmRG8TZidgjUjwtvIo=; b=EukbmAIJamNsedhM9xeGIUSiMYx0kdJ3GJ8Vb+zL6i3L7OhPpWS1/28a3s+4+kIU1Q N7fsyph/8q107NfJzMfv//9pmT2VqByaGZmdYjARLLq2UXB5giDiZkwyoPZ660pgs6JB CM7B1vBxz6wHFMopscgvnbUk/TVHmwqIKPkjYWmLu7ftlZd8/tYC0mdihYoIHdYijx1Z bPsznaWbDeT5taJ3BlBCNxXUoemtlHOOTe1IUoUmbfz8GWmn4D+xFvlYLiUwmiaFHumB leHB00cchwRboyhFGks0ZXW3v6GM5/yIvXqvb49jl4bCy1Q5hFHGq7RIp6GCSVCebuT9 xhIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787582847; x=1788187647; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0ep6UPzJTnZmCqVomVSsdJlCYmmRG8TZidgjUjwtvIo=; b=lwSXIAGg45tWmseUp4QNsZ1Vb/Oe16zs5T/jbe/b2iDm89LOBpz0WDL8m43Qme+zhB XUt3zjvSpX8LOhoJRGD0GRixdc9vFb7qOgfShWVEF6f5u0bqyvfX99HJ9ThQXOwCfaw0 lIISeL57eN/LqfsX7mgziTd41e6bkoNefc4aIZNX1FfMVSBejRPGli2uv92zqqopQ+co A0Ld6pXqykwTdjHJdjmdAl4HXQ7cs3q71F8pjk6zKX8Y+r29262cN18u4NzwfP31ZQI1 wgIgrub6LfTX/aUxu9GU+pzW016Xdw7UoSTzerrFLRDg0c2FCQRYL9F5bzO1v0jVqcEd 1/Yw== X-Forwarded-Encrypted: i=1; AHgh+RqYxblFfvKvCsTAZU8t/XcGLRAq2db2CZSLYw+ZVjo2tkCTtz2xlzyTNDZmjYMqAVWR9IfVTH57FRmbWPI=@vger.kernel.org X-Gm-Message-State: AFuF++mhZh/8a1gdpqd5drn8AyS6aB7sODQHafAW0IbJOInhYw4a4ru8 b9VkQDpFei9T/M6MSkysrEMdHxMs19lwygJC1xRvpKxwPVznjE0RKoA81J07Sqdp2w== X-Gm-Gg: AR+sD113y5WvD0jGLPhnZnBoIJ7gFLvjeG6zPZifOgh8YUt9PCuVatDcP24x5RGGhby kKhi9AzP3YJq+nzGDz4gcxPBTwndQWkAb4E9PnqjH0BtJdisHBz/78hbepZm8P6cvPn3U0P10lh hc7dZbc14AAXJQCXrz3ankl/9fhCK2/nYFC3R27igsbM2RTxgYAH9pBj3fjt66XEZobdJfjZDAV T3H1MMfWMBnKjHao/lFi/h3Ni88ndM3/nGNyMDQrP9VSlqiTVgagFRVfYe66HjHYgxcMs4ljdgl +8AZGuIBU9GyieekSz4Th2Y1ltQqZJdSFu6ovABD1cXmlfWdE/IVh0+/Nq5pZIVHXZT9NyTCMny q/7ASWHhj/akUqd+7DgZBqHrStWwLsGS+SxLwvpkzwZXAFgHGpypaQBpKJCJLX8rXm5zUr1HWLb 2uRosmGM/5ldJenebjNq+XVOEob84VVNa/7EeseaicMdvY+RMoj3OK2lbM3Di6zuqxd2UY6k3CQ JT/vF6KOC3SX6Smx+8DFambPmVkVfmpJyXeNYfLO3htqsc7 X-Received: by 2002:a05:690c:e013:10b0:7fd:a7b6:8d87 with SMTP id 00721157ae682-849f5a102b1mr80844207b3.25.1787582845685; Mon, 24 Aug 2026 07:47:25 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84caabb9e88sm34369867b3.29.2026.08.24.07.47.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:47:24 -0700 (PDT) Date: Mon, 24 Aug 2026 07:47:20 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 23/25] fs,mm/fbatch: use invalidate_bh_lrus() not invalidate_bh_lrus_cpu() In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Whilst there can be a case for including an invalidate_bh_lrus_cpu() in lru_add_drain_all()'s workqueued visits to remote CPUs, wouldn't it now be a preferable cleanup to remove that alternative, and stick with the smp_call_function_many_cond()-based invalidate_bh_lrus() throughout? And fix the UP lru_add_drain_all() to include an invalidate_bh_lrus(), which went missing when 5.15 commit 243418e3925d ("mm: fs: invalidate bh_lrus for only cold path") took that out of lru_add_drain(). Signed-off-by: Hugh Dickins --- fs/buffer.c | 16 +--------------- include/linux/buffer_head.h | 4 ---- mm/folio.c | 25 ++++++------------------- 3 files changed, 7 insertions(+), 38 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index ed966fa73b1b..7d114e5b9c62 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1427,7 +1427,7 @@ static void invalidate_bh_lru(void *arg) put_cpu_var(bh_lrus); } =20 -bool has_bh_in_lru(int cpu, void *dummy) +static bool has_bh_in_lru(int cpu, void *dummy) { struct bh_lru *b =3D per_cpu_ptr(&bh_lrus, cpu); int i; @@ -1446,20 +1446,6 @@ void invalidate_bh_lrus(void) } EXPORT_SYMBOL_GPL(invalidate_bh_lrus); =20 -/* - * It's called from workqueue context so we need a bh_lru_lock to close - * the race with preemption/irq. - */ -void invalidate_bh_lrus_cpu(void) -{ - struct bh_lru *b; - - bh_lru_lock(); - b =3D this_cpu_ptr(&bh_lrus); - __invalidate_bh_lrus(b); - bh_lru_unlock(); -} - void folio_set_bh(struct buffer_head *bh, struct folio *folio, unsigned long offset) { diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index fd2c7115c054..f19f9e80be8f 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -518,8 +518,6 @@ bool mmb_has_buffers(struct mapping_metadata_bhs *mmb); void mmb_invalidate(struct mapping_metadata_bhs *mmb); int mmb_sync(struct mapping_metadata_bhs *mmb); void invalidate_bh_lrus(void); -void invalidate_bh_lrus_cpu(void); -bool has_bh_in_lru(int cpu, void *dummy); extern int buffer_heads_over_limit; =20 #else /* CONFIG_BUFFER_HEAD */ @@ -528,8 +526,6 @@ static inline void buffer_init(void) {} static inline bool try_to_free_buffers(struct folio *folio) { return true;= } static inline int mmb_sync(struct mapping_metadata_bhs *mmb) { return 0; } static inline void invalidate_bh_lrus(void) {} -static inline void invalidate_bh_lrus_cpu(void) {} -static inline bool has_bh_in_lru(int cpu, void *dummy) { return false; } #define buffer_heads_over_limit 0 =20 #endif /* CONFIG_BUFFER_HEAD */ diff --git a/mm/folio.c b/mm/folio.c index 782b8245d213..3212c7a58623 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -748,21 +748,6 @@ void lru_add_drain(void) mlock_drain_local(); } =20 -/* - * It's called from per-cpu workqueue context in SMP case so - * lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on - * the same cpu. It shouldn't be a problem in !SMP case since - * the core is only one and the locks will disable preemption. - */ -static void lru_add_and_bh_lrus_drain(void) -{ - local_lock(&cpu_fbatches.lock); - lru_add_drain_cpu(smp_processor_id()); - local_unlock(&cpu_fbatches.lock); - invalidate_bh_lrus_cpu(); - mlock_drain_local(); -} - void lru_add_drain_cpu_zone(struct zone *zone) { local_lock(&cpu_fbatches.lock); @@ -778,7 +763,7 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain= _work); =20 static void lru_add_drain_per_cpu(struct work_struct *dummy) { - lru_add_and_bh_lrus_drain(); + lru_add_drain(); } =20 static bool cpu_needs_drain(unsigned int cpu) @@ -791,8 +776,7 @@ static bool cpu_needs_drain(unsigned int cpu) folio_batch_count(&fbatches->lru_move_tail) || folio_batch_count(&fbatches->lru_deactivate_file) || folio_batch_count(&fbatches->lru_deactivate) || - need_mlock_drain(cpu)) || - has_bh_in_lru(cpu, NULL); + need_mlock_drain(cpu)); } =20 /* @@ -891,6 +875,8 @@ static inline void __lru_add_drain_all(bool force_all_c= pus) } } =20 + invalidate_bh_lrus(); + for_each_cpu(cpu, &has_work) flush_work(&per_cpu(lru_add_drain_work, cpu)); =20 @@ -906,6 +892,7 @@ void lru_add_drain_all(void) void lru_add_drain_all(void) { lru_add_drain(); + invalidate_bh_lrus(); } #endif /* CONFIG_SMP */ =20 @@ -939,7 +926,7 @@ void lru_cache_disable(void) #ifdef CONFIG_SMP __lru_add_drain_all(true); #else - lru_add_and_bh_lrus_drain(); + lru_add_drain_all(); #endif } =20 --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AC252F7EFC for ; Mon, 24 Aug 2026 14:49:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582964; cv=none; b=tmTFFpC8kW1XlDT+P/u3i6Uu1IA9UJIoApiXL4H1E6oEU6NA/5g27D+yVEw4AEJ5XSTgx3EvbQIq6I49nyYxCUViT5nHWwY8n+VTJRPuLM00njocCBxE1imCF6mNVReR5BjLrLen4Zvss9WYuG1caWVq6m19Xa8ssUReYrV3YhU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582964; c=relaxed/simple; bh=yGiXozn38J+0tD/OI9++1VPv9Nhfk2FT1fAC8DE0XR4=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=ZV9+h9fNPhTqa/tSSVBlHH4hO/FsmzZicyV+g2FRoPiyr0cjt0fFEuHSmZHITOL2l+EGII2pbM9d3WvXbep0UoRX75EHypyyYpJfQHWrYJNKx/wDdl1oCh9JV2tyNxgIeczgIuNY0jHZKYUDjv11I1b89ErD7s6QUgqImMC8zsY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=snsaeIzq; arc=none smtp.client-ip=209.85.128.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="snsaeIzq" Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-836c5b01e82so47910317b3.0 for ; Mon, 24 Aug 2026 07:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787582960; x=1788187760; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=bJghNqQ8GHnzdXaTAtv+ShGeZ67uYvDYnO7pv3lrswI=; b=snsaeIzqoa9XDrrwKZ9DFRIMy416HkPopHZNYIh3wEXbGPy/RsgJQH+Q+lyvWdhnB3 SfT+43eH3MDQMO18l0ocDU7VgxFxxiRi15apzlqSMUy5yt/oCzqpmNI2y/Dn6gakzaN4 v9k3fzJY6s9m1E/vBodZoHZfzK+JlDdT+Q6NrPRFjD70ax9piMpmOwDZgMJiihbp3cUt mrDeUzyNI7aEV6cvJYLjGJEa9fZj7JRLMifSOGfMuVKlA1SE53jafAJVlSeXM7TrCAik DlPjAuRP8IHltQRgbjzcGQIyi1gMkEpND7+22WxyAwL6s934IzpQi7Sb+CJ2IfWD5KKy I8Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787582960; x=1788187760; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bJghNqQ8GHnzdXaTAtv+ShGeZ67uYvDYnO7pv3lrswI=; b=GXN5JI/mVQ1DjQ4i4ZyI1AKk2FuHE7LNLIwUEUQiteajzWuVOfxrvl0hcaBGFhdt78 ax/cSkz5pPiVswJcWfE7W17lEjmUiTlyYbtB6B6hXCL4zfVk/emT4yFO3+v3JUJgEy/D pbTrJ246tmA5AUgmFfGsHMW0URxV+RWYEBZWbnqtNZPBtvV9BW/TVy/SyvdaXSfJlyBz 5Q+2CHk5zOj8G+2OscYBs1c+9DMgE8XBV4QUjNkas8dC8HKJGyo+gPkbk0kGy9qfkGDk qrxfOvG9ZmIOEUxRZ/UyJQ4orcf0AFWG4iHBmzRu6GJCV7cdE1rUj7LXCdwttRoVlgb+ z/Vg== X-Forwarded-Encrypted: i=1; AHgh+RqQ9/QkbrSil+hXLAHpdKRzyhzBdlOy9Y1TGeR/2Cln79D4BkH8UoBWsheFLlQHCUZKTzr8N43I4J8MXgY=@vger.kernel.org X-Gm-Message-State: AFuF++nVjvmz+cNjFC0vUj9UGd8usu/kXA6ImVHD4HFK53cwZ74f4sfT wcWWW0/gz+nQBGq3ASu/bz6d/s4VCAQcqpa2GmVk1QqB7vDif1prFLeohAnMPj8qsg== X-Gm-Gg: AR+sD101wtpA8zUVEJGJWrXmiHEPlMWETY2yIcDY6sTGK7donmPW1uvkmggN/bmqUEp VL3BGCwAZ04nLZdbRxFB2+fwp8bUrrT4NJVYgdnX6dc/PIa6O+2xPjyAoAjOo74ZRk3sD4AXDY1 MRomErymBBm2P5l4t4LmkhRyAvxLVTzyn8vSKJI4E/T2/6SWeD4dbf/FHPpG+x86iBTxMmLRVAR pZsg6jHkSrZJQb/aRTJ7B9/4Ay4Y7gyO18pHPDUHWcscpTDlnfCDWSRC8C9nZAU6U14Lg+NSgDF rGw9Y5ylDDcTCbxCxLXNC2G0Bo58DqlGUyjlDjUnZlHMFbGztrp+egQAON77iu0Iwvw/0UfELUg uab9AwLNP55Fy/wqgh9I0kVIvSHTUFnkP3GhYBbz1BQX095VhnqZi+fKxmnkusksoKoXF7zjhBF 0NJaIH3gtQub3D0D50FYUfhFWD2bE7J/D4jXDB6b0/5xYz9VIhAKMdzoCbxvU402RMycZayaKI2 82V6EeusYan1iV9kxrxHvqMNYwz9P7aTZE1+PN3r0WaJcpzP6agp/MiIHY= X-Received: by 2002:a05:690c:e1c4:10b0:820:1281:8deb with SMTP id 00721157ae682-849f08113c4mr88243297b3.8.1787582959923; Mon, 24 Aug 2026 07:49:19 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cab854483sm34302067b3.33.2026.08.24.07.49.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:49:19 -0700 (PDT) Date: Mon, 24 Aug 2026 07:49:14 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 24/25] fs,mm/fbatch: lru_cache_disable() keep off buffer_head lrus only In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <9dde166b-c9b8-9f09-03ef-6c15a26cbc77@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the per-cpu fbatch folio references are gone, there seems to be no excuse for lru_cache_disable() there - other than offline_pages() needing to lru_add_drain_all() to erase stale pointers from the fbatches. Remove lru_cache_disabled() checks from all except bh_lru_install(): assuming that lru_cache_disable() might have value in preventing repeated calls to invalidate_bh_lrus() when migrating folios in memory hotremoval. So move all that from mm/folio.c to fs/buffer.c: but I can't see how any of the paranoid synchronize_rcu_expedited() business is needed now (or even before) - leave it out. And back at the mm end, lru_add_drain_all() does not need that force_all_cpus either - it stopped forcing all cpus in 5.18 commit ff042f4a9b05 ("mm: lru_cache_disable: replace work queue synchronization with synchronize_rcu"). Signed-off-by: Hugh Dickins --- fs/buffer.c | 24 +++++++++++++++--- include/linux/buffer_head.h | 4 +++ include/linux/swap.h | 7 ------ mm/folio.c | 49 ++++--------------------------------- mm/internal.h | 6 ----- mm/memory_hotplug.c | 4 +++ mm/mlock.c | 4 +-- 7 files changed, 36 insertions(+), 62 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 7d114e5b9c62..7455a11dfc4a 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1200,6 +1200,24 @@ static inline void check_irqs_on(void) #endif } =20 +static atomic_t lru_disable_count =3D ATOMIC_INIT(0); + +void lru_cache_disable(void) +{ + if (atomic_inc_return(&lru_disable_count) =3D=3D 1) + invalidate_bh_lrus(); +} + +static inline bool lru_cache_disabled(void) +{ + return atomic_read(&lru_disable_count); +} + +void lru_cache_enable(void) +{ + atomic_dec(&lru_disable_count); +} + /* * Install a buffer_head into this cpu's LRU. If not already in the LRU, = it is * inserted at the front, and the buffer_head at the back if any is evicte= d. @@ -1215,9 +1233,9 @@ static void bh_lru_install(struct buffer_head *bh) bh_lru_lock(); =20 /* - * the refcount of buffer_head in bh_lru prevents dropping the - * attached page(i.e., try_to_free_buffers) so it could cause - * failing page migration. + * The refcount of buffer_head in bh_lru prevents dropping the + * attached page (i.e., try_to_free_buffers), so it could cause + * repeated calls to invalidate_bh_lrus() during page migration. * Skip putting upcoming bh into bh_lru until migration is done. */ if (lru_cache_disabled() || cpu_is_isolated(smp_processor_id())) { diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index f19f9e80be8f..3b6a41b7932f 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -517,6 +517,8 @@ void mmb_init(struct mapping_metadata_bhs *mmb, struct = address_space *mapping); bool mmb_has_buffers(struct mapping_metadata_bhs *mmb); void mmb_invalidate(struct mapping_metadata_bhs *mmb); int mmb_sync(struct mapping_metadata_bhs *mmb); +void lru_cache_disable(void); +void lru_cache_enable(void); void invalidate_bh_lrus(void); extern int buffer_heads_over_limit; =20 @@ -525,6 +527,8 @@ extern int buffer_heads_over_limit; static inline void buffer_init(void) {} static inline bool try_to_free_buffers(struct folio *folio) { return true;= } static inline int mmb_sync(struct mapping_metadata_bhs *mmb) { return 0; } +static inline void lru_cache_disable(void) {} +static inline void lru_cache_enable(void) {} static inline void invalidate_bh_lrus(void) {} #define buffer_heads_over_limit 0 =20 diff --git a/include/linux/swap.h b/include/linux/swap.h index f21e1dd6febc..ddfd9b7bb861 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -305,13 +305,6 @@ void lru_add_drain_all(void); /* linux/mm/folio-compat.c */ void mark_page_accessed(struct page *page); =20 -extern atomic_t lru_disable_count; - -static inline bool lru_cache_disabled(void) -{ - return atomic_read(&lru_disable_count); -} - extern unsigned long shrink_all_memory(unsigned long nr_pages); extern int vm_swappiness; long remove_mapping(struct address_space *mapping, struct folio *folio); diff --git a/mm/folio.c b/mm/folio.c index 3212c7a58623..dac2f2d5dcc1 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -173,7 +173,7 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, local_lock(&cpu_fbatches.lock); =20 if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + !folio_may_be_lru_cached(folio)) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 if (disable_irq) @@ -501,7 +501,7 @@ void __folio_add_lru(struct folio *folio, bool mlockit) smp_mb__before_atomic(); folio_set_lru(folio); =20 - if (full || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + if (full || !folio_may_be_lru_cached(folio)) folio_batch_move_lru(fbatch, lru_add); =20 local_unlock(&cpu_fbatches.lock); @@ -786,7 +786,7 @@ static bool cpu_needs_drain(unsigned int cpu) * Calling this function with cpu hotplug locks held can actually lead * to obscure indirect dependencies via WQ context. */ -static inline void __lru_add_drain_all(bool force_all_cpus) +void lru_add_drain_all(void) { /* * lru_drain_gen - Global pages generation number @@ -810,7 +810,7 @@ static inline void __lru_add_drain_all(bool force_all_c= pus) if (WARN_ON(!mm_percpu_wq)) return; =20 - trace_mm_lru_add_drain_all_tp(force_all_cpus); + trace_mm_lru_add_drain_all_tp(false); =20 /* * Guarantee folio_batch counter stores visible by this CPU @@ -837,7 +837,7 @@ static inline void __lru_add_drain_all(bool force_all_c= pus) * (C) Exit the draining operation if a newer generation, from another * lru_add_drain_all(), was already scheduled for draining. Check (A). */ - if (unlikely(this_gen !=3D lru_drain_gen && !force_all_cpus)) + if (unlikely(this_gen !=3D lru_drain_gen)) goto done; =20 /* @@ -883,11 +883,6 @@ static inline void __lru_add_drain_all(bool force_all_= cpus) done: mutex_unlock(&lock); } - -void lru_add_drain_all(void) -{ - __lru_add_drain_all(false); -} #else void lru_add_drain_all(void) { @@ -896,40 +891,6 @@ void lru_add_drain_all(void) } #endif /* CONFIG_SMP */ =20 -atomic_t lru_disable_count =3D ATOMIC_INIT(0); - -/* - * lru_cache_disable() needs to be called before we start compiling - * a list of folios to be migrated using folio_isolate_lru(). - * It drains folios on LRU cache and then disable on all cpus until - * lru_cache_enable is called. - * - * Must be paired with a call to lru_cache_enable(). - */ -void lru_cache_disable(void) -{ - atomic_inc(&lru_disable_count); - /* - * Readers of lru_disable_count are protected by either disabling - * preemption or rcu_read_lock: - * - * preempt_disable, local_irq_disable [bh_lru_lock()] - * rcu_read_lock [rt_spin_lock CONFIG_PREEMPT_RT] - * preempt_disable [local_lock !CONFIG_PREEMPT_RT] - * - * Since v5.1 kernel, synchronize_rcu() is guaranteed to wait on - * preempt_disable() regions of code. So any CPU which sees - * lru_disable_count =3D 0 will have exited the critical - * section when synchronize_rcu() returns. - */ - synchronize_rcu_expedited(); -#ifdef CONFIG_SMP - __lru_add_drain_all(true); -#else - lru_add_drain_all(); -#endif -} - /** * folios_put_refs - Reduce the reference count on a batch of folios. * @folios: The folios. diff --git a/mm/internal.h b/mm/internal.h index 9a25552cbd83..7a301c401d39 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -56,12 +56,6 @@ static inline bool folio_may_be_lru_cached(struct folio = *folio) return !folio_test_large(folio); } =20 -static inline void lru_cache_enable(void) -{ - atomic_dec(&lru_disable_count); -} - -void lru_cache_disable(void); void lru_add_drain(void); void lru_add_drain_cpu(int cpu); void lru_add_drain_cpu_zone(struct zone *zone); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 226ab9cb078a..19756c45b5e7 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -2095,6 +2096,9 @@ int offline_pages(unsigned long start_pfn, unsigned l= ong nr_pages, =20 } while (ret); =20 + /* Remove any instances of the freed pages from per-cpu fbatches. */ + lru_add_drain_all(); + /* Mark all sections offline and remove free pages from the buddy. */ managed_pages =3D __offline_isolated_pages(start_pfn, end_pfn); pr_debug("Offlined Pages %ld\n", nr_pages); diff --git a/mm/mlock.c b/mm/mlock.c index 971430e6251e..a3cfdb274fc7 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -247,7 +247,7 @@ void mlock_folio(struct folio *folio) local_lock(&mlock_fbatch.lock); fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); if (!folio_batch_add(fbatch, mlock_flagged(folio)) || - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + !folio_may_be_lru_cached(folio)) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); } @@ -278,7 +278,7 @@ void munlock_folio(struct folio *folio) local_lock(&mlock_fbatch.lock); fbatch =3D this_cpu_ptr(&mlock_fbatch.fbatch); if (!folio_batch_add(fbatch, folio) || - !folio_may_be_lru_cached(folio) || lru_cache_disabled()) + !folio_may_be_lru_cached(folio)) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); } --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yx1-f47.google.com (mail-yx1-f47.google.com [74.125.224.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 341F93644D4 for ; Mon, 24 Aug 2026 14:51:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787583082; cv=none; b=Sg+87oztLljmagza8xwlnjq2yYsnCGrCResFeT6Bk24TLDQWShyaWLI43IvwHDyv+vD++hVQAr3w1oYA7mOe9G2ODhbUlMFhE8JCRy41O3n+6GIxFopzbor59NA5VWP9cSBwx4cCGbE79Ro8jwzOiZVCYfFcK9/g1TFghGMkrTA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787583082; c=relaxed/simple; bh=7NJT4klpwyNAMnizim7gBiFUSAGUskks23qiOn/LHJk=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=BNH6ZhDGL0GJ9hsCATqyj5+FEIWPf99MWqhTJKff7CLxUPCaUxeTgldsdGU9uFuNOoWGtrX5GuqGINOef+DrNPcBr2WBYgG1ABHWKAEHJHIqOaRowIkvC2t04CVDs1bbhww//kXCN9zk3I/YSFgYnnW4ok9Ys7ir0BTPfOsN6ek= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=S6ki/Kff; arc=none smtp.client-ip=74.125.224.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="S6ki/Kff" Received: by mail-yx1-f47.google.com with SMTP id 956f58d0204a3-66c67a73eb7so6826423d50.0 for ; Mon, 24 Aug 2026 07:51:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787583080; x=1788187880; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=oBC/SAP50I4KxaPevcH0bVL/Jm4cuaV8N0/RCICjbNo=; b=S6ki/KffP0CvW4GdH1OievddahR9ZfFf8YPANe+JjHk1n5WhyjAXSp2eLSrrF81MS0 vewun7rPdck7G1hU9NpG7nNY167RPfM1oK02Yvkx/obU985dXLQ4KuMUrZylgGAtAsRE BdgUc1mSGWpwfTcTETXOExnNb2IQXWIA9t16VebWo3C1cjx1tpxC9KJ/QaGsyXVQyU86 63RPry+8FEZ+a7ZkTh4+JnbYefZf4LH92gCAcbfMDzflxBOzD9DD+EE6XWDDUN6TwYbo plugGwqaZYZJ+gm9flXqaCswhY89IaRl09bCJ1PGT4/tskvt4wx1XxeJwFWCT/DQw9Ed QUEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787583080; x=1788187880; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=oBC/SAP50I4KxaPevcH0bVL/Jm4cuaV8N0/RCICjbNo=; b=gkZnmij4VZXWTI4ba7V/0HUzEzPdpRVKNVooZo86i5HtRouJCffqj3xRz+WUVzxPSe MwRMadzWziYBxbJfzpmQtSfSkNsi+BTEdFH345frZc8gA8ujchJ3xxDFumbFwkrLBxSh AEZWTWexiQYltdIorB09ohLq+KgZol6Z0k1UAuMfTFmadWc9Goxcknnidb6zLx1hUW/Z iUOiRj55sBQe5c8VNY6d4zUSwF2iKR2V2SQAd9CYK2ZEOkwgdqBt/r6+l/9xdIphKsFs xqGiVho1BhQCyK/mph1DVAaU5A4pVdaY2pNHsdqeQ0/jtc9wpuuIBzXUSmff0KqbAnnc LnWg== X-Forwarded-Encrypted: i=1; AHgh+RpNsOqvOmW5lH8tTGiQAUOwWXwZnew63akp18JoLPmO/TVa/cpt4Jq54+aQhM9x8y1RKjlP0QQaQTebZZI=@vger.kernel.org X-Gm-Message-State: AFuF++kUAsJNVZl0NGeZeO9+RmLsgatle1ADPQPkWbLQQd+pbK6+7dAF JO+vf0OPaBfD/qFkiGZ2odFFYpX1zE5kFoyhwxjSV7fXAw7T64ciohPyWcqcx8s0QA== X-Gm-Gg: AR+sD12tULABW3m5mkLuNglLnda8w+kW7mMjclAsjbxIy+JvdG3CsuTNlhlVIcqAf1o nTCv/NwtmJK94BTM4ZVS+pwS1b9o+RR77CiVXuPzTFe+4JxzUCP0eh4ypqjOSIwi4/k1cjLt4r4 AmSUYTD8Bhcu9B9LmmSXS7OeMRuyzLiTTTevVtTeSM68o3fOHv8AZZug6yhkMOCNuJrYpG8SGMj V7cerZ1a9N1UZ2MrKyzlgQE0T4+GdCH9B8/0eTrlgaEt3Q19uqKPTfl2C2XHmBCj+UCAvOAOE0+ zCpnnka5qFfqbcA59vMzu3Eew1f1dZx+MYHT/dJsNf3ZR4dV4kLVTn4bgqsvLIMQ4yuaDZE8WBN EkDNSUPu08B5HXN/PCskJepuk+cKYkBx7FZLKueAwFdVj8sY/X899QKEeY37IktCD5orDD2VWSZ N1HuTAVUdEHdJDYq7+uXwbC7BUurVFMJDa9aa7Ga8d4yLc9uR5mSYYmZsVnh04akspKIPknxnBK eECiNkBDaCorYDA1UkRr8xtSxtsWxTpQZHpdj+noJnDNBCT8MWPTGFaAfk= X-Received: by 2002:a05:690e:4843:b0:66c:e378:bd2 with SMTP id 956f58d0204a3-66ce3780eecmr7793232d50.22.1787583079315; Mon, 24 Aug 2026 07:51:19 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84cac01868dsm34817937b3.42.2026.08.24.07.51.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2026 07:51:18 -0700 (PDT) Date: Mon, 24 Aug 2026 07:51:12 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 25/25] mm/fbatch: move lru_add_drain_all() declaration to mm/internal.h In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <552d4644-995b-e605-175d-3268b67adb66@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Only mm now uses lru_add_drain_all(), in forced reclaim and in memory hotremoval: help to keep it that way by moving its declaration alongside that of lru_add_drain() in mm/internal.h. Signed-off-by: Hugh Dickins --- include/linux/swap.h | 1 - mm/internal.h | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index ddfd9b7bb861..94b694328ee7 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -300,7 +300,6 @@ static inline void folio_add_lru(struct folio *folio) __folio_add_lru(folio, false); } void folio_mark_accessed(struct folio *folio); -void lru_add_drain_all(void); =20 /* linux/mm/folio-compat.c */ void mark_page_accessed(struct page *page); diff --git a/mm/internal.h b/mm/internal.h index 7a301c401d39..73d618254a3b 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -57,6 +57,7 @@ static inline bool folio_may_be_lru_cached(struct folio *= folio) } =20 void lru_add_drain(void); +void lru_add_drain_all(void); void lru_add_drain_cpu(int cpu); void lru_add_drain_cpu_zone(struct zone *zone); void folio_deactivate(struct folio *folio); --=20 2.51.0 From nobody Mon Sep 28 08:02:23 2026 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 82FB82475D0 for ; Wed, 2 Sep 2026 03:54:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788321294; cv=none; b=UnEwBNB+PWY6W6mBVJ+cuiLLe9sbIMbRHerbRWyy6sc1DwBf3hIHhEp4KWv155OmVDQlGibDeQJ4078qSvAhGhYENElsJhg24V9G744rja05whY6OQZcHhY7swYFAs5+PdCgfg3K/mBO8hJAVLSqedxc7pTBf3+h2iajpK6hMQo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788321294; c=relaxed/simple; bh=fRhuNFDdOevX6AnwPPrw0lvkQt/C1Q5JAeS99ARfKsg=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=pRt14ZsCGDr5hKx9TDkpHduGdRNWAlHzKhG1u5MbwK452Ck1cb/VnLA9tyNln7NpkCaQQrrw4oFz1Vlv15UIyHd7tMBNTARbc2cuR69IJ3vuh9qcE+XCCCwviaEAewr5oCKPX4gd/6HwVRqb1EHe07Q9wibZVsV0/q2Pngj0MNM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GfBLQ3HC; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GfBLQ3HC" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-85a50f6a7f7so9292137b3.2 for ; Tue, 01 Sep 2026 20:54:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788321291; x=1788926091; darn=vger.kernel.org; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=y3C3YFkFN20q0dqMhyWIalQhIX+NjYlCT3NZSRQXbv0=; b=GfBLQ3HCJAEBkM3S20uzlPqJx0t9g7goeoft1HoZDyfkUTCw9eRKP0a7Ey9ozYuohA bab6Un0U9LfZOl2D7lI0WIDG7i4nX8mjXgGjDC/IBV2XxxN8dzLwQvoIPml7mhwx4GZ3 t7c29WfgOiR2Ov8PR1qOPr1iM5mATUgZJrv/f2hs+m2Epdl+hiwc91nIt2HeyIhJ2nu2 0wZfGBNTVYtG4HCVJPCxGQByDgmOCR+AViIAOPsg+ucnZrz01IAigiMIVg+Bq9CfkI8F LYYAB33Z66kqO50JpCtkvIM9ZtudZ41X6XbqwbFjKMks1UcSiJOuYenkoJdAzscOHhYI ayOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788321291; x=1788926091; h=content-type:mime-version:references:message-id:in-reply-to:subject :cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=y3C3YFkFN20q0dqMhyWIalQhIX+NjYlCT3NZSRQXbv0=; b=DGoEUnm+ok9OfR1xeVO7D5CL0/cZI3eprmeVPX6FcDnI/gg/PcCMTavbyuBeijF8YI HfbX61M4BcBAktkpPtOTFJesvvJ9fXsz1KYdlr2aXT6DgQoYoV2XxFkh+vGIxbqErCkb X3VCAvk2vM9vXdXJQNvxEM/J0OY3DTLtHwxtbAfn9nkKNmHx83dsTEJ8XzI6FknKpB8u mcloxwV3qHhrof7yoFjMDdcDY1BuGU2gEbay0G1dS7nr/RNOECjMLwF+TLnWQo/Jp1rH BuBVRwRC7aXc5aYA6AJeL+3xpexDe3PVjidbbsHeO3qlhqDCaBkU88uJdB6rvsqaB6BW 0jNg== X-Forwarded-Encrypted: i=1; AKwUvBw8p8Ni6VqLmlmGH7dnqMbkM37UEDJ37725jR2pPMETGn/vRVEVccEJeHw6ATCy3fNJ1wuO5swVzl6ixnc=@vger.kernel.org X-Gm-Message-State: AFuF++k0vWjIQNEF0WpgMpw077msLnQym6k6LmarMZs8f5SbrsG7THrL eypV4e2JQXItsof4v/E5mMrkVmxcZ4q4TbQe3DOxVcCAvvV7rCtOjaF2+CE136P+9A== X-Gm-Gg: AYBFou36NH5PeJmQO69kMUfJiPjAjTmUdr9KeWD+/4EqQVcQCm49UUtspt19a9HDIna hagQsD5azChcyul3qgWTJS+wQxTomQZmpjSCzOKn2eq7ROdIghk3S/+PhLRfcy/4z7Odw/yt4Yf Mre+9MCcmPjJyQDKIBwTi7bYijjNfz3WSHJzSAU+Oq7Z5+3UcxLQaqUOBfSywNtx8Fwb6sYHj3o EpZsiahc5hHVM16bfjX5btb6DgkgH6PXuRZnDzOv0Mn7NWETAMMGTpvkUAdavX0Ir3AWJ96YS1j 1v1jjK49ZvqtDuKcR0i1oSb+GG2wpwFgnNkGQbtR5ax7BtPmCWzV3G5b5o1DYNllsE0AVlpm+qC JCxC5u/bmpzhrdUJLSXewqk7Ua7bWXOWhEFt/Iq1RKxLcRLZKDNr41+wh1eBjPvk/eF5sCpG2eW ZrT0gW4dZRMvBoDksps65mYO1vT7kCqIhBiCiIADP/QNffEuchuskRothG+BYKjwtad3IEItEww w/REJKRI3ftXhyk91il4N3atph5WI6jcZf4YCvgFJOOl5hB X-Received: by 2002:a05:690e:264e:b0:66f:92a7:1b28 with SMTP id 956f58d0204a3-66f9bfd8205mr445722d50.49.1788321290911; Tue, 01 Sep 2026 20:54:50 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66f985611casm1051256d50.8.2026.09.01.20.54.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Sep 2026 20:54:49 -0700 (PDT) Date: Tue, 1 Sep 2026 20:54:33 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Ackerley Tng , Alexander Viro , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , JP Kobryn , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 26/25] mm/fbatch: drop reference inside the loop when draining In-Reply-To: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Message-ID: <6d24ca2c-d535-cbdf-dab1-97046a3914ce@google.com> References: <14a16945-529b-8bc0-ab38-3ea97e54e223@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" folio_batch_move_lru() and mlock_folio_batch() used folios_put_refs() after their loop: but that's counter-productive, to batch up dropping all the references acquired within the loop. Now folio_put_testzero() inside the loop, where we also already hold (bar races) the right lock to remove the folio from its lruvec. This should be much friendlier to compaction, in the common case when the folio is still in use, to bring it back to expected refs sooner; but I suspect that when the folio is freed, compaction won't accept it until it gets to be PageBuddy later on? Respect the comment in mm/vmscan.c move_folios_to_lru(): could be done differently, but it's not worth optimizing the case when we cleared lru, since it also has to handle the case when we failed to clear it. Signed-off-by: Hugh Dickins --- I shall have to rebase the series, but this is an important afterthought which is best added into the review now. mm/folio.c | 23 +++++++++++++++++------ mm/mlock.c | 27 +++++++++++++++++++++------ 2 files changed, 38 insertions(+), 12 deletions(-) diff --git a/mm/folio.c b/mm/folio.c index 55ca799a9dcd..7c545e4c090d 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -128,20 +128,18 @@ static void lru_add(struct lruvec *lruvec, struct fol= io *folio) =20 static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t mov= e_fn) { - int i; + int i, j =3D 0; struct lruvec *lruvec =3D NULL; unsigned long flags =3D 0; =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { struct folio *folio =3D fbatch->folios[i]; =20 - if (!folio_try_get(folio)) { - fbatch->folios[i] =3D NULL; + if (!folio_try_get(folio)) continue; - } =20 if (!folio_test_clear_lru(folio)) - continue; + goto restored_lru; =20 /* Do not add to LRU if it has already been added */ if (move_fn =3D=3D lru_add && !lru_add_del_folio(folio)) @@ -155,11 +153,24 @@ static void folio_batch_move_lru(struct folio_batch *= fbatch, move_fn_t move_fn) lruvec_add_folio(lruvec, folio); restore_lru: folio_set_lru(folio); + /* See mm/vmscan.c move_folios_to_lru() comment on ordering */ +restored_lru: + if (unlikely(folio_put_testzero(folio))) { + folio_unqueue_deferred_split(folio); + __page_cache_release(folio, &lruvec, &flags); + fbatch->folios[j++] =3D folio; + } } =20 if (lruvec) lruvec_unlock_irqrestore(lruvec, flags); - folios_put_refs(fbatch, NULL); + if (unlikely(j)) { + fbatch->nr =3D j; + mem_cgroup_uncharge_folios(fbatch); + free_unref_folios(fbatch); + } else { + folio_batch_reinit(fbatch); + } } =20 static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch, diff --git a/mm/mlock.c b/mm/mlock.c index a3cfdb274fc7..c528cf7135bf 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -27,6 +27,7 @@ #include =20 #include "internal.h" +#include "page_alloc.h" =20 struct mlock_fbatch { local_lock_t lock; @@ -170,28 +171,42 @@ static void mlock_folio_batch(struct folio_batch *fba= tch) struct lruvec *lruvec =3D NULL; unsigned long mlock; struct folio *folio; - int i; + int i, j =3D 0; =20 for (i =3D 0; i < folio_batch_count(fbatch); i++) { folio =3D fbatch->folios[i]; mlock =3D (unsigned long)folio & MLOCK_FLAG; folio =3D (struct folio *)((unsigned long)folio - mlock); - fbatch->folios[i] =3D folio; =20 - if (!folio_try_get(folio)) { - fbatch->folios[i] =3D NULL; + if (!folio_try_get(folio)) continue; - } =20 if (mlock) lruvec =3D __mlock_folio(folio, lruvec); else lruvec =3D __munlock_folio(folio, lruvec); + + if (unlikely(folio_put_testzero(folio))) { + folio_unqueue_deferred_split(folio); + /* __page_cache_release() without irqflags */ + if (folio_test_lru(folio)) { + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + lruvec_del_folio(lruvec, folio); + __folio_clear_lru_flags(folio); + } + fbatch->folios[j++] =3D folio; + } } =20 if (lruvec) lruvec_unlock_irq(lruvec); - folios_put_refs(fbatch, NULL); + if (unlikely(j)) { + fbatch->nr =3D j; + mem_cgroup_uncharge_folios(fbatch); + free_unref_folios(fbatch); + } else { + folio_batch_reinit(fbatch); + } } =20 void mlock_drain_local(void) --=20 2.51.0